xs

package
v1.4.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jan 15, 2026 License: MIT Imports: 48 Imported by: 0

Documentation

Overview

Package xs is a collection of eXtended actions (xactions), including multi-object operations, list-objects, (cluster) rebalance and (target) resilver, ETL, and more.

  • Copyright (c) 2021-2026, NVIDIA CORPORATION. All rights reserved.

Package xs is a collection of eXtended actions (xactions), including multi-object operations, list-objects, (cluster) rebalance and (target) resilver, ETL, and more.

  • Copyright (c) 2024-2026, NVIDIA CORPORATION. All rights reserved.

Package xs is a collection of eXtended actions (xactions), including multi-object operations, list-objects, (cluster) rebalance and (target) resilver, ETL, and more.

  • Copyright (c) 2018-2025, NVIDIA CORPORATION. All rights reserved.

Package xs is a collection of eXtended actions (xactions), including multi-object operations, list-objects, (cluster) rebalance and (target) resilver, ETL, and more.

  • Copyright (c) 2018-2025, NVIDIA CORPORATION. All rights reserved.

Package xs is a collection of eXtended actions (xactions), including multi-object operations, list-objects, (cluster) rebalance and (target) resilver, ETL, and more.

  • Copyright (c) 2018-2026, NVIDIA CORPORATION. All rights reserved.

Package xs is a collection of eXtended actions (xactions), including multi-object operations, list-objects, (cluster) rebalance and (target) resilver, ETL, and more.

  • Copyright (c) 2018-2025, NVIDIA CORPORATION. All rights reserved.

Package xs is a collection of eXtended actions (xactions), including multi-object operations, list-objects, (cluster) rebalance and (target) resilver, ETL, and more.

  • Copyright (c) 2018-2026, NVIDIA CORPORATION. All rights reserved.

Package xs is a collection of eXtended actions (xactions), including multi-object operations, list-objects, (cluster) rebalance and (target) resilver, ETL, and more.

  • Copyright (c) 2018-2025, NVIDIA CORPORATION. All rights reserved.

Package xs is a collection of eXtended actions (xactions), including multi-object operations, list-objects, (cluster) rebalance and (target) resilver, ETL, and more.

  • Copyright (c) 2018-2026, NVIDIA CORPORATION. All rights reserved.

Package xs is a collection of eXtended actions (xactions), including multi-object operations, list-objects, (cluster) rebalance and (target) resilver, ETL, and more.

  • Copyright (c) 2018-2026, NVIDIA CORPORATION. All rights reserved.

Package xs is a collection of eXtended actions (xactions), including multi-object operations, list-objects, (cluster) rebalance and (target) resilver, ETL, and more.

  • Copyright (c) 2018-2026, NVIDIA CORPORATION. All rights reserved.

Package xs is a collection of eXtended actions (xactions), including multi-object operations, list-objects, (cluster) rebalance and (target) resilver, ETL, and more.

  • Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.

Package xs is a collection of eXtended actions (xactions), including multi-object operations, list-objects, (cluster) rebalance and (target) resilver, ETL, and more.

  • Copyright (c) 2018-2025, NVIDIA CORPORATION. All rights reserved.

Package xs is a collection of eXtended actions (xactions), including multi-object operations, list-objects, (cluster) rebalance and (target) resilver, ETL, and more.

  • Copyright (c) 2018-2026, NVIDIA CORPORATION. All rights reserved.

Package xs is a collection of eXtended actions (xactions), including multi-object operations, list-objects, (cluster) rebalance and (target) resilver, ETL, and more.

  • Copyright (c) 2022-2025, NVIDIA CORPORATION. All rights reserved.

Package xs is a collection of eXtended actions (xactions), including multi-object operations, list-objects, (cluster) rebalance and (target) resilver, ETL, and more.

  • Copyright (c) 2018-2026, NVIDIA CORPORATION. All rights reserved.

Package xs is a collection of eXtended actions (xactions), including multi-object operations, list-objects, (cluster) rebalance and (target) resilver, ETL, and more.

  • Copyright (c) 2024-2026, NVIDIA CORPORATION. All rights reserved.

Package xs is a collection of eXtended actions (xactions), including multi-object operations, list-objects, (cluster) rebalance and (target) resilver, ETL, and more.

  • Copyright (c) 2018-2026, NVIDIA CORPORATION. All rights reserved.

Package xs is a collection of eXtended actions (xactions), including multi-object operations, list-objects, (cluster) rebalance and (target) resilver, ETL, and more.

  • Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.

Package xact provides core functionality for the AIStore eXtended Actions (xactions).

  • Copyright (c) 2021-2025, NVIDIA CORPORATION. All rights reserved.

Package xs is a collection of eXtended actions (xactions), including multi-object operations, list-objects, (cluster) rebalance and (target) resilver, ETL, and more.

  • Copyright (c) 2021-2025, NVIDIA CORPORATION. All rights reserved.

Package xs is a collection of eXtended actions (xactions), including multi-object operations, list-objects, (cluster) rebalance and (target) resilver, ETL, and more.

  • Copyright (c) 2018-2026, NVIDIA CORPORATION. All rights reserved.

Package xs is a collection of eXtended actions (xactions), including multi-object operations, list-objects, (cluster) rebalance and (target) resilver, ETL, and more.

  • Copyright (c) 2021-2026, NVIDIA CORPORATION. All rights reserved.

Package xs contains most of the supported eXtended actions (xactions) with some exceptions that include certain storage services (mirror, EC) and extensions (downloader, lru).

  • Copyright (c) 2022-2026, NVIDIA CORPORATION. All rights reserved.

Package xs contains most of the supported eXtended actions (xactions) with some exceptions that include certain storage services (mirror, EC) and extensions (downloader, lru).

  • Copyright (c) 2018-2025, NVIDIA CORPORATION. All rights reserved.

Package xs is a collection of eXtended actions (xactions), including multi-object operations, list-objects, (cluster) rebalance and (target) resilver, ETL, and more.

  • Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.

Index

Constants

View Source 
const PrefixTcoID = "tco-"

Variables

View Source 
var (
	ErrGone = errors.New("gone")
)

Functions

func FreeCOI added in v1.3.26 

func FreeCOI(a *CoiParams)

func Preg added in v1.3.26 

func Preg()

for additional startup-time reg-s see lru, downloader, ec

func PrmObjName added in v1.3.22 

func PrmObjName(objfqn, dirfqn, prefix string) (_ string, err error)

func RenewBlobDl added in v1.3.22 

func RenewBlobDl(xid string, params *core.BlobParams, oa *cmn.ObjAttrs) xreg.RenewRes
   Main Thread                        Worker Goroutines (N)           Backend
(XactBlobDl.Run())                   (chunkDownloader.run())              │
   │                                        │                             │
   │     chunkWi{sgl, roff} -> workCh       │                             │
   ├──────────────────────────────────────> │                             │
   │                                        ├─> GetObjReader(roff, size) ─┤
   │                                        │                             │
   │                                        |<── copy to a chunk file <───┤
   │                                        │
   │                                        ├── add chunk to manifest
   │     doneCh <- chunkDone{sgl, roff}     │
   │ <──────────────────────────────────────┤
   │                                        │
   ├── handle out-of-order (pending queue)  │
   │                                        │
   ├─── sequential write to RespWriter      │
   │       (for streaming GET)              │
   │                                        │
   │        schedule the next chunk         │
   │   chunkWi{sgl, nextRoff} -> workCh     │
   │──────────────────────────────────────> │
   │                                        │
   (loop until woff >= fullSize)

===================================================================================== Consumer: `chunkDownloader.run()` - worker goroutines Each worker continuously: 1. Pull work item from workCh

  • roff indicates byte range [roff, roff+chunkSize)
  • SGL is a reusable buffer provided by main thread

2. Performs HTTP range request to remote backend 3. Persist chunk:

  • Create chunk metadata and chunk file from LOM
  • Copy data from backend reader to multi-writer: chunk file + checksum (+ SGL if RespWriter set)
  • Add the chunk to the manifest

4. Notify completion: Send chunkDone{sgl, written, roff, err} to doneCh 5. Repeat: Wait for next work item on workCh

Producer: XactBlobDl - main thread ------------------------------------------------------------------------------------------------ PHASE 1: `blobFactory.Start()` and `XactBlobDl.start()` - Initialize worker and chunk manifest - Seed initial work items (chunks) to workCh (one per worker) ------------------------------------------------------------------------------------------------ PHASE 2: `XactBlobDl.Run()` - coordination loop Main thread continuously: 1. Receive chunk completion from doneCh 2. Handle out-of-order: If chunk arrived early (roff > woff), add to pending queue 3. Sequential write: Copy SGL to RespWriter if streaming, reset SGL, advance woff 4. Schedule next chunk: Send workCh <- chunkWi{sgl, nextRoff} (reuse SGL) 5. Plug holes: Walk pending queue, write chunks that are now in-order (roff == woff) 6. Check completion: If woff >= fullSize, exit loop ------------------------------------------------------------------------------------------------ PHASE 3: `XactBlobDl.Run()` fin label - finalization - Close workCh and wait for workers - On success: Compute checksum, persist manifest via lom.CompleteUfest(), update stats - On error/abort: Clean up via manifest.Abort(), update error stats ===================================================================================== Abort Flow: ----------- Abort() signals via Base.Abort() (sends to ChanAbort) and returns immediately. It does NOT wait for workers - cleanup happens in the main thread:

Worker mode (runWorkers):

  1. Main thread receives <-ChanAbort(), jumps to cleanup
  2. close(workCh) signals workers to exit
  3. wg.Wait() blocks until all workers complete their current do() call
  4. Returns error -> finalize() -> manifest.Abort() cleans up chunk files

Serial mode (runSerial):

  1. IsAborted() checked between chunks -> returns error
  2. finalize() -> manifest.Abort() cleans up chunk files

Note: manifest.Abort() is called only AFTER all I/O has stopped.

=====================================================================================

func Treg added in v1.3.26 

func Treg(coi COI)

Types

type BckRename added in v1.4.0 

type BckRename struct {
	*XactTCB
}

func (*BckRename) CtlMsg added in v1.4.1 

func (r *BckRename) CtlMsg() string

func (*BckRename) FromTo added in v1.4.0 

func (r *BckRename) FromTo() (*meta.Bck, *meta.Bck)

func (*BckRename) Name added in v1.4.0 

func (r *BckRename) Name() string

func (*BckRename) Run added in v1.4.0 

func (r *BckRename) Run(wg *sync.WaitGroup)

BckRename xaction is a wrapper around XactTCB that adds the following: - if not aborted; call BMDVersionFixup to piggyback bucket renaming to remove bckFrom from BMD (see `whatRenamedLB` in proxy.go) -

func (*BckRename) Snap added in v1.4.0 

func (r *BckRename) Snap() (snap *core.Snap)

func (*BckRename) String added in v1.4.0 

func (r *BckRename) String() string

type COI added in v1.3.26 

type COI interface {
	CopyObject(lom *core.LOM, dm *bundle.DM, coi *CoiParams) CoiRes
}

type CoiParams added in v1.3.26 

type CoiParams struct {
	Xact   core.Xact
	OAH    cos.OAH // object attributes after applying core.GetROC
	Config *cmn.Config
	BckTo  *meta.Bck
	core.GetROC
	core.PutWOC
	ETLArgs         *core.ETLArgs
	ObjnameTo       string
	Buf             []byte
	OWT             cmn.OWT
	Finalize        bool // copies and EC (as in poi.finalize())
	DryRun          bool // no changes
	LatestVer       bool // can be used without changing bucket's 'versioning.validate_warm_get'; see also: QparamLatestVer
	Sync            bool // see core.GetROC at core/ldp.go
	ContinueOnError bool // when false, a failure to copy triggers abort
}

func AllocCOI added in v1.3.26 

func AllocCOI() *CoiParams

type CoiRes added in v1.3.28 

type CoiRes struct {
	Err   error
	Lsize int64
	Ecode int
	RGET  bool // when reading source via backend.GetObjReader
}

type Election 

type Election struct {
	xact.Base
}

func (*Election) CtlMsg added in v1.4.1 

func (*Election) CtlMsg() string

func (*Election) Run 

func (*Election) Run(*sync.WaitGroup)

func (*Election) Snap added in v1.3.16 

func (r *Election) Snap() *core.Snap

type LsoRsp 

type LsoRsp struct {
	Err    error
	Lst    *cmn.LsoRes
	Status int
}

`on-demand` per list-objects request

type LsoXact 

type LsoXact struct {
	// contains filtered or unexported fields
}

`on-demand` per list-objects request

func (*LsoXact) Abort 

func (r *LsoXact) Abort(err error) (ok bool)

skip on-demand idleness check

func (*LsoXact) CtlMsg added in v1.4.1 

func (r *LsoXact) CtlMsg() string

func (*LsoXact) Do 

func (r *LsoXact) Do(msg *apc.LsoMsg) *LsoRsp

func (*LsoXact) Run 

func (r *LsoXact) Run(wg *sync.WaitGroup)

func (*LsoXact) Snap 

func (r *LsoXact) Snap() *core.Snap

func (*LsoXact) String 

func (r *LsoXact) String() (s string)

func (*LsoXact) TxnAbort 

func (r *LsoXact) TxnAbort(err error)

limited pre-run abort

type Rebalance 

type Rebalance struct {
	Args *xreg.RebArgs
	xact.Base
}

func (*Rebalance) CtlMsg added in v1.4.1 

func (xreb *Rebalance) CtlMsg() string

func (*Rebalance) RebID 

func (xreb *Rebalance) RebID() int64

func (*Rebalance) Run 

func (*Rebalance) Run(*sync.WaitGroup)

func (*Rebalance) Snap 

func (xreb *Rebalance) Snap() (snap *core.Snap)

type Resilver 

type Resilver struct {
	Args *xreg.ResArgs
	// runtime
	Nbusy atomic.Int64

	// base
	xact.Base
	// contains filtered or unexported fields
}

func (*Resilver) CtlMsg added in v1.4.1 

func (xres *Resilver) CtlMsg() string

func (*Resilver) Run 

func (*Resilver) Run(*sync.WaitGroup)

func (*Resilver) SetJgroup added in v1.4.2 

func (xres *Resilver) SetJgroup(jgroup *mpather.Jgroup)

func (*Resilver) Snap 

func (xres *Resilver) Snap() *core.Snap

func (*Resilver) String 

func (xres *Resilver) String() string

type TestBmvFactory 

type TestBmvFactory = bmvFactory

type XactArch 

type XactArch struct {
	// contains filtered or unexported fields
}

func (*XactArch) BeginMsg added in v1.3.28 

func (r *XactArch) BeginMsg(msg *cmn.ArchiveBckMsg, archlom *core.LOM) (err error)

func (*XactArch) CtlMsg added in v1.4.1 

func (r *XactArch) CtlMsg() string

func (*XactArch) DoMsg added in v1.3.28 

func (r *XactArch) DoMsg(msg *cmn.ArchiveBckMsg)

func (*XactArch) FromTo added in v1.3.16 

func (r *XactArch) FromTo() (src, dst *meta.Bck)

func (*XactArch) Name added in v1.3.16 

func (r *XactArch) Name() (s string)

func (*XactArch) Run 

func (r *XactArch) Run(wg *sync.WaitGroup)

func (*XactArch) Snap 

func (r *XactArch) Snap() (snap *core.Snap)

func (*XactArch) String 

func (r *XactArch) String() (s string)

func (*XactArch) TxnAbort 

func (r *XactArch) TxnAbort(err error)

limited pre-run abort

type XactBlobDl added in v1.3.22 

type XactBlobDl struct {
	xact.Base
	// contains filtered or unexported fields
}

func (*XactBlobDl) CtlMsg added in v1.4.1 

func (r *XactBlobDl) CtlMsg() string

func (*XactBlobDl) Name added in v1.3.22 

func (r *XactBlobDl) Name() string

func (*XactBlobDl) Run added in v1.3.22 

func (r *XactBlobDl) Run(wg *sync.WaitGroup)

func (*XactBlobDl) Size added in v1.3.26 

func (r *XactBlobDl) Size() int64

func (*XactBlobDl) Snap added in v1.3.22 

func (r *XactBlobDl) Snap() (snap *core.Snap)

func (*XactBlobDl) String added in v1.3.26 

func (r *XactBlobDl) String() string

type XactDirPromote 

type XactDirPromote struct {
	xact.BckJog
	// contains filtered or unexported fields
}

func (*XactDirPromote) CtlMsg added in v1.4.1 

func (r *XactDirPromote) CtlMsg() string

func (*XactDirPromote) Run 

func (r *XactDirPromote) Run(wg *sync.WaitGroup)

func (*XactDirPromote) SetFshare 

func (r *XactDirPromote) SetFshare(v bool)

func (*XactDirPromote) Snap added in v1.3.16 

func (r *XactDirPromote) Snap() *core.Snap

type XactMoss added in v1.3.30 

type XactMoss struct {
	xact.DemandBase
	// contains filtered or unexported fields
}

func (*XactMoss) Abort added in v1.3.30 

func (r *XactMoss) Abort(err error) bool

func (*XactMoss) Assemble added in v1.3.30 

func (r *XactMoss) Assemble(req *apc.MossReq, w http.ResponseWriter, wid string) error

gather other requested data (local and remote); emit resulting archive (phase 3)

func (*XactMoss) BcastAbort added in v1.3.30 

func (r *XactMoss) BcastAbort(err error)

func (*XactMoss) CtlMsg added in v1.4.1 

func (r *XactMoss) CtlMsg() string

read from StatsUpdater and report: 1) _global_ target metrics, and 2) this xaction's just-in-time _pending_ and bewarm

func (*XactMoss) DecPending added in v1.3.30 

func (r *XactMoss) DecPending()

func (*XactMoss) IncPending added in v1.3.30 

func (r *XactMoss) IncPending()

func (*XactMoss) PrepRx added in v1.3.30 

func (r *XactMoss) PrepRx(req *apc.MossReq, smap *meta.Smap, wid string, receiving, usingPrev bool) error

(phase 1)

func (*XactMoss) RecvObj added in v1.3.30 

func (r *XactMoss) RecvObj(hdr *transport.ObjHdr, reader io.Reader, err error) error

demux -> wi.recv() note convention: received hdr.ObjName is `nameInArch` (ie., filename in resulting TAR)

func (*XactMoss) Run added in v1.3.30 

func (*XactMoss) Run(*sync.WaitGroup)

func (*XactMoss) Send added in v1.3.30 

func (r *XactMoss) Send(req *apc.MossReq, smap *meta.Smap, dt *meta.Snode, wid string, usingPrev bool) error

send all requested local data => DT (`tsi`) (phase 2)

func (*XactMoss) Snap added in v1.3.30 

func (r *XactMoss) Snap() *core.Snap

type XactNsumm added in v1.3.21 

type XactNsumm struct {
	xact.BckJog
	// contains filtered or unexported fields
}

func (*XactNsumm) CtlMsg added in v1.4.1 

func (r *XactNsumm) CtlMsg() string

func (*XactNsumm) Name added in v1.3.21 

func (r *XactNsumm) Name() string

func (*XactNsumm) Result added in v1.3.21 

func (r *XactNsumm) Result() (cmn.AllBsummResults, error)

func (*XactNsumm) Run added in v1.3.21 

func (r *XactNsumm) Run(started *sync.WaitGroup)

func (*XactNsumm) Snap added in v1.3.21 

func (r *XactNsumm) Snap() *core.Snap

func (*XactNsumm) String added in v1.3.21 

func (r *XactNsumm) String() string

type XactTCB added in v1.3.22 

type XactTCB struct {

	// mountpath joggers
	xact.BckJog
	// contains filtered or unexported fields
}

func (*XactTCB) Abort added in v1.3.28 

func (r *XactTCB) Abort(err error) bool

func (*XactTCB) Args added in v1.3.22 

func (r *XactTCB) Args() *xreg.TCBArgs

func (*XactTCB) CtlMsg added in v1.4.1 

func (r *XactTCB) CtlMsg() string

func (*XactTCB) FromTo added in v1.3.22 

func (r *XactTCB) FromTo() (*meta.Bck, *meta.Bck)

func (*XactTCB) Name added in v1.3.22 

func (r *XactTCB) Name() string

func (*XactTCB) Run added in v1.3.22 

func (r *XactTCB) Run(wg *sync.WaitGroup)

func (*XactTCB) Snap added in v1.3.22 

func (r *XactTCB) Snap() (snap *core.Snap)

func (*XactTCB) String added in v1.3.22 

func (r *XactTCB) String() string

func (*XactTCB) TxnAbort added in v1.3.22 

func (r *XactTCB) TxnAbort(err error)

limited pre-run abort

type XactTCO added in v1.3.28 

type XactTCO struct {
	// contains filtered or unexported fields
}

func (*XactTCO) BeginMsg added in v1.3.28 

func (r *XactTCO) BeginMsg(msg *cmn.TCOMsg)

func (*XactTCO) ContMsg added in v1.3.28 

func (r *XactTCO) ContMsg(msg *cmn.TCOMsg)

func (*XactTCO) CtlMsg added in v1.4.1 

func (r *XactTCO) CtlMsg() string

func (*XactTCO) FromTo added in v1.3.28 

func (r *XactTCO) FromTo() (*meta.Bck, *meta.Bck)

func (*XactTCO) Name added in v1.3.28 

func (r *XactTCO) Name() string

func (*XactTCO) Run added in v1.3.28 

func (r *XactTCO) Run(wg *sync.WaitGroup)

func (*XactTCO) Snap added in v1.3.28 

func (r *XactTCO) Snap() (snap *core.Snap)

func (*XactTCO) String added in v1.3.28 

func (r *XactTCO) String() string

func (*XactTCO) TxnAbort added in v1.3.28 

func (r *XactTCO) TxnAbort(err error)

limited pre-run abort

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.