package cimfs

import "github.com/Microsoft/hcsshim/pkg/cimfs"

This package provides simple go wrappers on top of the win32 CIMFS APIs.

Details about CimFS & related win32 APIs can be found here: https://learn.microsoft.com/en-us/windows/win32/api/_cimfs/

Details about how CimFS is being used in containerd can be found here: https://github.com/containerd/containerd/issues/8346

CIM types: Currently we support 2 types of CIMs:

Standard CIMs store all the contents of a CIM in one or more region & objectID files. This means a single CIM is made up of a `.cim` file, one or more region files and one or more objectID files. All of these files MUST be present in the same directory in order for that CIM to work. Block CIMs store all the data of a CIM in a single block device. A VHD can be such a block device. For convenience CimFS also allows using a block formatted file as a block device.

Standard CIMs can be created with the `func Create(imagePath string, oldFSName string, newFSName string) (_ *CimFsWriter, err error)` function defined in this package, whereas block CIMs can be created with the `func CreateBlockCIM(blockPath, oldName, newName string, blockType BlockCIMType) (_ *CimFsWriter, err error)` function.

Forking & Merging CIMs: In container world, CIMs are used for storing container image layers. Usually, one layer is stored in one CIM. This means we need a way to combine multiple CIMs to create the rootfs of a container. This can be achieved either by forking the CIMs or merging the CIMs.

Forking CIMs: Forking means every time a CIM is created for a non-base layer, we fork it off of a parent layer CIM. This ensures that contents that are written to this CIM are merged with that of parent layer CIMs at the time of CIM creation itself. When such a CIM is mounted we get a combined view of the contents of this CIM as well as the parent CIM from which this CIM was forked. However, this means that all the CIMs MUST be stored in the same directory in order for forked CIMs to work. And every non-base layer CIM is dependent on all of its parent layer CIMs.

Merging CIMs: If we create one or more CIMs without forking them at the time of creation, we can still merge those CIMs later to create a new special type of CIM called merged CIM. When mounted, this merged CIM provides a view of the combined contents of all the layers that were merged. The advantage of this approach is that each layer CIM (also referred to as source CIMs in the context of merging CIMs) can be created & stored independent of its parent CIMs. (Currently we only support merging block CIMs).

In order to create a merged CIM we need at least 2 non-forked block CIMs (we can not merge forked & non-forked CIMs), these CIMs are also referred to as source CIMs. We first create a new CIM (for storing the merge) via the `CreateBlockCIM` API, then call `CimAddFsToMergedImage2` repeatedly to add the source CIMs one by one to the merged CIM. Closing the handle on this new CIM commits it automatically. The order in which source CIMs are added matters. A source CIM that was added before another source CIM takes precedence when merging the CIM contents. Crating this merged CIM only combines the metadata of all the source CIMs, however the actual data isn't copied to the merged CIM. This is why when mounting the merged CIM, we still need to provide paths to the source CIMs.

`CimMergeMountImage` is used to mount a merged CIM. This API expects an array of paths of the merged CIM and all the source CIMs. Note that the array MUST include the merged CIM path at the 0th index and all the source CIMs in the same order in which they were added at the time of creation of the merged CIM. For example, if we merged CIMs 1.cim & 2.cim by first adding 1.cim (via CimAddFsToMergedImage) and then adding 2.cim, then the array should be [merged.cim, 1.cim, 2.cim]

Merged CIM specific APIs.

`CimTombstoneFile`: is used for creating a tombstone file in a CIM. Tombstone file is similar to a whiteout file used in case of overlayFS. A tombstone's primary use case is for merged CIMs. When multiple source CIMs are merged, a tombstone file/directory ensures that any files with the same path in the lower layers (i.e source CIMs that are added after the CIM that has a tombstone) do not show up in the mounted filesystem view. For example, imagine 1.cim has a file at path `foo/bar.txt` and 2.cim has a tombstone at path `foo/bar.txt`. If a merged CIM is created by first adding 2.cim (via CimAddFsToMergedImage) and then adding 1.cim and then when that merged CIM is mounted, `foo/bar.txt` will not show up in the mounted filesystem. A tombstone isn't required when using forked CIMs, because we can just call `CimDeletePath` to remove a file from the lower layers in that case. However, that doesn't work for merged CIMs since at the time of writing one of the source CIMs, we can't delete files from other source CIMs.

`CimCreateMergeLink`: is used to create a file link that is resolved at the time of merging CIMs. This is required if we want to create a hardlink in one source CIM that points to a file in another source CIM. Such a hardlink can not be resolved at the time of writing the source CIM. It can only be resolved at the time of merge. This API allows us to create such cross layer hard links.

Index

Constants 

const (
	BlockCIMTypeNone BlockCIMType = iota
	BlockCIMTypeSingleFile
	BlockCIMTypeDevice

	CimMountFlagNone       uint32 = 0x0
	CimMountFlagEnableDax  uint32 = 0x2
	CimMountBlockDeviceCim uint32 = 0x10
	CimMountSingleFileCim  uint32 = 0x20

	CimCreateFlagNone                uint32 = 0x0
	CimCreateFlagDoNotExpandPEImages uint32 = 0x1
	CimCreateFlagFixedSizeChunks     uint32 = 0x2
	CimCreateFlagBlockDeviceCim      uint32 = 0x4
	CimCreateFlagSingleFileCim       uint32 = 0x8
	CimCreateFlagConsistentCim       uint32 = 0x10

	CimMergeFlagNone        uint32 = 0x0
	CimMergeFlagSingleFile  uint32 = 0x1
	CimMergeFlagBlockDevice uint32 = 0x2
)

Functions

func DestroyCim 

func DestroyCim(ctx context.Context, cimPath string) (retErr error)

DestroyCim finds out the region files, object files of this cim and then delete the region files, object files and the <layer-id>.cim file itself. Note that any other CIMs that were forked off of this CIM would become unusable after this operation. This should not be used for block CIMs, os.Remove is sufficient for block CIMs.

func GetCimUsage 

func GetCimUsage(ctx context.Context, cimPath string) (uint64, error)

GetCimUsage returns the total disk usage in bytes by the cim at path `cimPath`.

func IsBlockCimSupported 

func IsBlockCimSupported() bool

IsBlockCimSupported returns true if block formatted CIMs (i.e block device CIM & single file CIM) are supported on the current OS build.

func IsCimFSSupported 

func IsCimFSSupported() bool

func IsMergedCimSupported 

func IsMergedCimSupported() bool

func MergeBlockCIMs 

func MergeBlockCIMs(mergedCIM *BlockCIM, sourceCIMs []*BlockCIM) (err error)

MergeBlockCIMs creates a new merged BlockCIM from the provided source BlockCIMs. CIM at index 0 is considered to be topmost CIM and the CIM at index `length-1` is considered the base CIM. (i.e file with the same path in CIM at index 0 will shadow files with the same path at all other CIMs) When mounting this merged CIM the source CIMs MUST be provided in the exact same order.

func Mount 

func Mount(cimPath string, volumeGUID guid.GUID, mountFlags uint32) (string, error)

Mount mounts the given cim at a volume with given GUID. Returns the full volume path if mount is successful.

func MountMergedBlockCIMs 

func MountMergedBlockCIMs(mergedCIM *BlockCIM, sourceCIMs []*BlockCIM, mountFlags uint32, volumeGUID guid.GUID) (string, error)

MountMergedBlockCIMs mounts the given merged BlockCIM (usually created with `MergeBlockCIMs`) at a volume with given GUID. The `sourceCIMs` MUST be identical to the `sourceCIMs` passed to `MergeBlockCIMs` when creating this merged CIM.

func Unmount 

func Unmount(volumePath string) error

Unmount unmounts the cim at mounted at path `volumePath`.

Types

type BlockCIM 

type BlockCIM struct {
	Type BlockCIMType
	// BlockPath is a path to the block device or the single file which contains the
	// CIM.
	BlockPath string
	// Since a block device CIM or a single file CIM can container multiple CIMs, we
	// refer to an individual CIM using its name.
	CimName string
}

BlockCIM represents a CIM stored in a block formatted way.

A CIM usually is made up of a .cim file and multiple region & objectID files. Currently, all of these files are stored together in the same directory. To refer to such a CIM, we provide the path to the `.cim` file and the corresponding region & objectID files are assumed to be present right next to it. In this case the directory on the host's filesystem which holds one or more such CIMs is the container for those CIMs.

Using multiple files for a single CIM can be very limiting. (For example, if you want to do a remote mount for a CIM layer, you now need to mount multiple files for a single layer). In such cases having a single container which contains all of the CIM related data is a great option. For this reason, CimFS has added support for a new type of a CIM named BlockCIM. A BlockCIM is a CIM for which the container used to store all of the CIM files is a block device or a binary file formatted like a block device. Such a block device (or a binary file) doesn't have a separate filesystem (like NTFS or FAT32) on it. Instead it is formatted in such a way that CimFS driver can read the blocks and find out which CIMs are present on that block device. The CIMs stored on a raw block device are sometimes referred to as block device CIMs and CIMs stored on the block formatted single file are referred as single file CIMs.

func (*BlockCIM) String 

func (b *BlockCIM) String() string

added for logging convenience

type BlockCIMType 

type BlockCIMType uint32

type CimFsWriter 

type CimFsWriter struct {
	// contains filtered or unexported fields
}

CimFsWriter represents a writer to a single CimFS filesystem instance. On disk, the image is composed of a filesystem file and several object ID and region files. Note: The CimFsWriter isn't thread safe!

func Create 

func Create(imagePath string, oldFSName string, newFSName string) (_ *CimFsWriter, err error)

Create creates a new cim image. The CimFsWriter returned can then be used to do operations on this cim. If `oldFSName` is provided the new image is "forked" from the CIM with name `oldFSName` located under `imagePath`.

func CreateBlockCIM 

func CreateBlockCIM(blockPath, name string, blockType BlockCIMType) (_ *CimFsWriter, err error)

Create creates a new block CIM and opens it for writing. The CimFsWriter returned can then be used to add/remove files to/from this CIM.

func (*CimFsWriter) AddFile 

func (c *CimFsWriter) AddFile(path string, info *winio.FileBasicInfo, fileSize int64, securityDescriptor []byte, extendedAttributes []byte, reparseData []byte) error

AddFile adds a new file to the image. The file is added at the specified path. After calling this function, the file is set as the active stream for the image, so data can be written by calling `Write`. 

func (c *CimFsWriter) AddLink(oldPath string, newPath string) error

AddLink adds a hard link at `newPath` that points to `oldPath`. 

func (c *CimFsWriter) AddMergedLink(oldPath string, newPath string) error

AddMergedLink adds a hard link at `newPath` that points to `oldPath` in the image. However unlike AddLink this link is resolved at merge time. This allows us to create links to files that are in other CIMs.

func (*CimFsWriter) AddTombstone 

func (c *CimFsWriter) AddTombstone(path string) error

Adds a tombstone at given path. This ensures that when the the CIMs are merged, the file at this path from lower layers won't show up in a mounted CIM. In case of Unlink, the file from the lower layers still shows up after merge.

func (*CimFsWriter) Close 

func (c *CimFsWriter) Close() (err error)

Close closes the CimFS filesystem.

func (*CimFsWriter) CreateAlternateStream 

func (c *CimFsWriter) CreateAlternateStream(path string, size uint64) (err error)

CreateAlternateStream creates alternate stream of given size at the given path inside the cim. This will replace the current active stream. Always, finish writing current active stream and then create an alternate stream. 

func (c *CimFsWriter) Unlink(path string) error

Unlink deletes the file at `path` from the image. Note that the file MUST have been already added to the image.

func (*CimFsWriter) Write 

func (c *CimFsWriter) Write(p []byte) (int, error)

Write writes bytes to the active stream.

type LinkError 

type LinkError struct {
	Cim string
	Op  string
	Old string
	New string
	Err error
}

func (*LinkError) Error 

func (e *LinkError) Error() string

type MountError 

type MountError struct {
	Cim        string
	Op         string
	VolumeGUID guid.GUID
	Err        error
}

func (*MountError) Error 

func (e *MountError) Error() string

type OpError 

type OpError struct {
	Cim string
	Op  string
	Err error
}

func (*OpError) Error 

func (e *OpError) Error() string

type PathError 

type PathError struct {
	Cim  string
	Op   string
	Path string
	Err  error
}

PathError is the error type returned by most functions in this package.

func (*PathError) Error 

func (e *PathError) Error() string

Source Files

cim_writer_windows.go cimfs.go common.go doc.go mount_cim.go

Directories

PathSynopsis
pkg/cimfs/formatformat package maintains some basic structures to allows us to read header of a cim file.
Version
v0.13.0 (latest)
Published
Apr 21, 2025
Platform
windows/amd64
Imports
17 packages
Last checked
2 hours ago –

Tools for package owners.