zea.data.chunk_cache

zea.data.chunk_cache

On-disk cache of compressed HDF5 chunks fetched over the network.

Streaming re-fetches the same bytes every time: reading the same 5 frames of an hf:// file three times costs the same ~0.9 s three times. The chunks are immutable and we already fetch at chunk granularity, so they cache exactly.

Keyed by the file’s content hash (HF hands us lfs.sha256 on open, for free) rather than its URL, because an hf:// resolve URL points at a mutable ref: re-uploading a file changes what /resolve/main/ returns while the URL stays the same. Keying on content means a re-upload simply misses, instead of silently serving stale bytes.

Compressed bytes are stored, not decoded arrays: it is the network that costs, and decoding is fast and parallel (see zea.data.chunk_reader).

Module Attributes

ENABLED	Cache is enabled unless ZEA_CHUNK_CACHE=0.
MAX_BYTES	Byte budget for the whole cache.
PRUNE_EVERY	Prune this often (in writes) rather than on every one — the check has to stat the tree.

Functions

cache_for(details[, root])	A cache for the file described by details (an fsspec info mapping).
clear([root])	Remove every cached chunk.
prune([root, max_bytes])	Delete least-recently-used chunks until the cache fits in max_bytes.

Classes

CachedFile(fileobj, cache[, block_size])	A read-only file object that serves h5py's metadata reads from the chunk cache.
ChunkCache(content_id[, root])	Compressed chunks of one file, on disk, addressed by (offset, size).

class zea.data.chunk_cache.CachedFile(fileobj, cache, block_size=None)

Bases: object

A read-only file object that serves h5py’s metadata reads from the chunk cache.

Opening a streamed file re-reads its HDF5 metadata (superblock, group and chunk B-trees) over the network every time — and HF’s CDN intermittently stalls ~3 s on the first range request to a cold object, which is then most of the cost of an open. Those metadata bytes are as cacheable as the chunks are, so they go in the same cache.

Reads are served in aligned blocks, since h5py’s metadata reads are many and small; a block is the unit that gets stored. Chunks and blocks share one keyspace safely: a key is (offset, size) into an immutable file, so the same key always means the same bytes.

block_size defaults to the streamed file object’s own block size, so a cached open and an uncached one fetch the same ranges — a smaller block here would split each of the underlying reads into several requests.

close()

Return type:

None

property closed: bool

details

this is where the chunk fetcher reads the file’s content hash from.

Type:

Passed through

read(length=-1)

Return type:

bytes

readable()

Return type:

bool

seek(offset, whence=0)

Return type:

int

seekable()

Return type:

bool

tell()

Return type:

int

writable()

Return type:

bool

class zea.data.chunk_cache.ChunkCache(content_id, root=None)

Bases: object

Compressed chunks of one file, on disk, addressed by (offset, size).

get(offset, size)

The cached bytes of this chunk, or None on a miss.

Return type:

bytes | None

put(offset, size, data)

Store this chunk. Failure to cache is never an error — it just costs a re-fetch.

Return type:

None

zea.data.chunk_cache.ENABLED = True

Cache is enabled unless ZEA_CHUNK_CACHE=0.

zea.data.chunk_cache.MAX_BYTES = 10737418240

Byte budget for the whole cache. Overrun is pruned oldest-first (by access time).

zea.data.chunk_cache.PRUNE_EVERY = 64

Prune this often (in writes) rather than on every one — the check has to stat the tree.

zea.data.chunk_cache.cache_for(details, root=None)

A cache for the file described by details (an fsspec info mapping).

None when caching is off, or when nothing in details identifies the content. A weaker key (the path, say) would go stale on re-upload, and a stale chunk cache is worse than no chunk cache: it corrupts reads silently.

Return type:

ChunkCache | None

zea.data.chunk_cache.clear(root=None)

Remove every cached chunk.

Return type:

None

zea.data.chunk_cache.prune(root=None, max_bytes=10737418240)

Delete least-recently-used chunks until the cache fits in max_bytes.

Returns the bytes freed. Uses access time, so chunks a training run keeps re-reading survive while one-off reads age out.

Return type:

int