zea.data.chunk_reader

zea.data.chunk_reader

Concurrent chunk reads for zea HDF5 files, bypassing h5py’s serial read path.

h5py reads one chunk at a time and decodes them under a global lock, so N chunks cost N decodes back to back — and, over HTTP, N round trips. But h5py does hand us the chunk manifest (get_chunk_info_by_coord: byte offset, size, filter mask), so we can fetch the compressed bytes ourselves — concurrently when remote, from the file descriptor when local — and decode them in a thread pool (Blosc and zlib release the GIL). Measured on a 201 MB read of 16 chunks: 31 ms against h5py’s 291 ms locally, 126 ms against 863 ms over HTTP.

A pure optimisation, and treated as one: anything the fast path does not fully understand (an unknown codec, a contiguous dataset, an exotic selection) falls back to dset[selection], and h5py stays the reader for everything else in the file. Wired in through ChunkedDataset, so file.data.raw_data[0:8] gets it for free.

Two details carry most of the win and are easy to lose in a refactor:

• Bytes are read with os.pread, not read_direct_chunk — that takes h5py’s global lock, serialising the fetch and copying every chunk an extra time.

• Chunks are decompressed straight into the output array. Decoding to a temporary and copying it in costs more than the decode itself (121 ms of copy against 26 ms of decode) and the copy is serial, so it caps everything.

Module Attributes

Ticker

Called with the byte size of each chunk as it arrives.

Functions

eligible(dset, fetcher)	Whether dset can be read through the fast path at all.
fetcher_for(file)	The fetcher for an open File, or None if it has none.
filter_ids(dset)	Filter ids of the dataset's pipeline, in the order HDF5 applied them on write.
read(dset, selection, fetcher[, progress])	Read selection from dset, concurrently, falling back to h5py when unsure.

Classes

Fetcher()	Source of raw (still-compressed) chunk bytes for one open file.
HTTPFetcher(url[, token, cache])	Reads chunk bytes over HTTP range requests, all of them concurrently.
LocalFetcher(path)	Reads chunk bytes from the file descriptor.

class zea.data.chunk_reader.Fetcher

Bases: object

Source of raw (still-compressed) chunk bytes for one open file.

The two backends want opposite things. A local file wants per_chunk: one chunk at a time from inside a decode worker, so the next read overlaps the last decode. HTTP wants the ranges batched into one call so they go out together — N ranges for one round trip, which a chunk-at-a-time fetch would throw away.

close()

Release whatever the fetcher holds open.

Return type:

None

fetch(ranges, on_bytes=None)

Return the bytes of each (offset, size) range, in order.

on_bytes is called with the size of each range as it arrives (for progress reporting). It runs on whichever thread completed the fetch, so it must be thread-safe.

Return type:

list[bytes]

pending_bytes(ranges)

Bytes of ranges that fetch() would actually have to stream or download.

Defaults to the full total: only HTTPFetcher can know better, by checking its on-disk cache. Used solely to decide whether a progress bar has anything to show — a read served entirely from cache does not stream or download, so it gets no bar.

Return type:

int

per_chunk = False

Whether fetching one chunk on its own is cheap (see above).

progress: bool | Callable[[int], None] | None = False

Progress reporting for reads through this file (set by File).

source: str | None = None

Human-readable file identifier (the local path or hf:// source url), used in fallback messages instead of a dataset’s internal HDF5 path (set by fetcher_for()).

class zea.data.chunk_reader.HTTPFetcher(url, token=None, cache=None)

Bases: Fetcher

Reads chunk bytes over HTTP range requests, all of them concurrently.

Deliberately fsspec’s async HTTPFileSystem and not HfFileSystem, whose cat_ranges is serial: the same 16 ranges took 2745 ms through HfFileSystem against 177 ms here — sixteen round trips against one. The whole remote win rests on this.

The ranges are issued as individual _cat_file coroutines gathered on fsspec’s event loop, rather than through cat_ranges. That is both faster and steadier — cat_ranges does its own batching and periodically stalls (measured at 20 ms/request: 32 ranges in 1.06 s against 0.06 s here) — and it is what makes per-chunk progress possible at all, since cat_ranges only returns once every range is done.

fetch(ranges, on_bytes=None)

Return the bytes of each (offset, size) range, in order.

on_bytes is called with the size of each range as it arrives (for progress reporting). It runs on whichever thread completed the fetch, so it must be thread-safe.

Return type:

list[bytes]

pending_bytes(ranges)

Bytes of ranges that fetch() would actually have to stream or download.

Defaults to the full total: only HTTPFetcher can know better, by checking its on-disk cache. Used solely to decide whether a progress bar has anything to show — a read served entirely from cache does not stream or download, so it gets no bar.

Return type:

int

class zea.data.chunk_reader.LocalFetcher(path)

Bases: Fetcher

Reads chunk bytes from the file descriptor.

os.pread is positional, so it needs no seek and no lock: reading a raw OS descriptor touches none of h5py/HDF5’s state, so the decode workers each read the descriptor themselves, overlapping I/O with decoding (31 ms against 46 ms for a 16-chunk read).

os.pread is Unix-only, though. On Windows it is absent, so we fall back to a lock-guarded lseek + read: the workers share the fd’s file position, so the reads serialise there, but decoding still overlaps across chunks. (read_direct_chunk under a lock would serve here too, but it is slower and drags in HDF5’s chunk API for no gain.)

close()

Release whatever the fetcher holds open.

fetch(ranges, on_bytes=None)

Return the bytes of each (offset, size) range, in order.

on_bytes is called with the size of each range as it arrives (for progress reporting). It runs on whichever thread completed the fetch, so it must be thread-safe.

Return type:

list[bytes]

per_chunk = True

Whether fetching one chunk on its own is cheap (see above).

zea.data.chunk_reader.Ticker

Called with the byte size of each chunk as it arrives. Runs on worker threads.

alias of Callable[[int], None] | None

zea.data.chunk_reader.eligible(dset, fetcher)

Whether dset can be read through the fast path at all.

Cheap and conservative: chunked storage, a decodable filter pipeline, a fetcher, and a plain numeric dtype (no vlen strings, no compound types — h5py handles those, and they are never the bulk arrays this exists for).

Return type:

bool

zea.data.chunk_reader.fetcher_for(file)

The fetcher for an open File, or None if it has none.

A file zea streamed from hf:// reads over HTTP; a file on disk reads from its descriptor. Anything else (an in-memory file, a driver we do not recognise) has no fast path, and its datasets fall back to h5py.

Return type:

Fetcher | None

zea.data.chunk_reader.filter_ids(dset)

Filter ids of the dataset’s pipeline, in the order HDF5 applied them on write.

Return type:

list[int]

zea.data.chunk_reader.read(dset, selection, fetcher, progress=False)

Read selection from dset, concurrently, falling back to h5py when unsure.

The contract is equality: this returns exactly what dset[selection] returns.

Any read that misses the concurrent path — for its storage layout (unchunked or a codec zea cannot decode) or its selection — logs a one-time note naming the cause and the fix, regardless of progress, since the serial fallback is slower on disk too (see _resave_note(), _selection_note()).

Parameters:

• dset (Dataset) – The dataset to read from.

• selection (Any) – Any NumPy-style index. Ints, unit-step slices and increasing index lists take the fast path; anything else is handed to h5py.

• fetcher (Fetcher | None) – Source of the chunk bytes for this file (see fetcher_for()). None disables the fast path.

• progress (Union[bool, Callable[[int], None], None]) – True shows a tqdm bar over the compressed bytes; a callable is invoked with each chunk’s size as it arrives. Reads served by h5py (an lzf file, a strided selection) report no per-chunk progress: h5py fetches the whole selection in one opaque call, so there is nothing to observe.

Returns:

The selected data.

Return type:

ndarray