zea.Dataset

class zea.Dataset(file_paths, validate=True, directory_splits=None, revision=None, lazy=True, file_filter=None, _suggest_lazy=True, **kwargs)

Bases: H5FileHandleCache

Dataset class managing multiple File objects.

Initializes the Dataset.

Parameters:

• file_paths (Union[Sequence[str | Path], str, Path]) – (list of) path(s) to the folder(s) containing the HDF5 file(s) or list of HDF5 file paths. Can be a mixed list of folders and files.

• validate (bool) – Whether to validate the discovered files against the zea format, raising if any of them is not a valid zea file. Defaults to True. The verdict is cached, so only the first call over a given dataset opens every file. Lazy hf:// files are validated when first opened instead.

• directory_splits (list | None) – List of directory split by. Is a list of floats between 0 and 1, with the same length as the number of file_paths given. If none, all files in file_paths are used.

• revision (str | None) – HuggingFace revision (branch, tag, or commit hash). Only used when file_paths contains hf:// paths. Defaults to None (uses HuggingFace Hub default, i.e. the main branch).

• lazy (bool) – If True, hf:// files are not downloaded at init — each file is downloaded on first access. len(ds) returns the number of files (not total frames). Defaults to True.

• file_filter (Union[Callable[[File], bool], dict, None]) – Keep only files whose content matches a predicate. Either a callable File -> bool (a file is kept when it returns True), or a declarative dotted-path dict mapping a path on the File (e.g. "metadata.subject.fat_percentage", "scan.center_frequency") to a condition: the EXISTS() helper (field must be present), a plain value (equality), or a callable on the resolved value. All dict entries are ANDed. Files whose predicate raises (e.g. no metadata group) are excluded, except that I/O errors (OSError from HDF5/network reads) propagate rather than being treated as a mismatch. Evaluating the predicate reads each file; with lazy=True remote (hf://) files are streamed for just the metadata/scan bytes rather than downloaded in full, and the surviving files stay lazy. Defaults to None (no filtering).

__call__()

Call self as a function.

copy(to_path, key, mode=None)

Copy the data for all or a specific key to a new location.

Works for a dataset built from a single file, a list of files, or a folder. Each file is written under to_path, mirroring its location relative to the common parent directory of the dataset’s files (for a single file, its own name is used).

Has the option to copy all keys or only a specific key. By default, it only copies if the destination file does not already contain the key. You can change the mode to ‘w’ to overwrite the destination file. Will always copy metadata such as dataset attributes and scan object.

Parameters:

• to_path (str | Path) – The destination folder where files will be copied.

• key (str) – The key to copy from the source files. If ‘all’ or ‘*’, all keys will be copied.

• mode (str | None) – The mode in which to open the destination files. Defaults to ‘a’ (append mode), and ‘w’ (write mode) if key is ‘all’ or ‘*’. See: https://docs.h5py.org/en/stable/high/file.html#opening-creating-files

find_files(paths)

Expand folders, hf:// paths and single files into a flat list of files.

Return type:

List[str]

classmethod from_config(path, user=None, **kwargs)

Creates a Dataset from a config file.

Return type:

Self

load_file_shapes(key, metadata_paths=())

Load the shapes of the datasets in each file.

Parameters:

• key (str) – HDF5 dataset key whose shape to read.

• metadata_paths (Sequence[str]) – Dotted metadata paths to check for while each file is open. Default is () (no check).

Returns:

(file_shapes, metadata_gaps, metadata_signatures). metadata_gaps maps a file path to the requested paths that file lacks – only files missing something appear, so an empty dict means every file can answer. metadata_signatures maps every file path to its {leaf path: shape} mapping, for checking that the files agree well enough to be batched together.

Return type:

tuple

property n_files

Return number of files in dataset.

remote_bytes

Bytes held by the files this dataset streams rather than downloads. Filled by find_files from the repo listing, so it costs nothing extra to know.

property total_frames

Return total number of frames in dataset.