zea.data.datasets

zea.data.datasets

This module provides classes and utilities for loading, validating, and managing ultrasound datasets stored in HDF5 format. It supports both local and Hugging Face Hub datasets, and offers efficient file handle caching for large collections of files.

Main Classes

• H5FileHandleCache: Caches open HDF5 file handles to optimize repeated access.

• Folder: Represents a group of HDF5 files in a directory, with optional validation.

• Dataset: Provides an iterable interface over multiple HDF5 files or folders, with

  support for directory-based splitting and validation.

Functions

• split_files_by_directory: Splits files among directories according to specified ratios.

• count_samples_per_directory: Counts the number of files per directory.

Features

• Validation of dataset integrity with flag files and error logging.

• Support for Hugging Face Hub datasets with local caching.

• Utilities for dataset splitting and sample counting.

• Example usage provided in the module’s main block.

Functions

EXISTS(value)	Condition for compile_file_filter() dict filters: field is present.
compile_file_filter(file_filter)	Normalize file_filter into a File -> bool predicate (or None).
count_samples_per_directory(file_names, ...)	Count number of samples per directory.
split_files_by_directory(file_names, ...)	Split files according to their parent directories and given split ratios.

Classes

Dataset(file_paths[, validate, ...])	Dataset class managing multiple File objects.
Folder(folder_path[, validate, ...])	Group of HDF5 files in a folder that can be validated.
H5FileHandleCache([file_handle_cache_capacity])	Cache for HDF5 file handles.

zea.data.datasets.EXISTS(value)

Condition for compile_file_filter() dict filters: field is present.

Use as the condition for a dotted path to keep only files where that field is present (i.e. resolves to a non-None value). For example {"metadata.subject.fat_percentage": EXISTS} keeps only files that record a subject fat percentage.

It is a plain callable (so it survives module reloads and needs no special casing), evaluated like any other value-level predicate.

Return type:

bool

class zea.data.datasets.Folder(folder_path, validate=False, hf_cache_dir=PosixPath('/home/docs/.cache/zea/huggingface/datasets'), revision=None)

Bases: object

Group of HDF5 files in a folder that can be validated. Mostly used internally, you might want to use the Dataset class instead.

copy(to_path, key, mode=None)

Copy the data for all or a specific key to a new location.

Has the option to copy all keys or only a specific key. By default, it only copies if the destination file does not already contain the key. You can change the mode to ‘w’ to overwrite the destination file. Will always copy metadata such as dataset attributes and scan object.

Parameters:

• to_path (str | Path) – The destination path where files will be copied.

• key (str) – The key to copy from the source files. If ‘all’ or ‘*’, all keys will be copied.

• mode (str | None) – The mode in which to open the destination files. Defaults to ‘a’ (append mode), and ‘w’ (write mode) if key is ‘all’ or ‘*’. See: https://docs.h5py.org/en/stable/high/file.html#opening-creating-files

find_h5_files()

Return type:

List[str]

static get_data_types(file_path)

Get data types from file.

load_file_shapes(key)

Load the shapes of the datasets in each file.

property n_files

Return number of files in dataset.

validate_folder()

Validate dataset contents.

If a validation file exists, it checks if the dataset was validated on the same date. If the validation file was corrupted, it raises an error. If the validation file was not corrupted and validated, it prints a message and returns.

class zea.data.datasets.H5FileHandleCache(file_handle_cache_capacity=128)

Bases: object

Cache for HDF5 file handles.

This class manages a cache of HDF5 file handles to avoid reopening files multiple times. It uses an OrderedDict to maintain the order of file access and closes the least recently used file when the cache reaches its capacity.

close()

Close all cached file handles.

get_file(file_path)

Open an HDF5 file and cache it.

Return type:

File

pop(file_path)

Pop a file from the cache and close it.

zea.data.datasets.compile_file_filter(file_filter)

Normalize file_filter into a File -> bool predicate (or None).

Accepts either a callable predicate over an open File, or a declarative dotted-path dict (see _compile_dict_filter()). Returns None when file_filter is None (no filtering).

Return type:

Optional[Callable[[File], bool]]

zea.data.datasets.count_samples_per_directory(file_names, directories)

Count number of samples per directory.

Parameters:

• file_names (list) – List of file paths

• directories (str or list) – Directory or list of directories

Returns:

Dictionary with directory paths as keys and sample counts as values

Return type:

dict

zea.data.datasets.split_files_by_directory(file_names, directory_list, directory_splits)

Split files according to their parent directories and given split ratios.

Parameters:

• file_names (list) – List of file paths.

• directory_list (list) – List of directory paths to split by.

• directory_splits (list) – List of split ratios (0-1) for each directory.

Returns:

(split_file_names, split_file_shapes)

Return type:

tuple