zea.datapaths

Utility functions for handling local data paths.

This module provides utilities for managing local and remote data paths in zea projects. It supports user- and machine-specific configuration via a users.yaml file, allowing dynamic resolution of data roots for portable and reproducible workflows.

The main entry point is set_data_paths(), which resolves the data_root and output paths for the current user and machine. format_data_path() then turns the relative paths used in zea configs into absolute ones. To set up a users.yaml interactively, run zea datapaths (see create_new_user()).

See the notebook Using local data paths with zea for an extensive example of how to set up your local data paths.

Example usage

>>> import yaml
>>> from zea.datapaths import set_data_paths

>>> user_config = {"data_root": "/path/to/data", "output": "/path/to/output"}
>>> with open("users.yaml", "w", encoding="utf-8") as file:
...     yaml.dump(user_config, file)

>>> user = set_data_paths("users.yaml")
>>> print(user.data_root)
/path/to/data

Functions

create_new_user([user_config_path, local])	Creates a new user profile in users.yaml if one does not already exist.
format_data_path(path[, user])	Resolve a dataset path against the user's data_root.
set_data_paths([user_config, local, verify])	Get data paths (absolute paths to location of data).

Exceptions

NoYamlFileError	Raised when the users.yaml file is not found.
UnknownHostnameWarning	Custom Warning indicating that the hostname was not found for this user in the user.yaml file
UnknownLocalRemoteWarning(message[, field])	Custom Warning indicating that the path corresponding to the local or remote key was not found in the user.yaml file
UnknownUsernameWarning	Custom Warning indicating that the username was not found in the user.yaml file

exception zea.datapaths.NoYamlFileError

Bases: Warning

Raised when the users.yaml file is not found.

exception zea.datapaths.UnknownHostnameWarning

Bases: UserWarning

Custom Warning indicating that the hostname was not found for this user in the user.yaml file

exception zea.datapaths.UnknownLocalRemoteWarning(message, field=None)

Bases: UserWarning

Custom Warning indicating that the path corresponding to the local or remote key was not found in the user.yaml file

Carries the field it was raised for (data_root or output), so that create_new_user() fills in the key that is actually missing.

exception zea.datapaths.UnknownUsernameWarning

Bases: UserWarning

Custom Warning indicating that the username was not found in the user.yaml file

zea.datapaths.create_new_user(user_config_path=None, local=None)

Creates a new user profile in users.yaml if one does not already exist.

Parameters:

• user_config_path (str | Path | None) – Path that points to yaml file with user info. Defaults to None. In that case ./users.yaml is taken.

• local (bool | None) – Use the local dataset, or the one at the remote location. Per machine, the data_root can be set to a local or remote path. Each user can also have a different data_root for each machine. Default is None, which means that the data_root is shared for either local or remote (i.e. this parameter is ignored), see doc set_data_paths().

Returns:

The data paths for the (possibly newly created) user profile,

as returned by set_data_paths().

Return type:

Config

zea.datapaths.format_data_path(path, user=None)

Resolve a dataset path against the user’s data_root.

Absolute paths and hf:// paths are returned as-is, relative paths are interpreted relative to user.data_root.

Parameters:

• path (str | Path | HFPath) – Path to the dataset. Can be absolute, relative to the user’s data_root, or a hf:// path.

• user (Config | None) – User config as returned by set_data_paths(). Only required for relative paths.

Returns:

The resolved path. An HFPath is returned for hf:// paths, a pathlib.Path otherwise.

Return type:

Path | HFPath

Raises:

AssertionError – If path is relative and no user is provided.

zea.datapaths.set_data_paths(user_config=None, local=True, verify=True)

Get data paths (absolute paths to location of data).

Parameters:

• user_config (Union[str, dict, None]) – Path to a YAML file with user info. If None, uses ./users.yaml as the default file. Can also be a dictionary structured as shown below.

• local (bool | None) – Whether to pick the local or the remote path for entries that define both (e.g. a local disk and a remote share). Default is True. Set to None when every path that applies is a plain string, i.e. shared between local and remote.

• verify (bool) – Verify that the paths exist and are directories. Default is True.

Example YAML structure:

data_root: ...
output: ...

You can also specify different data_root for different users and machines:

my_username:
  my_hostname:
    system: windows
    data_root: ...
    output: ...
  other_hostname:
    system: linux
    data_root:
      local: ...
      remote: ...
  # If both my_hostname and other_hostname are not matching, fallback to:
  system: linux
  data_root: ...

other_username:
  data_root: ...

The machine section takes precedence over the user section, which takes precedence over the userless and machineless one at the bottom. Precedence is per key, so a section only needs to set what it changes: a machine that pins just system still inherits data_root and output from the levels above it.

Returns:

Absolute paths to location of data. Stores the following parameters:

data_root, zea_root, output, system, username, hostname

Return type:

Config

Raises:

ValueError – If user_config is not a string, dictionary or None.

Note

When user_config points to a YAML file that does not exist yet, an empty one is created and – when running interactively – you are offered to set up a profile, see create_new_user(). When no data_root can be resolved for the current user and machine, a warning is raised and a default path for the current operating system is used. output is optional and stays None when the file does not set one.