config_handler#

This module provides a class to load/write configuration files from the ClearMap configuration directory.

The configuration files are used to store the parameters for the ClearMap processing steps. Supported formats are .cfg (ConfigObj), .yml/.yaml (YAML) and .json (JSON). Other formats (TOML, INI, XML, etc.) could be supported in the future through a simple plugin function to this module.

class ConfigAlternativesRegistry[source]#

Bases: object

Registry of alternative names for configuration files.

static get_channel_sections() → tuple[str, ...][source]#

get_alternatives(cfg_name: str) → list[str][source]#

Get the list of alternative names for a given configuration name.

Parameters:: cfg_name (str) – The base name of the configuration file (without params and extension).

Returns

list[str]: A list of alternative names for the configuration file.

Raises:: ValueError – If no alternatives are found for the given configuration name.

get_spec(cfg_name: str) → ConfigSpec[source]#

Get the ConfigSpec for a given configuration name.

Parameters:: cfg_name (str) – The base name of the configuration file (without params and extension).

Returns

ConfigSpec: The configuration specification.

Raises:: KeyError – If no specification is found for the given configuration name.

is_global_cfg(cfg_name: str) → bool[source]#: Check if the given config name is a global config (i.e. machine, display, preferences).

is_legacy_cfg(cfg_name: str) → bool[source]#: Check if the given config name is a legacy config (i.e. alignment, processing).

is_local_file(cfg_name: str) → bool[source]#: Check if the given config name is a tab file (i.e. has alternatives).

pipeline_to_section_name(pipeline_name: str) → str[source]#

Convert a pipeline name to a configuration section name. E.g. ‘TubeMap’ -> ‘vasculature’

Parameters:: pipeline_name (str) – The name of the pipeline.

Returns

str: The corresponding configuration section name.

to_canonical(cfg_name)[source]#

property alternative_names: list[list[str]]#

Get all alternative names registered.

Returns

list[str]: A list of all alternative configuration names.

property canonical_config_names#

Get the canonical configuration names (first in each alternative group).

Returns

list[str]: A list of canonical configuration names.

property canonical_global_config_names#

Get the canonical configuration names for global sections.

Returns

list[str]: A list of canonical configuration names for global sections.

property canonical_group_config_names#

Get the canonical configuration names for group sections.

Returns

list[str]: A list of canonical configuration names for group sections.

property canonical_pipeline_config_names#

Get the canonical configuration names for pipeline sections.

Returns

list[str]: A list of canonical configuration names for pipeline sections.

class ConfigHandler(src_dir: Path | str)[source]#

Bases: object

Resolves logical config names to files and loads them via readers; writes using writer_functions (atomic).

Class Attributes#

loader_functions: Dict[str, Callable[[Path, bool], Optional[ConfigProxy]]]: A dictionary mapping file extensions to their corresponding loader functions.
writer_functions: Dict[str, Callable[[Path, dict], None]]: A dictionary mapping file extensions to their corresponding writer functions.
user_defaults_dir: Path: The default directory where configuration files are located.
supported_exts: tuple: A tuple of supported file extensions for configuration files.

src_dir#

The source directory where configuration files are located.

Type:: Path

classmethod dump(path: Path, data: dict) → None[source]#

Write the given data to the specified path using the appropriate writer function. The function is determined by the file extension. The write is atomic (writes to a temp file then renames).

Parameters:

path (Path) – The path to the output configuration file
data (dict) – The data to write

Raises:

ValueError – If no writer function is found for the given file extension.

classmethod get_cfg_from_path(cfg_path: str | Path) → ConfigProxy[source]#

Load a configuration file from the given path using the appropriate loader function. The function is determined by the file extension.

Warning

The file must exist.

Parameters:: cfg_path (str | Path) – The path to the configuration file. Must exist.

Returns

ConfigProxy: The configuration file as a ConfigProxy object.

classmethod get_default_cfg(cfg_name)[source]#

classmethod get_default_path(cfg_name: str, must_exist: bool = True, *, base_dir: Path | None = None, version: str | None = None, from_package: bool = False) → Path | None[source]#

Get the path to the default configuration file with the given name. Several extensions are tried in order of preference. If None is found, the first possible option is returned if must_exist is False or a FileNotFoundError is raised if must_exist is True.

Parameters:

cfg_name (str) – The name (without params and extension) of the configuration file
must_exist (bool) – Whether the file must exist. If missing and True, a FileNotFoundError is raised
base_dir (Optional[Path]) – The base directory to look for the default config file. If specified, this overrides the from_package parameter.
version (Optional[str]) – The version string in the major.minor format to use for the folder structure. If None, the current ClearMap version is used.
from_package (bool) –
If True, look for the default config in the ClearMap package directory. Otherwise, look in the user’s ~/.clearmap directory.

Warning

This only affects the default path if no base_dir is specified.

Returns

Path: The path to the default configuration file

classmethod get_global_canonical_path(cfg_name: str) → Path[source]#

classmethod get_global_path(cfg_name: str, must_exist: bool = True, *, base_dir: Path | None = None, version: str | None = None) → Path | None[source]#

Get the path to the global configuration file with the given name. Several extensions are tried in order of preference. If None is found, the first possible option is returned if must_exist is False or a FileNotFoundError is raised if must_exist is True.

Parameters:

cfg_name (str) – The name (without params and extension) of the configuration file
must_exist (bool) – Whether the file must exist. If missing and True, a FileNotFoundError is raised
base_dir (Optional[Path]) – The base directory to look for the global config file. If None, the user’s ~/.clearmap directory is used.
version (Optional[str]) – The version string in the major.minor format to use for the folder structure.

Returns

Path: The path to the global configuration file

classmethod get_new_cfg_writer(cfg_path: str | Path, *, must_not_exist: bool = True) → ConfigProxy[source]#

Create a new, empty ConfigProxy for cfg_path, without reading from disk.

Parameters:

cfg_path (str | Path) – Target config path (may or may not exist).
must_not_exist (bool) – If True, raise if the file already exists.

Returns

ConfigProxy: An empty ConfigProxy bound to cfg_path with the proper writer.

classmethod get_patched_cfg_from_path(cfg_path: str | Path) → ConfigProxy[source]#

Load a configuration file from the given path and patch it with the default configuration. The default configuration is loaded from the ClearMap package directory. The function is determined by the file extension. The file must exist.

Parameters:: cfg_path (str | Path) – The path to the configuration file. Must exist.

Returns

ConfigProxy: The patched configuration file as a ConfigProxy object.

classmethod get_user_defaults_canonical_path(cfg_name: str) → Path[source]#

Canonical path for a user default config for the current ClearMap version.

No scanning / fallback across versions
Uses current clearmap_version
Always uses the given ext (default: .yml for 3.1+)

classmethod resolve_write_path(name: str, *, base_dir: Path) → Path[source]#: Resolve the target path to write a config by name. - Tab/experiment sections -> experiment base_dir - Global sections (machine/display/preferences) -> user global directory

static is_global(cfg_name: str) → bool[source]#

static is_local(cfg_name: str) → bool[source]#

static normalise_cfg_name(cfg_name: str) → str[source]#: Normalise config name by stripping params and version suffix.

static strip_params(cfg_name: str, params_pattern: str = '_params') → str[source]#: Strip params from config name, e.g. _v2_params from sample_v2_params

static strip_version_suffix(cfg_name: str) → str[source]#: Strip version suffix from config name, e.g. _v2_3 from sample_v2_3_params

get_canonical_path(cfg_name: str) → Path[source]#

Get the canonical path to the configuration file with the given name. The path is determined based on whether the config is local (tab/experiment) or global (machine/display/preferences). The rest is determined by the current ClearMap version and layout as well as preferred extension.

Parameters:: cfg_name (str) – The name (without params and extension) of the configuration file.

Returns

Path: The canonical path to the configuration file

get_cfg(cfg_name: str, must_exist: bool = True) → ConfigProxy | None[source]#

Get the configuration file with the given name. Several extensions are tried in order of preference. If None is found, None is returned if must_exist is False or a FileNotFoundError is raised if must_exist is True.

Parameters:

cfg_name (str) – The name (without params and extension) of the configuration file
must_exist (bool) – Whether the file must exist. If missing and True, a FileNotFoundError is raised

Returns

ConfigProxy | None: The configuration file as a ConfigProxy object or None if not found and must_exist is False

get_cfg_path(cfg_name: str, must_exist: bool = True) → Path[source]#

Get the path to the configuration file with the given name. Several extensions are tried in order of preference. If None is found, the canonical path is returned if must_exist is False or a FileNotFoundError is raised if must_exist is True.

Parameters:

cfg_name (str) – The name (without params and extension) of the configuration file. If present, they will be stripped.
must_exist (bool) – Whether the file must exist. If missing and True, a FileNotFoundError is raised.

Returns

Path: The path to the configuration file

get_local_canonical_path(cfg_name: str) → Path[source]#

loader_functions: Dict[str, Callable[[Path, bool], ConfigProxy | None]] = {'.cfg': <function get_configobj_cfg>, '.ini': <function get_configobj_cfg>, '.json': <function get_json_cfg>, '.yaml': <function get_yml_cfg>, '.yml': <function get_yml_cfg>}#

property src_dir#

supported_exts = ('.yml', '.yaml', '.cfg', '.ini', '.json')#

user_defaults_dir = PosixPath('/home/charly.rousseau/.clearmap')#

user_global_dir = PosixPath('/home/charly.rousseau/.clearmap/config')#

writer_functions: Dict[str, Callable[[Path, dict], None]] = {'.cfg': <function to_configobj>, '.ini': <function to_configobj>, '.json': <function to_json>, '.yaml': <function to_yml>, '.yml': <function to_yml>}#

class ConfigProxy(filename: str, _loader: Callable[[Path], ConfigProxy], _dumper: Callable[[Path, dict], None])[source]#

Bases: dict

Dict-like wrapper that remembers filename and provides write()/reload(). Used to return from get_*_cfg functions. This is basically just a dict with I/O methods attached. Compatible with ConfigObj interface for yml/json/cfg files.

reload() → ConfigProxy[source]#

write(outfile: Any | None = None) → None[source]#: Backwards-compatible signature. If outfile is given (a file-like), we still write to our filename atomically, ignoring outfile (kept for compatibility).

filename: str#

class ConfigSpec(names: tuple[str, ...], scope: ClearMap.config.config_handler.Scope, subdir: str = '')[source]#

Bases: object

names: tuple[str, ...]#

scope: Scope#

subdir: str = ''#

class FlowList(iterable=(), /)[source]#

Bases: list

Marker type for inline (flow-style) sequences.

class Scope(value)[source]#

Bases: Enum

EXPERIMENT = 'experiment'#

GLOBAL = 'global'#

GROUP = 'group'#

flatten_list(groups: list[list[str]]) → list[str][source]#: Flatten a list of lists of item groups into a single list.

get_cfg_reader_function(cfg_path: Path | str) → Callable[[Path, bool], ConfigProxy | None][source]#

Get the appropriate configuration reader function for the given file path. The function is determined by the file extension.

Parameters:: cfg_path (Path | str) – The path to the configuration file.

Returns

Callable[[Path, bool], Optional[ConfigProxy]]: The configuration reader function.

get_configobj_cfg(cfg_path: Path | str, must_exist: bool = True) → ConfigProxy | None[source]#

Load a .cfg/.ini file using ConfigObj and return as ConfigProxy.

Parameters:

cfg_path (Path | str) – Path to the .cfg/.ini file
must_exist (bool) – If True, raise an error if the file does not exist.

Returns

ConfigProxy | None: The configuration as a ConfigProxy object, or None if the file does not exist and must_exist is False.

Raises:: ConfigParsingError – If the file cannot be parsed.

get_configs(cfg_path: str | Path, processing_params_path: str | Path, machine_cfg_path: str | Path | None = None)[source]#

Get the machine, sample and processing configurations from the given paths. If machine_cfg_path is None, the default machine config path is used.

Parameters:

cfg_path (str | Path) – Path to the sample configuration file
processing_params_path (str | Path) – Path to the processing parameters configuration file
machine_cfg_path (str | Path | None) – Path to the machine configuration file. If None, the default machine config path is used.

Returns

Tuple[ConfigProxy, ConfigProxy, ConfigProxy]: A tuple containing the machine, sample and processing configurations as ConfigProxy objects.

get_json_cfg(cfg_path: Path, must_exist: bool = True) → ConfigProxy | None[source]#

Load a .json file using the json module and return as ConfigProxy.

Parameters:

cfg_path (Path) – Path to the .json file
must_exist (bool) – If True, raise an error if the file does not exist.

Returns

ConfigProxy | None: The configuration as a ConfigProxy object, or None if the file does not exist and must_exist is False.

get_yml_cfg(cfg_path: Path, must_exist: bool = True) → ConfigProxy | None[source]#

Load a .yml/.yaml file using PyYAML and return as ConfigProxy.

Parameters:

cfg_path (Path) – Path to the .yml/.yaml file
must_exist (bool) – If True, raise an error if the file does not exist.

Returns

ConfigProxy | None: The configuration as a ConfigProxy object, or None if the file does not exist and must_exist is False.

is_all_scalars(seq)[source]#

mark_inline_sequences(obj, max_items=3)[source]#: Inline short sequences of scalars; convert tuples to lists (portable YAML).

patch_cfg(cfg, default_cfg)[source]#: Recursively patch cfg with missing keys from default_cfg. :param cfg: The configuration to patch. :type cfg: dict-like :param default_cfg: The default configuration to use for patching. :type default_cfg: dict-like

prioritize_top_keys(obj, first=('clearmap_version', 'clearmap_schema'))[source]#: Return a shallowly-reordered top-level mapping.

represent_flow_list(dumper, data)[source]#

scan_folder_for_experiments(folder: Path | str, *, exclude_dir_names: Sequence[str] = ('configs_backup', 'config_snapshots')) → Set[Path][source]#

Discover experiment folders under folder by scanning recursively for sample config files.

A folder is considered an experiment root if it contains a config file whose base name matches one of the ‘sample’ alternatives and whose extension is supported by ConfigHandler.

Excludes any match located under directories listed in exclude_dir_names.

to_configobj(path: Path, data: dict) → None[source]#

Write a dict to a .cfg/.ini file using ConfigObj.

Parameters:

path (Path) – Path to the output .cfg/.ini file
data (dict) – Data to write

to_json(path: Path, data: dict) → None[source]#

Write a dict to a .json file using the json module.

Parameters:

path (Path) – Path to the output .json file
data (dict) – Data to write

to_yml(path: Path, data: dict) → None[source]#

Write a dict to a .yml/.yaml file using PyYAML.

Parameters:

path (Path) – Path to the output .yml/.yaml file
data (dict) – Data to write

Raises:

RuntimeError – If PyYAML is not installed.