omics_utils.Step#
- class omics_utils.Step(base_dir, cluster=None, scale=None)#
Bases:
ABCBase preprocessing step in a modular ETL pipeline. Also see PreprocessingPipeline.
- __init__(base_dir, cluster=None, scale=None)#
Initializes an instance of a base preprocessing step.
- Parameters:
base_dir (
str) – Folder where a subfolder is created for each preprocessing step.cluster (
Optional[Partial[JobQueueCluster|LocalCluster]]) – Configured cluster to use for distributed preprocessing using dask.scale (
int|None) – Number of workers to use in the cluster. Only relevant when cluster is given.
Methods
__init__(base_dir[, cluster, scale])Initializes an instance of a base preprocessing step.
Context manager for cluster lifecycle management.
read(*args)Read the data from the previous step or the input data.
run()Run the whole preprocessing step by sequentially running read, transform and write.
run_step()Run the pipeline step.
transform(*args)Transform the data.
write(*args)Save the result of the transformation.
Attributes
Name of the preprocessing step preceding this one or None if this is the first.
Name of this preprocessing step.
- cluster_context()#
Context manager for cluster lifecycle management.
- Yields:
A dask Client connected to the configured cluster.
- Return type:
Generator[Client,None,None]
- abstract property input_step: str | None#
Name of the preprocessing step preceding this one or None if this is the first.
- abstract property output_step: str#
Name of this preprocessing step.
- abstractmethod read(*args)#
Read the data from the previous step or the input data.
- Parameters:
args (
Any) – The input data, e.g. file paths.- Returns:
The data ready for transformation.
- Return type:
str
- abstractmethod run()#
Run the whole preprocessing step by sequentially running read, transform and write.
- Return type:
None
- run_step()#
Run the pipeline step.
- Return type:
None
- abstractmethod transform(*args)#
Transform the data.
- Parameters:
args (
Any) – The input data.- Return type:
Any- Returns:
The transformed output.
- abstractmethod write(*args)#
Save the result of the transformation.
- Parameters:
args (
Any) – The transformed data.- Return type:
None