omics_utils.Step#

class omics_utils.Step(base_dir, cluster=None, scale=None)#

Bases: ABC

Base preprocessing step in a modular ETL pipeline. Also see PreprocessingPipeline.

__init__(base_dir, cluster=None, scale=None)#

Initializes an instance of a base preprocessing step.

Parameters:
  • base_dir (str) – Folder where a subfolder is created for each preprocessing step.

  • cluster (Optional[Partial[JobQueueCluster | LocalCluster]]) – Configured cluster to use for distributed preprocessing using dask.

  • scale (int | None) – Number of workers to use in the cluster. Only relevant when cluster is given.

Methods

__init__(base_dir[, cluster, scale])

Initializes an instance of a base preprocessing step.

cluster_context()

Context manager for cluster lifecycle management.

read(*args)

Read the data from the previous step or the input data.

run()

Run the whole preprocessing step by sequentially running read, transform and write.

run_step()

Run the pipeline step.

transform(*args)

Transform the data.

write(*args)

Save the result of the transformation.

Attributes

input_step

Name of the preprocessing step preceding this one or None if this is the first.

output_step

Name of this preprocessing step.

cluster_context()#

Context manager for cluster lifecycle management.

Yields:

A dask Client connected to the configured cluster.

Return type:

Generator[Client, None, None]

abstract property input_step: str | None#

Name of the preprocessing step preceding this one or None if this is the first.

abstract property output_step: str#

Name of this preprocessing step.

abstractmethod read(*args)#

Read the data from the previous step or the input data.

Parameters:

args (Any) – The input data, e.g. file paths.

Returns:

The data ready for transformation.

Return type:

str

abstractmethod run()#

Run the whole preprocessing step by sequentially running read, transform and write.

Return type:

None

run_step()#

Run the pipeline step.

Return type:

None

abstractmethod transform(*args)#

Transform the data.

Parameters:

args (Any) – The input data.

Return type:

Any

Returns:

The transformed output.

abstractmethod write(*args)#

Save the result of the transformation.

Parameters:

args (Any) – The transformed data.

Return type:

None