omics_utils.Pipeline#

class omics_utils.Pipeline(steps, output_dir=None, force_recompute=False, cluster=None, scale=None)#

Bases: object

Handles a sequence of preprocessing steps in an ETL style pipeline.

Can be used for the preprocessing, concatenation, stratification, splitting and saving of tabular omics datasets.

__init__(steps, output_dir=None, force_recompute=False, cluster=None, scale=None)#

Initializes an instance of a preprocessing pipeline.

Parameters:
  • steps (list[Partial[Step]]) – Preprocessing steps to perform in sequence

  • output_dir (str | None) – Output path to save intermediate and final preprocessing outputs in. By default, this is the hydra output directory (“outputs”).

  • force_recompute (bool) – Whether to recompute already existing intermediate results.

  • cluster (Optional[Partial[JobQueueCluster | LocalCluster]]) – Configured cluster to use for distributed preprocessing using dask. This config will be used for all steps in the pipeline.

  • scale (int | None) – Number of jobs to start on the cluster.

Methods

__init__(steps[, output_dir, ...])

Initializes an instance of a preprocessing pipeline.

run()

Run the preprocessing pipeline on the data.

run()#

Run the preprocessing pipeline on the data.

Return type:

None