omics_utils.Partitioner#

class omics_utils.Partitioner(*args, input_dir, converter, filterer, n_outer_partitions=50, n_inner_partitions=10, n_debug=None, subset=None, **kwargs)#

Bases: Step, ABC

Partitioner for AnnData files.

__init__(*args, input_dir, converter, filterer, n_outer_partitions=50, n_inner_partitions=10, n_debug=None, subset=None, **kwargs)#
Parameters:
  • input_dir (str) – Folder where the input CellXGene AnnData files are saved.

  • converter (Partial[AnnDataConverter]) – Converter to convert AnnData into a DataFrame format.

  • filterer (Partial[AnnDataFilterer]) – Filter to apply to each individual AnnData file.

  • n_debug (int | None) – Take the first n input files only for debugging purposes.

  • n_outer_partitions (int) – Number of output partitions.

  • n_inner_partitions (int) – Number of output partitions within each outer partition.

  • subset (list[str] | None) – Subset of .h5ad file names to process.

  • only_primary – Whether to only process cells marked as primary data.

Methods

__init__(*args, input_dir, converter, filterer)

cluster_context()

Context manager for cluster lifecycle management.

read(input_files)

Read and transform the data from the the input data.

run()

Run the whole preprocessing step by sequentially running read and write.

run_step()

Run the pipeline step.

transform(file_path)

Preprocess a single AnnData file.

write(data_df)

Load/save the result of the transformation.

Attributes

input_step

Has no preceding preprocessing step.

meta_df

Empty DataFrame used by Dask to determine the output of transform.

output_step

Name of this preprocessing step.

cluster_context()#

Context manager for cluster lifecycle management.

Yields:

A dask Client connected to the configured cluster.

Return type:

Generator[Client, None, None]

property input_step: str | None#

Has no preceding preprocessing step.

property meta_df: DataFrame#

Empty DataFrame used by Dask to determine the output of transform.

property output_step: str#

Name of this preprocessing step.

read(input_files)#

Read and transform the data from the the input data.

NOTE: In this case the read step contains the transform step for compatibility with dask.

Parameters:

input_files (list[Path]) – File paths to process in parallel.

Return type:

DataFrame

Returns:

The data ready for transformation.

run()#

Run the whole preprocessing step by sequentially running read and write.

Return type:

None

run_step()#

Run the pipeline step.

Return type:

None

transform(file_path)#

Preprocess a single AnnData file.

Parameters:

file_path (Path) – Path to the AnnData file to be transformed.

Return type:

list[Delayed]

Returns:

A list of Delayed tasks for preprocessing.

write(data_df)#

Load/save the result of the transformation.

Parameters:

data_df (DataFrame) – A DataFrame containing the patient/donor metadata and the matrix X.

Return type:

None