omics_utils.Partitioner#
- class omics_utils.Partitioner(*args, input_dir, converter, filterer, n_outer_partitions=50, n_inner_partitions=10, n_debug=None, subset=None, **kwargs)#
Bases:
Step,ABCPartitioner for AnnData files.
- __init__(*args, input_dir, converter, filterer, n_outer_partitions=50, n_inner_partitions=10, n_debug=None, subset=None, **kwargs)#
- Parameters:
input_dir (
str) – Folder where the input CellXGene AnnData files are saved.converter (
Partial[AnnDataConverter]) – Converter to convert AnnData into a DataFrame format.filterer (
Partial[AnnDataFilterer]) – Filter to apply to each individual AnnData file.n_debug (
int|None) – Take the first n input files only for debugging purposes.n_outer_partitions (
int) – Number of output partitions.n_inner_partitions (
int) – Number of output partitions within each outer partition.subset (
list[str] |None) – Subset of .h5ad file names to process.only_primary – Whether to only process cells marked as primary data.
Methods
__init__(*args, input_dir, converter, filterer)Context manager for cluster lifecycle management.
read(input_files)Read and transform the data from the the input data.
run()Run the whole preprocessing step by sequentially running read and write.
run_step()Run the pipeline step.
transform(file_path)Preprocess a single AnnData file.
write(data_df)Load/save the result of the transformation.
Attributes
Has no preceding preprocessing step.
Empty DataFrame used by Dask to determine the output of transform.
Name of this preprocessing step.
- cluster_context()#
Context manager for cluster lifecycle management.
- Yields:
A dask Client connected to the configured cluster.
- Return type:
Generator[Client,None,None]
- property input_step: str | None#
Has no preceding preprocessing step.
- property meta_df: DataFrame#
Empty DataFrame used by Dask to determine the output of transform.
- property output_step: str#
Name of this preprocessing step.
- read(input_files)#
Read and transform the data from the the input data.
NOTE: In this case the read step contains the transform step for compatibility with dask.
- Parameters:
input_files (
list[Path]) – File paths to process in parallel.- Return type:
DataFrame- Returns:
The data ready for transformation.
- run()#
Run the whole preprocessing step by sequentially running read and write.
- Return type:
None
- run_step()#
Run the pipeline step.
- Return type:
None
- transform(file_path)#
Preprocess a single AnnData file.
- Parameters:
file_path (
Path) – Path to the AnnData file to be transformed.- Return type:
list[Delayed]- Returns:
A list of Delayed tasks for preprocessing.
- write(data_df)#
Load/save the result of the transformation.
- Parameters:
data_df (
DataFrame) – A DataFrame containing the patient/donor metadata and the matrix X.- Return type:
None