omics_utils.Limiter#
- class omics_utils.Limiter(*args, **kwargs)#
Bases:
StepCalculates the size of the output partitions for compatibility with the cellarium-ml dataloader.
- __init__(*args, **kwargs)#
Initializes an instance of a base preprocessing step.
- Parameters:
base_dir – Folder where a subfolder is created for each preprocessing step.
cluster – Configured cluster to use for distributed preprocessing using dask.
scale – Number of workers to use in the cluster. Only relevant when cluster is given.
Methods
__init__(*args, **kwargs)Initializes an instance of a base preprocessing step.
Context manager for cluster lifecycle management.
read()Get the file paths from the previous step.
run()Calculate the partition limits.
run_step()Run the pipeline step.
transform(input_files)Compute the partition sizes for each fold from the stratification step.
write(limits)Save the computed limits to json.
Attributes
Name of the preceding preprocessing step.
Name of this preprocessing step.
- cluster_context()#
Context manager for cluster lifecycle management.
- Yields:
A dask Client connected to the configured cluster.
- Return type:
Generator[Client,None,None]
- property input_step: str#
Name of the preceding preprocessing step.
- property output_step: str#
Name of this preprocessing step.
- read()#
Get the file paths from the previous step.
- Return type:
list[str]- Returns:
List of file paths to process.
- run()#
Calculate the partition limits.
- Return type:
None
- run_step()#
Run the pipeline step.
- Return type:
None
- transform(input_files)#
Compute the partition sizes for each fold from the stratification step.
- Return type:
dict[int,int]- Returns:
Dictionary containing the file limits for each fold.
- write(limits)#
Save the computed limits to json.
- Parameters:
limits (
dict) – Dictionary containing the file limits.- Return type:
None