omics_utils.Limiter#

class omics_utils.Limiter(*args, **kwargs)#

Bases: Step

Calculates the size of the output partitions for compatibility with the cellarium-ml dataloader.

__init__(*args, **kwargs)#

Initializes an instance of a base preprocessing step.

Parameters:
  • base_dir – Folder where a subfolder is created for each preprocessing step.

  • cluster – Configured cluster to use for distributed preprocessing using dask.

  • scale – Number of workers to use in the cluster. Only relevant when cluster is given.

Methods

__init__(*args, **kwargs)

Initializes an instance of a base preprocessing step.

cluster_context()

Context manager for cluster lifecycle management.

read()

Get the file paths from the previous step.

run()

Calculate the partition limits.

run_step()

Run the pipeline step.

transform(input_files)

Compute the partition sizes for each fold from the stratification step.

write(limits)

Save the computed limits to json.

Attributes

input_step

Name of the preceding preprocessing step.

output_step

Name of this preprocessing step.

cluster_context()#

Context manager for cluster lifecycle management.

Yields:

A dask Client connected to the configured cluster.

Return type:

Generator[Client, None, None]

property input_step: str#

Name of the preceding preprocessing step.

property output_step: str#

Name of this preprocessing step.

read()#

Get the file paths from the previous step.

Return type:

list[str]

Returns:

List of file paths to process.

run()#

Calculate the partition limits.

Return type:

None

run_step()#

Run the pipeline step.

Return type:

None

transform(input_files)#

Compute the partition sizes for each fold from the stratification step.

Return type:

dict[int, int]

Returns:

Dictionary containing the file limits for each fold.

write(limits)#

Save the computed limits to json.

Parameters:

limits (dict) – Dictionary containing the file limits.

Return type:

None