Readme#

🧬 omics-utils#

A modern toolkit for handling larger-than-memory omics datasets using hydra-zen and dask on a Slurm cluster with Docker/Apptainer containerization.

Python 3.12 Docker Hydra Zen Dask

🌟 Key Features#

  • πŸ§ͺ Concatenation of multiple sub-datasets/cohorts into one larger dataset

  • πŸ”€ Assigning train/val dataset splits and shuffling

  • πŸ’Ύ Sharding of the folds across multiple smaller-than-memory files

  • πŸš€ Dask + Slurm integration for efficient parallelization

  • 🧩 Reproducibility and modular type-checked configs via hydra-zen

πŸ“‹ Table of Contents#

πŸ”‘ Container Registry Authentication#

Generate Token#

  1. Create a new GitHub token at Settings β†’ Developer settings β†’ Personal access tokens with:

    • read:packages permission

    • write:packages permission

Log In#

With Apptainer:

apptainer remote login --username <your GitHub username> docker://ghcr.io

With Docker:

docker login ghcr.io -u <your GitHub username>

When prompted, enter your token as the password.

🐳 Container Setup#

Choose one of the following methods to set up your environment:

Option 1: Apptainer (Slurm Cluster)#

  1. Install VSCode Remote Tunnels Extension

    First, install the Remote Tunnels extension in VSCode.

  2. Connect to compute resources

    For CPU resources:

    srun --partition=cpu-2h --pty bash
    

    For GPU resources:

    srun --partition=gpu-2h --gpus-per-task=1 --pty bash
    
  3. Launch container

    To open a tunnel to connect your local VSCode to the container on the cluster:

    apptainer run --nv --writable-tmpfs oras://ghcr.io/marvinsxtr/omics-utils:latest-sif code tunnel
    

    πŸ’‘ You can specify a version tag (e.g., v0.0.1) instead of latest. Available versions are listed at GitHub Container Registry.

    In VSCode press Shift+Alt+P (Windows/Linux) or Shift+Cmd+P (Mac), type β€œconnect to tunnel”, select GitHub and select your named node on the cluster. Your IDE is now connected to the cluster.

    To open a shell in the container on the cluster:

    apptainer run --nv --writable-tmpfs oras://ghcr.io/marvinsxtr/omics-utils:latest-sif /bin/bash
    

    πŸ’‘ This may take a few minutes on the first run as the container image is downloaded.

Option 2: Docker (Local Machine)#

  1. Install VSCode Dev Containers Extension

    First, install the Dev Containers extension in VSCode.

  2. Open the Repository in the Dev Container

    Click the Reopen in Container button in the pop-up that appears once you open the repository in VSCode.

    Alternatively, open the command palette in VSCode by pressing Shift+Alt+P (Windows/Linux) or Shift+Cmd+P (Mac), and type Dev Containers: Reopen in Container.

Using Slurm within Apptainer#

In order to access Slurm with submitit from within the container, you first need to set up passwordless SSH to the login node.

On the cluster, create a new SSH key pair in case you don’t have one yet:

ssh-keygen -t ed25519 -C "your_email@example.com"

and add your public key to the authorized_keys:

cat ~/.ssh/id_ed25519.pub >> ~/.ssh/authorized_keys

You can verify that this works by running

ssh $USER@$HOST exit

which should return without any prompt.

🧬 Running Preprocessing#

Make sure you followed the previous sections beforehand and run these commands in the container.

The preprocessing pipeline supports multiple omics datasets:

CellXGene Preprocessing#

Before running the CellXGene preprocessing, download the h5ad files from AWS into a folder as described in the documentation and set DATA_DIR to point to this folder.

The current version was tested on the 2024-07-01 release of CellXGene.

python omics_utils/preprocess.py pipeline=cellxgene

HLCA Preprocessing#

python omics_utils/preprocess.py pipeline=hlca

πŸ“¦ Package Management#

  1. Update dependencies

    This project uses uv for Python dependency management.

    Inside the container (!):

    # Add a specific package
    uv add <package-name>
    
    # Update all dependencies from pyproject.toml
    uv sync
    
  2. Commit changes to the repository:

    Use tags for versioning:

    git add pyproject.toml uv.lock
    git commit -m "Updated dependencies"
    git tag v0.0.1
    git push && git push --tags
    
  3. Use the updated image:

    The GitHub Actions workflow automatically builds a new image when changes are pushed.

    With Apptainer:

    apptainer run --nv --writable-tmpfs oras://ghcr.io/marvinsxtr/omics-utils:v0.0.1-sif /bin/bash
    

    With Docker:

    docker run -it --rm --platform=linux/amd64 ghcr.io/marvinsxtr/omics-utils:v0.0.1 /bin/bash
    

πŸ“š Documentation#

To build the documentation locally, run the following command in your container:

uv run sphinx-build docs/source docs/build

πŸ‘₯ Contributions#

Contributions to this project are very welcome! Feel free to open a PR or reach out with suggestions.