Build branch openpipeline_spatial/niche-compass with version niche-compass to openpipeline_spatial on branch niche-compass (0c1677b)

Build pipeline: openpipelines-bio.openpipeline-spatial.niche-compass-z8ftz

Source commit: 0c1677bb93

Source message: trigger ci

.gitignore

@@ -0,0 +1,63 @@
+# IDEs and editors
+/.idea
+.project
+.classpath
+*.launch
+.settings/
+.vscode
+
+# Temp
+gitignore
+test_results
+
+# System Files
+.DS_Store
+Thumbs.db
+
+# Nextflow
+work
+.nextflow*
+
+# viash
+check_results/
+out/
+output*
+output_log/
+resources_test
+/viash_tools/
+/test/
+
+# jupyter notebook
+/.ipynb_checkpoints/
+*.ipynb
+
+# compress
+/__MACOSX/
+
+# python
+*__pycache__*
+
+# Python virtual environments
+.venv
+
+# temporary files related
+temp
+
+# NextFlow
+work/
+.nextflow.log
+.nextflow*
+out/ 
+trace*.txt
+
+# Macos
+.DS_Store
+
+# vscode
+.vscode/launch.json
+.vscode/settings.json
+
+# linting
+renv.lock
+.Rprofile
+renv/

.pre-commit-config.yaml

@@ -0,0 +1,24 @@
+
+repos:
+- repo: https://github.com/astral-sh/ruff-pre-commit
+  # Ruff version.
+  rev: v0.12.1
+  hooks:
+    - id: ruff-check
+      args: [ --fix ]
+    - id: ruff-format
+- repo: local
+  hooks:
+  - id: run_styler
+    name: run_styler
+    language: r
+    description: style files with {styler}
+    entry: "Rscript -e 'styler::style_file(commandArgs(TRUE))'"
+    files: '(\.[rR]profile|\.[rR]|\.[rR]md|\.[rR]nw|\.[qQ]md)$'
+    additional_dependencies: 
+      - styler
+      - knitr
+- repo: https://github.com/lorenzwalthert/precommit
+  rev: v0.4.3.9012 
+  hooks:
+    - id: lintr

CHANGELOG.md

@@ -0,0 +1,45 @@
+# openpipeline_spatial x.x.x
+
+## MINOR CHANGES
+
+* Add `scope` to component and workflow configurations (PR #22).
+
+* Bump version of spatialdata-io to 0.3.0 and spatialdata to 0.5.0. Pin version of pyarrow to 18.0.0 for compatibility (PR #24).
+
+## BUG FIXES
+
+* `convert/from_cosmx_to_h5mu`: Fixed an issue where parent directories of the cosmx output bundle were duplicated when reading in data (PR #25).
+
+# openpipeline_spatial 0.1.1
+
+## MINOR CHANGES
+
+* Add a README (PR #21).
+
+## NEW FUNCTIONALITY
+
+* `convert`: Updated multiple components to accept spatial output bundles in .zip format (for CosMx, Xenium and Aviti) as input (PR #19, PR #20).
+
+* `convert/from_cosmx_to_h5mu`: Updated component to handle CosMx output bundles generated with AtoMx SIP versions < v1.3.2 (PR #25).
+
+# openpipeline_spatial 0.1.0
+
+## NEW FUNCTIONALITY
+
+* `filter/subset_cosmx`: Added a component to subset COSMX data (PR #3, PR #9).
+
+* `convert/from_cosmx_to_h5mu`: Added converter component for COSMX data (PR #3, PR #9).
+
+* `mapping/spaceranger_count`: Added a spaceranger count component (PR #2).
+
+* `convert/from_spatialdata_to_h5mu`, `convert/from_xenium_to_spatialdata`, `convert/from_xenium_to_h5mu`: Added converter components for xenium data (PR #1, #10).
+
+* `convert/from_xenium_to_spatialexperiment`, `convert/from_cosmx_to_spatialexperiment`: Added converter components for Xenium or CosMx data to SpatialExperiment objects (PR #9).
+
+* `convert/from_cells2stats_to_h5mu`: Added a component to convert data resulting from Aviti Teton sequencers processed by Cells2Stats into an H5MU file (PR #15).
+
+* `workflows/qc/qc`: Added a pipeline for calculating qc metrics of spatial omics samples (PR #5).
+
+* `workflows/multiomics/spatial_process_samples`: Added a pipeline to pre-process multiple spatial omics samples (PR #7).
+
+* `convert/from_h5mu_to_spatialexperiment`: Added converter component for H5MU data to SpatialExperiment objects (PR #15).

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 openpipelines-bio
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -0,0 +1,55 @@
+# OpenPipeline Spatial
+
+Extensible spatial single cell analysis pipelines for reproducible and large-scale spatial single cell processing using Viash and Nextflow.
+
+OpenPipeline Spatial extends the [OpenPipeline](https://github.com/openpipelines-bio/openpipeline/) ecosystem with specialized workflows and components for spatial transcriptomics analysis. It provides standardized, reproducible pipelines that are technology-agnostic and can be used for processing spatial omics data from various technologies and platforms.
+
+[![ViashHub](https://img.shields.io/badge/ViashHub-openpipeline_spatial-7a4baa.svg)](https://www.viash-hub.com/packages/openpipeline_spatial)
+[![GitHub](https://img.shields.io/badge/GitHub-viash--hub%2Fopenpipeline_spatial-blue.svg)](https://github.com/openpipelines-bio/openpipeline_spatial)
+[![GitHub
+License](https://img.shields.io/github/license/openpipelines-bio/openpipeline_spatial.svg)](https://github.com/openpipelines-bio/openpipeline_spatial/blob/main/LICENSE)
+[![GitHub
+Issues](https://img.shields.io/github/issues/openpipelines-bio/openpipeline_spatial.svg)](https://github.com/openpipelines-bio/openpipeline_spatial/issues)
+[![Viash
+version](https://img.shields.io/badge/Viash-v0.9.3-blue.svg)](https://viash.io)
+
+## Functionality
+
+OpenPipeline Spatial executes a list of predefined tasks specifically designed for spatial omics data. These discrete steps are also provided as standalone components that can be executed individually with a standardized interface.
+
+The following spatial-specific workflows are provided:
+
+- [Ingestion](https://www.viash-hub.com/packages/openpipeline_spatial/latest/components?search=mapping): Whereas many technologies generate count matrices on-instrument, functionality is provided for the mapping & quantification of 10X Visum data.
+- [Interoperability](https://www.viash-hub.com/packages/openpipeline_spatial/latest/components?search=convert): To make sure all spatial workflows are technology-agnostic, functionality is provided to convert count matrices from different technologies (e.g. Xenium, CosMx, AtoMx, Aviti) into a common format (H5MU). In addition, functionality is provided to convert between various Spatial data formats (e.g. Seurat, SpatialExperiment, MuData, SpatialData).
+- [QC](https://www.viash-hub.com/packages/openpipeline_spatial/latest/components?search=spatial_qc): Calculation of comprehensive quality control metrics.
+- [Sample Processing](https://www.viash-hub.com/packages/openpipeline_spatial/latest/components?search=spatial_process_samples): Batch processing of multiple spatial samples, including count-based filtering, normalisation and dimensionality reduction.
+
+## Extended functionality
+
+Whereas this package only provides spatial-specific functionality, it is designed to work seamlessly with the core [OpenPipeline package](https://github.com/openpipelines-bio/openpipeline/). This means that all core OpenPipeline workflows and components can be used in conjunction with the spatial-specific ones. For example, the [**integration**](https://www.viash-hub.com/packages/openpipeline/latest/components?search=workflows%2Fintegration) and [**cell type annotation**](https://www.viash-hub.com/packages/openpipeline/latest/components?search=workflows%2Fannotation) workflows can be applied to spatial data after it has been processed using the spatial-specific workflows.
+
+``` mermaid lang="mermaid"
+flowchart LR
+  demultiplexing["Step 1: Ingestion"]
+  ingestion["Step 2: QC"]
+  process_samples["Step 3: Process Samples"]
+  integration["Step 4: Integration"]
+  downstream["Step 5: Downstream Analysis"]
+  demultiplexing-->ingestion-->process_samples-->integration-->downstream
+```
+
+## Execution via CLI or Seqera Cloud
+
+The openpipeline_spatial package is available via [Viash
+Hub](https://www.viash-hub.com/packages/openpipeline_spatial/latest/), where
+you can receive instructions on how to run the end-to-end workflow as
+well as individual subworkflows or components.
+
+It’s possible to run the workflow directly from Seqera Cloud. The necessary Nextflow schema files have been [built and provided with the workflows](https://packages.viash-hub.com/vsh/openpipeline_spatial/-/tree/build/main/target/nextflow?ref_type=heads) in order to use the form-based input. However, Seqera Cloud can not deal with multiple-value parameters for batch processing of multiple samples. Therefore, it’s better to use Viash Hub also here for launching the workflow on Seqera Cloud.
+
+* Navigate to the [Viash Hub package page](https://www.viash-hub.com/packages/openpipeline_spatial/latest/), select the workflow you want to launch and click the `launch` button.
+* Select the execution environment of choice (e.g. `Seqera Cloud`, `CLI` or `Executable`)
+* Fill in the form with the required parameters and launch the workflow.
+
+## Support
+For issues specific to spatial analysis, please use the [GitHub issues tracker](https://github.com/openpipelines-bio/openpipeline_spatial/issues). For general OpenPipeline questions, refer to the main [OpenPipeline documentation](https://openpipelines.bio/).

_viash.yaml

@@ -0,0 +1,22 @@
+viash_version: 0.9.4
+source: src
+target: target
+name: openpipeline_spatial
+organization: vsh
+links:
+  repository: https://github.com/openpipelines-bio/openpipeline_spatial
+  docker_registry: ghcr.io
+repositories:
+  - name: openpipeline
+    repo: openpipeline
+    type: vsh
+    tag: v3.0.0
+info:
+  test_resources:
+    - type: s3
+      path: s3://openpipelines-bio/openpipeline_spatial/resources_test
+      dest: resources_test
+config_mods: |-
+  .resources += {path: '/src/workflows/utils/labels.config', dest: 'nextflow_labels.config'}
+  .runners[.type == 'nextflow'].config.script := 'includeConfig("nextflow_labels.config")'
+version: niche-compass

resources_test_scripts/aviti_teton_tiny.sh

@@ -0,0 +1,122 @@
+#!/bin/bash
+
+set -eo pipefail
+
+# get the root of the directory
+REPO_ROOT=$(git rev-parse --show-toplevel)
+
+# ensure that the command below is run from the root of the repository
+cd "$REPO_ROOT"
+
+ID=aviti
+DIR=resources_test/$ID/
+OUT=$DIR/teton_cells2stats_tiny/
+
+# Create directories
+[ -d "$DIR" ] || mkdir -p "$DIR"
+[ -d "$OUT" ] || mkdir -p "$OUT"
+
+echo "> Downloading Aviti Teton data"
+wget "https://go.elementbiosciences.com/l/938263/28kddnj7/d59cp" -O "${DIR}/PLUT-0105.tar.gz"
+tar -xzf "${DIR}/PLUT-0105.tar.gz" -C "$DIR"
+rm "${DIR}/PLUT-0105.tar.gz"
+
+echo "> Processing and subsetting Aviti Teton data"
+python <<HEREDOC
+import os
+import shutil
+import pandas as pd
+import glob
+import json
+
+src_dir = "${DIR}/PLUT-0105"
+dest_dir = "${OUT}"
+subset_image_dirs = False
+wells_to_keep = ["A1"]
+max_cells_per_well = 1000
+
+os.makedirs(dest_dir, exist_ok=True)
+
+print(f"Processing data from {src_dir} to {dest_dir}")
+
+# Copy images
+if subset_image_dirs:
+    image_dirs = ["CellSegmentation", "Projection"]
+    for image_dir in image_dirs:
+        image_dir_path = os.path.join(src_dir, image_dir)
+        if not os.path.exists(image_dir_path):
+            print(f"Warning: Image directory not found: {image_dir_path}")
+            continue
+        if not os.path.isdir(image_dir_path):
+            print(f"Warning: Path exists but is not a directory: {image_dir_path}")
+            continue
+        print(f"Processing image directory: {image_dir}")
+        
+        for well in wells_to_keep:
+            dest_path = f"{dest_dir}/{image_dir}/Well{well}"
+            os.makedirs(dest_path, exist_ok=True)
+            src_path = glob.glob(os.path.join(src_dir, image_dir, f"Well{well}"))
+            if len(src_path) != 1:
+                print(f"Warning: Expected 1 path for Well{well}, found {len(src_path)}")
+                continue
+            shutil.copytree(src_path[0], os.path.join(dest_path), dirs_exist_ok=True)
+
+# Copy count matrix
+src_path = os.path.join(src_dir, "Cytoprofiling", "Instrument", "RawCellStats.parquet")
+if os.path.exists(src_path):
+    print(f"Processing count matrix: {src_path}")
+    df = pd.read_parquet(src_path)
+    print(f"Original data: {len(df)} rows")
+    
+    # Filter by wells
+    df = df[df["Well"].isin(wells_to_keep)]
+    print(f"After well filtering: {len(df)} rows")
+    
+    if max_cells_per_well:
+        # Limit the number of cells per well
+        df = df.head(max_cells_per_well)
+        print(f"After cell limit: {len(df)} rows")
+
+    dest_path = os.path.join(dest_dir, "Cytoprofiling", "Instrument")
+    os.makedirs(dest_path, exist_ok=True)
+    dest_file = os.path.join(dest_path, "RawCellStats.parquet")
+    df.to_parquet(dest_file, engine="pyarrow")
+    print(f"Saved processed count matrix to {dest_file}")
+else:
+    print(f"Warning: Count matrix not found at {src_path}")
+
+# Copy Panel Metadata
+panel_src_path = os.path.join(src_dir, "Panel.json")
+if os.path.exists(panel_src_path):
+    panel_dest_path = os.path.join(dest_dir, "Panel.json")
+    shutil.copy2(panel_src_path, panel_dest_path)
+    print(f"Copied Panel.json")
+else:
+    print(f"Warning: Panel.json not found at {panel_src_path}")
+print("Processing complete!")
+HEREDOC
+
+echo "> Removing original aviti_teton folder"
+rm -rf "$DIR/PLUT-0105"
+
+echo "> Aviti Teton tiny dataset created successfully at $OUT"
+
+viash run src/convert/from_cells2stats_to_h5mu/config.vsh.yaml -- \
+    --input "${OUT}" \
+    --output "$DIR/aviti_teton_tiny.h5mu" \
+    --output_compression "gzip"
+
+echo "> Conversion to H5MU complete"
+
+viash run src/neighbors/spatial_neighborhood_graph/config.vsh.yaml -- \
+    --input "$DIR/aviti_teton_tiny.h5mu" \
+    --output "$DIR/aviti_teton_tiny.h5mu"
+
+echo "> Spatial neighbor graph calculation complete"
+
+aws s3 sync \
+    --profile di \
+    "$DIR" \
+    s3://openpipelines-bio/openpipeline_spatial/resources_test/aviti \
+    --delete \
+    --dryrun

resources_test_scripts/cosmx_tiny.sh

@@ -0,0 +1,58 @@
+#!/bin/bash
+
+set -eo pipefail
+
+# get the root of the directory
+REPO_ROOT=$(git rev-parse --show-toplevel)
+
+# ensure that the command below is run from the root of the repository
+cd "$REPO_ROOT"
+
+DIR="resources_test/cosmx"
+ID="Lung5_Rep2"
+OUT="$DIR/$ID/"
+
+# create tempdir
+MY_TEMP="${VIASH_TEMP:-/tmp}"
+TMPDIR=$(mktemp -d "$MY_TEMP/$ID-XXXXXX")
+function clean_up {
+  [[ -d "$TMPDIR" ]] && rm -r "$TMPDIR"
+}
+trap clean_up EXIT
+
+if [ ! -d "$OUT" ]; then
+    flat_dataset="https://nanostring-public-share.s3.us-west-2.amazonaws.com/SMI-Compressed/Lung5_Rep2/Lung5_Rep2+SMI+Flat+data.tar.gz"
+    wget  "$flat_dataset" -O "$TMPDIR/Lung5_Rep2.tar.gz"
+    mkdir -p "$TMPDIR/Lung5_Rep2"
+    tar -xzf "$TMPDIR/Lung5_Rep2.tar.gz" -C "$TMPDIR/Lung5_Rep2"
+    mkdir -p "$OUT"
+    mv "$TMPDIR/Lung5_Rep2/Lung5_Rep2/Lung5_Rep2-Flat_files_and_images/"* "$OUT/"
+fi
+
+viash run src/filter/subset_cosmx/config.vsh.yaml -- \
+    --input "$OUT" \
+    --num_fovs 3 \
+    --subset_transcripts_file True \
+    --subset_polygons_file False \
+    --output "${DIR}/${ID}_tiny"
+
+viash run src/convert/from_cosmx_to_h5mu/config.vsh.yaml -- \
+    --input ${DIR}/${ID}_tiny \
+    --output "$DIR/${ID}_tiny.h5mu" \
+    --output_compression "gzip"
+
+viash run src/neighbors/spatial_neighborhood_graph/config.vsh.yaml -- \
+    --input "$DIR/${ID}_tiny.h5mu" \
+    --output "$DIR/${ID}_tiny.h5mu"
+
+echo "> Spatial neighbor graph calculation complete"
+
+rm -rf "$OUT"
+
+# Sync to S3
+aws s3 sync \
+    --profile di \
+    "$DIR" \
+    s3://openpipelines-bio/openpipeline_spatial/resources_test/cosmx \
+    --delete \
+    --dryrun

resources_test_scripts/reference_tiny.sh

@@ -0,0 +1,19 @@
+#!/bin/bash
+
+set -eo pipefail
+
+# get the root of the directory
+REPO_ROOT=$(git rev-parse --show-toplevel)
+
+# ensure that the command below is run from the root of the repository
+cd "$REPO_ROOT"
+DIR="resources_test/GRCh38"
+
+mkdir -p $DIR
+
+aws s3 sync \
+    --profile di \
+    s3://openpipelines-bio/openpipeline_spatial/resources_test/GRCh38 \
+    "$DIR" \
+    --delete \
+    --dryrun

resources_test_scripts/visium_tiny.sh

@@ -0,0 +1,35 @@
+#!/bin/bash
+
+set -eo pipefail
+
+# get the root of the directory
+REPO_ROOT=$(git rev-parse --show-toplevel)
+
+# Define absolute directory path
+DIR="$REPO_ROOT/resources_test/visium"
+
+# from https://www.10xgenomics.com/resources/datasets/human-ovarian-cancer-1-standard
+mkdir -p "$DIR"
+
+# Input Files - download to the specific directory
+curl -o "$DIR/Visium_FFPE_Human_Ovarian_Cancer_fastqs.tar" https://cf.10xgenomics.com/samples/spatial-exp/1.3.0/Visium_FFPE_Human_Ovarian_Cancer/Visium_FFPE_Human_Ovarian_Cancer_fastqs.tar
+curl -o "$DIR/Visium_FFPE_Human_Ovarian_Cancer_image.jpg" https://cf.10xgenomics.com/samples/spatial-exp/1.3.0/Visium_FFPE_Human_Ovarian_Cancer/Visium_FFPE_Human_Ovarian_Cancer_image.jpg
+curl -o "$DIR/Visium_FFPE_Human_Ovarian_Cancer_probe_set.csv" https://cf.10xgenomics.com/samples/spatial-exp/1.3.0/Visium_FFPE_Human_Ovarian_Cancer/Visium_FFPE_Human_Ovarian_Cancer_probe_set.csv
+
+# Extract in the specific directory
+tar xvf "$DIR/Visium_FFPE_Human_Ovarian_Cancer_fastqs.tar" -C "$DIR"
+
+# Create subsampled dataset with ImageMagick
+# https://imagemagick.org/index.php
+mkdir -p "$DIR/subsampled"
+convert "$DIR/Visium_FFPE_Human_Ovarian_Cancer_image.jpg" -resize 2000x2000 "$DIR/subsampled/Visium_FFPE_Human_Ovarian_Cancer_image.jpg"
+for f in "$DIR"/Visium_FFPE_Human_Ovarian_Cancer_fastqs/*L001*R*; do 
+  gzip -cdf "$f" | head -n 40000 | gzip -c > "$DIR/subsampled/$(basename "$f")"; 
+done
+
+aws s3 sync \
+    --profile di \
+    "$DIR" \
+    s3://openpipelines-bio/openpipeline_spatial/resources_test/visium \
+    --delete \
+    --dryrun

resources_test_scripts/xenium_tiny.sh

@@ -0,0 +1,49 @@
+#!/bin/bash
+
+set -eo pipefail
+
+# get the root of the directory
+REPO_ROOT=$(git rev-parse --show-toplevel)
+
+# Define absolute directory paths
+DIR="$REPO_ROOT/resources_test/xenium"
+ID="xenium_tiny"
+OUT="$DIR/$ID"
+
+# create tempdir
+MY_TEMP="${VIASH_TEMP:-/tmp}"
+TMPDIR=$(mktemp -d "$MY_TEMP/$ID-XXXXXX")
+function clean_up {
+  [[ -d "$TMPDIR" ]] && rm -r "$TMPDIR"
+}
+trap clean_up EXIT
+
+if [ ! -d "$OUT" ]; then
+    tiny_dataset="https://raw.githubusercontent.com/nf-core/test-datasets/spatialxe/Xenium_Prime_Mouse_Ileum_tiny_outs.zip"
+    wget "$tiny_dataset" -O "$TMPDIR/xenium_tiny.zip"
+
+    unzip -q "$TMPDIR/xenium_tiny.zip" -d "$TMPDIR/xenium_tiny"
+    mkdir -p "$OUT"
+    mv "$TMPDIR/xenium_tiny/Xenium_Prime_Mouse_Ileum_tiny_outs/"* "$OUT/"
+fi
+
+viash run "$REPO_ROOT/src/convert/from_xenium_to_spatialdata/config.vsh.yaml" -- \
+    --input "$OUT" \
+    --output "$DIR/$ID.zarr"
+
+viash run "$REPO_ROOT/src/convert/from_spatialdata_to_h5mu/config.vsh.yaml" -- \
+    --input "$DIR/$ID.zarr" \
+    --output "$DIR/$ID.h5mu"
+
+
+viash run src/neighbors/spatial_neighborhood_graph/config.vsh.yaml -- \
+    --input "$DIR/$ID.h5mu" \
+    --output "$DIR/$ID.h5mu"
+
+# Sync to S3
+aws s3 sync \
+    --profile di \
+    "$DIR" \
+    s3://openpipelines-bio/openpipeline_spatial/resources_test/xenium \
+    --delete \
+    --dryrun

ruff.toml

@@ -0,0 +1,43 @@
+# Exclude a variety of commonly ignored directories.
+exclude = [
+    ".git",
+    ".pyenv",
+    ".pytest_cache",
+    ".ruff_cache",
+    ".venv",
+    ".vscode",
+    "__pypackages__",
+    "_build",
+    "build",
+    "dist",
+    "node_modules",
+    "site-packages",
+]
+
+builtins = ["meta"]
+
+
+
+
+[format]
+# Like Black, use double quotes for strings.
+quote-style = "double"
+
+# Like Black, indent with spaces, rather than tabs.
+indent-style = "space"
+
+# Like Black, respect magic trailing commas.
+skip-magic-trailing-comma = false
+
+# Like Black, automatically detect the appropriate line ending.
+line-ending = "auto"
+
+[lint.flake8-pytest-style]
+fixture-parentheses = false
+mark-parentheses = false
+
+[lint]
+ignore = [
+  # module level import not at top of file
+  "E402"
+]

src/authors/dorien_roosen.yaml

@@ -0,0 +1,11 @@
+name: Dorien Roosen
+info:
+  role: Core Team Member
+  links:
+    email: dorien@data-intuitive.com
+    github: dorien-er
+    linkedin: dorien-roosen
+  organizations:
+    - name: Data Intuitive
+      href: https://www.data-intuitive.com
+      role: Data Scientist

src/authors/dries_schaumont.yaml

@@ -0,0 +1,12 @@
+name: Dries Schaumont
+info:
+  role: Core Team Member
+  links:
+    email: dries@data-intuitive.com
+    github: DriesSchaumont
+    orcid: "0000-0002-4389-0440"
+    linkedin: dries-schaumont
+  organizations:
+    - name: Data Intuitive
+      href: https://www.data-intuitive.com
+      role: Data Scientist

src/authors/jakub_majercik.yaml

@@ -0,0 +1,11 @@
+name: Jakub Majercik
+info:
+  role: Contributor
+  links:
+    email: jakub@data-intuitive.com
+    github: jakubmajercik
+    linkedin: jakubmajercik
+  organizations:
+    - name: Data Intuitive
+      href: https://www.data-intuitive.com
+      role: Bioinformatics Engineer

src/authors/robrecht_cannoodt.yaml

@@ -0,0 +1,15 @@
+name: Robrecht Cannoodt
+info:
+  role: Core Team Member
+  links:
+    email: robrecht@data-intuitive.com
+    github: rcannood
+    orcid: "0000-0003-3641-729X"
+    linkedin: robrechtcannoodt
+  organizations:
+    - name: Data Intuitive
+      href: https://www.data-intuitive.com
+      role: Data Science Engineer
+    - name: Open Problems
+      href: https://openproblems.bio
+      role: Core Member

src/authors/weiwei_schultz.yaml

@@ -0,0 +1,6 @@
+name: Weiwei Schultz
+info:
+  role: Contributor
+  organizations:
+    - name: Janssen R&D US
+      role: Associate Director Data Sciences

src/base/h5_compression_argument.yaml

@@ -0,0 +1,9 @@
+arguments:
+  - name: "--output_compression"
+    description: |
+      Compression format to use for the output AnnData and/or Mudata objects.
+      By default no compression is applied.
+    type: string
+    choices: ["gzip", "lzf"]
+    required: false
+    example: "gzip"

src/base/requirements/anndata.yaml

@@ -0,0 +1,2 @@
+packages:
+  - anndata~=0.11.1

src/base/requirements/anndata_mudata.yaml

@@ -0,0 +1,5 @@
+__merge__: [/src/base/requirements/anndata.yaml, .]
+packages:
+  - mudata~=0.3.1
+script: |
+  exec("try:\n  import awkward\nexcept ModuleNotFoundError:\n  exit(0)\nelse:  exit(1)")

src/base/requirements/openpipeline_testutils.yaml

@@ -0,0 +1,2 @@
+github:
+  - openpipelines-bio/core#subdirectory=packages/python/openpipeline_testutils

src/base/requirements/python_test_setup.yaml

@@ -0,0 +1,8 @@
+test_setup:
+  - type: apt
+    packages:
+      - git
+  - type: python
+    __merge__:
+      - /src/base/requirements/viashpy.yaml
+      - /src/base/requirements/openpipeline_testutils.yaml

src/base/requirements/scanpy.yaml

@@ -0,0 +1,2 @@
+packages:
+  - scanpy~=1.10.4

src/base/requirements/spatialdata-io.yaml

@@ -0,0 +1,3 @@
+packages:
+  - spatialdata-io~=0.3.0
+__merge__: [ ., /src/base/requirements/spatialdata.yaml ]

src/base/requirements/spatialdata.yaml

@@ -0,0 +1,3 @@
+packages:
+  - spatialdata~=0.5.0
+  - pyarrow~=18.0.0

src/base/requirements/squidpy.yaml

@@ -0,0 +1,3 @@
+__merge__: [/src/base/requirements/spatialdata.yaml, .]
+packages:
+  - squidpy~=1.6.5

src/base/requirements/viashpy.yaml

@@ -0,0 +1,2 @@
+packages:
+  - viashpy==0.9.0

src/convert/from_cells2stats_to_h5mu/config.vsh.yaml

@@ -0,0 +1,137 @@
+name: from_cells2stats_to_h5mu
+namespace: convert
+scope: public
+description: |
+  Convert spatial data resulting from Aviti Teton sequencers that have been processed by the Element Biosciences cells2stats workflow to H5MU format.
+  
+  This component processes cells2stats count matrices to create a standardized H5MU file for downstream analysis.
+  
+  The component reads:
+  - Parquet file containing the count matrix and metadata
+  - Panel.json with target and batch information
+  
+  And outputs an H5MU file with:
+  - Count data as the main .X matrix
+  - Spatial coordinates in obsm
+  - Cell Paint intensities in obsm (optional)
+  - Nuclear count data as a layer (optional)
+  - CellProfiler morphology metrics in obsm (optional)
+  - Unassigned targets in obsm (optional)
+
+authors:
+  - __merge__: /src/authors/dorien_roosen.yaml
+    roles: [ maintainer ]
+
+argument_groups:
+  - name: Inputs
+    arguments:
+      - name: --input
+        type: file
+        direction: input
+        required: true
+        description: |
+          Path to the cells2stats output bundle. 
+          Expected folder structure (showing required files only):
+          ├── Cytoprofiling/
+          │   └── Instrument/
+          │       └── RawCellStats.parquet
+          └── Panel.json
+        example: path/to/aviti_output/
+
+  - name: Outputs
+    arguments:
+      - name: --output
+        type: file
+        direction: output
+        required: true
+        description: Output H5MU file path.
+        example: output.h5mu
+    __merge__: [., /src/base/h5_compression_argument.yaml]
+
+  - name: Options
+    arguments:
+      - name: --modality
+        type: string
+        default: rna
+        description: The modality to which the processed data will be written to in the H5MU file.
+      - name: --obsm_coordinates
+        type: string
+        description: |
+          Key name to store the spatial coordinates (in pixels) in obsm.
+          If present, spatial coordinates in micrometers will be stored under {obsm_coordinates}_um.
+          The column names will be stored in uns.
+        default: spatial
+      - name: --layer_nuclear_counts
+        type: string
+        description: |
+          Name for nuclear counts layer. If specified, nuclear count data 
+          will be stored as a separate layer in the AnnData object.
+        example: nuclear_counts
+      - name: --obsm_cell_paint
+        type: string
+        description: |
+          Key name for storing Cell Paint target intensities in obsm. 
+          If provided, Cell Paint target intensity data will be stored as a separate matrix in the obsm field.
+          The column names will be stored in uns.
+        example: cell_paint
+      - name: --obsm_cell_paint_nuclear
+        type: string
+        description: |
+          Key name for storing Nuclear Cell Paint target intensities in obsm.
+          If provided, Nuclear Cell Paint target intensity data will be stored as a separate matrix in the obsm field.
+          The column names will be stored in uns.
+        example: cell_paint_nuclear
+      - name: --obsm_cell_profiler
+        type: string
+        description: |
+          Key name for storing CellProfiler morphology metrics in obsm.
+          If provided, CellProfiler morphology metrics will be stored as a separate matrix in the obsm field.
+          The column names will be stored in uns.
+        example: cell_profiler
+      - name: --obsm_unassigned_targets
+        type: string
+        description: |
+          Key name for storing any unassigned target data in obsm.
+          If provided, unassigned target data will be stored as a separate matrix in the obsm field.
+          The column names will be stored in uns.
+        example: cell_profiler
+
+resources:
+  - type: python_script
+    path: script.py
+  - path: /src/utils/setup_logger.py
+  - path: /src/utils/unzip_archived_folder.py
+
+test_resources:
+  - type: python_script
+    path: test.py
+  - path: /resources_test/aviti/
+
+engines:
+  - type: docker
+    image: python:3.13-slim
+    setup:
+      - type: apt
+        packages: 
+          - procps
+          - build-essential
+          - zlib1g-dev
+          - git
+      - type: python
+        __merge__: [/src/base/requirements/anndata_mudata.yaml, .]
+        packages: [ pyarrow ]
+        # Windows explorer uses DEFLATE64 compression for large ZIP files,
+        # which is not supported by most standard library zipfile module
+        git: [ https://codeberg.org/miurahr/zipfile-inflate64.git@v0.2 ]
+    test_setup:
+      - type: apt
+        packages:
+          - zip
+      - type: python
+        __merge__: [ /src/base/requirements/viashpy.yaml, .]
+
+runners:
+  - type: executable
+  - type: nextflow
+    directives:
+      label: [lowmem, lowcpu]

src/convert/from_cells2stats_to_h5mu/script.py

@@ -0,0 +1,305 @@
+import sys
+from pathlib import Path
+import scipy.sparse as sp
+import pandas as pd
+import mudata as mu
+import anndata as ad
+import re
+import json
+import zipfile_inflate64 as zipfile
+import os
+
+## VIASH START
+par = {
+    "input": "./resources_test/aviti/aviti_teton_tiny_2",
+    "modality": "rna",
+    "output": "aviti_tiny_test.h5mu",
+    "output_compression": "gzip",
+    "layer_nuclear_counts": "nuclear_counts",
+    "obsm_coordinates": "spatial",
+    "obsm_cell_paint": "cell_paint",
+    "obsm_cell_paint_nuclear": "cell_paint_nuclear",
+    "obsm_cell_profiler": "cell_profiler",
+    "obsm_unassigned_targets": "unassigned_targets",
+}
+meta = {"resources_dir": "src/utils"}
+## VIASH END
+
+sys.path.append(meta["resources_dir"])
+from setup_logger import setup_logger
+from unzip_archived_folder import extract_selected_files_from_zip
+
+
+logger = setup_logger()
+
+
+def assert_matching_order(var_names, count_columns, split_pattern=None):
+    for var, col in zip(var_names, count_columns):
+        count_var = col if not split_pattern else col.split("_Nuclear")[0]
+        assert var == count_var, "Orders do not match"
+
+
+def categorize_columns(column_list, target_panel):
+    # Extract imaging and barcoding information from Panel.json
+    imaging_batches = [tube["BatchName"] for tube in target_panel["ImagingPrimerTubes"]]
+    barcoding_batches = [
+        tube["BatchName"] for tube in target_panel["BarcodingPrimerTubes"]
+    ]
+
+    # Extract target information
+    cellpaint_targets = [target["Target"] for target in target_panel["ImagingTargets"]]
+    barcoding_targets = [
+        target["Target"] for target in target_panel["BarcodingTargets"]
+    ]
+
+    # METADATA (for .obs and .obsm)
+    # Fixed columns
+    columns_fixed = [
+        "Area",
+        "AreaUm",
+        "Cell",
+        "NuclearArea",
+        "NuclearAreaUm",
+        "Tile",
+        "Well",
+        "WellLabel",
+    ]
+    obs_columns_fixed = list(set(columns_fixed) & set(column_list))
+
+    # Coordinate columns
+    coordinate_columns = ["X", "Y", "Xum", "Yum"]
+    obsm_coordinate_columns = list(set(coordinate_columns) & set(column_list))
+
+    # Cell Paint target intensity columns (format: {cell_paint_target.batch})
+    cell_paint_columns = [
+        col
+        for col in column_list
+        if any(
+            col.startswith(f"{target}.") and col.endswith(f".{batch}")
+            for target in cellpaint_targets
+            for batch in imaging_batches
+        )
+    ]
+
+    # Cell Paint nuclear target intensity columns (format: {cell_paint_target_Nuclear.batch})
+    cell_paint_nuclear_columns = [
+        col
+        for col in column_list
+        if any(
+            col.startswith(f"{target}_Nuclear") and col.endswith(f".{batch}")
+            for target in cellpaint_targets
+            for batch in imaging_batches
+        )
+    ]
+
+    # CellProfiler morphology metrics
+    morphology_patterns = [
+        r"^AreaShape_",
+        r"^Granularity_",
+        r"^Texture_",
+        r"^Intensity_",
+        r"^Location_",
+        r"^RadialDistribution_",
+    ]
+    cell_profiler_columns = [
+        col
+        for col in column_list
+        for pattern in morphology_patterns
+        if re.match(pattern, col)
+    ]
+
+    # COUNT MATRICES (for .X and layers)
+    # Feature Count Matrix - barcoding targets (format: {target.batch})
+    # Includes cellular and nuclear counts
+    count_columns = [
+        col
+        for col in column_list
+        if any(
+            col.startswith(f"{target}.") and col.endswith(f".{batch}")
+            for target in barcoding_targets
+            for batch in barcoding_batches
+        )
+    ]
+
+    # Nuclear Feature Count Matrix - barcoding targets (format: {target_Nuclear.batch})
+    # Includes only nuclear counts
+    nuclear_count_columns = [
+        col
+        for col in column_list
+        if any(
+            col.startswith(f"{target}_Nuclear") and col.endswith(f".{batch}")
+            for target in barcoding_targets
+            for batch in barcoding_batches
+        )
+    ]
+
+    # Unassigned columns (format: {Unassigned_*.*})
+    unassigned_columns = [col for col in column_list if col.startswith("Unassigned")]
+
+    # Make sure all columns have been categorized and have expected sizes
+    assert len(count_columns) == len(nuclear_count_columns), (
+        "Cellular and nuclear count columns do not match."
+    )
+    all_categorized_columns = (
+        obs_columns_fixed
+        + obsm_coordinate_columns
+        + cell_paint_columns
+        + cell_paint_nuclear_columns
+        + cell_profiler_columns
+        + count_columns
+        + nuclear_count_columns
+        + unassigned_columns
+    )
+    assert len(column_list) == len(all_categorized_columns), (
+        "Column categorization incomplete."
+    )
+
+    return (
+        obs_columns_fixed,
+        obsm_coordinate_columns,
+        cell_paint_columns,
+        cell_paint_nuclear_columns,
+        cell_profiler_columns,
+        count_columns,
+        nuclear_count_columns,
+        unassigned_columns,
+    )
+
+
+def retrieve_input_data(cells2stats_output_bundle):
+    # Expected folder structure (showing only relevant files):
+    # ├── Cytoprofiling/
+    # │   └── Instrument/
+    # │       └── RawCellStats.parquet
+    # └── Panel.json
+
+    required_file_patterns = {
+        "target_panel": "**/Panel.json",
+        "count_matrix": "**/Cytoprofiling/Instrument/RawCellStats.parquet",
+    }
+
+    if zipfile.is_zipfile(cells2stats_output_bundle):
+        cells2stats_output_bundle = extract_selected_files_from_zip(
+            cells2stats_output_bundle, members=required_file_patterns.values()
+        )
+    else:
+        cells2stats_output_bundle = Path(cells2stats_output_bundle)
+
+    assert os.path.isdir(cells2stats_output_bundle), (
+        "Input is expected to be a (compressed) directory."
+    )
+
+    input_data = {}
+    for key, pattern in required_file_patterns.items():
+        file = list(cells2stats_output_bundle.glob(pattern))
+        assert len(file) == 1, (
+            f"Expected exactly one file matching pattern {pattern}, found {len(file)}."
+        )
+        input_data[key] = file[0]
+
+    return input_data
+
+
+def main():
+    logger.info("Reading input data...")
+    input_data = retrieve_input_data(par["input"])
+    with open(input_data["target_panel"], "r") as f:
+        target_panel = json.load(f)
+    df = pd.read_parquet(input_data["count_matrix"], engine="pyarrow")
+    df_columns = df.columns.tolist()
+
+    logger.info("Categorizing input data...")
+    (
+        obs_columns_fixed,
+        coordinate_columns,
+        cell_paint_columns,
+        cell_paint_nuclear_columns,
+        cell_profiler_columns,
+        count_columns,
+        nuclear_count_columns,
+        unassigned_columns,
+    ) = categorize_columns(df_columns, target_panel)
+
+    df = df.set_index(df["Cell"].astype(str), drop=False)
+    df.index_name = None
+
+    # var and obs names
+    var_names = [var.split(".")[0] for var in count_columns]
+    obs_names = df["Cell"].astype(str).tolist()
+
+    # Count matrix
+    logger.info("Creating count matrix...")
+    count_df = df[count_columns].copy()
+    count_matrix_sparse = sp.csr_matrix(count_df.values)
+
+    # Obs field
+    logger.info(f"Creating obs field with columns {obs_columns_fixed}")
+    obs_df = df[obs_columns_fixed].copy()
+
+    # Create AnnData object
+    logger.info("Creating AnnData object...")
+    adata = ad.AnnData(
+        X=count_matrix_sparse,
+        obs=obs_df,
+        var=pd.DataFrame(index=var_names),
+    )
+
+    adata.obs_names = obs_names
+    adata.var_names = var_names
+
+    # Spatial coordinates
+    coordinate_sets = {
+        par["obsm_coordinates"]: ["X", "Y"],
+        f"{par['obsm_coordinates']}_um": ["Xum", "Yum"],
+    }
+
+    for obsm_key, coord_cols in coordinate_sets.items():
+        if all(col in coordinate_columns for col in coord_cols):
+            coordinates = df[coord_cols].copy()
+            adata.obsm[obsm_key] = coordinates.values
+            adata.uns[obsm_key] = coord_cols
+            logger.info(f"Added {obsm_key} coordinates ({coord_cols}) to obsm")
+        else:
+            missing_cols = [col for col in coord_cols if col not in coordinate_columns]
+            logger.warning(
+                f"Skipping {obsm_key}: missing coordinate columns {missing_cols}"
+            )
+
+    # Add (optional) .obsm fields
+    if par["obsm_cell_paint"]:
+        logger.info(f"Adding {par['obsm_cell_paint']} to obsm")
+        adata.obsm[par["obsm_cell_paint"]] = df[cell_paint_columns].copy()
+        adata.uns[par["obsm_cell_paint"]] = cell_paint_columns
+    if par["obsm_cell_paint_nuclear"]:
+        logger.info(f"Adding {par['obsm_cell_paint_nuclear']} to obsm")
+        adata.obsm[par["obsm_cell_paint_nuclear"]] = df[
+            cell_paint_nuclear_columns
+        ].copy()
+        adata.uns[par["obsm_cell_paint_nuclear"]] = cell_paint_nuclear_columns
+    if par["obsm_cell_profiler"]:
+        logger.info(f"Adding {par['obsm_cell_profiler']} to obsm")
+        adata.obsm[par["obsm_cell_profiler"]] = df[cell_profiler_columns].copy()
+        adata.uns[par["obsm_cell_profiler"]] = cell_profiler_columns
+    if par["obsm_unassigned_targets"]:
+        logger.info(f"Adding {par['obsm_unassigned_targets']} to obsm")
+        adata.obsm["unassigned_targets"] = df[unassigned_columns].copy()
+        adata.uns["unassigned_targets"] = unassigned_columns
+
+    # Add (optional) nuclear count layer
+    if par["layer_nuclear_counts"]:
+        assert_matching_order(
+            var_names, nuclear_count_columns, split_pattern="_Nuclear"
+        )
+        logger.info(f"Adding {par['layer_nuclear_counts']} to layers")
+        nuclear_count_df = df[nuclear_count_columns].copy()
+        nuclear_count_matrix_sparse = sp.csr_matrix(nuclear_count_df.values)
+        adata.layers[par["layer_nuclear_counts"]] = nuclear_count_matrix_sparse
+
+    # Write output MuData
+    logger.info("Writing MuData object...")
+    mdata = mu.MuData({par["modality"]: adata})
+    mdata.write_h5mu(par["output"], compression=par["output_compression"])
+
+
+if __name__ == "__main__":
+    main()

src/convert/from_cells2stats_to_h5mu/test.py

@@ -0,0 +1,158 @@
+import pytest
+import sys
+import mudata as mu
+import subprocess
+
+## VIASH START
+meta = {
+    "executable": "./target/executable/convert/from_cells2stats_to_h5mu/from_cells2stats_to_h5mu",
+    "resources_dir": "resources_test/aviti/",
+}
+## VIASH END
+
+input = f"{meta['resources_dir']}/aviti/teton_cells2stats_tiny/"
+
+
+def test_simple_execution(run_component, tmp_path):
+    output = tmp_path / "aviti.h5mu"
+
+    # run component
+    run_component(
+        ["--input", input, "--output", str(output), "--output_compression", "gzip"]
+    )
+
+    assert output.is_file(), "output file was not created"
+
+    mdata = mu.read_h5mu(output)
+    assert list(mdata.mod.keys()) == ["rna"], "Expected modality rna"
+    adata = mdata.mod["rna"]
+
+    assert adata.X.dtype.kind == "f"
+    expected_obs_keys = [
+        "AreaUm",
+        "Area",
+        "Tile",
+        "WellLabel",
+        "Well",
+        "Cell",
+        "NuclearAreaUm",
+        "NuclearArea",
+    ]
+    assert all([obs in expected_obs_keys for obs in adata.obs.columns])
+    obs_counts = ["Area", "Cell", "NuclearArea"]
+    assert all([adata.obs[obs].dtype.kind == "u" for obs in obs_counts])
+    obs_areas = ["AreaUm", "NuclearAreaUm"]
+    assert all([adata.obs[obs].dtype.kind == "f" for obs in obs_areas])
+    obs_categories = ["Tile", "WellLabel", "Well"]
+    assert all([adata.obs[obs].dtype.kind == "O" for obs in obs_categories])
+
+    expected_obsm_keys = ["spatial", "spatial_um"]
+    assert list(adata.obsm.keys()) == expected_obsm_keys
+    assert list(adata.uns.keys()) == expected_obsm_keys
+    assert all(adata.obsm[obsm].dtype.kind == "f" for obsm in expected_obsm_keys)
+
+
+def test_compressed_input(run_component, tmp_path):
+    output = tmp_path / "aviti.h5mu"
+    zipped_input = tmp_path / "aviti.zip"
+
+    subprocess.run(
+        ["zip", "-r", str(zipped_input), "aviti"], cwd=meta["resources_dir"], check=True
+    )
+
+    # run component
+    run_component(
+        [
+            "--input",
+            zipped_input,
+            "--output",
+            str(output),
+            "--output_compression",
+            "gzip",
+        ]
+    )
+
+    assert output.is_file(), "output file was not created"
+
+    mdata = mu.read_h5mu(output)
+    assert list(mdata.mod.keys()) == ["rna"], "Expected modality rna"
+    adata = mdata.mod["rna"]
+
+    assert adata.X.dtype.kind == "f"
+    expected_obs_keys = [
+        "AreaUm",
+        "Area",
+        "Tile",
+        "WellLabel",
+        "Well",
+        "Cell",
+        "NuclearAreaUm",
+        "NuclearArea",
+    ]
+    assert all([obs in expected_obs_keys for obs in adata.obs.columns])
+    obs_counts = ["Area", "Cell", "NuclearArea"]
+    assert all([adata.obs[obs].dtype.kind == "u" for obs in obs_counts])
+    obs_areas = ["AreaUm", "NuclearAreaUm"]
+    assert all([adata.obs[obs].dtype.kind == "f" for obs in obs_areas])
+    obs_categories = ["Tile", "WellLabel", "Well"]
+    assert all([adata.obs[obs].dtype.kind == "O" for obs in obs_categories])
+
+    expected_obsm_keys = ["spatial", "spatial_um"]
+    assert list(adata.obsm.keys()) == expected_obsm_keys
+    assert list(adata.uns.keys()) == expected_obsm_keys
+    assert all(adata.obsm[obsm].dtype.kind == "f" for obsm in expected_obsm_keys)
+
+
+def test_extended_parameters(run_component, tmp_path):
+    output = tmp_path / "aviti_ext.h5mu"
+
+    # run component
+    run_component(
+        [
+            "--input",
+            input,
+            "--modality",
+            "mod1",
+            "--output",
+            str(output),
+            "--layer_nuclear_counts",
+            "nuclear_counts",
+            "--obsm_coordinates",
+            "coords",
+            "--obsm_cell_paint",
+            "cell_paint",
+            "--obsm_cell_paint_nuclear",
+            "cell_paint_nuclear",
+            "--obsm_cell_profiler",
+            "cell_profiler",
+            "--obsm_unassigned_targets",
+            "unassigned_targets",
+            "--output_compression",
+            "gzip",
+        ]
+    )
+
+    assert output.is_file(), "output file was not created"
+
+    mdata = mu.read_h5mu(output)
+    assert list(mdata.mod.keys()) == ["mod1"]
+    adata = mdata.mod["mod1"]
+
+    assert list(adata.layers) == ["nuclear_counts"]
+    assert adata.layers["nuclear_counts"].dtype.kind == "f"
+
+    expected_obsm_keys = [
+        "cell_paint",
+        "cell_paint_nuclear",
+        "cell_profiler",
+        "coords",
+        "coords_um",
+        "unassigned_targets",
+    ]
+
+    assert list(adata.uns.keys()) == expected_obsm_keys
+    assert list(adata.obsm.keys()) == expected_obsm_keys
+
+
+if __name__ == "__main__":
+    sys.exit(pytest.main([__file__]))

src/convert/from_cosmx_to_h5mu/config.vsh.yaml

@@ -0,0 +1,75 @@
+name: "from_cosmx_to_h5mu"
+namespace: "convert"
+scope: "public"
+description: |
+  Converts the output from NanoString experiment into a MuData objcet.
+    - `<dataset_id>_exprMat_file.csv`: File containing the counts.
+    - `<dataset_id>`_metadata_file: File containing the spatial coordinates and additional cell-level metadata.
+    - `<dataset_id>_fov_file.csv`: File containing the coordinates of all the fields of view.
+  In addition to reading the regular Nanostring output, it loads CellComposite and CellLabels directories, if present,
+  containing the images.
+
+authors:
+  - __merge__: /src/authors/dorien_roosen.yaml
+    roles: [ maintainer ]
+  - __merge__: /src/authors/weiwei_schultz.yaml
+    roles: [ contributor ]
+arguments:
+  - name: "--input"
+    alternatives: ["-i"]
+    type: file
+    description: Input folder. Must contain the output from a NanoString CosMx run.
+    example: cosmx_data
+    direction: input
+    required: true
+  - name: "--modality"
+    type: string
+    default: rna
+  - name: "--output"
+    alternatives: ["-o"]
+    type: file
+    description: The output h5mu file.
+    example: "output.h5mu"
+    direction: output
+  - name: "--output_compression"
+    type: string
+    choices: ["gzip", "lzf"]
+    required: false
+    example: "gzip"
+
+resources:
+  - type: python_script
+    path: script.py
+  - path: /src/utils/setup_logger.py
+  - path: /src/utils/unzip_archived_folder.py
+
+test_resources:
+  - type: python_script
+    path: test.py
+  - path: /resources_test/cosmx/Lung5_Rep2_tiny/
+engines:
+  - type: docker
+    image: python:3.12-slim
+    setup:
+      - type: apt
+        packages: 
+          - procps
+          - build-essential
+          - zlib1g-dev
+          - git
+      - type: python
+        __merge__: [/src/base/requirements/anndata_mudata.yaml, /src/base/requirements/squidpy.yaml, .]
+        packages: [ pyarrow ]
+        # Windows explorer uses DEFLATE64 compression for large ZIP files,
+        # which is not supported by most standard library zipfile module
+        git: [ https://codeberg.org/miurahr/zipfile-inflate64.git@v0.2 ]
+    test_setup:
+      - type: apt
+        packages:
+          - zip
+    __merge__: [ /src/base/requirements/python_test_setup.yaml, . ]
+runners:
+  - type: executable
+  - type: nextflow
+    directives:
+      label: [lowmem, singlecpu]

src/convert/from_cosmx_to_h5mu/script.py

@@ -0,0 +1,98 @@
+import sys
+import os
+import squidpy as sq
+import mudata as mu
+import zipfile_inflate64 as zipfile
+from pathlib import Path
+
+## VIASH START
+par = {
+    "input": "./resources_test/cosmx/Lung5_Rep2_tiny",
+    "output": "./resources_test/cosmx/Lung5_Rep2_tiny.h5mu",
+    "modality": "rna",
+    "output_compression": None,
+}
+meta = {"resources_dir": "src/utils"}
+## VIASH END
+
+sys.path.append(meta["resources_dir"])
+from setup_logger import setup_logger
+from unzip_archived_folder import extract_selected_files_from_zip
+
+logger = setup_logger()
+
+
+def retrieve_input_data(cosmx_output_bundle):
+    # Expected folder structure (showing only relevant files):
+    # ├── *_exprMat_file.csv
+    # ├── *_fov_positions_file.csv
+    # └── *_metadata_file.csv
+
+    required_file_patterns = {
+        "counts_file": "**/*exprMat_file.csv",
+        "fov_file": "**/*fov_positions_file.csv",
+        "meta_file": "**/*metadata_file.csv",
+    }
+    if zipfile.is_zipfile(cosmx_output_bundle):
+        cosmx_output_bundle = extract_selected_files_from_zip(
+            cosmx_output_bundle, members=required_file_patterns.values()
+        )
+    else:
+        cosmx_output_bundle = Path(cosmx_output_bundle)
+
+    assert os.path.isdir(cosmx_output_bundle), (
+        "Input is expected to be a (compressed) directory."
+    )
+
+    input_data = {}
+    for key, pattern in required_file_patterns.items():
+        file = list(cosmx_output_bundle.glob(pattern))
+        assert len(file) == 1, f"Expected one file for {key}, found {len(file)}."
+        input_data[key] = file[0]
+
+    parent_dirs = {file.parent for file in input_data.values()}
+    assert len(parent_dirs) == 1, (
+        f"Input files are expected to be in the same directory."
+        f"Found files in {len(parent_dirs)} different directories: {parent_dirs}"
+    )
+
+    return input_data
+
+
+def main():
+    logger.info("Reading in CosMx data...")
+    input_files = retrieve_input_data(par["input"])
+
+    try:
+        adata = sq.read.nanostring(
+            path=input_files["counts_file"].parent,
+            counts_file=input_files["counts_file"].name,
+            meta_file=input_files["meta_file"].name,
+            fov_file=input_files["fov_file"].name,
+        )
+    except ValueError as e:
+        if "Index fov invalid" in str(e):
+            # CosMx experiments processed with AtoMx SIP <v1.3.2 has 'FOV' index column in fov_file
+            # see https://nanostring-biostats.github.io/CosMx-Analysis-Scratch-Space/posts/flat-file-exports/flat-files-compare.html
+            import pandas as pd
+
+            df = pd.read_csv(input_files["fov_file"])
+            df.rename(columns={"FOV": "fov"}, inplace=True)
+            df.to_csv(input_files["fov_file"], index=False)
+
+            adata = sq.read.nanostring(
+                path=input_files["counts_file"].parent,
+                counts_file=input_files["counts_file"].name,
+                meta_file=input_files["meta_file"].name,
+                fov_file=input_files["fov_file"].name,
+            )
+        else:
+            raise e
+
+    logger.info("Writing output MuData object...")
+    mdata = mu.MuData({par["modality"]: adata})
+    mdata.write_h5mu(par["output"], compression=par["output_compression"])
+
+
+if __name__ == "__main__":
+    main()

src/convert/from_cosmx_to_h5mu/test.py

@@ -0,0 +1,130 @@
+import pytest
+import sys
+import mudata as mu
+import subprocess
+import pandas as pd
+
+
+def test_simple_execution(run_component, tmp_path):
+    output = tmp_path / "cosmx_tiny.h5mu"
+
+    run_component(
+        [
+            "--input",
+            meta["resources_dir"] + "/Lung5_Rep2_tiny",
+            "--output",
+            output,
+        ]
+    )
+    assert output.is_file(), "output file was not created"
+
+    mdata = mu.read_h5mu(output)
+    assert list(mdata.mod.keys()) == ["rna"], "Expected modality rna"
+
+    adata = mdata.mod["rna"]
+
+    assert list(adata.obs.keys()) == [
+        "fov",
+        "Area",
+        "AspectRatio",
+        "CenterX_global_px",
+        "CenterY_global_px",
+        "Width",
+        "Height",
+        "Mean.MembraneStain",
+        "Max.MembraneStain",
+        "Mean.PanCK",
+        "Max.PanCK",
+        "Mean.CD45",
+        "Max.CD45",
+        "Mean.CD3",
+        "Max.CD3",
+        "Mean.DAPI",
+        "Max.DAPI",
+        "cell_ID",
+    ]
+
+    assert list(adata.uns.keys()) == ["spatial"]
+    assert list(adata.obsm.keys()) == ["spatial", "spatial_fov"]
+
+    assert adata.obsm["spatial"].dtype == "int"
+    assert adata.obsm["spatial_fov"].dtype == "float"
+
+
+def test_compressed_input(run_component, tmp_path):
+    output = tmp_path / "cosmx_tiny.h5mu"
+    zipped_input = tmp_path / "Lung5_Rep2_tiny.zip"
+
+    subprocess.run(
+        ["zip", "-r", str(zipped_input), "Lung5_Rep2_tiny"],
+        cwd=meta["resources_dir"],
+        check=True,
+    )
+
+    run_component(
+        [
+            "--input",
+            zipped_input,
+            "--output",
+            output,
+        ]
+    )
+    assert output.is_file(), "output file was not created"
+
+    mdata = mu.read_h5mu(output)
+    assert list(mdata.mod.keys()) == ["rna"], "Expected modality rna"
+
+    adata = mdata.mod["rna"]
+
+    assert list(adata.obs.keys()) == [
+        "fov",
+        "Area",
+        "AspectRatio",
+        "CenterX_global_px",
+        "CenterY_global_px",
+        "Width",
+        "Height",
+        "Mean.MembraneStain",
+        "Max.MembraneStain",
+        "Mean.PanCK",
+        "Max.PanCK",
+        "Mean.CD45",
+        "Max.CD45",
+        "Mean.CD3",
+        "Max.CD3",
+        "Mean.DAPI",
+        "Max.DAPI",
+        "cell_ID",
+    ]
+
+    assert list(adata.uns.keys()) == ["spatial"]
+    assert list(adata.obsm.keys()) == ["spatial", "spatial_fov"]
+
+    assert adata.obsm["spatial"].dtype == "int"
+    assert adata.obsm["spatial_fov"].dtype == "float"
+
+
+def test_legacy_atomx_input(run_component, tmp_path):
+    output = tmp_path / "cosmx_tiny.h5mu"
+
+    # mimic legacy AtoMx SIP output structure
+    fov_file = (
+        meta["resources_dir"] + "/Lung5_Rep2_tiny/Lung5_Rep2_fov_positions_file.csv"
+    )
+    df = pd.read_csv(fov_file)
+    df.rename(columns={"fov": "FOV"}, inplace=True)
+    df.to_csv(fov_file, index=False)
+
+    run_component(
+        [
+            "--input",
+            meta["resources_dir"] + "/Lung5_Rep2_tiny",
+            "--output",
+            output,
+        ]
+    )
+    assert output.is_file(), "output file was not created"
+
+
+if __name__ == "__main__":
+    sys.exit(pytest.main([__file__]))

src/convert/from_cosmx_to_spatialexperiment/config.vsh.yaml

@@ -0,0 +1,83 @@
+name: "from_cosmx_to_spatialexperiment"
+namespace: "convert"
+scope: "public"
+description: |
+  Creates a SpatialExperiment object from the downloaded unzipped CosMx directory for Nanostring
+  CosMx spatial gene expression data, and saves it as a SpatialExperiment object.
+  The constructor assumes the downloaded unzipped CosMx Folder has the following structure:
+
+  Mandatory files
+  · | — *_exprMat_file.csv
+  · | — *_metadata_file.csv
+  Optional files, by default added to the metadata() as a list of paths (will be converted to parquet):
+  · | —*_fov_positions_file.csv
+  · | — *_tx_file.csv
+  · | — *_polygons.csv
+
+authors:
+  - __merge__: /src/authors/dorien_roosen.yaml
+    roles: [ author, maintainer ]
+arguments:
+  - name: "--input"
+    alternatives: ["-i"]
+    type: file
+    description: Input CosMx directory
+    direction: input
+    required: true
+    example: path/to/cosmx_bundle
+  - name: "--add_tx_path"
+    type: boolean
+    default: true
+    description: |
+      Whether to add parquet paths to the metadata. 
+      If True, `*_tx_file.csv` file will be converted to .parquet and added to the metadata. 
+  - name: "--add_polygon_path"
+    type: boolean
+    default: true
+    description: |
+      Whether to add polygon path to the metadata.
+      If True, `*_polygons.csv` file will be converted to .parquet and be added to the metadata.
+  - name: "--add_fov_positions"
+    type: boolean
+    default: true
+    description: |
+      Whether to add fov positions to the metadata.
+      If True, `fov_positions_file.csv` will be added to the metadata.
+  - name: "--alternative_experiment_features"
+    type: string
+    multiple: true
+    description: Feature names containing these strings will be moved to altExps(sxe) slots as separate SpatialExperiment objects.
+    default: [NegPrb, Negative, SystemControl, FalseCode]
+  - name: "--output"
+    alternatives: ["-o"]
+    type: file
+    description: Output SpatialExperiment file
+    direction: output
+    required: true
+    example: output.rds
+resources:
+  - type: r_script
+    path: script.R
+  - path: /src/utils/unzip_archived_folder.R
+test_resources:
+  - type: r_script
+    path: test.R
+  - path: /resources_test/cosmx/Lung5_Rep2_tiny
+engines:
+  - type: docker
+    image: rocker/r2u:24.04
+    setup:
+      - type: apt
+        packages: 
+          - libhdf5-dev
+          - libgeos-dev
+      - type: r
+        bioc: [ SpatialExperimentIO ]
+    test_setup:
+      - type: r
+        cran: [ testthat ]
+runners:
+  - type: executable
+  - type: nextflow
+    directives:
+      label: [lowmem, singlecpu]

src/convert/from_cosmx_to_spatialexperiment/script.R

@@ -0,0 +1,62 @@
+library(SpatialExperimentIO)
+
+### VIASH START
+par <- list(
+  input = "resources_test/cosmx/test2.zip",
+  add_tx_path = TRUE,
+  add_polygon_path = FALSE,
+  add_fov_positions = TRUE,
+  alternative_experiment_features = c(
+    "NegPrb", "Negative", "SystemControl", "FalseCode"
+  ),
+  output = "spe_cosmx_test.rds"
+)
+meta <- list(
+  resources_dir = "src/utils/"
+)
+### VIASH END
+
+source(paste0(meta$resources_dir, "/unzip_archived_folder.R"))
+
+cat("Reading input data...")
+if (tools::file_ext(par$input) == "zip") {
+  expected_file_patterns <- c(
+    "*.csv",
+    "*.parquet"
+  )
+  tmp_dir <- extract_selected_files(
+    par$input,
+    members = expected_file_patterns
+  )
+  cosmx_output_bundle <- file.path(
+    tmp_dir,
+    tools::file_path_sans_ext(basename(par$input))
+  )
+} else {
+  cosmx_output_bundle <- par$input
+}
+
+cat("Setting parameters...")
+if (par$add_polygon_path == FALSE && par$add_tx_path == FALSE) {
+  add_parquet_paths <- FALSE
+} else {
+  add_parquet_paths <- TRUE
+}
+
+cat("Converting to SpatialExperiment...")
+spe <- readCosmxSXE(
+  dirName = cosmx_output_bundle,
+  returnType = "SPE",
+  countMatPattern = "exprMat_file.csv",
+  metaDataPattern = "metadata_file.csv",
+  coordNames = c("CenterX_global_px", "CenterY_global_px"),
+  addFovPos = par$add_fov_positions,
+  fovPosPattern = "fov_positions_file.csv",
+  addParquetPaths = add_parquet_paths,
+  addPolygon = par$add_polygon_path,
+  addTx = par$add_tx_path,
+  altExps = par$alternative_experiment_features
+)
+
+cat("Saving output...")
+saveRDS(spe, file = par$output)

src/convert/from_cosmx_to_spatialexperiment/test.R

@@ -0,0 +1,182 @@
+library(testthat, warn.conflicts = FALSE)
+library(SpatialExperimentIO)
+library(SpatialExperiment)
+
+## VIASH START
+meta <- list(
+  executable = "./from_cosmx_to_spatialexperiment",
+  resources_dir = "resources_test/cosmx/",
+  name = "from_cosmx_to_spatialexperiment"
+)
+## VIASH END
+
+cat("> Checking simple execution\n")
+
+spe <- paste0(
+  meta[["resources_dir"]],
+  "/Lung5_Rep2_tiny"
+)
+out_rds <- "output.rds"
+
+cat("> Running ", meta[["name"]], "\n", sep = "")
+out <- processx::run(
+  meta[["executable"]],
+  c(
+    "--input", spe,
+    "--add_tx_path", TRUE,
+    "--add_polygon_path", FALSE,
+    "--output", out_rds
+  )
+)
+
+cat("> Checking whether output file exists\n")
+expect_equal(out$status, 0)
+expect_true(file.exists(out_rds))
+
+cat("> Reading output file\n")
+obj <- readRDS(file = out_rds)
+
+cat("> Checking whether Seurat object is in the right format\n")
+# Object type
+expect_is(obj, "SpatialExperiment")
+# Assay structure
+expect_equal(names(slot(obj, "assays")), "counts")
+# Spatial coordinates
+expect_equal(
+  spatialCoordsNames(obj),
+  c("CenterX_global_px", "CenterY_global_px")
+)
+# Alternative experiments
+expect_equal(altExpNames(obj), c("NegPrb"))
+# Metadata components
+expect_named(
+  metadata(obj),
+  c("fov_positions", "transcripts"),
+  ignore.order = TRUE
+)
+# Parquet paths
+expect_true(grepl("\\.parquet$", metadata(obj)[["transcripts"]]))
+# Dimensions
+input <- readCosmxSXE(
+  dirName = spe,
+  addParquetPaths = FALSE,
+  returnType = "SPE"
+)
+
+dim_rds <- dim(obj)
+dim_input <- dim(input)
+
+expect_equal(dim_rds, dim_input)
+
+
+cat("> Checking execution with compressed input\n")
+
+spe <- paste0(meta[["resources_dir"]], "/Lung5_Rep2_tiny")
+out_rds <- "output.rds"
+
+create_folder_archive <- function(
+  folder_path,
+  archive = "Lung5_Rep2_tiny.zip"
+) {
+  old_wd <- getwd()
+  on.exit(setwd(old_wd))
+  setwd(meta$resources_dir)
+  system2("zip", c("-r", archive, "Lung5_Rep2_tiny"))
+  paste0(meta$resources_dir, "/", archive)
+}
+
+zipped_spe <- create_folder_archive(spe)
+
+cat("> Running ", meta[["name"]], "\n", sep = "")
+out <- processx::run(
+  meta[["executable"]],
+  c(
+    "--input", zipped_spe,
+    "--add_tx_path", TRUE,
+    "--add_polygon_path", FALSE,
+    "--output", out_rds
+  )
+)
+
+cat("> Checking whether output file exists\n")
+expect_equal(out$status, 0)
+expect_true(file.exists(out_rds))
+
+cat("> Reading output file\n")
+obj <- readRDS(file = out_rds)
+
+cat("> Checking whether Seurat object is in the right format\n")
+# Object type
+expect_is(obj, "SpatialExperiment")
+# Assay structure
+expect_equal(names(slot(obj, "assays")), "counts")
+# Spatial coordinates
+expect_equal(
+  spatialCoordsNames(obj),
+  c("CenterX_global_px", "CenterY_global_px")
+)
+# Alternative experiments
+expect_equal(altExpNames(obj), c("NegPrb"))
+# Metadata components
+expect_named(
+  metadata(obj),
+  c("fov_positions", "transcripts"),
+  ignore.order = TRUE
+)
+# Parquet paths
+expect_true(grepl("\\.parquet$", metadata(obj)[["transcripts"]]))
+# Dimensions
+input <- readCosmxSXE(
+  dirName = spe,
+  addParquetPaths = FALSE,
+  returnType = "SPE"
+)
+
+dim_rds <- dim(obj)
+dim_input <- dim(input)
+
+expect_equal(dim_rds, dim_input)
+
+
+cat("> Checking parameter functionality\n")
+
+out_rds_ext <- "output_ext.rds"
+
+cat("> Running ", meta[["name"]], "\n", sep = "")
+out_ext <- processx::run(
+  meta[["executable"]],
+  c(
+    "--input", spe,
+    "--add_fov_positions", FALSE,
+    "--add_tx_path", FALSE,
+    "--add_polygon_path", FALSE,
+    "--alternative_experiment_features", c("Negative"),
+    "--output", out_rds_ext
+  )
+)
+
+cat("> Checking whether output file exists\n")
+expect_equal(out_ext$status, 0)
+expect_true(file.exists(out_rds_ext))
+
+cat("> Reading output file\n")
+obj_ext <- readRDS(file = out_rds_ext)
+
+cat("> Checking whether Seurat object is in the right format\n")
+# Object type
+expect_is(obj_ext, "SpatialExperiment")
+# Assay structure
+expect_equal(names(slot(obj_ext, "assays")), "counts")
+# Spatial coordinates
+expect_equal(
+  spatialCoordsNames(obj_ext),
+  c("CenterX_global_px", "CenterY_global_px")
+)
+# Alternative experiments
+expect_length(altExpNames(obj_ext), 0)
+# Metadata components
+expect_length(metadata(obj_ext), 0)
+
+dim_rds_ext <- dim(obj_ext)
+expect_true(identical(dim_rds_ext[2], dim_input[2]))
+expect_false(identical(dim_rds_ext[1], dim_input[1]))

src/convert/from_h5mu_to_spatialexperiment/config.vsh.yaml

@@ -0,0 +1,75 @@
+name: "from_h5mu_to_spatialexperiment"
+namespace: "convert"
+scope: "public"
+description: |
+  Converts an h5mu file into a SpatialExperiment object.
+authors:
+  - __merge__: /src/authors/dorien_roosen.yaml
+    roles: [ author ]
+arguments:
+  - name: "--input"
+    alternatives: ["-i"]
+    type: file
+    description: Input h5mu file
+    direction: input
+    required: true
+    example: input.h5mu
+  - name: "--modality"
+    type: string
+    required: true
+    default: "rna"
+    description: Name of the modality to be converted.
+  - name: "--obsm_spatial_coordinates"
+    type: string
+    required: false
+    description: |
+      Key in the .obsm field that contains the spatial coordinates. 
+      Will be mapped to spatialCoords in the SpatialExperiment object.
+  - name: "--output"
+    alternatives: ["-o"]
+    type: file
+    description: Output SpatialExperiment file
+    direction: output
+    required: true
+    example: output.rds
+resources:
+  - type: r_script
+    path: script.R
+test_resources:
+  - type: r_script
+    path: test.R
+  - path: /resources_test/aviti/aviti_teton_tiny.h5mu
+  - path: /resources_test/cosmx/Lung5_Rep2_tiny.h5mu
+  - path: /resources_test/xenium/xenium_tiny.h5mu
+
+engines:
+  - type: docker
+    image: rocker/r2u:22.04
+    setup:
+      - type: apt
+        packages: 
+          - libhdf5-dev
+          - libgeos-dev
+      - type: r
+        cran: [ hdf5r, SpatialExperiment ]
+        github: scverse/anndataR@36f3caad9a7f360165c1510bbe0c62657580415a
+    test_setup:
+      - type: docker
+        env: 
+          - RETICULATE_PYTHON=/usr/bin/python
+      - type: apt
+        packages:
+          - python3
+          - python3-pip
+          - python3-dev
+          - python-is-python3
+      - type: r
+        cran: [ reticulate, testthat ]
+      - type: python
+        __merge__: /src/base/requirements/anndata_mudata.yaml
+
+runners:
+  - type: executable
+  - type: nextflow
+    directives:
+      label: [lowmem, singlecpu]

src/convert/from_h5mu_to_spatialexperiment/script.R

@@ -0,0 +1,113 @@
+library(SpatialExperiment)
+library(SingleCellExperiment)
+library(hdf5r)
+library(Matrix)
+library(hdf5r)
+
+## VIASH START
+par <- list(
+  input = "resources_test/xenium/xenium_tiny.h5mu",
+  output = "xenium_test.rds",
+  modality = "rna",
+  obsm_spatial_coordinates = "spatial"
+)
+## VIASH END
+
+
+h5mu_to_h5ad <- function(h5mu_path, modality_name) {
+  tmp_path <- tempfile(fileext = ".h5ad")
+  mod_location <- paste("mod", modality_name, sep = "/")
+  h5src <- hdf5r::H5File$new(h5mu_path, "r")
+  h5dest <- hdf5r::H5File$new(tmp_path, "w")
+  # Copy over the child objects and the child attributes from root
+  # Root cannot be copied directly because it always exists and
+  # copying does not allow overwriting.
+  children <- hdf5r::list.objects(h5src,
+    path = mod_location,
+    full.names = FALSE, recursive = FALSE
+  )
+  for (child in children) {
+    h5dest$obj_copy_from(
+      h5src, paste(mod_location, child, sep = "/"),
+      paste0("/", child)
+    )
+  }
+  # Also copy the root attributes
+  root_attrs <- hdf5r::h5attr_names(x = h5src)
+  for (attr in root_attrs) {
+    h5a <- h5src$attr_open(attr_name = attr)
+    robj <- h5a$read()
+    h5dest$create_attr_by_name(
+      attr_name = attr,
+      obj_name = ".",
+      robj = robj,
+      space = h5a$get_space(),
+      dtype = h5a$get_type()
+    )
+  }
+  h5src$close()
+  h5dest$close()
+
+  tmp_path
+}
+
+read_spatial_coordinates <- function(sce, spatial_coordinates_name) {
+  # Check if the specified spatial coordinates exist in reducedDims
+  reduced_dims <- SingleCellExperiment::reducedDims(sce)
+  if (par$obsm_spatial_coordinates %in% names(reduced_dims)) {
+    spatial_coords <- reduced_dims[[par$obsm_spatial_coordinates]]
+    if (ncol(spatial_coords) != 2) {
+      stop(
+        "Spatial coordinates must have 2 columns, but found ",
+        ncol(spatial_coords), " columns"
+      )
+    }
+    # Set proper column names for spatial coordinates
+    colnames(spatial_coords) <- c("x", "y")
+  } else {
+    warning(
+      "Spatial coordinates '", par$obsm_spatial_coordinates,
+      "' not found in reducedDims. Available dimensions: ",
+      paste(names(reduced_dims), collapse = ", ")
+    )
+    spatial_coords <- NULL
+  }
+  spatial_coords
+}
+
+main <- function() {
+  # Convert to AnnData
+  cat("Converting H5MU file to H5AD...\n")
+  h5file <- h5mu_to_h5ad(par$input, par$modality)
+
+  # Convert to SpatialExperiment
+  cat("Converting to SingleCellExperiment...\n")
+  sce <- anndataR::read_h5ad(h5file, as = "SingleCellExperiment")
+
+  # Extract spatial coordinates if specified
+  if (
+    !is.null(par$obsm_spatial_coordinates) &&
+      length(par$obsm_spatial_coordinates) > 0
+  ) {
+    cat("Reading in spatial coordinates...\n")
+    spatial_coords <- read_spatial_coordinates(
+      sce, par$obsm_spatial_coordinates
+    )
+    SingleCellExperiment::reducedDims(sce)[[
+      par$obsm_spatial_coordinates
+    ]] <- NULL
+  } else {
+    spatial_coords <- NULL
+  }
+
+  # Converting SingleCellExperiment to SpatialExperiment
+  cat("Converting to SpatialExperiment...\n")
+  spe <- as(sce, "SpatialExperiment")
+  SpatialExperiment::spatialCoords(spe) <- spatial_coords
+
+  # Saving SpatialExperiment object
+  cat("Saving SpatialExperiment object to:", par$output, "\n")
+  saveRDS(spe, file = par$output, compress = FALSE)
+}
+
+main()

src/convert/from_h5mu_to_spatialexperiment/test.R

@@ -0,0 +1,475 @@
+library(testthat)
+library(SpatialExperiment)
+library(SingleCellExperiment)
+library(hdf5r)
+library(Matrix)
+library(reticulate)
+
+mu <- reticulate::import("mudata")
+ad <- reticulate::import("anndata")
+
+## VIASH START
+meta <- list(
+  resources_dir = "resources_test"
+)
+## VIASH END
+
+# Helper function to create mock H5MU test data
+create_mock_h5mu <- function(path) {
+  n_obs <- 5
+  n_var_mod1 <- 4
+  n_var_mod2 <- 3
+
+  # ============== MOD1 MODALITY ==============
+
+  mod1_x_data <- matrix(c(
+    1, 2, 3, 0,
+    4, 5, 6, 2,
+    0, 1, 2, 3,
+    2, 0, 1, 4,
+    1, 3, 0, 2
+  ), nrow = n_obs, ncol = n_var_mod1, byrow = TRUE)
+
+
+  # Create obs dataframe
+  mod1_obs <- data.frame(
+    Obs1 = c("A", "B", "A", "C", "B"),
+    Obs2 = c(0.9, 0.8, 0.95, 0.7, 0.85),
+    Obs3 = c(FALSE, FALSE, TRUE, FALSE, FALSE),
+    row.names = paste0("cell_", 1:n_obs),
+    stringsAsFactors = FALSE
+  )
+  # Create var dataframe
+  mod1_var <- data.frame(
+    Feat1 = c("A", "B", "C", "D"),
+    Feat2 = c(TRUE, FALSE, TRUE, FALSE),
+    Feat3 = c(1.6, 2.2, 1.2, 1.8),
+    row.names = paste0("gene_", 1:n_var_mod1),
+    stringsAsFactors = FALSE
+  )
+
+  # Create layers
+  mod1_layers <- list(
+    counts = mod1_x_data * 2
+  )
+
+  # Create obsm
+  obsm_1 <- matrix(c(
+    100.5, 200.3,
+    150.2, 180.7,
+    120.8, 220.1,
+    180.4, 160.9,
+    200.1, 190.5
+  ), nrow = n_obs, ncol = 2, byrow = TRUE)
+
+  obsm_2 <- matrix(c(
+    -1.2, 0.8, 0.3,
+    1.1, -0.5, -0.2,
+    0.3, 1.2, 0.7,
+    -0.8, -0.3, 1.1,
+    0.9, 0.2, -0.9
+  ), nrow = n_obs, ncol = 3, byrow = TRUE)
+
+  mod1_obsm <- list(
+    Obsm1 = obsm_1,
+    Obsm2 = obsm_2
+  )
+
+  # Create uns (unstructured metadata)
+  mod1_uns <- list(
+    experiment_info = "metadata"
+  )
+
+  # Create AnnData object for mod1 using AnnDataR
+  ad_mod1 <- ad$AnnData(
+    X = mod1_x_data,
+    obs = mod1_obs,
+    var = mod1_var,
+    layers = mod1_layers,
+    obsm = mod1_obsm,
+    uns = mod1_uns
+  )
+
+  # ============== MOD2 MODALITY ==============
+
+  # Create expression matrix
+  mod2_x_data <- matrix(c(
+    10, 20, 15,
+    25, 30, 18,
+    12, 22, 20,
+    18, 25, 12,
+    20, 28, 16
+  ), nrow = n_obs, ncol = n_var_mod2, byrow = TRUE)
+
+  # Create obs dataframe
+  mod2_obs <- data.frame(
+    Obs = c("C", "D", "C", "E", "D"),
+    row.names = paste0("cell_", 1:n_obs),
+    stringsAsFactors = FALSE
+  )
+
+  # Create var dataframe
+  mod2_var <- data.frame(
+    Feat = c("d", "e", "g"),
+    row.names = paste0("protein_", 1:n_var_mod2),
+    stringsAsFactors = FALSE
+  )
+
+  # Create AnnData object for mod2
+  ad_mod2 <- ad$AnnData(
+    X = mod2_x_data,
+    obs = mod2_obs,
+    var = mod2_var
+  )
+
+  # ============== CREATE MUDATA ==============
+
+  # Create MuData object using reticulate
+  mdata <- mu$MuData(list(
+    mod1 = ad_mod1,
+    mod2 = ad_mod2
+  ))
+
+  # Write Mudata to path
+  mdata$write_h5mu(path)
+  path
+}
+
+# Main test
+test_simple_execution <- function() {
+  cat("> > Testing Simple Conversion\n")
+  cat("> Creating mock H5MU file\n")
+
+  # Create mock H5MU file
+  test_h5mu <- tempfile(fileext = ".h5mu")
+  create_mock_h5mu(test_h5mu)
+
+  # Output file
+  out_rds <- tempfile(fileext = ".rds")
+
+  # Run conversion
+  cat("> Running conversion\n")
+  out <- processx::run(
+    meta[["executable"]],
+    c(
+      "--input", test_h5mu,
+      "--modality", "mod1",
+      "--output", out_rds,
+      "--obsm_spatial_coordinates", "Obsm1"
+    )
+  )
+
+  cat("> Checking execution status\n")
+  testthat::expect_equal(out$status, 0)
+  testthat::expect_true(file.exists(out_rds))
+
+  cat("> Reading output file\n")
+  spe <- readRDS(file = out_rds)
+  testthat::expect_s4_class(spe, "SpatialExperiment")
+
+  cat("> Opening input file for comparison\n")
+  mod1 <- mu$read_h5ad(test_h5mu, mod = "mod1")
+
+  cat("> Testing dimensions\n")
+  dim_spe <- dim(spe)
+  dim_h5mu <- dim(mod1$X)
+
+  testthat::expect_equal(dim_spe[1], dim_h5mu[2])
+  testthat::expect_equal(dim_spe[2], dim_h5mu[1])
+  testthat::expect_equal(nrow(spe), 4)
+  testthat::expect_equal(ncol(spe), 5)
+
+  cat("> Testing colData (obs) transfer and data types\n")
+  col_data <- SummarizedExperiment::colData(spe)
+  coldata_cols <- colnames(col_data)
+  obs_cols <- colnames(mod1$obs)
+  testthat::expect_true(all(obs_cols %in% coldata_cols))
+
+  # Test data types in colData
+  testthat::expect_true(is.factor(col_data$Obs1))
+  testthat::expect_true(is.numeric(col_data$Obs2))
+  testthat::expect_true(is.logical(col_data$Obs3))
+
+  cat("> Testing rowData (var) transfer and data types\n")
+  row_data <- SummarizedExperiment::rowData(spe)
+  row_names <- colnames(row_data)
+  var_cols <- colnames(mod1$var)
+  testthat::expect_true(all(var_cols %in% row_names))
+
+  # Test data types in rowData
+  testthat::expect_true(is.character(row_data$Feat1))
+  testthat::expect_true(is.logical(row_data$Feat2))
+  testthat::expect_true(is.numeric(row_data$Feat3))
+
+  cat("> Testing spatialCoords\n")
+  spatial_coords <- SpatialExperiment::spatialCoords(spe)
+  testthat::expect_false(is.null(spatial_coords))
+  testthat::expect_equal(ncol(spatial_coords), 2)
+  testthat::expect_equal(nrow(spatial_coords), ncol(spe))
+  testthat::expect_identical(colnames(spatial_coords), c("x", "y"))
+
+  # Test spatial coordinate data types and values
+  testthat::expect_true(is.numeric(spatial_coords[, "x"]))
+  testthat::expect_true(is.numeric(spatial_coords[, "y"]))
+
+  # Compare with original spatial coordinates
+  original_spatial <- mod1$obsm[["Obsm1"]]
+  testthat::expect_equal(
+    as.numeric(original_spatial),
+    as.numeric(spatial_coords)
+  )
+
+  cat("> Testing assay data\n")
+  counts_matrix <- SummarizedExperiment::assays(spe)[["counts"]]
+  testthat::expect_true(is(counts_matrix, "Matrix") || is.matrix(counts_matrix))
+  testthat::expect_true(all(counts_matrix >= 0))
+  testthat::expect_equal(dim(counts_matrix), c(4, 5))
+
+  cat("> Testing reducedDims\n")
+  # PCA should not be in reducedDims since we only specified spatial
+  red_dims <- SingleCellExperiment::reducedDims(spe)
+  testthat::expect_false(is.null(red_dims))
+  testthat::expect_equal(names(red_dims), c("Obsm2"))
+  testthat::expect_equal(dim(red_dims$Obsm2), c(5, 3))
+  testthat::expect_true(is.numeric(red_dims$Obsm2))
+
+  # Compare with original spatial coordinates
+  original_dimred <- mod1$obsm[["Obsm2"]]
+  testthat::expect_equal(
+    as.numeric(red_dims$Obsm2),
+    as.numeric(original_dimred)
+  )
+
+  # Clean up
+  unlink(c(test_h5mu, out_rds))
+}
+
+test_xenium_execution <- function() {
+  cat("> > Testing Xenium Conversion\n")
+  xenium_h5mu <- paste0(
+    meta[["resources_dir"]],
+    "/xenium_tiny.h5mu"
+  )
+
+  # Output file
+  out_rds <- tempfile(fileext = ".rds")
+
+  # Run conversion
+  cat("> Running conversion\n")
+  out <- processx::run(
+    meta[["executable"]],
+    c(
+      "--input", xenium_h5mu,
+      "--modality", "rna",
+      "--output", out_rds,
+      "--obsm_spatial_coordinates", "spatial"
+    )
+  )
+
+  cat("> Checking execution status\n")
+  testthat::expect_equal(out$status, 0)
+  testthat::expect_true(file.exists(out_rds))
+
+  cat("> Reading output file\n")
+  xenium_spe <- readRDS(file = out_rds)
+  testthat::expect_s4_class(xenium_spe, "SpatialExperiment")
+
+  cat("> Opening input file for comparison\n")
+  rna_mod <- mu$read_h5ad(xenium_h5mu, mod = "rna")
+
+  cat("> Testing dimensions\n")
+  dim_spe <- dim(xenium_spe)
+  dim_h5mu <- dim(rna_mod$X)
+
+  testthat::expect_equal(dim_spe[1], dim_h5mu[2])
+  testthat::expect_equal(dim_spe[2], dim_h5mu[1])
+
+  cat("> Testing colData (obs) transfer and data types\n")
+  col_data <- SummarizedExperiment::colData(xenium_spe)
+  coldata_cols <- colnames(col_data)
+  obs_cols <- colnames(rna_mod$obs)
+  testthat::expect_true(all(obs_cols %in% coldata_cols))
+
+  cat("> Testing rowData (var) transfer and data types\n")
+  row_data <- SummarizedExperiment::rowData(xenium_spe)
+  row_names <- colnames(row_data)
+  var_cols <- colnames(rna_mod$var)
+  testthat::expect_true(all(var_cols %in% row_names))
+
+  cat("> Testing spatialCoords\n")
+  spatial_coords <- SpatialExperiment::spatialCoords(xenium_spe)
+  testthat::expect_false(is.null(spatial_coords))
+  testthat::expect_equal(ncol(spatial_coords), 2)
+  testthat::expect_equal(nrow(spatial_coords), ncol(xenium_spe))
+  testthat::expect_identical(colnames(spatial_coords), c("x", "y"))
+
+  # Test spatial coordinate data types and values
+  testthat::expect_true(is.numeric(spatial_coords[, "x"]))
+  testthat::expect_true(is.numeric(spatial_coords[, "y"]))
+
+  # Compare with original spatial coordinates
+  original_spatial <- rna_mod$obsm[["spatial"]]
+  testthat::expect_equal(
+    as.numeric(original_spatial),
+    as.numeric(spatial_coords)
+  )
+
+  # Clean up
+  unlink(c(xenium_h5mu, out_rds))
+}
+
+test_aviti_execution <- function() {
+  cat("> > Testing Aviti Conversion\n")
+  aviti_h5mu <- paste0(
+    meta[["resources_dir"]],
+    "/aviti_teton_tiny.h5mu"
+  )
+
+  # Output file
+  out_rds <- tempfile(fileext = ".rds")
+
+  # Run conversion
+  cat("> Running conversion\n")
+  out <- processx::run(
+    meta[["executable"]],
+    c(
+      "--input", aviti_h5mu,
+      "--modality", "rna",
+      "--output", out_rds,
+      "--obsm_spatial_coordinates", "spatial"
+    )
+  )
+
+  cat("> Checking execution status\n")
+  testthat::expect_equal(out$status, 0)
+  testthat::expect_true(file.exists(out_rds))
+
+  cat("> Reading output file\n")
+  aviti_spe <- readRDS(file = out_rds)
+  testthat::expect_s4_class(aviti_spe, "SpatialExperiment")
+
+  cat("> Opening input file for comparison\n")
+  rna_mod <- mu$read_h5ad(aviti_h5mu, mod = "rna")
+
+  cat("> Testing dimensions\n")
+  dim_spe <- dim(aviti_spe)
+  dim_h5mu <- dim(rna_mod$X)
+
+  testthat::expect_equal(dim_spe[1], dim_h5mu[2])
+  testthat::expect_equal(dim_spe[2], dim_h5mu[1])
+
+  cat("> Testing colData (obs) transfer and data types\n")
+  col_data <- SummarizedExperiment::colData(aviti_spe)
+  coldata_cols <- colnames(col_data)
+  obs_cols <- colnames(rna_mod$obs)
+  testthat::expect_true(all(obs_cols %in% coldata_cols))
+
+  cat("> Testing rowData (var) transfer and data types\n")
+  row_data <- SummarizedExperiment::rowData(aviti_spe)
+  row_names <- colnames(row_data)
+  var_cols <- colnames(rna_mod$var)
+  testthat::expect_true(all(var_cols %in% row_names))
+
+  cat("> Testing spatialCoords\n")
+  spatial_coords <- SpatialExperiment::spatialCoords(aviti_spe)
+  testthat::expect_false(is.null(spatial_coords))
+  testthat::expect_equal(ncol(spatial_coords), 2)
+  testthat::expect_equal(nrow(spatial_coords), ncol(aviti_spe))
+  testthat::expect_identical(colnames(spatial_coords), c("x", "y"))
+
+  # Test spatial coordinate data types and values
+  testthat::expect_true(is.numeric(spatial_coords[, "x"]))
+  testthat::expect_true(is.numeric(spatial_coords[, "y"]))
+
+  # Compare with original spatial coordinates
+  original_spatial <- rna_mod$obsm[["spatial"]]
+  testthat::expect_equal(
+    as.numeric(original_spatial),
+    as.numeric(spatial_coords)
+  )
+
+  # Clean up
+  unlink(c(aviti_h5mu, out_rds))
+}
+
+test_cosmx_execution <- function() {
+  cat("> > Testing CosMx Conversion\n")
+  cosmx_h5mu <- paste0(
+    meta[["resources_dir"]],
+    "/Lung5_Rep2_tiny.h5mu"
+  )
+
+  # Output file
+  out_rds <- tempfile(fileext = ".rds")
+
+  # Run conversion
+  cat("> Running conversion\n")
+  out <- processx::run(
+    meta[["executable"]],
+    c(
+      "--input", cosmx_h5mu,
+      "--modality", "rna",
+      "--output", out_rds,
+      "--obsm_spatial_coordinates", "spatial"
+    )
+  )
+
+  cat("> Checking execution status\n")
+  testthat::expect_equal(out$status, 0)
+  testthat::expect_true(file.exists(out_rds))
+
+  cat("> Reading output file\n")
+  cosmx_spe <- readRDS(file = out_rds)
+  testthat::expect_s4_class(cosmx_spe, "SpatialExperiment")
+
+  cat("> Opening input file for comparison\n")
+  rna_mod <- mu$read_h5ad(cosmx_h5mu, mod = "rna")
+
+  cat("> Testing dimensions\n")
+  dim_spe <- dim(cosmx_spe)
+  dim_h5mu <- dim(rna_mod$X)
+
+  testthat::expect_equal(dim_spe[1], dim_h5mu[2])
+  testthat::expect_equal(dim_spe[2], dim_h5mu[1])
+
+  cat("> Testing colData (obs) transfer and data types\n")
+  col_data <- SummarizedExperiment::colData(cosmx_spe)
+  coldata_cols <- colnames(col_data)
+  obs_cols <- colnames(rna_mod$obs)
+  testthat::expect_true(all(obs_cols %in% coldata_cols))
+
+  cat("> Testing rowData (var) transfer and data types\n")
+  row_data <- SummarizedExperiment::rowData(cosmx_spe)
+  row_names <- colnames(row_data)
+  var_cols <- colnames(rna_mod$var)
+  testthat::expect_true(all(var_cols %in% row_names))
+
+  cat("> Testing spatialCoords\n")
+  spatial_coords <- SpatialExperiment::spatialCoords(cosmx_spe)
+  testthat::expect_false(is.null(spatial_coords))
+  testthat::expect_equal(ncol(spatial_coords), 2)
+  testthat::expect_equal(nrow(spatial_coords), ncol(cosmx_spe))
+  testthat::expect_identical(colnames(spatial_coords), c("x", "y"))
+
+  # Test spatial coordinate data types and values
+  testthat::expect_true(is.numeric(spatial_coords[, "x"]))
+  testthat::expect_true(is.numeric(spatial_coords[, "y"]))
+
+  # Compare with original spatial coordinates
+  original_spatial <- rna_mod$obsm[["spatial"]]
+  testthat::expect_equal(
+    as.numeric(original_spatial),
+    as.numeric(spatial_coords)
+  )
+
+  # Clean up
+  unlink(c(cosmx_h5mu, out_rds))
+}
+
+cat("Starting tests...")
+test_simple_execution()
+test_xenium_execution()
+test_aviti_execution()
+test_cosmx_execution()
+
+cat("All tests completed!\n")

src/convert/from_spatialdata_to_h5mu/config.vsh.yaml

@@ -0,0 +1,56 @@
+name: "from_spatialdata_to_h5mu"
+namespace: "convert"
+scope: "public"
+description: |
+  Reads in the Tables field stored in a SpatialData object and converts it to an h5mu file.
+    
+authors:
+  - __merge__: /src/authors/dorien_roosen.yaml
+    roles: [ maintainer ]
+  - __merge__: /src/authors/weiwei_schultz.yaml
+    roles: [ contributor ]
+arguments:
+  - name: "--input"
+    alternatives: ["-i"]
+    type: file
+    description: Input zarr folder where the SpatialData object is stored.
+    example: input.zarr
+    direction: input
+    required: true
+  - name: "--modality"
+    type: string
+    default: rna
+  - name: "--output"
+    alternatives: ["-o"]
+    type: file
+    description: The output h5mu file.
+    example: "output.h5mu"
+    direction: output
+  - name: "--output_compression"
+    type: string
+    choices: ["gzip", "lzf"]
+    required: false
+    example: "gzip"
+resources:
+  - type: python_script
+    path: script.py
+  - path: /src/utils/setup_logger.py
+test_resources:
+  - type: python_script
+    path: test.py
+  - path: /resources_test/xenium/xenium_tiny.zarr
+engines:
+  - type: docker
+    image: python:3.12-slim
+    setup:
+      - type: apt
+        packages: 
+          - procps
+      - type: python
+        __merge__: [/src/base/requirements/anndata_mudata.yaml, /src/base/requirements/spatialdata.yaml]
+    __merge__: [ /src/base/requirements/python_test_setup.yaml, .]
+runners:
+  - type: executable
+  - type: nextflow
+    directives:
+      label: [lowmem, singlecpu]

src/convert/from_spatialdata_to_h5mu/script.py

@@ -0,0 +1,28 @@
+import sys
+import spatialdata as sd
+import mudata as mu
+
+## VIASH START
+par = {
+    "input": "./resources_test/xenium/xenium_tiny.zarr",
+    "output": "./resources_test/xenium/xenium_tiny.h5mu",
+    "modality": "rna",
+    "output_compression": None,
+}
+meta = {"resources_dir": "src/utils"}
+## VIASH END
+
+sys.path.append(meta["resources_dir"])
+from setup_logger import setup_logger
+
+logger = setup_logger()
+
+logger.info("Reading in Xenium data...")
+sdata = sd.read_zarr(par["input"])
+
+logger.info("Fetching AnnData table from SpatialData object...")
+adata = sdata.tables["table"]
+
+logger.info("Writing output MuData object...")
+mdata = mu.MuData({par["modality"]: adata})
+mdata.write_h5mu(par["output"], compression=par["output_compression"])

src/convert/from_spatialdata_to_h5mu/test.py

@@ -0,0 +1,52 @@
+import pytest
+import sys
+import mudata as mu
+
+
+def test_simple_execution(run_component, tmp_path):
+    output = tmp_path / "output.h5mu"
+
+    run_component(
+        [
+            "--input",
+            meta["resources_dir"] + "/xenium_tiny.zarr",
+            "--output",
+            output,
+        ]
+    )
+    assert output.is_file(), "output file was not created"
+
+    mdata = mu.read_h5mu(output)
+    assert list(mdata.mod.keys()) == ["rna"], "Expected modality rna"
+
+    adata = mdata.mod["rna"]
+
+    # TODO: update what is checked here when spatialdata from other experimental set-ups are tested (e.g. cosmx, visium)
+    assert list(adata.obs.keys()) == [
+        "cell_id",
+        "transcript_counts",
+        "control_probe_counts",
+        "genomic_control_counts",
+        "control_codeword_counts",
+        "unassigned_codeword_counts",
+        "deprecated_codeword_counts",
+        "total_counts",
+        "cell_area",
+        "nucleus_area",
+        "nucleus_count",
+        "segmentation_method",
+        "region",
+        "z_level",
+        "cell_labels",
+    ]
+
+    assert list(adata.uns.keys()) == ["spatialdata_attrs"]
+    assert list(adata.obsm.keys()) == ["spatial"]
+    assert list(adata.var.keys()) == ["gene_ids", "feature_types", "genome"]
+
+    assert all(adata.var["feature_types"] == "Gene Expression")
+    assert adata.obsm["spatial"].dtype == "float"
+
+
+if __name__ == "__main__":
+    sys.exit(pytest.main([__file__]))

src/convert/from_xenium_to_h5mu/config.vsh.yaml

@@ -0,0 +1,78 @@
+name: "from_xenium_to_h5mu"
+namespace: "convert"
+scope: "public"
+description: |
+  Converts the output from Xenium to a single .h5mu file, where the count matrix is written to the `rna` modality.
+  The following files are expected to be present in the Xenium output bundle:
+  ├── cell_feature_matrix.h5
+  ├── cells.parquet
+  ├── experiment.xenium
+  └── metrics_summary.csv
+
+authors:
+  - __merge__: /src/authors/dorien_roosen.yaml
+    roles: [ maintainer ]
+arguments:
+  - name: "--input"
+    alternatives: ["-i"]
+    type: file
+    description: Input folder. Must contain the output from a Xenium run.
+    example: xenium_output_bundle
+    direction: input
+    required: true
+  - name: "--output"
+    alternatives: ["-o"]
+    type: file
+    description: Output .h5mu file.
+    example: "xenium.h5mu"
+    direction: output
+  - name: "--obsm_coordinates"
+    type: string
+    description: Name of the .obsm slot under which to store the cell centroid coordinates.
+    default: "spatial"
+  - name: "--uns_experiment"
+    type: string
+    description: Name of the .uns slot under which to store the Xenium experiment specifications.
+    default: "xenium_experiment"
+  - name: "--uns_metrics"
+    type: string
+    description: Name of the .uns slot under which to store the summary QC metrics.
+    default: "xenium_metrics"
+__merge__: [., /src/base/h5_compression_argument.yaml]
+
+resources:
+  - type: python_script
+    path: script.py
+  - path: /src/utils/setup_logger.py
+  - path: /src/utils/unzip_archived_folder.py
+test_resources:
+  - type: python_script
+    path: test.py
+  - path: /resources_test/xenium/xenium_tiny
+engines:
+  - type: docker
+    image: python:3.12-slim
+    setup:
+      - type: apt
+        packages: 
+          - procps
+          - build-essential
+          - zlib1g-dev
+          - git
+      - type: python
+        __merge__: [/src/base/requirements/anndata_mudata.yaml, /src/base/requirements/scanpy.yaml, .]
+        packages: [ pyarrow ]
+        # Windows explorer uses DEFLATE64 compression for large ZIP files,
+        # which is not supported by most standard library zipfile module
+        git: [ https://codeberg.org/miurahr/zipfile-inflate64.git@v0.2 ]
+    test_setup:
+      - type: apt
+        packages:
+          - zip
+      - type: python
+        __merge__: [ /src/base/requirements/viashpy.yaml, .]
+runners:
+  - type: executable
+  - type: nextflow
+    directives:
+      label: [lowmem, singlecpu]

src/convert/from_xenium_to_h5mu/script.py

@@ -0,0 +1,100 @@
+import sys
+from pathlib import Path
+import scanpy as sc
+import pandas as pd
+import mudata as mu
+import zipfile_inflate64 as zipfile
+import json
+import os
+
+## VIASH START
+par = {
+    "input": "test/xenium_tiny.zip",
+    "output": "xenium_tiny_test.h5mu",
+    "output_compression": "gzip",
+    "obsm_coordinates": "spatial",
+    "uns_experiment": "xenium_experiment",
+    "uns_metrics": "xenium_metrics",
+}
+meta = {"resources_dir": "src/utils"}
+## VIASH END
+
+sys.path.append(meta["resources_dir"])
+from setup_logger import setup_logger
+from unzip_archived_folder import extract_selected_files_from_zip
+
+logger = setup_logger()
+
+
+def _retrieve_input_data(xenium_output_bundle):
+    # Expected folder structure (showing only relevant files):
+    # ├── cell_feature_matrix.h5
+    # ├── cells.parquet
+    # ├── experiment.xenium
+    # └── metrics_summary.csv
+
+    required_file_patterns = {
+        "count_matrix": "**/cell_feature_matrix.h5",
+        "cells_metadata": "**/cells.parquet",
+        "experiment": "**/experiment.xenium",
+        "metrics_summary": "**/metrics_summary.csv",
+    }
+
+    if zipfile.is_zipfile(xenium_output_bundle):
+        xenium_output_bundle = extract_selected_files_from_zip(
+            xenium_output_bundle,
+            members=[pattern for pattern in required_file_patterns.values()],
+        )
+    else:
+        xenium_output_bundle = Path(xenium_output_bundle)
+
+    assert os.path.isdir(xenium_output_bundle), (
+        "Input is expected to be a (compressed) directory."
+    )
+
+    input_data = {}
+    for key, pattern in required_file_patterns.items():
+        file = list(xenium_output_bundle.glob(pattern))
+        assert len(file) == 1, (
+            f"Expected exactly one file matching pattern {pattern}, found {len(file)}."
+        )
+        input_data[key] = file[0]
+
+    return input_data
+
+
+def _format_cell_id_column(cell_id_column: pd.Series) -> pd.Series:
+    """Convert cell IDs to string format, decoding bytes if necessary."""
+    return cell_id_column.apply(
+        lambda x: x.decode("utf-8") if isinstance(x, bytes) else str(x)
+    )
+
+
+# Read data from Xenium output bundle
+logger.info("Reading input data...")
+
+input_data = _retrieve_input_data(par["input"])
+
+adata = sc.read_10x_h5(input_data["count_matrix"])
+metadata = pd.read_parquet(input_data["cells_metadata"], engine="pyarrow")
+with open(input_data["experiment"], "r") as f:
+    specs = json.load(f)
+metrics_summary = pd.read_csv(
+    input_data["metrics_summary"], decimal=".", quotechar='"', thousands=","
+)
+
+# Extract and format required columns
+cell_ids = _format_cell_id_column(metadata["cell_id"])
+coordinates = metadata[["x_centroid", "y_centroid"]].to_numpy()
+metadata.drop(["cell_id", "x_centroid", "y_centroid"], axis=1, inplace=True)
+
+# Updata AnnData with metadata
+adata.obs = metadata
+adata.obs_names = cell_ids
+adata.obsm[par["obsm_coordinates"]] = coordinates
+adata.uns[par["uns_experiment"]] = specs
+adata.uns[par["uns_metrics"]] = metrics_summary
+
+# Write output MuData
+mdata = mu.MuData({"rna": adata})
+mdata.write_h5mu(par["output"], compression=par["output_compression"])

src/convert/from_xenium_to_h5mu/test.py

@@ -0,0 +1,161 @@
+import pytest
+import sys
+import subprocess
+import mudata as mu
+
+## VIASH START
+meta = {
+    "executable": "./target/executable/convert/from_xenium_to_h5mu/from_xenium_to_h5mu",
+    "resources_dir": "resources_test/",
+    "config": "src/convert/from_xenium_to_h5mu/config.vsh.yaml",
+}
+## VIASH END
+
+input = f"{meta['resources_dir']}/xenium_tiny"
+
+
+def test_simple_execution(run_component, tmp_path):
+    output = tmp_path / "xenium.h5mu"
+
+    # run component
+    run_component(
+        ["--input", input, "--output", str(output), "--output_compression", "gzip"]
+    )
+
+    assert output.is_file(), "output file was not created"
+
+    mdata = mu.read_h5mu(output)
+    assert list(mdata.mod.keys()) == ["rna"], "Expected modality rna"
+    adata = mdata.mod["rna"]
+
+    assert list(adata.obs.keys()) == [
+        "transcript_counts",
+        "control_probe_counts",
+        "genomic_control_counts",
+        "control_codeword_counts",
+        "unassigned_codeword_counts",
+        "deprecated_codeword_counts",
+        "total_counts",
+        "cell_area",
+        "nucleus_area",
+        "nucleus_count",
+        "segmentation_method",
+    ]
+
+    assert list(adata.uns.keys()) == ["xenium_experiment", "xenium_metrics"]
+    assert list(adata.obsm.keys()) == ["spatial"]
+    assert list(adata.var.keys()) == ["gene_ids", "feature_types", "genome"]
+
+    assert adata.X.dtype.kind == "f"
+    assert all(adata.var["feature_types"] == "Gene Expression")
+    assert adata.obsm["spatial"].dtype == "float"
+    obs_counts = [
+        "transcript_counts",
+        "control_probe_counts",
+        "genomic_control_counts",
+        "unassigned_codeword_counts",
+        "deprecated_codeword_counts",
+        "total_counts",
+        "nucleus_count",
+    ]
+    assert all([adata.obs[obs].dtype == "int" for obs in obs_counts])
+    obs_areas = ["cell_area", "nucleus_area"]
+    assert all([adata.obs[obs].dtype == "float" for obs in obs_areas])
+
+
+def test_compressed_input(run_component, tmp_path):
+    output = tmp_path / "xenium.h5mu"
+    zipped_input = tmp_path / "xenium_tiny.zip"
+
+    subprocess.run(
+        ["zip", "-r", str(zipped_input), "xenium_tiny"],
+        cwd=meta["resources_dir"],
+        check=True,
+    )
+
+    # run component
+    run_component(
+        [
+            "--input",
+            zipped_input,
+            "--output",
+            str(output),
+            "--output_compression",
+            "gzip",
+        ]
+    )
+
+    assert output.is_file(), "output file was not created"
+
+    mdata = mu.read_h5mu(output)
+    assert list(mdata.mod.keys()) == ["rna"], "Expected modality rna"
+    adata = mdata.mod["rna"]
+
+    assert list(adata.obs.keys()) == [
+        "transcript_counts",
+        "control_probe_counts",
+        "genomic_control_counts",
+        "control_codeword_counts",
+        "unassigned_codeword_counts",
+        "deprecated_codeword_counts",
+        "total_counts",
+        "cell_area",
+        "nucleus_area",
+        "nucleus_count",
+        "segmentation_method",
+    ]
+
+    assert list(adata.uns.keys()) == ["xenium_experiment", "xenium_metrics"]
+    assert list(adata.obsm.keys()) == ["spatial"]
+    assert list(adata.var.keys()) == ["gene_ids", "feature_types", "genome"]
+
+    assert adata.X.dtype.kind == "f"
+    assert all(adata.var["feature_types"] == "Gene Expression")
+    assert adata.obsm["spatial"].dtype == "float"
+    obs_counts = [
+        "transcript_counts",
+        "control_probe_counts",
+        "genomic_control_counts",
+        "unassigned_codeword_counts",
+        "deprecated_codeword_counts",
+        "total_counts",
+        "nucleus_count",
+    ]
+    assert all([adata.obs[obs].dtype == "int" for obs in obs_counts])
+    obs_areas = ["cell_area", "nucleus_area"]
+    assert all([adata.obs[obs].dtype == "float" for obs in obs_areas])
+
+
+def test_rename_fields(run_component, tmp_path):
+    output = tmp_path / "xenium.h5mu"
+
+    # run component
+    run_component(
+        [
+            "--input",
+            input,
+            "--output",
+            str(output),
+            "--obsm_coordinates",
+            "test_coord",
+            "--uns_experiment",
+            "test_experiment",
+            "--uns_metrics",
+            "test_metrics",
+            "--output_compression",
+            "gzip",
+        ]
+    )
+
+    assert output.is_file(), "output file was not created"
+
+    mdata = mu.read_h5mu(output)
+    assert list(mdata.mod.keys()) == ["rna"]
+    adata = mdata.mod["rna"]
+
+    assert list(adata.uns.keys()) == ["test_experiment", "test_metrics"]
+    assert list(adata.obsm.keys()) == ["test_coord"]
+
+
+if __name__ == "__main__":
+    sys.exit(pytest.main([__file__]))

src/convert/from_xenium_to_spatialdata/config.vsh.yaml

@@ -0,0 +1,110 @@
+name: "from_xenium_to_spatialdata"
+namespace: "convert"
+scope: "public"
+description: |
+  Converts the output from 10X Genomics Xenium dataset into a SpatialData objcet.
+  By default, the following files will be converted:
+    - `experiment.xenium`: File containing specifications.
+    - `nucleus_boundaries.parquet`: Polygons of nucleus boundaries.
+    - `cell_boundaries.parquet`: Polygons of cell boundaries.
+    - `transcripts.parquet`: File containing transcripts.
+    - `cell_feature_matrix.h5`: File containing cell feature matrix.
+    - `cells.parquet`: File containing cell metadata.
+    - `morphology_mip.ome.tif`: File containing morphology mip.
+    - `morphology_focus.ome.tif`: File containing morphology focus.
+authors:
+  - __merge__: /src/authors/dorien_roosen.yaml
+    roles: [ maintainer ]
+  - __merge__: /src/authors/weiwei_schultz.yaml
+    roles: [ contributor ]
+arguments:
+  - name: "--input"
+    alternatives: ["-i"]
+    type: file
+    description: Input folder. Must contain the output from a xenium run.
+    example: xenium_data
+    direction: input
+    required: true
+  - name: "--output"
+    alternatives: ["-o"]
+    type: file
+    description: Zarr directory where the SpatialData object will be stored
+    example: "xenium_data.zarr"
+    direction: output
+  - name: "--cells_boundaries"
+    type: boolean
+    default: True
+    description: Whether to read cell boundaries (polygons).
+  - name: "--nucleus_boundaries"
+    type: boolean
+    default: True
+    description: Whether to read nucleus boundaries (polygons).
+  - name: "--cells_as_circles"
+    type: boolean_true
+    description: Whether to read cells also as circles (the center and the radius of each circle is computed from the corresponding labels cell).
+  - name: "--cells_labels"
+    type: boolean
+    default: True
+    description: Whether to read cell labels (raster). The polygonal version of the cell labels are simplified for visualization purposes, and using the raster version is recommended for analysis.
+  - name: "--transcripts"
+    type: boolean
+    default: True
+    description: Whether to read transcripts.
+  - name: "--nucleus_labels"
+    type: boolean
+    default: True
+    description: Whether to read nucleus labels (raster). The polygonal version of the nucleus labels are simplified for visualization purposes, and using the raster version is recommended for analysis.
+  - name: "--morphology_mip"
+    type: boolean
+    default: True
+    description: Whether to read the morphology mip image (available in versions < 2.0.0).
+  - name: "--morphology_focus"
+    type: boolean
+    default: True
+    description: Whether to read the morphology focus image.
+  - name: "--aligned_images"
+    type: boolean
+    default: True
+    description: Whether to also parse, when available, additional H&E or IF aligned images. For more control over the aligned images being read, in particular, to specify the axes of the aligned images, please set this parameter to False and use the xenium_aligned_image function directly.
+  - name: "--cells_table"
+    type: boolean
+    default: True
+    description: Whether to read the cell annotations in the AnnData table.
+  - name: "--n_jobs"
+    type: integer
+    default: 1
+
+resources:
+  - type: python_script
+    path: script.py
+  - path: /src/utils/setup_logger.py
+  - path: /src/utils/unzip_archived_folder.py
+test_resources:
+  - type: python_script
+    path: test.py
+  - path: /resources_test/xenium/xenium_tiny/
+engines:
+  - type: docker
+    image: python:3.12-slim
+    setup:
+      - type: apt
+        packages: 
+          - procps
+          - build-essential
+          - zlib1g-dev
+          - git
+      - type: python
+        # Windows explorer uses DEFLATE64 compression for large ZIP files,
+        # which is not supported by most standard library zipfile module
+        git: [ https://codeberg.org/miurahr/zipfile-inflate64.git@v0.2 ]
+        __merge__: [ /src/base/requirements/spatialdata-io.yaml, . ]
+    test_setup:
+      - type: apt
+        packages:
+          - zip
+    __merge__: [ /src/base/requirements/python_test_setup.yaml, .]
+runners:
+  - type: executable
+  - type: nextflow
+    directives:
+      label: [lowmem, singlecpu]

src/convert/from_xenium_to_spatialdata/script.py

@@ -0,0 +1,65 @@
+import sys
+from spatialdata_io import xenium
+import zipfile_inflate64 as zipfile
+from pathlib import Path
+
+## VIASH START
+par = {
+    "input": "resources_test/xenium/xenium_tiny",
+    "output": "./test/xenium_tiny.zarr",
+    "cells_boundaries": True,
+    "nucleus_boundaries": True,
+    "cells_as_circles": None,
+    "cells_labels": True,
+    "nucleus_labels": True,
+    "transcripts": True,
+    "morphology_mip": True,
+    "morphology_focus": True,
+    "aligned_images": True,
+    "cells_table": True,
+    "n_jobs": 1,
+}
+meta = {"resources_dir": "src/utils"}
+## VIASH END
+
+sys.path.append(meta["resources_dir"])
+from setup_logger import setup_logger
+from unzip_archived_folder import unzip_archived_folder
+
+logger = setup_logger()
+
+logger.info("Reading in Xenium data...")
+
+if zipfile.is_zipfile(par["input"]):
+    required_file_patterns = [
+        "**/experiment.xenium",
+        "**/nucleus_boundaries.parquet",
+        "**/cell_boundaries.parquet",
+        "**/transcripts.parquet",
+        "**/cell_feature_matrix.h5",
+        "**/cells.parquet",
+        "**/morphology_mip.ome.tif",
+        "**/morphology_focus.ome.tif",
+    ]
+    xenium_output_bundle = unzip_archived_folder(par["input"])
+else:
+    xenium_output_bundle = Path(par["input"])
+
+sdata = xenium(
+    xenium_output_bundle,
+    cells_boundaries=par["cells_boundaries"],
+    nucleus_boundaries=par["nucleus_boundaries"],
+    cells_as_circles=par["cells_as_circles"],
+    cells_labels=par["cells_labels"],
+    nucleus_labels=par["nucleus_labels"],
+    transcripts=par["transcripts"],
+    morphology_mip=par["morphology_mip"],  # only available in version < 2.0.0
+    morphology_focus=par["morphology_focus"],
+    aligned_images=par["aligned_images"],
+    cells_table=par["cells_table"],
+    n_jobs=par["n_jobs"],
+)
+
+
+logger.info("Writing out SpatialData object to Zarr...")
+sdata.write(par["output"], overwrite=True)

src/convert/from_xenium_to_spatialdata/test.py

@@ -0,0 +1,69 @@
+import pytest
+import os
+import sys
+import spatialdata as sd
+import subprocess
+
+
+def test_simple_execution(run_component, tmp_path):
+    output_sd_path = tmp_path / "sd"
+
+    run_component(
+        [
+            "--input",
+            meta["resources_dir"] + "/xenium_tiny",
+            "--output",
+            output_sd_path,
+        ]
+    )
+
+    assert os.path.exists(output_sd_path), "Output zarr folder was not created"
+
+    sdata = sd.read_zarr(output_sd_path)
+    assert isinstance(sdata, sd.SpatialData), (
+        "the generated output is not a SpatialData object"
+    )
+
+    assert os.path.exists(output_sd_path / "images"), "images folder was not created"
+    assert os.path.exists(output_sd_path / "labels"), "labels folder was not created"
+    assert os.path.exists(output_sd_path / "points"), "images folder was not created"
+    assert os.path.exists(output_sd_path / "shapes"), "shapes folder was not created"
+    assert os.path.exists(output_sd_path / "tables"), "tables folder was not created"
+    assert (output_sd_path / "zmetadata").is_file(), "zmetadata file was not created"
+
+
+def test_compressed_input(run_component, tmp_path):
+    output_sd_path = tmp_path / "sd"
+    zipped_input = tmp_path / "xenium_tiny.zip"
+
+    subprocess.run(
+        ["zip", "-r", str(zipped_input), "xenium_tiny"],
+        cwd=meta["resources_dir"],
+        check=True,
+    )
+    run_component(
+        [
+            "--input",
+            zipped_input,
+            "--output",
+            output_sd_path,
+        ]
+    )
+
+    assert os.path.exists(output_sd_path), "Output zarr folder was not created"
+
+    sdata = sd.read_zarr(output_sd_path)
+    assert isinstance(sdata, sd.SpatialData), (
+        "the generated output is not a SpatialData object"
+    )
+
+    assert os.path.exists(output_sd_path / "images"), "images folder was not created"
+    assert os.path.exists(output_sd_path / "labels"), "labels folder was not created"
+    assert os.path.exists(output_sd_path / "points"), "images folder was not created"
+    assert os.path.exists(output_sd_path / "shapes"), "shapes folder was not created"
+    assert os.path.exists(output_sd_path / "tables"), "tables folder was not created"
+    assert (output_sd_path / "zmetadata").is_file(), "zmetadata file was not created"
+
+
+if __name__ == "__main__":
+    sys.exit(pytest.main([__file__]))

src/convert/from_xenium_to_spatialexperiment/config.vsh.yaml

@@ -0,0 +1,76 @@
+name: "from_xenium_to_spatialexperiment"
+namespace: "convert"
+scope: "public"
+description: |
+  Creates a SpatialExperiment object from the downloaded unzipped Xenium Output Bundle directory
+  for 10x Genomics Xenium spatial gene expression data, and saves it as a SpatialExperiment object.
+  The constructor assumes the downloaded unzipped Xenium Output Bundle has the following structure:
+
+  Mandatory files
+  · | — cell_feature_matrix.h5
+  · | — cells.parquet
+  Optional files, by default added to the metadata() as a list of paths (will be converted to parquet):
+  · | — transcripts.parquet
+  · | — cell_boundaries.parquet
+  · | — nucleus_boundaries.parquet
+  · | — experiment.xenium
+
+authors:
+  - __merge__: /src/authors/dorien_roosen.yaml
+    roles: [ author, maintainer ]
+arguments:
+  - name: "--input"
+    alternatives: ["-i"]
+    type: file
+    description: Input Xenium Output Bundle
+    direction: input
+    required: true
+    example: path/to/xenium_bundle
+  - name: "--add_experiment_xenium"
+    type: boolean
+    default: true
+    description: Whether to add xenium.experiment parameters to the metadata.
+  - name: "--add_parquet_paths"
+    type: boolean
+    default: true
+    description: |
+      Whether to add parquet paths to the metadata. 
+      If True, `transcripts.parquet`, `cell_boundaries.parquet`, `nucleus_boundaries.parquet` will be added to the metadata. 
+  - name: "--alternative_experiment_features"
+    type: string
+    multiple: true
+    description: Feature names containing these strings will be moved to altExps(sxe) slots as separate SpatialExperiment objects.
+    default: [NegControlProbe, UnassignedCodeword, NegControlCodeword, antisense, BLANK]
+  - name: "--output"
+    alternatives: ["-o"]
+    type: file
+    description: Output SpatialExperiment file
+    direction: output
+    required: true
+    example: output.rds
+resources:
+  - type: r_script
+    path: script.R
+  - path: /src/utils/unzip_archived_folder.R
+test_resources:
+  - type: r_script
+    path: test.R
+  - path: /resources_test/xenium/xenium_tiny
+engines:
+  - type: docker
+    image: rocker/r2u:24.04
+    setup:
+      - type: apt
+        packages: 
+          - libhdf5-dev
+          - libgeos-dev
+      - type: r
+        bioc: [ SpatialExperimentIO ]
+    test_setup:
+      - type: r
+        cran: [ testthat ]
+runners:
+  - type: executable
+  - type: nextflow
+    directives:
+      label: [lowmem, singlecpu]

src/convert/from_xenium_to_spatialexperiment/script.R

@@ -0,0 +1,53 @@
+library(SpatialExperimentIO)
+
+### VIASH START
+par <- list(
+  input = "resources_test/xenium/temp_dir.zip",
+  add_experiment_xenium = TRUE,
+  add_parquet_paths = TRUE,
+  alternative_experiment_features = c(
+    "NegControlProbe", "UnassignedCodeword",
+    "NegControlCodeword", "antisense", "BLANK"
+  ),
+  output = "spe_test.rds"
+)
+meta <- list(
+  resources_dir = "src/utils/"
+)
+### VIASH END
+
+source(paste0(meta$resources_dir, "/unzip_archived_folder.R"))
+
+cat("Reading input data...")
+if (tools::file_ext(par$input) == "zip") {
+  required_file_patterns <- c(
+    "**/cell_feature_matrix.h5",
+    "**/*.parquet",
+    "**/experiment.xenium"
+  )
+  tmp_dir <- extract_selected_files(
+    par$input,
+    members = required_file_patterns
+  )
+  xenium_output_bundle <- file.path(
+    tmp_dir,
+    tools::file_path_sans_ext(basename(par$input))
+  )
+} else {
+  xenium_output_bundle <- par$input
+}
+
+cat("Converting to SpatialExperiment")
+spe <- readXeniumSXE(
+  dirName = xenium_output_bundle,
+  returnType = "SPE",
+  countMatPattern = "cell_feature_matrix.h5",
+  metaDataPattern = "cells.parquet",
+  coordNames = c("x_centroid", "y_centroid"),
+  addExperimentXenium = par$add_experiment_xenium,
+  addParquetPaths = par$add_parquet_paths,
+  altExps = par$alternative_experiment_features
+)
+
+cat("Saving output...")
+saveRDS(spe, file = par$output)

src/convert/from_xenium_to_spatialexperiment/test.R

@@ -0,0 +1,182 @@
+library(testthat, warn.conflicts = FALSE)
+library(SpatialExperimentIO)
+library(SpatialExperiment)
+
+## VIASH START
+meta <- list(
+  executable = "./from_xenium_to_spatialexperiment",
+  resources_dir = "resources_test/xenium",
+  name = "from_xenium_to_spatial_experiment"
+)
+## VIASH END
+
+cat("> Checking simple execution\n")
+
+spe <- paste0(
+  meta[["resources_dir"]],
+  "/xenium_tiny"
+)
+out_rds <- "output.rds"
+
+cat("> Running ", meta[["name"]], "\n", sep = "")
+out <- processx::run(
+  meta[["executable"]],
+  c(
+    "--input", spe,
+    "--output", out_rds
+  )
+)
+
+cat("> Checking whether output file exists\n")
+expect_equal(out$status, 0)
+expect_true(file.exists(out_rds))
+
+cat("> Reading output file\n")
+obj <- readRDS(file = out_rds)
+
+cat("> Checking whether Seurat object is in the right format\n")
+# Object type
+expect_is(obj, "SpatialExperiment")
+# Assay structure
+expect_equal(names(slot(obj, "assays")), "counts")
+# Spatial coordinates
+expect_equal(spatialCoordsNames(obj), c("x_centroid", "y_centroid"))
+# Alternative experiments
+expect_equal(
+  altExpNames(obj),
+  c("NegControlProbe", "UnassignedCodeword", "NegControlCodeword")
+)
+# Metadata components
+metadata_components <- c(
+  "experiment.xenium", "transcripts", "cell_boundaries", "nucleus_boundaries"
+)
+expect_named(
+  metadata(obj),
+  metadata_components,
+  ignore.order = TRUE
+)
+# Parquet paths
+parquet_components <- c("transcripts", "cell_boundaries", "nucleus_boundaries")
+for (component in parquet_components) {
+  expect_true(grepl("\\.parquet$", metadata(obj)[[component]]))
+}
+# Dimensions
+input <- readXeniumSXE(
+  dirName = spe,
+  returnType = "SPE"
+)
+dim_rds <- dim(obj)
+dim_input <- dim(input)
+
+expect_equal(dim_rds, dim_input)
+
+
+cat("> Checking execution with compressed input\n")
+
+spe <- paste0(
+  meta[["resources_dir"]],
+  "/xenium_tiny"
+)
+out_rds <- "output.rds"
+
+create_folder_archive <- function(folder_path, archive = "xenium_tiny.zip") {
+  old_wd <- getwd()
+  on.exit(setwd(old_wd))
+  setwd(meta$resources_dir)
+  system2("zip", c("-r", archive, "xenium_tiny"))
+  paste0(meta$resources_dir, "/", archive)
+}
+
+zipped_spe <- create_folder_archive(spe)
+
+cat("> Running ", meta[["name"]], "\n", sep = "")
+out <- processx::run(
+  meta[["executable"]],
+  c(
+    "--input", zipped_spe,
+    "--output", out_rds
+  )
+)
+
+cat("> Checking whether output file exists\n")
+expect_equal(out$status, 0)
+expect_true(file.exists(out_rds))
+
+cat("> Reading output file\n")
+obj <- readRDS(file = out_rds)
+
+cat("> Checking whether Seurat object is in the right format\n")
+# Object type
+expect_is(obj, "SpatialExperiment")
+# Assay structure
+expect_equal(names(slot(obj, "assays")), "counts")
+# Spatial coordinates
+expect_equal(spatialCoordsNames(obj), c("x_centroid", "y_centroid"))
+# Alternative experiments
+expect_equal(
+  altExpNames(obj),
+  c("NegControlProbe", "UnassignedCodeword", "NegControlCodeword")
+)
+# Metadata components
+metadata_components <- c(
+  "experiment.xenium", "transcripts", "cell_boundaries", "nucleus_boundaries"
+)
+expect_named(
+  metadata(obj),
+  metadata_components,
+  ignore.order = TRUE
+)
+# Parquet paths
+parquet_components <- c("transcripts", "cell_boundaries", "nucleus_boundaries")
+for (component in parquet_components) {
+  expect_true(grepl("\\.parquet$", metadata(obj)[[component]]))
+}
+# Dimensions
+input <- readXeniumSXE(
+  dirName = spe,
+  returnType = "SPE"
+)
+dim_rds <- dim(obj)
+dim_input <- dim(input)
+
+expect_equal(dim_rds, dim_input)
+
+
+cat("> Checking parameter functionality\n")
+
+out_rds_ext <- "output_ext.rds"
+
+cat("> Running ", meta[["name"]], "\n", sep = "")
+out_ext <- processx::run(
+  meta[["executable"]],
+  c(
+    "--input", spe,
+    "--add_experiment_xenium", FALSE,
+    "--add_parquet_paths", FALSE,
+    "--alternative_experiment_features", c("NegControlProbe"),
+    "--output", out_rds_ext
+  )
+)
+
+cat("> Checking whether output file exists\n")
+expect_equal(out_ext$status, 0)
+expect_true(file.exists(out_rds_ext))
+
+cat("> Reading output file\n")
+obj_ext <- readRDS(file = out_rds_ext)
+
+cat("> Checking whether Seurat object is in the right format\n")
+# Object type
+expect_is(obj_ext, "SpatialExperiment")
+# Assay structure
+expect_equal(names(slot(obj_ext, "assays")), "counts")
+# Spatial coordinates
+expect_equal(spatialCoordsNames(obj_ext), c("x_centroid", "y_centroid"))
+# Alternative experiments
+expect_equal(altExpNames(obj_ext), c("NegControlProbe"))
+# Metadata components
+expect_true(length(metadata(obj_ext)) == 0)
+
+dim_rds_ext <- dim(obj_ext)
+expect_true(identical(dim_rds_ext[2], dim_input[2]))
+expect_false(identical(dim_rds_ext[1], dim_input[1]))

src/dataflow/obsp_block_concatenation/config.vsh.yaml

@@ -0,0 +1,96 @@
+name: obsp_block_concatenation
+namespace: "dataflow"
+scope: "public"
+description: |
+  Concatenate observations from samples in several (uni- and/or multi-modal) MuData files into a single file.
+  Performs block concatenation of `.obsp` matrices across samples for each modality.
+authors:
+  - __merge__: /src/authors/dries_schaumont.yaml
+    roles: [ author ]
+  - __merge__: /src/authors/dorien_roosen.yaml
+    roles: [ maintainer ]
+arguments:
+  - name: "--input"
+    alternatives: ["-i"]
+    type: file
+    multiple: true
+    description: Paths to the different samples to be concatenated.
+    required: true
+    example: sample_paths
+  - name: "--modality"
+    type: string
+    multiple: true
+    description: "Only output concatenated objects for the provided modalities. Outputs all modalities by default."
+    required: false
+  - name: "--input_id"
+    type: string
+    multiple: true
+    description: |
+      Names of the different samples that have to be concatenated.  Must be specified when using '--mode move'.
+      In this case, the ids will be used for the columns names of the dataframes registring the conflicts.
+      If specified, must be of same length as `--input`.
+    required: false
+  - name: "--output"
+    description: |
+      Output location for the concatenated MuData object file.
+    alternatives: ["-o"]
+    type: file
+    direction: output
+    example: "output.h5mu"
+  - name: "--other_axis_mode"
+    type: string
+    choices: [same, unique, first, only, concat, move]
+    default: move
+    description: |
+      How to handle the merging of other axis (var, obs, ...).
+
+        - None: keep no data
+        - same: only keep elements of the matrices which are the same in each of the samples
+        - unique: only keep elements for which there is only 1 possible value (1 value that can occur in multiple samples)
+        - first: keep the annotation from the first sample
+        - only: keep elements that show up in only one of the objects (1 unique element in only 1 sample)
+        - move: identical to 'same', but moving the conflicting values to .varm or .obsm
+  - name: "--uns_merge_mode"
+    description: |
+      How to handle the merging of .uns across modalities
+        - None: keep no data
+        - same: only keep elements of the matrices which are the same in each of the samples
+        - unique: only keep elements for which there is only 1 possible value (1 value that can occur in multiple samples)
+        - first: keep the annotation from the first sample
+        - only: keep elements that show up in only one of the objects (1 unique element in only 1 sample)
+        - make_unique: identical to 'unique', but keys which are not unique are made unique by prefixing them with the sample id.
+    type: string
+    choices: ["same", "unique", "first", "only", "make_unique"]
+    default: make_unique
+__merge__: [., /src/base/h5_compression_argument.yaml]
+
+resources:
+  - type: python_script
+    path: script.py
+  - path: /src/utils/setup_logger.py
+  - path: /src/utils/compress_h5mu.py
+test_resources:
+  - type: python_script
+    path: test.py
+  - path: /resources_test/concat_test_data/e18_mouse_brain_fresh_5k_filtered_feature_bc_matrix_subset_unique_obs.h5mu
+  - path: /resources_test/concat_test_data/human_brain_3k_filtered_feature_bc_matrix_subset_unique_obs.h5mu
+engines:
+  - type: docker
+    image: python:3.12-slim
+    setup:
+      - type: apt
+        packages: 
+          - procps
+      - type: python
+        __merge__: [/src/base/requirements/anndata_mudata.yaml, .]
+        packages:
+          - pandas~=2.1.1
+    __merge__: [ /src/base/requirements/python_test_setup.yaml, .]
+    test_setup:
+      - type: python
+        __merge__: [ /src/base/requirements/viashpy.yaml, .]
+runners:
+  - type: executable
+  - type: nextflow
+    directives:
+      label: [midcpu, highmem]

src/dataflow/obsp_block_concatenation/script.py

@@ -0,0 +1,450 @@
+from __future__ import annotations
+import sys
+import anndata
+import mudata as mu
+import pandas as pd
+import numpy as np
+from collections.abc import Iterable
+from multiprocessing import Pool
+from pathlib import Path
+from h5py import File as H5File
+from typing import Literal
+import shutil
+import scipy.sparse as sp
+
+### VIASH START
+par = {
+    "input": [
+        "sample1.h5mu",
+        "sample3.h5mu",
+    ],
+    "modality": None,
+    "output": "foo3.h5mu",
+    "input_id": ["sample1", "sample3"],
+    "other_axis_mode": "move",
+    "output_compression": "gzip",
+    "uns_merge_mode": "make_unique",
+}
+meta = {"cpus": 10, "resources_dir": "src/utils/"}
+### VIASH END
+
+sys.path.append(meta["resources_dir"])
+from compress_h5mu import compress_h5mu
+from setup_logger import setup_logger
+
+logger = setup_logger()
+
+
+def nunique(row):
+    unique = pd.unique(row)
+    unique_without_na = pd.core.dtypes.missing.remove_na_arraylike(unique)
+    return len(unique_without_na) > 1
+
+
+def any_row_contains_duplicate_values(n_processes: int, frame: pd.DataFrame) -> bool:
+    """
+    Check if any row contains duplicate values, that are not NA.
+    """
+    numpy_array = frame.to_numpy()
+    with Pool(n_processes) as pool:
+        is_duplicated = pool.map(nunique, iter(numpy_array))
+    return any(is_duplicated)
+
+
+def concatenate_matrices(
+    n_processes: int, matrices: dict[str, pd.DataFrame], align_to: pd.Index
+) -> tuple[
+    dict[str, pd.DataFrame], pd.DataFrame | None, dict[str, pd.core.dtypes.dtypes.Dtype]
+]:
+    """
+    Merge matrices by combining columns that have the same name.
+    Columns that contain conflicting values (e.i. the columns have different values),
+    are not merged, but instead moved to a new dataframe.
+    """
+    column_names = set(column_name for var in matrices.values() for column_name in var)
+    logger.debug("Trying to concatenate columns: %s.", ",".join(column_names))
+    if not column_names:
+        return {}, pd.DataFrame(index=align_to)
+    conflicts, concatenated_matrix = split_conflicts_and_concatenated_columns(
+        n_processes, matrices, column_names, align_to
+    )
+    concatenated_matrix = cast_to_writeable_dtype(concatenated_matrix)
+    conflicts = {
+        conflict_name: cast_to_writeable_dtype(conflict_df)
+        for conflict_name, conflict_df in conflicts.items()
+    }
+    return conflicts, concatenated_matrix
+
+
+def get_first_non_na_value_vector(df):
+    numpy_arr = df.to_numpy()
+    n_rows, n_cols = numpy_arr.shape
+    col_index = pd.isna(numpy_arr).argmin(axis=1)
+    flat_index = n_cols * np.arange(n_rows) + col_index
+    return pd.Series(numpy_arr.ravel()[flat_index], index=df.index, name=df.columns[0])
+
+
+def make_uns_keys_unique(mod_data, concatenated_data):
+    """
+    Check if the uns keys across samples are unique before adding them
+    to the final concatenated object. If a conflict occurs between the samples,
+    add the sample ID to make the key unique again.
+    """
+    all_uns_keys = {}
+    for sample_id, mod in mod_data.items():
+        for uns_key, _ in mod.uns.items():
+            all_uns_keys.setdefault(uns_key, []).append(sample_id)
+    for uns_key, samples_ids in all_uns_keys.items():
+        assert samples_ids
+        if len(samples_ids) == 1:
+            sample_id = samples_ids[0]
+            concatenated_data.uns[uns_key] = mod_data[sample_id].uns[uns_key]
+        else:
+            for sample_id in samples_ids:
+                concatenated_data.uns[f"{sample_id}_{uns_key}"] = mod_data[
+                    sample_id
+                ].uns[uns_key]
+    return concatenated_data
+
+
+def split_conflicts_and_concatenated_columns(
+    n_processes: int,
+    matrices: dict[str, pd.DataFrame],
+    column_names: Iterable[str],
+    align_to: pd.Index,
+) -> tuple[dict[str, pd.DataFrame], pd.DataFrame]:
+    """
+    Retrieve columns with the same name from a list of dataframes which are
+    identical across all the frames (ignoring NA values).
+    Columns which are not the same are regarded as 'conflicts',
+    which are stored in seperate dataframes, one per columns
+    with the same name that store conflicting values.
+    """
+    conflicts = {}
+    concatenated_matrix = []
+    for column_name in column_names:
+        columns = {
+            input_id: var[column_name]
+            for input_id, var in matrices.items()
+            if column_name in var
+        }
+        assert columns, "Some columns should have been found."
+        concatenated_columns = pd.concat(
+            columns.values(), axis=1, join="outer", sort=False
+        )
+        if any_row_contains_duplicate_values(n_processes, concatenated_columns):
+            concatenated_columns.columns = (
+                columns.keys()
+            )  # Use the sample id as column name
+            concatenated_columns = concatenated_columns.reindex(align_to, copy=False)
+            conflicts[f"conflict_{column_name}"] = concatenated_columns
+        else:
+            unique_values = get_first_non_na_value_vector(concatenated_columns)
+            concatenated_matrix.append(unique_values)
+    if not concatenated_matrix:
+        return conflicts, pd.DataFrame(index=align_to)
+    concatenated_matrix = pd.concat(
+        concatenated_matrix, join="outer", axis=1, sort=False
+    )
+    concatenated_matrix = concatenated_matrix.reindex(align_to, copy=False)
+    return conflicts, concatenated_matrix
+
+
+def cast_to_writeable_dtype(result: pd.DataFrame) -> pd.DataFrame:
+    """
+    Cast the dataframe to dtypes that can be written by mudata.
+    """
+    # dtype inferral workfs better with np.nan
+    result = result.replace({pd.NA: np.nan})
+
+    # MuData supports nullable booleans and ints
+    # ie. `IntegerArray` and `BooleanArray`
+    result = result.convert_dtypes(
+        infer_objects=True,
+        convert_integer=True,
+        convert_string=False,
+        convert_boolean=True,
+        convert_floating=False,
+    )
+
+    # Convert leftover 'object' columns to string
+    # However, na values are supported, so convert all values except NA's to string
+    object_cols = result.select_dtypes(include="object").columns.values
+    for obj_col in object_cols:
+        result[obj_col] = (
+            result[obj_col]
+            .where(result[obj_col].isna(), result[obj_col].astype(str))
+            .astype("category")
+        )
+    return result
+
+
+def split_conflicts_modalities(
+    n_processes: int, samples: dict[str, anndata.AnnData], output: anndata.AnnData
+) -> anndata.AnnData:
+    """
+    Merge .var and .obs matrices of the anndata objects. Columns are merged
+    when the values (excl NA) are the same in each of the matrices.
+    Conflicting columns are moved to a separate dataframe (one dataframe for each column,
+    containing all the corresponding column from each sample).
+    """
+    matrices_to_parse = ("var", "obs")
+    for matrix_name in matrices_to_parse:
+        matrices = {
+            sample_id: getattr(sample, matrix_name)
+            for sample_id, sample in samples.items()
+        }
+        output_index = getattr(output, matrix_name).index
+        conflicts, concatenated_matrix = concatenate_matrices(
+            n_processes, matrices, output_index
+        )
+        if concatenated_matrix.empty:
+            concatenated_matrix.index = output_index
+
+        # Even though we did not touch the varm and obsm matrices that were already present,
+        # the joining of observations might have caused a dtype change in these matrices as well
+        # so these also need to be casted to a writable dtype...
+        for multidim_name, multidim_data in getattr(output, f"{matrix_name}m").items():
+            new_data = (
+                cast_to_writeable_dtype(multidim_data)
+                if isinstance(multidim_data, pd.DataFrame)
+                else multidim_data
+            )
+            getattr(output, f"{matrix_name}m")[multidim_name] = new_data
+
+        # Write the conflicts to the output
+        for conflict_name, conflict_data in conflicts.items():
+            getattr(output, f"{matrix_name}m")[conflict_name] = conflict_data
+
+        # Set other annotation matrices in the output
+        setattr(output, matrix_name, concatenated_matrix)
+
+    return output
+
+
+def get_common_obsp_keys(mod_data: dict[str, anndata.AnnData]) -> set[str]:
+    """
+    Find `.obsp` keys that are present in all samples and whose matrices
+    are square and match the number of observations per sample.
+
+    This ensures we only build block-diagonal graphs when shapes are consistent.
+    """
+    if not mod_data:
+        return set()
+
+    key_sets = [set(adata.obsp.keys()) for adata in mod_data.values()]
+    common_keys = set.intersection(*key_sets) if key_sets else set()
+
+    if not common_keys:
+        logger.info("No suitable `.obsp` keys found for block-diagonal merge.")
+    else:
+        logger.info(
+            "Will merge `.obsp` keys block-diagonally: %s", ", ".join(common_keys)
+        )
+
+    return common_keys
+
+
+def merge_obsp_block_diag(
+    mod_data: dict[str, anndata.AnnData],
+    concatenated_data: anndata.AnnData,
+) -> anndata.AnnData:
+    """
+    Build block-diagonal `.obsp` matrices for all common keys across samples.
+    """
+    common_keys = get_common_obsp_keys(mod_data)
+    if not common_keys:
+        logger.info("Skipping `.obsp` block-diagonal merge.")
+        return concatenated_data
+
+    # Order of blocks must match the concat order
+    adatas_in_order = list(mod_data.values())
+
+    for key in common_keys:
+        logger.info("Building block-diagonal obsp['%s'] matrix.", key)
+        blocks = []
+        for ad in adatas_in_order:
+            # `.tocsr()` to ensure compatible format for block_diag
+            blocks.append(ad.obsp[key].tocsr())
+        concatenated_data.obsp[key] = sp.block_diag(blocks, format="csr")
+
+    return concatenated_data
+
+
+def concatenate_modality(
+    n_processes: int,
+    mod: str | None,
+    input_files: Iterable[str | Path],
+    other_axis_mode: str,
+    uns_merge_mode: str,
+    input_ids: tuple[str],
+) -> anndata.AnnData:
+    concat_modes = {
+        "move": "unique",
+    }
+    other_axis_mode_to_apply = concat_modes.get(other_axis_mode, other_axis_mode)
+
+    uns_merge_modes = {"make_unique": None}
+    uns_merge_mode_to_apply = uns_merge_modes.get(uns_merge_mode, uns_merge_mode)
+
+    mod_data = {}
+    mod_indices_combined = pd.Index([])
+    for input_id, input_file in zip(input_ids, input_files):
+        if mod is not None:
+            try:
+                data = mu.read_h5ad(input_file, mod=mod)
+                mod_data[input_id] = data
+                mod_indices_combined = mod_indices_combined.append(data.obs.index)
+            except KeyError as e:  # Modality does not exist for this sample, skip it
+                if (
+                    f"Unable to synchronously open object (object '{mod}' doesn't exist)"
+                    not in str(e)
+                ):
+                    raise e
+                pass
+        else:  # When mod=None, process the 'global' h5mu state
+            with H5File(input_file, "r") as input_h5:
+                if "uns" in input_h5.keys():
+                    uns_data = anndata.experimental.read_elem(input_h5["uns"])
+                    if uns_data:
+                        mod_data[input_id] = anndata.AnnData(uns=uns_data)
+
+    if not mod_indices_combined.is_unique:
+        raise ValueError("Observations are not unique across samples.")
+
+    if not mod_data:
+        return anndata.AnnData()
+
+    concatenated_data = anndata.concat(
+        mod_data.values(),
+        join="outer",
+        merge=other_axis_mode_to_apply,
+        uns_merge=uns_merge_mode_to_apply,
+    )
+
+    if other_axis_mode == "move":
+        concatenated_data = split_conflicts_modalities(
+            n_processes, mod_data, concatenated_data
+        )
+
+    if uns_merge_mode == "make_unique":
+        concatenated_data = make_uns_keys_unique(mod_data, concatenated_data)
+
+    concatenated_data = merge_obsp_block_diag(mod_data, concatenated_data)
+
+    return concatenated_data
+
+
+def concatenate_modalities(
+    n_processes: int,
+    modalities: list[str],
+    input_files: Path | str,
+    other_axis_mode: str,
+    uns_merge_mode: str,
+    output_file: Path | str,
+    compression: Literal["gzip"] | Literal["lzf"],
+    input_ids: tuple[str] | None = None,
+) -> None:
+    """
+    Join the modalities together into a single multimodal sample.
+    """
+    logger.info("Concatenating samples.")
+    output_file, input_files = (
+        Path(output_file),
+        [Path(input_file) for input_file in input_files],
+    )
+    output_file_uncompressed = output_file.with_name(
+        output_file.stem + "_uncompressed.h5mu"
+    )
+    output_file_uncompressed.touch()
+    # Create empty mudata file
+    mdata = mu.MuData({modality: anndata.AnnData() for modality in modalities})
+    mdata.write(output_file_uncompressed, compression=compression)
+
+    # Use "None" for the global slots (not assigned to any modality)
+    for mod_name in modalities + [
+        None,
+    ]:
+        new_mod = concatenate_modality(
+            n_processes,
+            mod_name,
+            input_files,
+            other_axis_mode,
+            uns_merge_mode,
+            input_ids,
+        )
+        if mod_name is None:
+            if new_mod.uns:
+                with H5File(output_file_uncompressed, "r+") as open_h5mu_file:
+                    anndata.experimental.write_elem(
+                        open_h5mu_file, "uns", dict(new_mod.uns)
+                    )
+            continue
+        logger.info(
+            "Writing out modality '%s' to '%s' with compression '%s'.",
+            mod_name,
+            output_file_uncompressed,
+            compression,
+        )
+        mu.write_h5ad(output_file_uncompressed, data=new_mod, mod=mod_name)
+
+    if compression:
+        compress_h5mu(output_file_uncompressed, output_file, compression=compression)
+        output_file_uncompressed.unlink()
+    else:
+        shutil.move(output_file_uncompressed, output_file)
+
+    logger.info("Concatenation successful.")
+
+
+def main() -> None:
+    # Get a list of all possible modalities
+    mods = set()
+    for path in par["input"]:
+        try:
+            with H5File(path, "r") as f_root:
+                mods = mods | set(f_root["mod"].keys())
+        except OSError:
+            raise OSError(f"Failed to load {path}. Is it a valid h5 file?")
+
+    input_ids = None
+    if par["input_id"]:
+        input_ids: tuple[str] = tuple(i.strip() for i in par["input_id"])
+        if len(input_ids) != len(par["input"]):
+            raise ValueError(
+                "The number of sample names must match the number of sample files."
+            )
+
+        if len(set(input_ids)) != len(input_ids):
+            raise ValueError("The sample names should be unique.")
+
+    logger.info("\nConcatenating data from paths:\n\t%s", "\n\t".join(par["input"]))
+
+    if par["other_axis_mode"] == "move" and not input_ids:
+        raise ValueError("--mode 'move' requires --input_ids.")
+
+    n_processes = meta["cpus"] if meta["cpus"] else 1
+
+    if par.get("modality"):
+        par["modality"] = set(par["modality"])
+        if not par["modality"].issubset(mods):
+            mods_joined, input_mods_joined = ", ".join(mods), ", ".join(par["modality"])
+            raise ValueError(
+                f"One of the modalities provided ({input_mods_joined}) is not available in the input data {mods_joined}"
+            )
+        mods = par["modality"]
+
+    concatenate_modalities(
+        n_processes,
+        list(mods),
+        par["input"],
+        par["other_axis_mode"],
+        par["uns_merge_mode"],
+        par["output"],
+        par["output_compression"],
+        input_ids=input_ids,
+    )
+
+
+if __name__ == "__main__":
+    main()

src/filter/subset_cosmx/config.vsh.yaml

@@ -0,0 +1,72 @@
+name: "subset_cosmx"
+scope: "private"
+namespace: "filter"
+description: |
+  Filters the output from NanoString experiment to keep only a subset of the fields of view.
+  Expected input folder structure:
+  path/to/dataset/
+      ├── CellComposite/
+      ├── CellLabels/
+      ├── CellOverlay/
+      ├── CompartmentLabels/
+      ├── <dataset_id>_exprMat_file.csv
+      ├── <dataset_id>_fov_positions_file.csv
+      ├── <dataset_id>_metadata_file.csv
+      └── <dataset_id>_tx_file.csv  
+
+authors:
+  - __merge__: /src/authors/dorien_roosen.yaml
+    roles: [ maintainer ]
+  - __merge__: /src/authors/weiwei_schultz.yaml
+    roles: [ contributor ]
+arguments:
+  - name: "--input"
+    alternatives: ["-i"]
+    type: file
+    description: Input folder. Must contain the output from a NanoString CosMx run.
+    example: cosmx_data
+    direction: input
+    required: true
+  - name: "--num_fovs"
+    type: integer
+    required: true
+    description: Number of fields of views to keep. Will keep only the first <num_fovs> fields of view.
+  - name: "--subset_transcripts_file"
+    type: boolean
+    default: true
+    description: Whether to subset the <dataset_id>_tx_file.csv file.
+  - name: "--subset_polygons_file"
+    type: boolean
+    default: true
+    description: Whether to subset the <dataset_id>_polygons.csv file.
+  - name: "--output"
+    alternatives: ["-o"]
+    type: file
+    description: The directory where the subset data will be stored.
+    example: "cosmx_data_tiny"
+    direction: output
+
+
+resources:
+  - type: python_script
+    path: script.py
+  - path: /src/utils/setup_logger.py
+test_resources:
+  - type: python_script
+    path: test.py
+  - path: /resources_test/cosmx/Lung5_Rep2_tiny/
+engines:
+  - type: docker
+    image: python:3.12-slim
+    setup:
+      - type: apt
+        packages: 
+          - procps
+      - type: python
+        __merge__: [ /src/base/requirements/squidpy.yaml ]
+    __merge__: [ /src/base/requirements/python_test_setup.yaml, .]
+runners:
+  - type: executable
+  - type: nextflow
+    directives:
+      label: [lowmem, singlecpu]

src/filter/subset_cosmx/script.py

@@ -0,0 +1,69 @@
+import os
+import shutil
+import pandas as pd
+import glob
+import sys
+
+
+## VIASH START
+par = {
+    "input": "./resources_test/cosmx/Lung5_Rep2",
+    "output": "./resources_test/cosmx/Lung5_Rep2_tiny/",
+    "subset_transcripts_file": True,
+    "subset_polygons_file": False,
+    "num_fovs": 5,
+}
+meta = {"resources_dir": "src/utils"}
+## VIASH END
+
+
+sys.path.append(meta["resources_dir"])
+from setup_logger import setup_logger
+
+logger = setup_logger()
+
+
+def find_matrix_file(suffix):
+    pattern = os.path.join(par["input"], f"*{suffix}")
+    files = glob.glob(pattern)
+    assert len(files) == 1, (
+        f"Only one file matching pattern {pattern} should be present"
+    )
+    return files[0]
+
+
+kept_fovs = list(range(1, par["num_fovs"] + 1))
+
+os.makedirs(par["output"], exist_ok=True)
+
+# Images
+image_dirs = ["CellComposite", "CellLabels", "CellOverlay", "CompartmentLabels"]
+
+for image_dir in image_dirs:
+    logger.info(f"Subsetting {image_dir}, keeping fovs {kept_fovs}")
+    os.makedirs(f"{par['output']}/{image_dir}", exist_ok=True)
+    for fov in kept_fovs:
+        fov_str = f"{image_dir}_F{fov:03d}.*"
+
+        file_path = glob.glob(os.path.join(par["input"], image_dir, fov_str))
+        assert len(file_path) == 1
+        shutil.copy2(file_path[0], os.path.join(par["output"], image_dir))
+
+# Matrices
+counts_file = find_matrix_file("exprMat_file.csv")
+fov_file = find_matrix_file("fov_positions_file.csv")
+meta_file = find_matrix_file("metadata_file.csv")
+
+matrices = [counts_file, fov_file, meta_file]
+if par["subset_transcripts_file"]:
+    tx_file = find_matrix_file("tx_file.csv")
+    matrices.append(tx_file)
+if par["subset_polygons_file"]:
+    polygons_file = find_matrix_file("polygons.csv")
+    matrices.append(polygons_file)
+
+for matrix in matrices:
+    logger.info(f"Subsetting {matrix}, keeping fovs {kept_fovs}")
+    data = pd.read_csv(matrix)
+    data_tiny = data[data["fov"].isin(kept_fovs)]
+    data_tiny.to_csv(os.path.join(par["output"], os.path.basename(matrix)), index=False)

src/filter/subset_cosmx/test.py

@@ -0,0 +1,48 @@
+import os
+import sys
+import pytest
+import pandas as pd
+
+
+def test_simple_execution(run_component, tmp_path):
+    output_path = tmp_path / "output"
+    dataset_id = "Lung5_Rep2"
+    run_component(
+        [
+            "--input",
+            meta["resources_dir"] + "/Lung5_Rep2_tiny",
+            "--subset_transcripts_file",
+            "True",
+            "--subset_polygons_file",
+            "False",
+            "--num_fovs",
+            "2",
+            "--output",
+            output_path,
+        ]
+    )
+
+    assert os.path.exists(output_path), "Output folder was not created"
+
+    counts_file = output_path / f"{dataset_id}_exprMat_file.csv"
+    fov_file = output_path / f"{dataset_id}_fov_positions_file.csv"
+    meta_file = output_path / f"{dataset_id}_metadata_file.csv"
+    tx_file = output_path / f"{dataset_id}_tx_file.csv"
+
+    matrices = [counts_file, fov_file, meta_file, tx_file]
+    images = ["CellComposite", "CellLabels", "CellOverlay", "CompartmentLabels"]
+
+    for image in images:
+        assert os.path.exists(output_path / image), f"{image} folder was not created"
+        assert len(os.listdir(output_path / image)) == 2, (
+            f"{image} folder should contain 2 files"
+        )
+
+    for matrix in matrices:
+        assert os.path.exists(matrix), f"{matrix} file was not created"
+        data = pd.read_csv(matrix)
+        data["fov"].value_counts().shape[0] == 2, f"{matrix} should contain 2 fovs"
+
+
+if __name__ == "__main__":
+    sys.exit(pytest.main([__file__]))

src/mapping/spaceranger_count/config.vsh.yaml

@@ -0,0 +1,209 @@
+name: spaceranger_count
+namespace: mapping
+scope: public
+description: Count gene expression and protein expression reads from a single capture area.
+keywords: [spaceranger]
+links:
+  documentation: https://www.10xgenomics.com/support/software/space-ranger/latest/analysis/running-pipelines/space-ranger-count
+authors:
+  - __merge__: /src/authors/jakub_majercik.yaml
+    roles: [ author ]
+argument_groups:
+  - name: Inputs
+    arguments:
+      - type: file
+        name: --gex_reference
+        required: true
+        description: Path of folder containing 10x-compatible reference
+        example: "/path/to/refdata-gex-GRCh38-2020-A"
+
+      - type: file
+        name: --input
+        required: true
+        description: | 
+          Path to a directory containing input FASTQ data. Individual FASTQ files should follow the naming convention of 10x Genomics:
+          [Sample Name]_S[Sample Number]_L[Lane Number]_[Read Type]_001.fastq.gz
+
+          Where:
+          [Sample Name] is the name assigned during sample preparation/sequencing
+          S[Sample Number] is the sample index (usually S1, S2, etc.)
+          L[Lane Number] identifies the sequencing lane (L001, L002, etc.)
+
+          [Read Type] will be one of:
+          R1 - Read 1 (contains the spatial barcode and UMI)
+          R2 - Read 2 (contains the actual cDNA sequence)
+          I1 - Index Read 1 (if applicable)
+          I2 - Index Read 2 (if applicable)
+        example: "/path/to/fastq_folder"
+
+      - type: file
+        name: --probe_set
+        required: true
+        description: CSV file specifying the probe set used
+        example: "Visium_Human_Transcriptome_Probe_Set_v2.0_GRCh38-2020-A.csv"
+
+      - type: file
+        name: --cytaimage
+        required: false
+        description: | 
+          Brightfield image generated by the CytAssist instrument. 
+          When using CytAssist workflow, either this or --image must be provided.
+        example: "cyta_image.tif"
+
+      - type: file
+        name: --image
+        required: false
+        description: | 
+          H&E or fluorescence microscope image in TIFF or JPG format. 
+          Required for standard Visium workflow, optional when using --cytaimage for CytAssist workflow.
+        example: "brightfield.tif"
+
+  - name: Outputs
+    arguments:
+      - type: file
+        name: --output
+        required: true
+        direction: output
+        description: The folder to store the alignment results
+        example: "/path/to/output"
+
+  - name: Slide Information
+    arguments:
+      - type: string
+        name: --slide
+        description: Visium slide serial number (e.g., 'V10J25-015')
+        required: false
+        example: "V10J25-015"
+
+      - type: string
+        name: --area
+        description: Visium capture area identifier (e.g., 'A1')
+        required: false
+        example: "A1"
+
+      - type: string
+        name: --unknown_slide
+        description: | 
+          Use this option if the slide serial number and area were entered incorrectly on the CytAssist 
+          instrument and the correct values are unknown. Not compatible with --slide, --area, or 
+          --slide-file options
+        required: false
+        choices: [visium-1, visium-2, visium-2-large, visium-hd]
+
+      - type: file
+        name: --slidefile
+        description: Slide design file for offline use
+        required: false
+        example: "slide_design.gpr"
+
+      - type: boolean_true
+        name: --override_id
+        description: Overrides the slide serial number and capture area provided in the Cytassist image metadata
+
+  - name: Image Options
+    arguments:
+      - type: file
+        name: --darkimage
+        description: Multi-channel, dark-background fluorescence image
+        required: false
+        example: "fluorescence.tif"
+
+      - type: file
+        name: --colorizedimage
+        description: Color image representing pre-colored dark-background fluorescence images
+        required: false
+        example: "colored_fluorescence.tif"
+
+      - type: integer
+        name: --dapi_index
+        description: Index of DAPI channel (1-indexed) of fluorescence image
+        required: false
+        example: 1
+        min: 1
+
+      - type: double
+        name: --image_scale
+        description: Microns per microscope image pixel
+        required: false
+        example: 0.65
+        min: 0.01
+        max: 10
+
+      - type: boolean
+        name: --reorient_images
+        default: true
+        description: Whether to rotate and mirror image to align fiducial pattern
+
+  - name: Processing Options
+    arguments:
+      - type: boolean
+        name: --create_bam
+        required: true
+        description: Enable or disable BAM file generation
+        default: true
+
+      - type: boolean_true
+        name: --nosecondary
+        description: Disable secondary analysis (e.g., clustering)
+
+      - type: integer
+        name: --r1_length
+        required: false
+        description: Hard trim the input Read 1 to this length before analysis
+        min: 1
+
+      - type: integer
+        name: --r2_length
+        required: false
+        description: Hard trim the input Read 2 to this length before analysis
+        min: 1
+
+      - type: boolean
+        name: --filter_probes
+        default: true
+        description: Whether to filter the probe set using the "included" column
+
+      - type: integer
+        name: --custom_bin_size
+        description: Bin Visium HD data to specified size in microns (4-100, even values only) in addition to the standard binning size (2 µm, 8 µm, 16 µm)
+        min: 4
+        max: 100
+
+  - name: Input Selection
+    arguments:
+      - type: string
+        name: --project
+        required: false
+        description: Project folder name within mkfastq output
+        
+      - type: string
+        name: --sample
+        required: false
+        description: Prefix of FASTQ filenames to select
+
+      - type: integer
+        name: --lanes
+        multiple: true
+        required: false
+        description: Only use FASTQs from selected lanes
+        example: [1,2,3]
+
+resources:
+  - type: bash_script
+    path: script.sh
+test_resources:
+  - type: bash_script
+    path: test.sh
+  - path: /resources_test/visium
+  - path: /resources_test/GRCh38
+engines:
+  - type: docker
+    image: ghcr.io/data-intuitive/spaceranger:3.1
+    setup:
+      - type: docker
+        run: |
+          DEBIAN_FRONTEND=noninteractive apt update && \
+          apt upgrade -y && apt install -y procps && rm -rf /var/lib/apt/lists/*
+runners:
+  - type: executable
+  - type: nextflow

src/mapping/spaceranger_count/script.sh

@@ -0,0 +1,45 @@
+#!/bin/bash
+
+set -eo pipefail
+
+unset_if_false=(
+    par_override_id
+    par_nosecondary
+)
+
+for par in ${unset_if_false[@]}; do
+    test_val="${!par}"
+    [[ "$test_val" == "false" ]] && unset $par
+done
+
+spaceranger count \
+  ${par_output:+--id="$par_output"} \
+  ${par_gex_reference:+--transcriptome="$par_gex_reference"} \
+  ${par_input:+--fastqs="$par_input"} \
+  ${par_probe_set:+--probe-set="$par_probe_set"} \
+  ${par_cytaimage:+--cytaimage="$par_cytaimage"} \
+  ${par_image:+--image="$par_image"} \
+  ${par_slide:+--slide="$par_slide"} \
+  ${par_area:+--area="$par_area"} \
+  ${par_unknown_slide:+--unknown-slide="$par_unknown_slide"} \
+  ${par_slidefile:+--slidefile="$par_slidefile"} \
+  ${par_override_id:+--override-id} \
+  ${par_darkimage:+--darkimage="$par_darkimage"} \
+  ${par_colorizedimage:+--colorizedimage="$par_colorizedimage"} \
+  ${par_dapi_index:+--dapi-index="$par_dapi_index"} \
+  ${par_image_scale:+--image-scale="$par_image_scale"} \
+  ${par_reorient_images:+--reorient-images="$par_reorient_images"} \
+  ${par_create_bam:+--create-bam="$par_create_bam"} \
+  ${par_nosecondary:+--nosecondary} \
+  ${par_r1_length:+--r1-length="$par_r1_length"} \
+  ${par_r2_length:+--r2-length="$par_r2_length"} \
+  ${par_filter_probes:+--filter-probes="$par_filter_probes"} \
+  ${par_custom_bin_size:+--custom-bin-size="$par_custom_bin_size"} \
+  ${par_project:+--project="$par_project"} \
+  ${par_sample:+--sample="$par_sample"} \
+  ${par_lanes:+--lanes="$par_lanes"} \
+  ${meta_cpus:+--localcores="$meta_cpus"} \
+  ${meta_memory_gb:+--localmem=$(($meta_memory_gb-2))}
+
+mv -f "$par_output"/outs/* "$par_output"/
+rm -rf "$par_output"/outs

src/mapping/spaceranger_count/test.sh

@@ -0,0 +1,47 @@
+#!/bin/bash
+
+set -eo pipefail
+
+## VIASH START
+meta_executable="target/native/spaceranger/spaceranger_count/spaceranger_count"
+meta_resources_dir="resources_test"
+## VIASH END
+
+test_data="$meta_resources_dir/visium"
+
+echo "> Default test run"
+"$meta_executable" \
+    --output test_spaceranger \
+    --gex_reference "$meta_resources_dir/GRCh38" \
+    --input "$test_data/subsampled" \
+    --probe_set "$test_data/Visium_FFPE_Human_Ovarian_Cancer_probe_set.csv" \
+    --image "$test_data/subsampled/Visium_FFPE_Human_Ovarian_Cancer_image.jpg" \
+    --unknown_slide visium-1 \
+    --create_bam false
+
+echo "> Checking outputs..."
+
+# Define output directory
+OUT_DIR="test_spaceranger"
+
+# Function to check if file exists and is non-empty
+check_file() {
+    local file=$1
+    local description=$2
+    echo -n "Checking $description... "
+    if [ ! -f "$file" ]; then
+        echo "FAIL (file not found)"
+        exit 1
+    elif [ ! -s "$file" ]; then
+        echo "FAIL (file is empty)"
+        exit 1
+    else
+        echo "OK"
+    fi
+}
+
+# Check essential files
+check_file "$OUT_DIR/web_summary.html" "web summary"
+check_file "$OUT_DIR/metrics_summary.csv" "metrics summary"
+
+echo "> All tests passed successfully!"

src/neighbors/spatial_neighborhood_graph/config.vsh.yaml

@@ -0,0 +1,93 @@
+name: neighbors
+namespace: spatial_neighborhood_graph
+scope: public
+description: Calculates a spatial neighborhood graph.
+
+authors:
+  - __merge__: /src/authors/dorien_roosen.yaml
+    roles: [ maintainer ]
+
+argument_groups:
+  - name: "Inputs"
+    arguments:
+      - name: "--input"
+        type: file
+        required: true
+        description: Input H5MU file
+      - name: "--modality"
+        description: |
+          Which modality from the input MuData file to process.
+        type: string
+        default: "rna"
+        required: false
+      - name: "--layer"
+        type: string
+        required: false
+        description: "Input layer to use. If None, X is used"
+      - name: "--input_obsm_spatial_coords"
+        type: string
+        default: "spatial"
+        description: "Key in adata.obsm where spatial coordinates are stored"
+
+  - name: "Spatial Neighbors Calculation"
+    arguments:
+      - name: "--coord_type"
+        type: string
+        choices: ["generic", "grid"]
+        description: |
+          Type of coordinate system. Valid options are:
+          `grid` - grid coordinates.
+          `generic` - generic coordinates.
+          If not provided, `grid` is used if `--input_obsm_spatial_coords` is in --input .uns with `--n_neighs` = 6 (Visium), otherwise `generic` is used.
+      - name: "--n_spatial_neighbors"
+        type: integer
+        default: 6
+        description: |
+          Depending on `--coord_type`:
+          `grid` - number of neighboring tiles.
+          `generic` - number of neighborhoods for non-grid data. Only used when `--delaunay False`.
+      - name: "--delaunay"
+        type: boolean
+        default: false
+        description: |
+          Whether to use Delaunay triangulation to determine spatial neighborhood graph.
+          Only used when `--coord_type generic`.
+
+  - name: Outputs
+    arguments:
+      - name: --output
+        type: file
+        direction: output
+        required: true
+        description: Output H5MU file path.
+        example: output.h5mu
+    __merge__: [., /src/base/h5_compression_argument.yaml]
+
+resources:
+  - type: python_script
+    path: script.py
+  - path: /src/utils/setup_logger.py
+
+test_resources:
+  - type: python_script
+    path: test.py
+  - path: /resources_test/xenium/xenium_tiny.h5mu
+  - path: /resources_test/cosmx/Lung5_Rep2_tiny.h5mu
+
+engines:
+  - type: docker
+    image: python:3.12-slim
+    setup:
+      - type: apt
+        packages: 
+          - procps
+      - type: python
+        __merge__: [ /src/base/requirements/squidpy.yaml, /src/base/requirements/anndata_mudata.yaml, . ]
+    __merge__: [ /src/base/requirements/python_test_setup.yaml, .]
+
+
+runners:
+- type: executable
+- type: nextflow
+  directives:
+    label: [lowcpu, lowmem, lowdisk]

src/neighbors/spatial_neighborhood_graph/script.py

@@ -0,0 +1,50 @@
+import sys
+import squidpy as sq
+import mudata as mu
+
+## VIASH START
+par = {
+    # Inputs
+    "input": "resources_test/cosmx/Lung5_Rep2_tiny.h5mu",
+    "modality": "rna",
+    "layer": None,
+    "input_gp_mask": "resources_test/niche/prior_knowledge_gp_mask.json",
+    "input_obsm_spatial_coords": "spatial",
+    ## Spatial neighbor calculation
+    "n_spatial_neighbors": 4,
+    "coord_type": "generic",
+    "delaunay": False,
+    "output": "foo.h5mu",
+}
+
+meta = {"resources_dir": "src/utils/"}
+## VIASH END
+
+sys.path.append(meta["resources_dir"])
+from setup_logger import setup_logger
+
+logger = setup_logger()
+
+## Read in data
+adata = mu.read_h5ad(par["input"], mod=par["modality"])
+
+## Compute spatial neighbor graph
+logger.info("Computing spatial neighbor graph...")
+sq.gr.spatial_neighbors(
+    adata,
+    coord_type=par["coord_type"],
+    spatial_key=par["input_obsm_spatial_coords"],
+    n_neighs=par["n_spatial_neighbors"],
+    delaunay=par["delaunay"],
+)
+
+# Making the connectivity matrix symmetric
+logger.info("Making the connectivity matrix symmetric...")
+adata.obsp["spatial_connectivities"] = adata.obsp["spatial_connectivities"].maximum(
+    adata.obsp["spatial_connectivities"].T
+)
+
+## Save model and data
+logger.info("Saving output data...")
+mdata = mu.MuData({par["modality"]: adata})
+mdata.write_h5mu(par["output"])

src/neighbors/spatial_neighborhood_graph/test.py

@@ -0,0 +1,46 @@
+import pytest
+import mudata as mu
+import sys
+
+## VIASH START
+meta = {
+    "executable": "./target/executable/nichecompass/nichecompass/nichecompass",
+}
+## VIASH END
+
+input_xenium = f"{meta['resources_dir']}/xenium_tiny.h5mu"
+input_cosmx = f"{meta['resources_dir']}/Lung5_Rep2_tiny.h5mu"
+gp_mask = f"{meta['resources_dir']}/prior_knowledge_gp_mask.json"
+
+
+def test_simple_execution_xenium(run_component, tmp_path):
+    output = tmp_path / "nc_xenium.h5mu"
+
+    # run component
+    run_component(
+        [
+            "--input",
+            input_xenium,
+            "--output",
+            str(output),
+            "--output_compression",
+            "gzip",
+        ]
+    )
+
+    assert output.is_file(), "output file was not created"
+    mdata = mu.read_h5mu(output)
+    assert list(mdata.mod.keys()) == ["rna"], "Expected modality rna"
+    adata = mdata.mod["rna"]
+
+    expected_obsp_keys = ["spatial_connectivities", "spatial_distances"]
+    assert all([obsp in expected_obsp_keys for obsp in adata.obsp.keys()]), (
+        "Not all expected obsp keys found"
+    )
+    assert all(adata.obsp[obsp].dtype.kind == "f" for obsp in expected_obsp_keys), (
+        "Expected obsp matrices to be float type"
+    )
+
+
+if __name__ == "__main__":
+    sys.exit(pytest.main([__file__]))

src/nichecompass/nichecompass/config.vsh.yaml

@@ -0,0 +1,492 @@
+name: nichecompass
+namespace: nichecompass
+scope: public
+description: Trains a NicheCompass model and generates the latent space for the provided input data.
+
+authors:
+  - __merge__: /src/authors/dorien_roosen.yaml
+    roles: [ maintainer ]
+
+argument_groups:
+  - name: "Inputs"
+    arguments:
+      - name: "--input"
+        type: file
+        required: true
+        description: Input H5MU file
+      - name: "--input_gp_mask"
+        type: file
+        required: true
+        description: |
+          JSON file containing a nested dictionary containing the gene programs,
+          with keys being gene program names and values being dictionaries with keys `targets` and `sources`,
+          where `targets` contains a list of the names of genes in the gene program for the reconstruction of the gene expression of the node itself (receiving node)
+          and `sources` contains a list of the names of genes in the gene program for the reconstruction of the gene expression of the node's neighbors (transmitting nodes).
+        example: prior_knowledge_gp_mask.json
+      - name: "--modality"
+        description: |
+          Which modality from the input MuData file to process.
+        type: string
+        default: "rna"
+        required: false
+      - name: "--layer"
+        type: string
+        required: false
+        description: "Input layer to use. If None, X is used"
+      - name: "--input_obsm_spatial_connectivities"
+        type: string
+        default: "spatial_connectivities"
+        description: "Key in adata.obsm where connectivities of the spatial neighborhood graph are stored"
+      - name: "--input_obs_covariates"
+        type: string
+        multiple: true
+        description: "Keys of the adata.obs fields to use as covariates."
+
+  - name: "Spatial Neighbors Calculation"
+    arguments:
+      - name: "--coord_type"
+        type: string
+        choices: ["generic", "grid"]
+        description: |
+          Type of coordinate system. Valid options are:
+          `grid` - grid coordinates.
+          `generic` - generic coordinates.
+          If not provided, `grid` is used if `--input_obsm_spatial_coords` is in --input .uns with `--n_neighs` = 6 (Visium), otherwise `generic` is used.
+      - name: "--n_spatial_neighbors"
+        type: integer
+        default: 6
+        description: |
+          Depending on `--coord_type`:
+          `grid` - number of neighboring tiles.
+          `generic` - number of neighborhoods for non-grid data. Only used when `--delaunay False`.
+      - name: "--delaunay"
+        type: boolean
+        default: false
+        description: |
+          Whether to use Delaunay triangulation to determine spatial neighborhood graph.
+          Only used when `--coord_type generic`.
+
+  - name: Gene Program Mask 
+    arguments:
+      - name: "--min_genes_per_gp"
+        type: integer
+        default: 1
+        min: 0
+        description: |
+          Minimum number of genes in a gene program inluding both target and source genes that need to be available in the adata (gene expression has been probed) for a gene program not to be discarded.
+      - name: "--min_source_genes_per_gp"
+        type: integer
+        default: 0
+        min: 0
+        description: |
+          Minimum number of source genes in a gene program that need to be available in the adata (gene expression has been probed) for a gene program not to be discarded.
+      - name: "--min_target_genes_per_gp"
+        type: integer
+        default: 0
+        min: 0
+        description: |
+          Minimum number of target genes in a gene program that need to be available in the adata (gene expression has been probed) for a gene program not to be discarded.
+      - name: "--max_genes_per_gp"
+        type: integer
+        min: 1
+        description: |
+          Maximum number of genes in a gene program inluding both target and source genes that can be available in the adata (gene expression has been probed) for a gene program not to be discarded.
+      - name: "--max_source_genes_per_gp"
+        type: integer
+        min: 1
+        description: |
+          Maximum number of source genes in a gene program that can be available in the adata (gene expression has been probed) for a gene program not to be discarded.
+      - name: "--max_target_genes_per_gp"
+        type: integer
+        min: 1
+        description: |
+          Maximum number of target genes in a gene program that can be available in the adata (gene expression has been probed) for a gene program not to be discarded.
+      - name: "--filter_genes_not_in_masks"
+        type: boolean_true
+        description: |
+          Whether to remove the genes that are not in the gp masks from the adata object.
+
+  - name: NicheCompass Model Architecture
+    arguments:
+      - name: "--covariate_edges"
+        type: boolean
+        multiple: true
+        description: |
+          List of booleans that indicate whether there can be edges between different categories of the categorical covariates.
+          If this is `True` for a specific categorical covariate, this covariate will be excluded from the edge reconstruction loss.
+          Needs to match the length and order of `--input_obs_covariates`.
+      - name: "--covariate_embedding_injection_layers"
+        type: string
+        multiple: true
+        choices: ["encoder", "gene_expr_decoder", "chrom_access_decoder"]
+        default: ["gene_expr_decoder", "chrom_access_decoder"]
+        description: |
+          List of VGPGAE modules in which the categorical covariates embeddings are injected.
+      - name: "--include_edge_recon_loss"
+        type: boolean
+        default: true
+        description: |
+          Whether to include the edge reconstruction loss in the backpropagation.
+      - name: "--include_gene_expr_recon_loss"
+        type: boolean
+        default: true
+        description: |
+          Whether to include the gene expression reconstruction loss in the backpropagation.
+      - name: "--include_cat_covariates_contrastive_loss"
+        type: boolean
+        default: true
+        description: |
+          Whether to include the categorical covariates contrastive loss in the backpropagation.
+      - name: "--gene_expr_recon_dist"
+        type: string
+        choices: ["nb", "zinb"]
+        default: "nb"
+        description: |
+          The distribution used for gene expression reconstruction. 
+          If `nb`, uses a negative binomial distribution. 
+          If `zinb`, uses a zero-inflated negative binomial distribution.
+      - name: "--log_variational"
+        type: boolean
+        default: true
+        description: |
+          Whether to transform x by log(x+1) prior to encoding for numerical stability (not for normalization).
+      - name: "--node_label_method"
+        type: string
+        choices: ["one-hop-norm", "two-hop-norm", "one-hop-attention"]
+        default: "one-hop-norm"
+        description: |
+          Node label method that will be used for omics reconstruction.
+          If `one-hop-sum`, uses a concatenation of the node's input features with the sum of the input features of all nodes in the node's one-hop neighborhood.
+          If `one-hop-norm`, uses a concatenation of the node's input features with the node's one-hop neighbors input features normalized as per Kipf, T. N. & Welling, M. Semi-Supervised Classification with Graph Convolutional Networks. arXiv [cs.LG] (2016).
+          If `one-hop-attention`, uses a concatenation of the node's input features with the node's one-hop neighbors input features weighted by an attention mechanism.
+      - name: "--active_gp_thresh_ratio"
+        type: double
+        default: 0.1
+        min: 0.0
+        max: 1.0
+        description: |
+          Ratio that determines which gene programs are considered active and are used in the latent representation after model training.
+          All inactive gene programs will be dropped during model training after a determined number of epochs.
+          Aggregations of the absolute values of the gene weights of the gene expression decoder per gene program are calculated.
+          The maximum value, i.e. the value of the gene program with the highest aggregated value will be used as a benchmark and all gene programs whose aggregated value is smaller than `--active_gp_thresh_ratio` times this maximum value will be set to inactive.
+          If set to 0, all gene programs will be considered active.
+      - name: "--active_gp_type"
+        type: string
+        choices: ["mixed", "separate"]
+        default: "separate"
+        description: |
+          Type to determine active gene programs. 
+          Can be `mixed`, in which case active gene programs are determined across prior and add-on gene programs jointly,
+          or `separate` in which case they are determined separately for prior and add-on gene programs.
+      - name: "--n_fc_layers_encoder"
+        type: integer
+        default: 1
+        min: 0
+        description: |
+          Number of fully connected layers in the encoder before message passing layers.
+      - name: "--n_layers_encoder"
+        type: integer
+        default: 1
+        min: 0
+        description: |
+          Number of message passing layers in the encoder.
+      - name: "--n_hidden_encoder"
+        type: integer
+        min: 0
+        description: |
+          Number of nodes in the encoder hidden layers.
+            If not provided, it will be determined automatically based on the number of input genes and gene programs.
+      - name: "--conv_layer_encoder"
+        type: string
+        choices: ["gcnconv", "gatv2conv"]
+        default: "gatv2conv"
+        description: |
+          Convolutional layer used as GNN in the encoder.
+      - name: "--encoder_n_attention_heads"
+        type: integer
+        default: 4
+        min: 0
+        description: |
+          Number of attention heads used in the GNN layers of the encoder.
+          Only relevant if `--conv_layer_encoder gatv2conv`. 
+      - name: "--encoder_use_bn"
+        type: boolean
+        default: false
+        description: |
+          Whether to use a batch normalization layer at the end of the encoder to normalize `mu`.
+      - name: "--dropout_rate_encoder"
+        type: double
+        default: 0.0
+        min: 0.0
+        max: 1.0
+        description: |
+          Probability that nodes will be dropped in the encoder during training.
+      - name: "--dropout_rate_graph_decoder"
+        type: double
+        default: 0.0
+        min: 0.0
+        max: 1.0
+        description: |
+          Probability that nodes will be dropped in the graph decoder during training.
+      - name: "--n_addon_gp"
+        type: integer
+        default: 100
+        min: 0
+        description: |
+          Number of addon gene programs (i.e. gene programs that are not included in masks but can be learned de novo).
+      - name: "--cat_covariates_embeds_nums"
+        type: integer
+        multiple: true
+        description: |
+          Number of embedding nodes for all categorical covariates.
+          Must be the same length as `--input_obs_covariates`.
+      - name: "--random_state"
+        default: 0
+        type: integer
+        min: 0
+        description: |
+          Random seed for reproducibility.
+
+  - name: NicheCompass Training Parameters
+    arguments:
+      - name: "--n_epochs"
+        type: integer
+        min: 1
+        default: 100
+        description: Number of training epochs
+      - name: "--n_epochs_all_gps"
+        type: integer
+        min: 0
+        default: 25
+        description: |
+          Number of epochs during which all gene programs are used for model training.
+          After that only active gene programs are retained.
+      - name: "--n_epochs_no_edge_recon"
+        type: integer
+        default: 0
+        min: 0
+        description: |
+          Number of epochs during which the edge reconstruction loss is excluded from backpropagation for pretraining using the other loss components.
+      - name: "--n_epochs_no_cat_covariates_contrastive"
+        type: integer
+        default: 5
+        min: 0
+        description: |
+          Number of epochs during which the categorical covariates contrastive loss is excluded from backpropagation for pretraining using the other loss components.
+      - name: "--lr"
+        type: double
+        default: 0.001
+        min: 0.0
+        max: 1.0
+        description: Learning rate
+      - name: "--weight_decay"
+        type: double
+        default: 0.001
+        description: Weight decay (L2 penalty).
+      - name: "--lambda_edge_recon"
+        type: double
+        default: 500000.0
+        description: |
+          Lambda (weighting factor) for the edge reconstruction loss.
+          If `>0`, this will enforce gene programs to be meaningful for edge reconstruction and, hence, to preserve spatial colocalization information.
+      - name: "--lambda_gene_expr_recon"
+        type: double
+        default: 300.0
+        description: |
+          Lambda (weighting factor) for the gene expression reconstruction loss.
+          If `>0`, this will enforce interpretable gene programs that can be combined in a linear way to reconstruct gene expression.
+      - name: "--lambda_cat_covariates_contrastive"
+        type: double
+        default: 0.0
+        description: |
+          Lambda (weighting factor) for the categorical covariates contrastive loss.
+          If `>0`, this will enforce observations with different categorical covariates categories with very similar latent representations to become more similar, and observations with different latent representations to become more different.
+      - name: "--contrastive_logits_pos_ratio"
+        type: double
+        default: 0.0
+        min: 0.0
+        max: 1.0
+        description: |
+          Ratio for determining the logits threshold of positive contrastive examples of node pairs from different categorical covariates categories.
+          The top (`contrastive_logits_pos_ratio` * 100)% logits of node pairs from different categorical covariates categories serve as positive labels for the contrastive loss.
+      - name: "--contrastive_logits_neg_ratio"
+        type: double
+        default: 0.0
+        min: 0.0
+        max: 1.0
+        description: |
+          Ratio for determining the logits threshold of negative contrastive examples of node pairs from different categorical covariates categories.
+          The bottom (`contrastive_logits_neg_ratio` * 100)% logits of node pairs from different categorical covariates categories serve as negative labels for the contrastive loss.
+      - name: "--lambda_group_lasso"
+        type: double
+        default: 0.0
+        description: |
+          Lambda (weighting factor) for the group lasso regularization loss of gene programs.
+          If `>0`, this will enforce sparsity of gene programs.
+      - name: "--lambda_l1_masked"
+        type: double
+        default: 0.0
+        description: |
+          Lambda (weighting factor) for the L1 regularization loss of genes in masked gene programs.
+          If `>0`, this will enforce sparsity of genes in masked gene programs.
+      - name: "--l1_targets_categories"
+        type: string
+        multiple: true
+        description: |
+          Gene program mask targets categories for which l1 regularization loss will be applied.
+      - name: "--l1_sources_categories"
+        type: string
+        multiple: true
+        description: |
+          Gene program mask sources categories for which l1 regularization loss will be applied.
+      - name: "--lambda_l1_addon"
+        type: double
+        default: 30.0
+        description: |
+          Lambda (weighting factor) for the L1 regularization loss of genes in addon gene programs.
+          If `>0`, this will enforce sparsity of genes in addon gene programs.
+      - name: "--edge_val_ratio"
+        type: double
+        default: 0.1
+        min: 0.0
+        max: 1.0
+        description: |
+          Fraction of the data that is used as validation set on edge-level. The rest of the data will be used as training set on edge-level.
+      - name: "--node_val_ratio"
+        type: double
+        default: 0.1
+        min: 0.0
+        max: 1.0
+        description: |
+          Fraction of the data that is used as validation set on node-level. The rest of the data will be used as training set on node-level.
+      - name: "--edge_batch_size"
+        type: integer
+        min: 1
+        default: 256
+        description: |
+          Batch size for the edge-level dataloaders.
+      - name: "--node_batch_size"
+        type: integer
+        min: 1
+        description: |
+          Batch size for the node-level dataloaders.
+          If not provided, is automatically determined based on `--edge_batch_size`.
+      - name: "--n_sampled_neighbors"
+        type: integer
+        default: -1
+        min: -1
+        description: |
+          Number of neighbors that are sampled during model training from the spatial neighborhood graph.
+          If set to -1, all direct neighbors are included.
+
+  - name: Outputs
+    arguments:
+      - name: "--output"
+        type: file
+        required: true
+        direction: output
+        description: Output H5MU file
+      - name: "--output_model"
+        type: file
+        required: true
+        direction: output
+        description: Directory to save the trained NicheCompass model
+      - name: "--output_obsm_embedding"
+        type: string
+        default: nichecompass_latent
+        description: |
+          Key of the obsm field where the latent / gene program representation of active gene programs will be stored after NicheCompass model training.
+      - name: "--output_varm_gp_targets_mask"
+        type: string
+        default: nichecompass_gp_targets
+        description: |
+          Key of the varm field where the binary gene program mask for target genes of a gene program will be stored (target genes are used for the reconstruction of the gene expression of the node itself (receiving node )).
+      - name: "--output_varm_gp_sources_mask"
+        type: string
+        default: nichecompass_gp_sources
+        description: |
+          Key of the varm field where the binary gene program mask for source genes of a gene program will be stored (source genes are used for the reconstruction of the gene expression of the node's neighbors (transmitting nodes)).
+      - name: "--output_uns_gp_names"
+        type: string
+        default: nichecompass_gp_names
+        description: |
+          Key of the uns field where the gene program names will be stored.
+      - name: "--output_uns_active_gp_names"
+        type: string
+        default: nichecompass_active_gp_names
+        description: |
+          Key of the uns field where the active gene program names will be stored.
+      - name: "--output_uns_gene_index"
+        type: string
+        default: nichecompass_gene_idx
+        description: |
+          Key of the uns field where the index of a concatenated vector of target and source genes that are in the gene program masks will be stored.
+      - name: "--output_uns_target_genes_index"
+        type: string
+        default: nichecompass_target_genes_idx
+        description: |
+          Key of the uns field where the index of the target genes that are in the gene program mask will be stored.
+      - name: "--output_uns_source_genes_index"
+        type: string
+        default: nichecompass_source_genes_idx
+        description: |
+          Key of the uns field where the index of the source genes that are in the gene program mask will be stored.
+      - name: "--output_uns_covariate_embeddings"
+        type: string
+        multiple: true
+        description: |
+          Key of the uns fields where the covariate embeddings will be stored.
+          Needs to match the length and order of `--input_obs_covariates`.
+      - name: "--output_obsp_reconstructed_adj_edge_proba"
+        type: string
+        default: nichecompass_recon_connectivities
+        description: |
+          Key of the obsp field where the reconstructed adjacency matrix edge probabilities will be stored.
+      - name: "--output_obsp_agg_weights"
+        type: string
+        default: nichecompass_agg_weights
+        description: |
+          Key of the obsp field where the aggregation weights of the node label aggregator will be stored.
+
+resources:
+  - type: python_script
+    path: script.py
+  - path: /src/utils/setup_logger.py
+
+test_resources:
+  - type: python_script
+    path: test.py
+  - path: /resources_test/niche/prior_knowledge_gp_mask.json
+  - path: /resources_test/xenium/xenium_tiny.h5mu
+  - path: /resources_test/cosmx/Lung5_Rep2_tiny.h5mu
+
+engines:
+- type: docker
+  image: nvidia/cuda:12.4.1-cudnn-devel-ubuntu22.04
+  setup:
+    - type: apt
+      packages:
+        - libhdf5-dev
+        - python3-pip
+        - python3-dev
+        - python-is-python3
+    - type: docker
+      run: |
+        pip install torch --index-url https://download.pytorch.org/whl/cu124 \
+        && pip install pyg_lib torch-scatter torch-sparse -f https://data.pyg.org/whl/torch-2.6.0+cu124.html   
+    - type: python
+      __merge__: [ /src/base/requirements/anndata_mudata.yaml, . ]
+    - type: python
+      packages:
+        - numpy<2
+        - nichecompass
+  test_setup:
+    - type: python
+      __merge__: [ /src/base/requirements/viashpy.yaml, .]
+
+runners:
+- type: executable
+  # docker_run_args: ["--gpus all"]
+- type: nextflow
+  directives:
+    label: [midcpu, midmem, gpu, highdisk]

src/nichecompass/nichecompass/script.py

@@ -0,0 +1,217 @@
+import sys
+import json
+import mudata as mu
+
+from nichecompass.models import NicheCompass
+from nichecompass.utils import add_gps_from_gp_dict_to_adata
+from torch.cuda import is_available as cuda_is_available
+
+## VIASH START
+par = {
+    # Inputs
+    "input": "resources_test/cosmx/Lung5_Rep2_tiny.h5mu",
+    "modality": "rna",
+    "layer": None,
+    "input_gp_mask": "resources_test/niche/prior_knowledge_gp_mask.json",
+    "input_obs_covariates": None,
+    "input_obsm_spatial_connectivities": "spatial_connectivities",
+    ## GP Mask
+    "min_genes_per_gp": 2,
+    "min_source_genes_per_gp": 1,
+    "min_target_genes_per_gp": 1,
+    "max_genes_per_gp": None,
+    "max_source_genes_per_gp": None,
+    "max_target_genes_per_gp": None,
+    "filter_genes_not_in_masks": False,
+    # outputs
+    "output": "nichecompass_output.h5mu",
+    "output_model": "nichecompass_model/",
+    "output_uns_gp_names": "nichecompass_gp_names",
+    "output_varm_gp_targets_mask": "nichecompass_gp_targets",
+    "output_varm_gp_sources_mask": "nichecompass_gp_sources",
+    "output_obsm_embedding": "nichecompass_latent",
+    "output_uns_covariate_embeddings": None,
+    "output_obsp_reconstructed_adj_edge_proba": "nichecompass_recon_connectivities",
+    "output_uns_active_gp_names": "nichecompass_active_gp_names",
+    "output_uns_gene_index": "nichecompass_genes_idx",
+    "output_uns_target_genes_index": "nichecompass_target_genes_idx",
+    "output_uns_source_genes_index": "nichecompass_source_genes_idx",
+    "output_obsp_agg_weights": "nichecompass_agg_weights",
+    # model architecture
+    "include_edge_recon_loss": True,
+    "include_gene_expr_recon_loss": True,
+    "include_cat_covariates_contrastive_loss": False,
+    "covariates_edges": None,
+    "covariate_embedding_injection_layers": ["gene_expr_decoder"],
+    "gene_expr_recon_dist": "nb",
+    "log_variational": True,
+    "node_label_method": "one-hop-norm",
+    "active_gp_thresh_ratio": 0.01,
+    "active_gp_type": "separate",
+    "n_fc_layers_encoder": 1,
+    "n_layers_encoder": 1,
+    "n_hidden_encoder": None,
+    "conv_layer_encoder": "gatv2conv",
+    "encoder_n_attention_heads": 4,
+    "encoder_use_bn": False,
+    "dropout_rate_encoder": 0.0,
+    "dropout_rate_graph_decoder": 0.0,
+    "cat_covariates_cats": None,
+    "n_addon_gp": 100,
+    "cat_covariates_embeds_nums": None,
+    "random_state": 0,
+    # model training
+    "n_epochs": 1,
+    "n_epochs_all_gps": 0,
+    "n_epochs_no_edge_recon": 0,
+    "n_epochs_no_cat_covariates_contrastive": 0,
+    "lr": 0.001,
+    "weight_decay": 0.0,
+    "lambda_edge_recon": 500000.0,
+    "lambda_gene_expr_recon": 300.0,
+    "lambda_cat_covariates_contrastive": 0.0,
+    "contrastive_logits_pos_ratio": 0.0,
+    "contrastive_logits_neg_ratio": 0.0,
+    "lambda_group_lasso": 0.0,
+    "lambda_l1_masked": 0.0,
+    "l1_targets_categories": ["target_gene"],
+    "l1_sources_categories": ["source_gene"],
+    "lambda_l1_addon": 30.0,
+    "edge_val_ratio": 0.1,
+    "node_val_ratio": 0.1,
+    "edge_batch_size": 256,
+    "node_batch_size": None,
+    "n_sampled_neighbors": -1,
+}
+
+meta = {"resources_dir": "src/utils/"}
+## VIASH END
+
+sys.path.append(meta["resources_dir"])
+from setup_logger import setup_logger
+
+logger = setup_logger()
+
+use_gpu = cuda_is_available()
+logger.info("GPU enabled? %s", use_gpu)
+
+## Read in data
+adata = mu.read_h5ad(par["input"], mod=par["modality"])
+
+# ## Compute spatial neighbor graph
+# logger.info("Computing spatial neighbor graph...")
+# # Compute connectivities and distances
+# sq.gr.spatial_neighbors(
+#     adata,
+#     coord_type=par["coord_type"],
+#     spatial_key=par["input_obsm_spatial_coords"],
+#     n_neighs=par["n_spatial_neighbors"],
+#     delaunay=par["delaunay"],
+# )
+
+# # Making the connectivity matrix symmetric
+# adata.obsp["spatial_connectivities"] = adata.obsp["spatial_connectivities"].maximum(
+#     adata.obsp["spatial_connectivities"].T
+# )
+
+## Add GP mask to data
+logger.info("Adding prior knowledge gene program mask to data...")
+with open(par["input_gp_mask"], "r") as f:
+    prior_knowledge_gp_mask = json.load(f)
+
+add_gps_from_gp_dict_to_adata(
+    gp_dict=prior_knowledge_gp_mask,
+    adata=adata,
+    gp_targets_mask_key=par["output_varm_gp_targets_mask"],
+    gp_sources_mask_key=par["output_varm_gp_sources_mask"],
+    gp_names_key=par["output_uns_gp_names"],
+    genes_idx_key=par["output_uns_gene_index"],
+    target_genes_idx_key=par["output_uns_target_genes_index"],
+    source_genes_idx_key=par["output_uns_source_genes_index"],
+    min_genes_per_gp=par["min_genes_per_gp"],
+    min_source_genes_per_gp=par["min_source_genes_per_gp"],
+    min_target_genes_per_gp=par["min_target_genes_per_gp"],
+    max_genes_per_gp=par["max_genes_per_gp"],
+    max_source_genes_per_gp=par["max_source_genes_per_gp"],
+    max_target_genes_per_gp=par["max_target_genes_per_gp"],
+    filter_genes_not_in_masks=par["filter_genes_not_in_masks"],
+)
+
+logger.info("Initializing NicheCompass model...")
+model = NicheCompass(
+    adata,
+    counts_key=par["layer"],
+    adj_key=par["input_obsm_spatial_connectivities"],
+    gp_names_key=par["output_uns_gp_names"],
+    active_gp_names_key=par["output_uns_active_gp_names"],
+    gp_targets_mask_key=par["output_varm_gp_targets_mask"],
+    gp_sources_mask_key=par["output_varm_gp_sources_mask"],
+    latent_key=par["output_obsm_embedding"],
+    cat_covariates_keys=par["input_obs_covariates"],
+    cat_covariates_no_edges=par["covariates_edges"],
+    cat_covariates_embeds_keys=par["output_uns_covariate_embeddings"],
+    cat_covariates_embeds_injection_layers=par["covariate_embedding_injection_layers"],
+    gene_idx_key=par["output_uns_gene_index"],
+    target_gene_idx_key=par["output_uns_target_genes_index"],
+    source_gene_idx_key=par["output_uns_source_genes_index"],
+    recon_adj_key=par["output_obsp_reconstructed_adj_edge_proba"],
+    agg_weights_key=par["output_obsp_agg_weights"],
+    include_edge_recon_loss=par["include_edge_recon_loss"],
+    include_gene_expr_recon_loss=par["include_gene_expr_recon_loss"],
+    include_cat_covariates_contrastive_loss=par[
+        "include_cat_covariates_contrastive_loss"
+    ],
+    gene_expr_recon_dist=par["gene_expr_recon_dist"],
+    log_variational=par["log_variational"],
+    node_label_method=par["node_label_method"],
+    active_gp_thresh_ratio=par["active_gp_thresh_ratio"],
+    active_gp_type=par["active_gp_type"],
+    n_fc_layers_encoder=par["n_fc_layers_encoder"],
+    n_layers_encoder=par["n_layers_encoder"],
+    n_hidden_encoder=par["n_hidden_encoder"],
+    conv_layer_encoder=par["conv_layer_encoder"],
+    encoder_n_attention_heads=par["encoder_n_attention_heads"],
+    encoder_use_bn=par["encoder_use_bn"],
+    dropout_rate_encoder=par["dropout_rate_encoder"],
+    dropout_rate_graph_decoder=par["dropout_rate_graph_decoder"],
+    cat_covariates_cats=par["cat_covariates_cats"],
+    n_addon_gp=par["n_addon_gp"],
+    cat_covariates_embeds_nums=par["cat_covariates_embeds_nums"],
+    seed=par["random_state"],
+    use_cuda_if_available=use_gpu,
+)
+
+logger.info("Training NicheCompass model...")
+model.train(
+    n_epochs=par["n_epochs"],
+    n_epochs_all_gps=par["n_epochs_all_gps"],
+    n_epochs_no_edge_recon=par["n_epochs_no_edge_recon"],
+    n_epochs_no_cat_covariates_contrastive=par[
+        "n_epochs_no_cat_covariates_contrastive"
+    ],
+    lr=par["lr"],
+    weight_decay=par["weight_decay"],
+    lambda_edge_recon=par["lambda_edge_recon"],
+    lambda_gene_expr_recon=par["lambda_gene_expr_recon"],
+    lambda_cat_covariates_contrastive=par["lambda_cat_covariates_contrastive"],
+    contrastive_logits_pos_ratio=par["contrastive_logits_pos_ratio"],
+    contrastive_logits_neg_ratio=par["contrastive_logits_neg_ratio"],
+    lambda_group_lasso=par["lambda_group_lasso"],
+    lambda_l1_masked=par["lambda_l1_masked"],
+    l1_targets_categories=par["l1_targets_categories"],
+    l1_sources_categories=par["l1_sources_categories"],
+    lambda_l1_addon=par["lambda_l1_addon"],
+    edge_val_ratio=par["edge_val_ratio"],
+    node_val_ratio=par["node_val_ratio"],
+    edge_batch_size=par["edge_batch_size"],
+    node_batch_size=par["node_batch_size"],
+    n_sampled_neighbors=par["n_sampled_neighbors"],
+    use_cuda_if_available=use_gpu,
+)
+
+## Save model and data
+logger.info("Saving NicheCompass model and data...")
+mdata = mu.MuData({par["modality"]: adata})
+mdata.write_h5mu(par["output"])
+
+model.save(par["output_model"], save_adata=False)

src/nichecompass/nichecompass/test.py

@@ -0,0 +1,86 @@
+import pytest
+import mudata as mu
+
+## VIASH START
+meta = {
+    "executable": "./target/executable/nichecompass/nichecompass/nichecompass",
+}
+## VIASH END
+
+input_xenium = f"{meta['resources_dir']}/xenium_tiny.h5mu"
+input_cosmx = f"{meta['resources_dir']}/Lung5_Rep2_tiny.h5mu"
+gp_mask = f"{meta['resources_dir']}/prior_knowledge_gp_mask.json"
+
+
+def test_simple_execution_xenium(run_component, tmp_path):
+    output = tmp_path / "nc_xenium.h5mu"
+
+    # run component
+    run_component(
+        [
+            "--input",
+            input_xenium,
+            "--input_gp_mask",
+            gp_mask,
+            "--n_epochs",
+            "1",
+            "n_epochs_all_gps",
+            "0",
+            "n_epochs_no_edge_recon",
+            "0",
+            "n_epochs_no_cat_covariates_contrastive",
+            "0--output",
+            str(output),
+            "--output_compression",
+            "gzip",
+        ]
+    )
+
+    assert output.is_file(), "output file was not created"
+    mdata = mu.read_h5mu(output)
+    assert list(mdata.mod.keys()) == ["rna"], "Expected modality rna"
+    adata = mdata.mod["rna"]
+
+    expected_uns_keys = [
+        "nichecompass_sources_categories_label_encoder",
+        "nichecompass_targets_categories_label_encoder",
+        "nichecompass_source_genes_idx",
+        "nichecompass_target_genes_idx",
+        "nichecompass_genes_idx",
+        "nichecompass_gp_names",
+        "nichecompass_active_gp_names",
+    ]
+    assert all([uns in expected_uns_keys for uns in adata.uns.keys()])
+    assert len(adata.uns["nichecompass_gp_names"]) > len(
+        adata.uns["nichecompass_active_gp_names"]
+    ), "Expected less active GP names than total GP names"
+    assert adata.uns["nichecompass_genes_idx"] == (
+        adata.uns["nichecompass_source_genes_idx"]
+        + adata.uns["nichecompass_target_genes_idx"]
+    ), "Expected genes idx to be union of source and target genes idx"
+
+    expected_obsm_keys = ["nichecompass_latent"]
+    assert all([obsm in expected_obsm_keys for obsm in adata.obsm.keys()]), (
+        "Not all expected obsm keys found"
+    )
+    assert all(adata.obsm[obsm].dtype.kind == "f" for obsm in expected_obsm_keys), (
+        "Expected obsm matrices to be float type"
+    )
+
+    expected_varm_keys = [
+        "nichecompass_gp_sources",
+        "nichecompass_gp_targets",
+        "nichecompass_gp_sources_categories",
+        "nichecompass_gp_targets_categories",
+    ]
+    assert all([varm in expected_varm_keys for varm in adata.varm.keys()]), (
+        "Not all expected varm keys found"
+    )
+    assert (
+        adata.varm["nichecompass_gp_targets"].shape
+        == adata.varm["nichecompass_gp_sources"].shape
+    ), "Expected GP targets and sources varm to have same shape"
+
+
+if __name__ == "__main__":
+    pytest.main([__file__])

src/utils/compress_h5mu.py

@@ -0,0 +1,87 @@
+import shutil
+from anndata import AnnData
+from mudata import write_h5ad
+from h5py import File as H5File
+from h5py import Group, Dataset
+from pathlib import Path
+from typing import Union, Literal
+from functools import partial
+
+
+def compress_h5mu(
+    input_path: Union[str, Path],
+    output_path: Union[str, Path],
+    compression: Union[Literal["gzip"], Literal["lzf"]],
+):
+    input_path, output_path = str(input_path), str(output_path)
+
+    def copy_attributes(in_object, out_object):
+        for key, value in in_object.attrs.items():
+            out_object.attrs[key] = value
+
+    def visit_path(
+        output_h5: H5File,
+        compression: Union[Literal["gzip"], Literal["lzf"]],
+        name: str,
+        object: Union[Group, Dataset],
+    ):
+        if isinstance(object, Group):
+            new_group = output_h5.create_group(name)
+            copy_attributes(object, new_group)
+        elif isinstance(object, Dataset):
+            # Compression only works for non-scalar Dataset objects
+            # Scalar objects dont have a shape defined
+            if not object.compression and object.shape not in [None, ()]:
+                new_dataset = output_h5.create_dataset(
+                    name, data=object, compression=compression
+                )
+                copy_attributes(object, new_dataset)
+            else:
+                output_h5.copy(object, name)
+        else:
+            raise NotImplementedError(
+                f"Could not copy element {name}, "
+                f"type has not been implemented yet: {type(object)}"
+            )
+
+    with (
+        H5File(input_path, "r") as input_h5,
+        H5File(output_path, "w", userblock_size=512) as output_h5,
+    ):
+        copy_attributes(input_h5, output_h5)
+        input_h5.visititems(partial(visit_path, output_h5, compression))
+
+    with open(input_path, "rb") as input_bytes:
+        # Mudata puts metadata like this in the first 512 bytes:
+        # MuData (format-version=0.1.0;creator=muon;creator-version=0.2.0)
+        # See mudata/_core/io.py, read_h5mu() function
+        starting_metadata = input_bytes.read(100)
+        # The metadata is padded with extra null bytes up until 512 bytes
+        truncate_location = starting_metadata.find(b"\x00")
+        starting_metadata = starting_metadata[:truncate_location]
+    with open(output_path, "br+") as f:
+        nbytes = f.write(starting_metadata)
+        f.write(b"\0" * (512 - nbytes))
+
+
+def write_h5ad_to_h5mu_with_compression(
+    output_file: Union[str, Path],
+    h5mu: Union[str, Path],
+    modality_name: str,
+    modality_data: AnnData,
+    output_compression=None,
+):
+    output_file = Path(output_file)
+    h5mu = Path(h5mu)
+    output_file_uncompressed = (
+        output_file.with_name(output_file.stem + "_uncompressed.h5mu")
+        if output_compression
+        else output_file
+    )
+    shutil.copyfile(h5mu, output_file_uncompressed)
+    write_h5ad(filename=output_file_uncompressed, mod=modality_name, data=modality_data)
+    if output_compression:
+        compress_h5mu(
+            output_file_uncompressed, output_file, compression=output_compression
+        )
+        output_file_uncompressed.unlink()

src/utils/setup_logger.py

@@ -0,0 +1,12 @@
+def setup_logger():
+    import logging
+    from sys import stdout
+
+    logger = logging.getLogger()
+    logger.setLevel(logging.INFO)
+    console_handler = logging.StreamHandler(stdout)
+    logFormatter = logging.Formatter("%(asctime)s %(levelname)-8s %(message)s")
+    console_handler.setFormatter(logFormatter)
+    logger.addHandler(console_handler)
+
+    return logger

src/utils/unzip_archived_folder.R

@@ -0,0 +1,22 @@
+extract_selected_files <- function(zip_path, members) {
+  # Create a temporary directory for extraction
+  temp_dir <- tempfile("unzip_dir_")
+  dir.create(temp_dir)
+
+  # List all files in the archive
+  all_files <- utils::unzip(zip_path, list = TRUE)$Name
+
+  # Find files matching any of the glob patterns in 'members'
+  selected <- unique(unlist(
+    lapply(members, function(pattern) {
+      regex <- glob2rx(pattern)
+      grep(regex, all_files, value = TRUE)
+    })
+  ))
+
+  # Extract only the selected files
+  utils::unzip(zip_path, files = selected, exdir = temp_dir)
+
+  # Return the path to the extracted folder
+  file.path(temp_dir)
+}

src/utils/unzip_archived_folder.py

@@ -0,0 +1,50 @@
+import fnmatch
+import tempfile
+from pathlib import Path
+from typing import Union
+import zipfile_inflate64 as zipfile
+
+
+def unzip_archived_folder(archived_folder: Union[str, Path]) -> Union[str, Path]:
+    """
+    Extracts a ZIP archive to a temporary directory and returns the path to the extracted folder.
+
+    Args:
+        zip_path (Union[str, Path]): Path to the ZIP archive.
+
+    Returns:
+        extracted_path (Union[str, Path]): Path to the extracted folder inside the temporary directory.
+    """
+
+    temp_dir = Path(tempfile.TemporaryDirectory().name)
+    with zipfile.ZipFile(archived_folder, "r") as archive:
+        archive.extractall(temp_dir)
+
+    return temp_dir / Path(archived_folder).stem
+
+
+def extract_selected_files_from_zip(
+    zip_path: Union[str, Path], members: list[Union[str, Path]]
+) -> Union[str, Path]:
+    """
+    Extracts selected files (supports glob patterns) from a ZIP archive to a temporary directory.
+
+    Args:
+        zip_path (Union[str, Path]): Path to the ZIP archive.
+        members (list[str]): List of file paths within the archive to extract.
+
+    Returns:
+        Path: Path to the extraction directory.
+    """
+
+    temp_dir = Path(tempfile.TemporaryDirectory().name)
+
+    with zipfile.ZipFile(zip_path, "r") as archive:
+        all_files = archive.namelist()
+        selected = set()
+        for pattern in members:
+            selected.update(fnmatch.filter(all_files, str(pattern)))
+        for member in selected:
+            archive.extract(member, temp_dir)
+
+    return temp_dir

src/workflows/multiomics/spatial_process_samples/config.vsh.yaml

@@ -0,0 +1,312 @@
+name: "spatial_process_samples"
+namespace: "workflows/multiomics"
+scope: "public"
+description: "A pipeline to pre-process multiple spatial omics samples."
+authors:
+  - __merge__: /src/authors/dries_schaumont.yaml
+    roles: [ author, maintainer ]
+  - __merge__: /src/authors/dorien_roosen.yaml
+    roles: [ contributor ]
+  - __merge__: /src/authors/weiwei_schultz.yaml
+    roles: [ contributor ]
+
+argument_groups:
+  - name: Inputs
+    arguments:
+      - name: "--id"
+        required: true
+        type: string
+        description: ID of the sample.
+        example: foo
+      - name: "--input"
+        alternatives: [-i]
+        description: Path to the sample.
+        required: true
+        example: input.h5mu
+        type: file
+      - name: "--rna_layer"
+        type: string
+        description: "Input layer for the gene expression modality. If not specified, .X is used."
+        required: false
+      - name: "--prot_layer"
+        type: string
+        description: "Input layer for the antibody capture modality. If not specified, .X is used."
+        required: false
+
+  - name: "Outputs"
+    arguments:
+      - name: "--output"
+        type: file
+        required: true
+        direction: output
+        description: Destination path to the output.
+        example: output.h5mu
+
+  - name: "Sample ID options"
+    description: |
+      Options for adding the id to .obs on the MuData object. Having a sample 
+      id present in a requirement of several components for this pipeline.
+    arguments:
+      - name: "--add_id_to_obs"
+        description: "Add the value passed with --id to .obs."
+        type: boolean
+        default: true
+      - name: --add_id_obs_output
+        description: |
+          .Obs column to add the sample IDs to. Required and only used when 
+          --add_id_to_obs is set to 'true'
+        type: string
+        default: "sample_id"
+      - name: "--add_id_make_observation_keys_unique"
+        type: boolean
+        description: |
+          Join the id to the .obs index (.obs_names). 
+          Only used when --add_id_to_obs is set to 'true'.
+        default: true
+
+  - name: "RNA filtering options"
+    arguments:
+      - name: "--rna_min_counts"
+        example: 200
+        min: 1
+        type: integer
+        description: Minimum number of counts captured per cell.
+      - name: "--rna_max_counts"
+        example: 5000000
+        min: 1
+        type: integer
+        description: Maximum number of counts captured per cell.
+      - name: "--rna_min_genes_per_cell"
+        type: integer
+        min: 1
+        example: 200
+        description: Minimum of non-zero values per cell.
+      - name: "--rna_max_genes_per_cell"
+        example: 1500000
+        min: 1
+        type: integer
+        description: Maximum of non-zero values per cell.
+      - name: "--rna_min_cells_per_gene"
+        example: 3
+        min: 1
+        type: integer
+        description: Minimum of non-zero values per gene.
+      - name: "--rna_min_fraction_mito"
+        example: 0
+        min: 0
+        max: 1
+        type: double
+        description: Minimum fraction of UMIs that are mitochondrial.
+      - name: "--rna_max_fraction_mito"
+        type: double
+        min: 0
+        max: 1
+        example: 0.2
+        description: Maximum fraction of UMIs that are mitochondrial.
+      - name: "--rna_min_fraction_ribo"
+        example: 0
+        min: 0
+        max: 1
+        type: double
+        description: Minimum fraction of UMIs that are mitochondrial.
+      - name: "--rna_max_fraction_ribo"
+        type: double
+        min: 0
+        max: 1
+        example: 0.2
+        description: Maximum fraction of UMIs that are mitochondrial.
+
+  - name: "Protein filtering options"
+    arguments:
+      - name: "--prot_min_counts"
+        description: Minimum number of counts per cell.
+        type: integer
+        min: 1
+        example: 3
+      - name: "--prot_max_counts"
+        description: Minimum number of counts per cell.
+        type: integer
+        min: 1
+        example: 5000000
+      - name: "--prot_min_proteins_per_cell"
+        type: integer
+        min: 1
+        example: 200
+        description: Minimum of non-zero values per cell.
+      - name: "--prot_max_proteins_per_cell"
+        description: Maximum of non-zero values per cell.
+        type: integer
+        min: 1
+        example: 100000000
+      - name: "--prot_min_cells_per_protein"
+        example: 3
+        min: 1
+        type: integer
+        description: Minimum of non-zero values per protein.
+
+  - name: "Highly variable features detection"
+    arguments:
+      - name: "--highly_variable_features_var_output"
+        alternatives: ["--filter_with_hvg_var_output"]
+        required: false
+        type: string
+        default: "filter_with_hvg"
+        description: In which .var slot to store a boolean array corresponding to the highly variable genes.
+      - name: "--highly_variable_features_obs_batch_key"
+        alternatives: ["--filter_with_hvg_obs_batch_key"]
+        type: string
+        default: "sample_id"
+        required: false
+        description: |
+          If specified, highly-variable genes are selected within each batch separately and merged. This simple 
+          process avoids the selection of batch-specific genes and acts as a lightweight batch correction method.
+  - name: "Mitochondrial & Ribosomal Gene Detection"
+    arguments:
+      - name: "--var_gene_names"
+        required: false
+        example: "gene_symbol"
+        type: string
+        description: |
+          .var column name to be used to detect mitochondrial/ribosomal genes instead of .var_names (default if not set).
+          Gene names matching with the regex value from --mitochondrial_gene_regex or --ribosomal_gene_regex will be 
+          identified as mitochondrial or ribosomal genes, respectively. 
+      - name: "--var_name_mitochondrial_genes"
+        type: string
+        required: false
+        description: |
+          In which .var slot to store a boolean array corresponding the mitochondrial genes.
+      - name: "--obs_name_mitochondrial_fraction"
+        type: string
+        required: false
+        description: |
+          When specified, write the fraction of counts originating from mitochondrial genes 
+          (based on --mitochondrial_gene_regex) to an .obs column with the specified name.
+          Requires --var_name_mitochondrial_genes.
+      - name: --mitochondrial_gene_regex
+        type: string
+        description: |
+          Regex string that identifies mitochondrial genes from --var_gene_names.
+          By default will detect human and mouse mitochondrial genes from a gene symbol.
+        required: false
+        default: "^[mM][tT]-"
+      - name: "--var_name_ribosomal_genes"
+        type: string
+        required: false
+        description: |
+          In which .var slot to store a boolean array corresponding the ribosomal genes.
+      - name: "--obs_name_ribosomal_fraction"
+        type: string
+        required: false
+        description: |
+          When specified, write the fraction of counts originating from ribosomal genes 
+          (based on --ribosomal_gene_regex) to an .obs column with the specified name.
+          Requires --var_name_ribosomal_genes.
+      - name: --ribosomal_gene_regex
+        type: string
+        description: |
+          Regex string that identifies ribosomal genes from --var_gene_names.
+          By default will detect human and mouse ribosomal genes from a gene symbol.
+        required: false
+        default: "^[Mm]?[Rr][Pp][LlSs]"
+
+  - name: "QC metrics calculation options"
+    arguments:
+      - name: "--var_qc_metrics"
+        description: |
+          Keys to select a boolean (containing only True or False) column from .var.
+          For each cell, calculate the proportion of total values for genes which are labeled 'True', 
+          compared to the total sum of the values for all genes. Defaults to the combined values specified for
+          --var_name_mitochondrial_genes and --highly_variable_features_var_output.
+        type: string
+        multiple: True
+        multiple_sep: ','
+        required: false
+        example: "ercc,highly_variable"
+      - name: "--top_n_vars"
+        type: integer
+        description: |
+          Number of top vars to be used to calculate cumulative proportions.
+          If not specified, proportions are not calculated. `--top_n_vars 20,50` finds
+          cumulative proportion to the 20th and 50th most expressed vars.
+        multiple: true
+        multiple_sep: ','
+        required: false
+        default: [50, 100, 200, 500]
+
+  - name: "PCA options"
+    arguments:
+      - name: "--pca_overwrite"
+        type: boolean_true
+        description: "Allow overwriting slots for PCA output."
+
+  - name: "CLR options"
+    arguments:
+      - name: "--clr_axis"
+        type: integer
+        description: "Axis to perform the CLR transformation on."
+        default: 0
+        required: false
+
+  - name: "RNA Scaling options"
+    description: |
+      Options for enabling scaling of the log-normalized data to unit variance and zero mean.
+      The scaled data will be output a different layer and representation with reduced dimensions
+      will be created and stored in addition to the non-scaled data.
+    arguments:
+      - name: "--rna_enable_scaling"
+        description: "Enable scaling for the RNA modality."
+        type: boolean_true
+      - name: "--rna_scaling_output_layer"
+        type: string
+        default: "scaled"
+        description: "Output layer where the scaled log-normalized data will be stored."
+      - name: "--rna_scaling_pca_obsm_output"
+        type: string
+        description: |
+          Name of the .obsm key where the PCA representation of the log-normalized
+          and scaled data is stored.
+        default: "scaled_pca"
+      - name: "--rna_scaling_pca_loadings_varm_output"
+        type: string
+        description: |
+          Name of the .varm key where the PCA loadings of the log-normalized and scaled
+          data is stored.
+        default: "scaled_pca_loadings"
+      - name: "--rna_scaling_pca_variance_uns_output"
+        type: string
+        description: |
+          Name of the .uns key where the variance and variance ratio will be stored as a map.
+          The map will contain two keys: variance and variance_ratio respectively.
+        default: "scaled_pca_variance"
+      - name: "--rna_scaling_umap_obsm_output"
+        type: string
+        description:
+          Name of the .obsm key where the UMAP representation of the log-normalized
+          and scaled data is stored.
+        default: "scaled_umap"
+      - name: "--rna_scaling_max_value"
+        description: "Clip (truncate) data to this value after scaling. If not specified, do not clip."
+        required: false
+        type: double
+      - name: "--rna_scaling_zero_center"
+        type: boolean_false
+        description: If set, omit zero-centering variables, which allows to handle sparse input efficiently."
+
+dependencies:
+  - name: workflows/multiomics/process_samples
+    alias: spatial_sample_processing
+    repository: openpipeline
+
+resources:
+  - type: nextflow_script
+    path: main.nf
+    entrypoint: run_wf
+
+test_resources:
+  - type: nextflow_script
+    path: test.nf
+    entrypoint: test_wf
+  - path: /resources_test/xenium/xenium_tiny.h5mu
+
+runners:
+  - type: nextflow

src/workflows/multiomics/spatial_process_samples/integration_test.sh

@@ -0,0 +1,17 @@
+#!/bin/bash
+
+set -eo pipefail
+
+# get the root of the directory
+REPO_ROOT=$(git rev-parse --show-toplevel)
+
+# ensure that the command below is run from the root of the repository
+cd "$REPO_ROOT"
+
+nextflow \
+  run . \
+  -main-script src/workflows/multiomics/spatial_process_samples/test.nf \
+  -entry test_wf \
+  -profile docker,no_publish \
+  -c src/workflows/utils/labels_ci.config \
+  -c src/workflows/utils/integration_tests.config

src/workflows/multiomics/spatial_process_samples/main.nf

@@ -0,0 +1,77 @@
+workflow run_wf {
+  take:
+    input_ch
+
+  main:
+    output_ch = input_ch
+      | map { id, state ->
+        def new_state = [
+          state.id, 
+          state + ["_meta": ["join_id": id], "workflow_output": state.output]
+        ]
+        new_state
+      }
+      | spatial_sample_processing.run(
+        fromState: { id, state -> [
+          "id": id,
+          "input": state.input,
+          "rna_layer": state.rna_layer,
+          "prot_layer": state.prot_layer,
+          "add_id_to_obs": state.add_id_to_obs,
+          "add_id_obs_output": state.add_id_obs_output,
+          "add_id_make_observation_keys_unique": state.add_id_make_observation_keys_unique,
+          "rna_min_counts": state.rna_min_counts,
+          "rna_max_counts": state.rna_max_counts,
+          "rna_min_genes_per_cell": state.rna_min_genes_per_cell,
+          "rna_max_genes_per_cell": state.rna_max_genes_per_cell,
+          "rna_min_cells_per_gene": state.rna_min_cells_per_gene,
+          "rna_min_fraction_mito": state.rna_min_fraction_mito,
+          "rna_max_fraction_mito": state.rna_max_fraction_mito,
+          "rna_min_fraction_ribo": state.rna_min_fraction_ribo,
+          "rna_max_fraction_ribo": state.rna_max_fraction_ribo,
+          "prot_min_counts": state.prot_min_counts,
+          "prot_max_counts": state.prot_max_counts,
+          "prot_min_proteins_per_cell": state.prot_min_proteins_per_cell,
+          "prot_max_proteins_per_cell": state.prot_max_proteins_per_cell,
+          "prot_min_cells_per_protein": state.prot_min_cells_per_protein,
+          "highly_variable_features_var_output": state.highly_variable_features_var_output,
+          "highly_variable_features_obs_batch_key": state.highly_variable_features_obs_batch_key,
+          "var_gene_names": state.var_gene_names,
+          "var_name_mitochondrial_genes": state.var_name_mitochondrial_genes,
+          "obs_name_mitochondrial_fraction": state.obs_name_mitochondrial_fraction,
+          "mitochondrial_gene_regex": state.mitochondrial_gene_regex,
+          "var_name_ribosomal_genes": state.var_name_ribosomal_genes,
+          "obs_name_ribosomal_fraction": state.obs_name_ribosomal_fraction,
+          "ribosomal_gene_regex": state.ribosomal_gene_regex,
+          "var_qc_metrics": state.var_qc_metrics,
+          "top_n_vars": state.top_n_vars,
+          "pca_overwrite": state.pca_overwrite,
+          "clr_axis": state.clr_axis,
+          "rna_enable_scaling": state.rna_enable_scaling,
+          "rna_scaling_output_layer": state.rna_scaling_output_layer,
+          "rna_scaling_pca_obsm_output": state.rna_scaling_pca_obsm_output,
+          "rna_scaling_pca_loadings_varm_output": state.rna_scaling_pca_loadings_varm_output,
+          "rna_scaling_pca_variance_uns_output": state.rna_scaling_pca_variance_uns_output,
+          "rna_scaling_umap_obsm_output": state.rna_scaling_umap_obsm_output,
+          "rna_scaling_max_value": state.rna_scaling_max_value,
+          "rna_scaling_zero_center": state.rna_scaling_zero_center,
+          "output": state.workflow_output
+        ]},
+        args: [
+          "skip_scrublet_doublet_detection": "true",
+        ],
+        toState: [
+          "output": "output"
+        ]
+      )
+
+      | setState( 
+        [
+          "_meta": "_meta",
+          "output": "output"
+        ]
+      )
+
+  emit:
+    output_ch
+}

src/workflows/multiomics/spatial_process_samples/nextflow.config

@@ -0,0 +1,10 @@
+manifest {
+  nextflowVersion = '!>=20.12.1-edge'
+}
+
+params {
+  rootDir = java.nio.file.Paths.get("$projectDir/../../../../").toAbsolutePath().normalize().toString()
+}
+
+// include common settings
+includeConfig("${params.rootDir}/src/workflows/utils/labels.config")

src/workflows/multiomics/spatial_process_samples/test.nf

@@ -0,0 +1,33 @@
+nextflow.enable.dsl=2
+targetDir = params.rootDir + "/target/nextflow"
+
+include { spatial_process_samples } from targetDir + "/workflows/multiomics/spatial_process_samples/main.nf"
+
+params.resources_test = params.rootDir + "/resources_test"
+
+workflow test_wf {
+
+  resources_test = file(params.resources_test)
+
+  output_ch = Channel.fromList([
+    [
+      id: "xenium",
+      input: resources_test.resolve("xenium/xenium_tiny.h5mu"),
+      publish_dir: "foo/",
+      output: "test.h5mu",
+    ]
+  ])
+  | map{ state -> [state.id, state] }
+  | spatial_process_samples
+  | view { output ->
+    assert output.size() == 2 : "outputs should contain two elements; [id, file]"
+    assert output[1].output.toString().endsWith("test.h5mu") : "Output file should be a h5mu file. Found: ${output[1].output}"
+    "Output: $output"
+  }
+  | toSortedList()
+  | map { output_list ->
+    assert output_list.size() == 1 : "output channel should contain one event"
+    assert output_list[0][0] == "merged" : "Output ID should be 'merged'"
+  }
+  
+}

src/workflows/qc/spatial_qc/config.vsh.yaml

@@ -0,0 +1,174 @@
+name: "spatial_qc"
+namespace: "workflows/qc"
+scope: "public"
+description: "A pipeline to add basic qc statistics to a MuData containing spatial data."
+authors:
+  - __merge__: /src/authors/dries_schaumont.yaml
+    roles: [ author, maintainer ]
+  - __merge__: /src/authors/dorien_roosen.yaml
+    roles: [ contributor ]
+  - __merge__: /src/authors/weiwei_schultz.yaml
+    roles: [ contributor ]
+info:
+  test_dependencies:
+    - name: qc_test
+      namespace: test_workflows/qc
+argument_groups:
+  - name: Inputs
+    arguments:
+      - name: "--id"
+        required: true
+        type: string
+        description: ID of the sample.
+        example: foo
+      - name: "--input"
+        alternatives: [-i]
+        description: Path to the sample.
+        required: true
+        example: input.h5mu
+        type: file
+      - name: "--modality"
+        description: Which modality to process.
+        type: string
+        default: "rna"
+        required: false
+      - name: "--layer"
+        description: "Use specified layer for calculation of qc metrics. If not specified, adata.X is used."
+        type: string
+        example: "raw_counts"
+        required: false
+  - name: "Mitochondrial & Ribosomal Gene Detection"
+    arguments:
+      - name: "--var_gene_names"
+        required: false
+        example: "gene_symbol"
+        type: string
+        description: |
+          .var column name to be used to detect mitochondrial/ribosomal genes instead of .var_names (default if not set).
+          Gene names matching with the regex value from --mitochondrial_gene_regex or --ribosomal_gene_regex will be 
+          identified as mitochondrial or ribosomal genes, respectively.
+      - name: "--var_name_mitochondrial_genes"
+        type: string
+        required: false
+        description: |
+          In which .var slot to store a boolean array corresponding the mitochondrial genes.
+      - name: "--obs_name_mitochondrial_fraction"
+        type: string
+        required: false
+        description: |
+          .Obs slot to store the fraction of reads found to be mitochondrial. Defaults to 'fraction_' suffixed by the value of --var_name_mitochondrial_genes
+      - name: --mitochondrial_gene_regex
+        type: string
+        description: |
+          Regex string that identifies mitochondrial genes from --var_gene_names.
+          By default will detect human and mouse mitochondrial genes from a gene symbol.
+        required: false
+        default: "^[mM][tT]-"
+      - name: "--var_name_ribosomal_genes"
+        type: string
+        required: false
+        description: |
+          In which .var slot to store a boolean array corresponding the ribosomal genes.
+      - name: "--obs_name_ribosomal_fraction"
+        type: string
+        required: false
+        description: |
+          When specified, write the fraction of counts originating from ribosomal genes 
+          (based on --ribosomal_gene_regex) to an .obs column with the specified name.
+          Requires --var_name_ribosomal_genes.
+      - name: --ribosomal_gene_regex
+        type: string
+        description: |
+          Regex string that identifies ribosomal genes from --var_gene_names.
+          By default will detect human and mouse ribosomal genes from a gene symbol.
+        required: false
+        default: "^[Mm]?[Rr][Pp][LlSs]"
+  - name: "QC metrics calculation options"
+    arguments:
+      - name: "--var_qc_metrics"
+        description: |
+          Keys to select a boolean (containing only True or False) column from .var.
+          For each cell, calculate the proportion of total values for genes which are labeled 'True', 
+          compared to the total sum of the values for all genes. Defaults to the value from
+          --var_name_mitochondrial_genes.
+        type: string
+        multiple: True
+        multiple_sep: ','
+        required: false
+        example: "ercc,highly_variable"
+      - name: "--top_n_vars"
+        type: integer
+        description: |
+          Number of top vars to be used to calculate cumulative proportions.
+          If not specified, proportions are not calculated. `--top_n_vars 20,50` finds
+          cumulative proportion to the 20th and 50th most expressed vars.
+        multiple: true
+        multiple_sep: ','
+        required: false
+        default: [50, 100, 200, 500]
+      - name: "--output_obs_num_nonzero_vars"
+        description: |
+          Name of column in .obs describing, for each observation, the number of stored values
+          (including explicit zeroes). In other words, the name of the column that counts
+          for each row the number of columns that contain data.
+        type: string
+        required: false
+        default: "num_nonzero_vars"
+      - name: "--output_obs_total_counts_vars"
+        description: |
+          Name of the column for .obs describing, for each observation (row),
+          the sum of the stored values in the columns.
+        type: string
+        required: false
+        default: total_counts
+      - name: "--output_var_num_nonzero_obs"
+        description: |
+          Name of column describing, for each feature, the number of stored values
+          (including explicit zeroes). In other words, the name of the column that counts
+          for each column the number of rows that contain data.
+        type: string
+        required: false
+        default: "num_nonzero_obs"
+      - name: "--output_var_total_counts_obs"
+        description: |
+          Name of the column in .var describing, for each feature (column),
+          the sum of the stored values in the rows.
+        type: string
+        required: false
+        default: total_counts
+      - name: "--output_var_obs_mean"
+        type: string
+        description: |
+          Name of the column in .obs providing the mean of the values in each row.
+        default: "obs_mean"
+        required: false
+      - name: "--output_var_pct_dropout"
+        type: string
+        default: "pct_dropout"
+        description: |
+          Name of the column in .obs providing for each feature the percentage of
+          observations the feature does not appear on (i.e. is missing). Same as `--output_var_num_nonzero_obs`
+          but percentage based.
+  - name: "Outputs"
+    arguments:
+      - name: "--output"
+        type: file
+        required: true
+        direction: output
+        description: Destination path to the output.
+        example: output.h5mu
+dependencies:
+  - name: workflows/qc/qc
+    alias: spatial_qc_workflow
+    repository: openpipeline
+resources:
+  - type: nextflow_script
+    path: main.nf
+    entrypoint: run_wf
+test_resources:
+  - type: nextflow_script
+    path: test.nf
+    entrypoint: test_wf
+  - path: /resources_test/xenium/xenium_tiny.h5mu
+runners:
+  - type: nextflow

src/workflows/qc/spatial_qc/integration_test.sh

@@ -0,0 +1,15 @@
+#!/bin/bash
+
+# get the root of the directory
+REPO_ROOT=$(git rev-parse --show-toplevel)
+
+# ensure that the command below is run from the root of the repository
+cd "$REPO_ROOT"
+
+nextflow \
+  run . \
+  -main-script src/workflows/qc/spatial_qc/test.nf \
+  -entry test_wf \
+  -profile docker,no_publish \
+  -c src/workflows/utils/labels_ci.config \
+  -c src/workflows/utils/integration_tests.config

src/workflows/qc/spatial_qc/main.nf

@@ -0,0 +1,38 @@
+workflow run_wf {
+  take:
+    input_ch
+
+  main:
+    output_ch = input_ch
+      | spatial_qc_workflow.run(
+        fromState: { id, state -> [
+          "id": id,
+          "input": state.input,
+          "modality": state.modality,
+          "layer": state.layer,
+          "var_gene_names": state.var_gene_names,
+          "var_name_mitochondrial_genes": state.var_name_mitochondrial_genes,
+          "obs_name_mitochondrial_fraction": state.obs_name_mitochondrial_fraction,
+          "mitochondrial_gene_regex": state.mitochondrial_gene_regex,
+          "var_name_ribosomal_genes": state.var_name_ribosomal_genes,
+          "obs_name_ribosomal_fraction": state.obs_name_ribosomal_fraction,
+          "ribosomal_gene_regex": state.ribosomal_gene_regex,
+          "var_qc_metrics": state.var_qc_metrics,
+          "top_n_vars": state.top_n_vars,
+          "output_obs_num_nonzero_vars": state.output_obs_num_nonzero_vars,
+          "output_obs_total_counts_vars": state.output_obs_total_counts_vars,
+          "output_var_num_nonzero_obs": state.output_var_num_nonzero_obs,
+          "output_var_total_counts_obs": state.output_var_total_counts_obs,
+          "output_var_obs_mean": state.output_var_obs_mean,
+          "output_var_pct_dropout": state.output_var_pct_dropout
+        ]},
+        toState: [
+          "output": "output"
+        ]
+      )
+
+      | setState(["output"])
+    
+  emit:
+    output_ch
+}

src/workflows/qc/spatial_qc/nextflow.config

@@ -0,0 +1,10 @@
+manifest {
+  nextflowVersion = '!>=20.12.1-edge'
+}
+
+params {
+  rootDir = java.nio.file.Paths.get("$projectDir/../../../../").toAbsolutePath().normalize().toString()
+}
+
+// include common settings
+includeConfig("${params.rootDir}/src/workflows/utils/labels.config")

src/workflows/qc/spatial_qc/test.nf

@@ -0,0 +1,40 @@
+nextflow.enable.dsl=2
+
+include { spatial_qc } from params.rootDir + "/target/nextflow/workflows/qc/spatial_qc/main.nf"
+
+params.resources_test = params.rootDir + "/resources_test"
+
+workflow test_wf {
+
+  resources_test = file(params.resources_test)
+
+  output_ch = 
+    Channel.fromList([
+      [
+        id: "xenium_test",
+        input: resources_test.resolve("xenium/xenium_tiny.h5mu"),
+        var_name_mitochondrial_genes: "mitochondrial",
+        var_name_ribosomal_genes: "ribosomal",
+      ]
+    ])
+    | map { state -> [state.id, state] }
+    | spatial_qc.run(
+      toState: { id, output, state -> output + [og_input: state.input] }
+    )
+
+    | view { output ->
+      assert output.size() == 2 : "Outputs should contain two elements; [id, state]"
+
+      // check id
+      def id = output[0]
+      assert id.endsWith("_test")
+
+      // check output
+      def state = output[1]
+      assert state instanceof Map : "State should be a map. Found: ${state}"
+      assert state.containsKey("output") : "Output should contain key 'output'."
+      assert state.output.isFile() : "'output' should be a file."
+      assert state.output.toString().endsWith(".h5mu") : "Output file should end with '.h5mu'. Found: ${state.output}"
+
+    }
+}

src/workflows/utils/integration_tests.config

@@ -0,0 +1,36 @@
+profiles {
+
+  // detect tempdir
+  tempDir = java.nio.file.Paths.get(
+    System.getenv('NXF_TEMP') ?:
+      System.getenv('VIASH_TEMP') ?: 
+      System.getenv('TEMPDIR') ?: 
+      System.getenv('TMPDIR') ?: 
+      '/tmp'
+  ).toAbsolutePath()
+
+  mount_temp {
+    docker.temp            = tempDir
+    podman.temp            = tempDir
+    charliecloud.temp      = tempDir
+  }
+
+  no_publish {
+    process {
+      withName: '.*' {
+        publishDir = [
+          enabled: false
+        ]
+      }
+    }
+  }
+
+  docker {
+    docker.enabled         = true
+    // docker.userEmulation   = true
+    singularity.enabled    = false
+    podman.enabled         = false
+    shifter.enabled        = false
+    charliecloud.enabled   = false
+  }
+}

src/workflows/utils/labels.config

@@ -0,0 +1,68 @@
+process {
+  // Default resources for components that hardly do any processing
+  memory = { 2.GB * task.attempt }
+  cpus = 1
+
+  // Retry for exit codes that have something to do with memory issues
+  errorStrategy = { task.exitStatus in 137..140 ? 'retry' : 'terminate' }
+  maxRetries = 3
+  maxMemory = null
+
+  // CPU resources
+  withLabel: singlecpu { cpus = 1 }
+  withLabel: lowcpu { cpus = 4 }
+  withLabel: midcpu { cpus = 10 }
+  withLabel: highcpu { cpus = 20 }
+  
+  // Memory resources
+  withLabel: lowmem { memory = { get_memory( 50.GB * task.attempt ) } }
+  withLabel: midmem { memory = { get_memory( 50.GB * task.attempt ) } }
+  withLabel: highmem { memory = { get_memory( 50.GB * task.attempt ) } }
+  withLabel: veryhighmem { memory = { get_memory( 75.GB * task.attempt ) } }
+
+  // Disk space
+  // Nextflow apparently can't handle empty directives, i.e.
+  // withLabel: lowdisk {}
+  // so for that reason we have to add a dummy directive
+  withLabel: lowdisk {
+    dummyDirective = "dummyValue"
+  }
+  withLabel: middisk {
+    dummyDirective = "dummyValue"
+  }
+  withLabel: highdisk {
+    dummyDirective = "dummyValue"
+  }
+  withLabel: veryhighdisk {
+    dummyDirective = "dummyValue"
+  }
+  // NOTE: The above labels intentionally do not have an effect by default.
+  // The user should set the disk space requirements by adding the following
+  // to the compute environment:
+  //
+  // withLabel: lowdisk { disk = { 20.GB * task.attempt } }
+  // withLabel: middisk { disk = { 100.GB * task.attempt } }
+  // withLabel: highdisk { disk = { 200.GB * task.attempt } }
+  // withLabel: veryhighdisk { disk = { 500.GB * task.attempt } }
+}
+
+def get_memory(to_compare) {
+  if (!process.containsKey("maxMemory") || !process.maxMemory) {
+    return to_compare
+  }
+
+  try {
+    if (process.containsKey("maxRetries") && process.maxRetries && task.attempt == (process.maxRetries as int)) {
+      return process.maxMemory
+    }
+    else if (to_compare.compareTo(process.maxMemory as nextflow.util.MemoryUnit) == 1) {
+      return max_memory as nextflow.util.MemoryUnit
+    }
+    else {
+      return to_compare
+    }  
+  } catch (all) {
+        println "Error processing memory resources. Please check that process.maxMemory '${process.maxMemory}' and process.maxRetries '${process.maxRetries}' are valid!"
+        System.exit(1)
+  }
+}

src/workflows/utils/labels_ci.config

@@ -0,0 +1,105 @@
+process {
+  withLabel: lowmem { memory = 13.Gb }
+  withLabel: lowcpu { cpus = 4 }
+  withLabel: midmem { memory = 13.Gb }
+  withLabel: midcpu { cpus = 4 }
+  withLabel: highmem { memory = 13.Gb }
+  withLabel: highcpu { cpus = 4 }
+  withLabel: veryhighmem { memory = 13.Gb }
+  withLabel: lowdisk {
+    disk = {process.disk ? process.disk : null}
+  }
+  withLabel: middisk {
+    disk = {process.disk ? process.disk : null}
+  }
+  withLabel: highdisk {
+    disk = {process.disk ? process.disk : null}
+  }
+  withLabel: veryhighdisk {
+    disk = {process.disk ? process.disk : null}
+  }
+}
+
+env.NUMBA_CACHE_DIR = '/tmp'
+
+trace {
+    enabled = true
+    overwrite = true
+}
+dag {
+  overwrite = true
+}
+
+process.maxForks = 1
+
+profiles {
+  // detect tempdir
+  tempDir = java.nio.file.Paths.get(
+    System.getenv('NXF_TEMP') ?:
+      System.getenv('VIASH_TEMP') ?: 
+      System.getenv('TEMPDIR') ?: 
+      System.getenv('TMPDIR') ?: 
+      '/tmp'
+  ).toAbsolutePath()
+
+  mount_temp {
+    docker.temp            = tempDir
+    podman.temp            = tempDir
+    charliecloud.temp      = tempDir
+  }
+
+  no_publish {
+    process {
+      withName: '.*' {
+        publishDir = [
+          enabled: false
+        ]
+      }
+    }
+  }
+
+  docker {
+    docker.fixOwnership    = true
+    docker.enabled         = true
+    singularity.enabled    = false
+    podman.enabled         = false
+    shifter.enabled        = false
+    charliecloud.enabled   = false
+  }
+
+  local {
+    // This config is for local processing.
+    process {
+        maxMemory = 25.GB
+        withLabel: verylowcpu { cpus = 2 }
+        withLabel: lowcpu { cpus = 4 }
+        withLabel: midcpu { cpus = 6 }
+        withLabel: highcpu { cpus = 12 }
+  
+        withLabel: lowmem { memory = { get_memory( 8.GB * task.attempt ) } }
+        withLabel: midmem { memory = { get_memory( 12.GB * task.attempt ) } }
+        withLabel: highmem { memory = { get_memory( 20.GB * task.attempt ) } }
+    }
+  }
+}
+
+def get_memory(to_compare) {
+  if (!process.containsKey("maxMemory") || !process.maxMemory) {
+    return to_compare
+  }
+
+  try {
+    if (process.containsKey("maxRetries") && process.maxRetries && task.attempt == (process.maxRetries as int)) {
+      return process.maxMemory
+    }
+    else if (to_compare.compareTo(process.maxMemory as nextflow.util.MemoryUnit) == 1) {
+      return max_memory as nextflow.util.MemoryUnit
+    }
+    else {
+      return to_compare
+    }  
+  } catch (all) {
+        println "Error processing memory resources. Please check that process.maxMemory '${process.maxMemory}' and process.maxRetries '${process.maxRetries}' are valid!"
+        System.exit(1)
+  }
+}

target/_private/executable/filter/subset_cosmx/.config.vsh.yaml

@@ -0,0 +1,259 @@
+name: "subset_cosmx"
+namespace: "filter"
+version: "niche-compass"
+authors:
+- name: "Dorien Roosen"
+  roles:
+  - "maintainer"
+  info:
+    role: "Core Team Member"
+    links:
+      email: "dorien@data-intuitive.com"
+      github: "dorien-er"
+      linkedin: "dorien-roosen"
+    organizations:
+    - name: "Data Intuitive"
+      href: "https://www.data-intuitive.com"
+      role: "Data Scientist"
+- name: "Weiwei Schultz"
+  roles:
+  - "contributor"
+  info:
+    role: "Contributor"
+    organizations:
+    - name: "Janssen R&D US"
+      role: "Associate Director Data Sciences"
+argument_groups:
+- name: "Arguments"
+  arguments:
+  - type: "file"
+    name: "--input"
+    alternatives:
+    - "-i"
+    description: "Input folder. Must contain the output from a NanoString CosMx run."
+    info: null
+    example:
+    - "cosmx_data"
+    must_exist: true
+    create_parent: true
+    required: true
+    direction: "input"
+    multiple: false
+    multiple_sep: ";"
+  - type: "integer"
+    name: "--num_fovs"
+    description: "Number of fields of views to keep. Will keep only the first <num_fovs>\
+      \ fields of view."
+    info: null
+    required: true
+    direction: "input"
+    multiple: false
+    multiple_sep: ";"
+  - type: "boolean"
+    name: "--subset_transcripts_file"
+    description: "Whether to subset the <dataset_id>_tx_file.csv file."
+    info: null
+    default:
+    - true
+    required: false
+    direction: "input"
+    multiple: false
+    multiple_sep: ";"
+  - type: "boolean"
+    name: "--subset_polygons_file"
+    description: "Whether to subset the <dataset_id>_polygons.csv file."
+    info: null
+    default:
+    - true
+    required: false
+    direction: "input"
+    multiple: false
+    multiple_sep: ";"
+  - type: "file"
+    name: "--output"
+    alternatives:
+    - "-o"
+    description: "The directory where the subset data will be stored."
+    info: null
+    example:
+    - "cosmx_data_tiny"
+    must_exist: true
+    create_parent: true
+    required: false
+    direction: "output"
+    multiple: false
+    multiple_sep: ";"
+resources:
+- type: "python_script"
+  path: "script.py"
+  is_executable: true
+- type: "file"
+  path: "setup_logger.py"
+- type: "file"
+  path: "nextflow_labels.config"
+  dest: "nextflow_labels.config"
+description: "Filters the output from NanoString experiment to keep only a subset\
+  \ of the fields of view.\nExpected input folder structure:\npath/to/dataset/\n \
+  \   ├── CellComposite/\n    ├── CellLabels/\n    ├── CellOverlay/\n    ├── CompartmentLabels/\n\
+  \    ├── <dataset_id>_exprMat_file.csv\n    ├── <dataset_id>_fov_positions_file.csv\n\
+  \    ├── <dataset_id>_metadata_file.csv\n    └── <dataset_id>_tx_file.csv  \n"
+test_resources:
+- type: "python_script"
+  path: "test.py"
+  is_executable: true
+- type: "file"
+  path: "Lung5_Rep2_tiny"
+info: null
+status: "enabled"
+scope:
+  image: "private"
+  target: "private"
+repositories:
+- type: "vsh"
+  name: "openpipeline"
+  repo: "openpipeline"
+  tag: "v3.0.0"
+links:
+  repository: "https://github.com/openpipelines-bio/openpipeline_spatial"
+  docker_registry: "ghcr.io"
+runners:
+- type: "executable"
+  id: "executable"
+  docker_setup_strategy: "ifneedbepullelsecachedbuild"
+- type: "nextflow"
+  id: "nextflow"
+  directives:
+    label:
+    - "lowmem"
+    - "singlecpu"
+    tag: "$id"
+  auto:
+    simplifyInput: true
+    simplifyOutput: false
+    transcript: false
+    publish: false
+  config:
+    labels:
+      mem1gb: "memory = 1000000000.B"
+      mem2gb: "memory = 2000000000.B"
+      mem5gb: "memory = 5000000000.B"
+      mem10gb: "memory = 10000000000.B"
+      mem20gb: "memory = 20000000000.B"
+      mem50gb: "memory = 50000000000.B"
+      mem100gb: "memory = 100000000000.B"
+      mem200gb: "memory = 200000000000.B"
+      mem500gb: "memory = 500000000000.B"
+      mem1tb: "memory = 1000000000000.B"
+      mem2tb: "memory = 2000000000000.B"
+      mem5tb: "memory = 5000000000000.B"
+      mem10tb: "memory = 10000000000000.B"
+      mem20tb: "memory = 20000000000000.B"
+      mem50tb: "memory = 50000000000000.B"
+      mem100tb: "memory = 100000000000000.B"
+      mem200tb: "memory = 200000000000000.B"
+      mem500tb: "memory = 500000000000000.B"
+      mem1gib: "memory = 1073741824.B"
+      mem2gib: "memory = 2147483648.B"
+      mem4gib: "memory = 4294967296.B"
+      mem8gib: "memory = 8589934592.B"
+      mem16gib: "memory = 17179869184.B"
+      mem32gib: "memory = 34359738368.B"
+      mem64gib: "memory = 68719476736.B"
+      mem128gib: "memory = 137438953472.B"
+      mem256gib: "memory = 274877906944.B"
+      mem512gib: "memory = 549755813888.B"
+      mem1tib: "memory = 1099511627776.B"
+      mem2tib: "memory = 2199023255552.B"
+      mem4tib: "memory = 4398046511104.B"
+      mem8tib: "memory = 8796093022208.B"
+      mem16tib: "memory = 17592186044416.B"
+      mem32tib: "memory = 35184372088832.B"
+      mem64tib: "memory = 70368744177664.B"
+      mem128tib: "memory = 140737488355328.B"
+      mem256tib: "memory = 281474976710656.B"
+      mem512tib: "memory = 562949953421312.B"
+      cpu1: "cpus = 1"
+      cpu2: "cpus = 2"
+      cpu5: "cpus = 5"
+      cpu10: "cpus = 10"
+      cpu20: "cpus = 20"
+      cpu50: "cpus = 50"
+      cpu100: "cpus = 100"
+      cpu200: "cpus = 200"
+      cpu500: "cpus = 500"
+      cpu1000: "cpus = 1000"
+    script:
+    - "includeConfig(\"nextflow_labels.config\")"
+  debug: false
+  container: "docker"
+engines:
+- type: "docker"
+  id: "docker"
+  image: "python:3.12-slim"
+  target_registry: "images.viash-hub.com"
+  target_tag: "niche-compass"
+  namespace_separator: "/"
+  setup:
+  - type: "apt"
+    packages:
+    - "procps"
+    interactive: false
+  - type: "python"
+    user: false
+    packages:
+    - "spatialdata~=0.5.0"
+    - "pyarrow~=18.0.0"
+    - "squidpy~=1.6.5"
+    upgrade: true
+  test_setup:
+  - type: "apt"
+    packages:
+    - "git"
+    interactive: false
+  - type: "python"
+    user: false
+    packages:
+    - "viashpy==0.9.0"
+    github:
+    - "openpipelines-bio/core#subdirectory=packages/python/openpipeline_testutils"
+    upgrade: true
+  entrypoint: []
+  cmd: null
+- type: "native"
+  id: "native"
+build_info:
+  config: "src/filter/subset_cosmx/config.vsh.yaml"
+  runner: "executable"
+  engine: "docker|native"
+  output: "target/_private/executable/filter/subset_cosmx"
+  executable: "target/_private/executable/filter/subset_cosmx/subset_cosmx"
+  viash_version: "0.9.4"
+  git_commit: "0c1677bb93680d39ec2fb2f6bc68a2fcfae0e831"
+  git_remote: "https://github.com/openpipelines-bio/openpipeline_spatial"
+package_config:
+  name: "openpipeline_spatial"
+  version: "niche-compass"
+  info:
+    test_resources:
+    - type: "s3"
+      path: "s3://openpipelines-bio/openpipeline_spatial/resources_test"
+      dest: "resources_test"
+  repositories:
+  - type: "vsh"
+    name: "openpipeline"
+    repo: "openpipeline"
+    tag: "v3.0.0"
+  viash_version: "0.9.4"
+  source: "src"
+  target: "target"
+  config_mods:
+  - ".resources += {path: '/src/workflows/utils/labels.config', dest: 'nextflow_labels.config'}\n\
+    .runners[.type == 'nextflow'].config.script := 'includeConfig(\"nextflow_labels.config\"\
+    )'"
+  - ".engines += { type: \"native\" }"
+  - ".engines[.type == 'docker'].target_registry := 'images.viash-hub.com'"
+  - ".engines[.type == 'docker'].target_tag := 'niche-compass'"
+  organization: "vsh"
+  links:
+    repository: "https://github.com/openpipelines-bio/openpipeline_spatial"
+    docker_registry: "ghcr.io"

target/_private/executable/filter/subset_cosmx/nextflow_labels.config

@@ -0,0 +1,68 @@
+process {
+  // Default resources for components that hardly do any processing
+  memory = { 2.GB * task.attempt }
+  cpus = 1
+
+  // Retry for exit codes that have something to do with memory issues
+  errorStrategy = { task.exitStatus in 137..140 ? 'retry' : 'terminate' }
+  maxRetries = 3
+  maxMemory = null
+
+  // CPU resources
+  withLabel: singlecpu { cpus = 1 }
+  withLabel: lowcpu { cpus = 4 }
+  withLabel: midcpu { cpus = 10 }
+  withLabel: highcpu { cpus = 20 }
+  
+  // Memory resources
+  withLabel: lowmem { memory = { get_memory( 50.GB * task.attempt ) } }
+  withLabel: midmem { memory = { get_memory( 50.GB * task.attempt ) } }
+  withLabel: highmem { memory = { get_memory( 50.GB * task.attempt ) } }
+  withLabel: veryhighmem { memory = { get_memory( 75.GB * task.attempt ) } }
+
+  // Disk space
+  // Nextflow apparently can't handle empty directives, i.e.
+  // withLabel: lowdisk {}
+  // so for that reason we have to add a dummy directive
+  withLabel: lowdisk {
+    dummyDirective = "dummyValue"
+  }
+  withLabel: middisk {
+    dummyDirective = "dummyValue"
+  }
+  withLabel: highdisk {
+    dummyDirective = "dummyValue"
+  }
+  withLabel: veryhighdisk {
+    dummyDirective = "dummyValue"
+  }
+  // NOTE: The above labels intentionally do not have an effect by default.
+  // The user should set the disk space requirements by adding the following
+  // to the compute environment:
+  //
+  // withLabel: lowdisk { disk = { 20.GB * task.attempt } }
+  // withLabel: middisk { disk = { 100.GB * task.attempt } }
+  // withLabel: highdisk { disk = { 200.GB * task.attempt } }
+  // withLabel: veryhighdisk { disk = { 500.GB * task.attempt } }
+}
+
+def get_memory(to_compare) {
+  if (!process.containsKey("maxMemory") || !process.maxMemory) {
+    return to_compare
+  }
+
+  try {
+    if (process.containsKey("maxRetries") && process.maxRetries && task.attempt == (process.maxRetries as int)) {
+      return process.maxMemory
+    }
+    else if (to_compare.compareTo(process.maxMemory as nextflow.util.MemoryUnit) == 1) {
+      return max_memory as nextflow.util.MemoryUnit
+    }
+    else {
+      return to_compare
+    }  
+  } catch (all) {
+        println "Error processing memory resources. Please check that process.maxMemory '${process.maxMemory}' and process.maxRetries '${process.maxRetries}' are valid!"
+        System.exit(1)
+  }
+}

target/_private/executable/filter/subset_cosmx/setup_logger.py

@@ -0,0 +1,12 @@
+def setup_logger():
+    import logging
+    from sys import stdout
+
+    logger = logging.getLogger()
+    logger.setLevel(logging.INFO)
+    console_handler = logging.StreamHandler(stdout)
+    logFormatter = logging.Formatter("%(asctime)s %(levelname)-8s %(message)s")
+    console_handler.setFormatter(logFormatter)
+    logger.addHandler(console_handler)
+
+    return logger

target/_private/nextflow/filter/subset_cosmx/.config.vsh.yaml

@@ -0,0 +1,259 @@
+name: "subset_cosmx"
+namespace: "filter"
+version: "niche-compass"
+authors:
+- name: "Dorien Roosen"
+  roles:
+  - "maintainer"
+  info:
+    role: "Core Team Member"
+    links:
+      email: "dorien@data-intuitive.com"
+      github: "dorien-er"
+      linkedin: "dorien-roosen"
+    organizations:
+    - name: "Data Intuitive"
+      href: "https://www.data-intuitive.com"
+      role: "Data Scientist"
+- name: "Weiwei Schultz"
+  roles:
+  - "contributor"
+  info:
+    role: "Contributor"
+    organizations:
+    - name: "Janssen R&D US"
+      role: "Associate Director Data Sciences"
+argument_groups:
+- name: "Arguments"
+  arguments:
+  - type: "file"
+    name: "--input"
+    alternatives:
+    - "-i"
+    description: "Input folder. Must contain the output from a NanoString CosMx run."
+    info: null
+    example:
+    - "cosmx_data"
+    must_exist: true
+    create_parent: true
+    required: true
+    direction: "input"
+    multiple: false
+    multiple_sep: ";"
+  - type: "integer"
+    name: "--num_fovs"
+    description: "Number of fields of views to keep. Will keep only the first <num_fovs>\
+      \ fields of view."
+    info: null
+    required: true
+    direction: "input"
+    multiple: false
+    multiple_sep: ";"
+  - type: "boolean"
+    name: "--subset_transcripts_file"
+    description: "Whether to subset the <dataset_id>_tx_file.csv file."
+    info: null
+    default:
+    - true
+    required: false
+    direction: "input"
+    multiple: false
+    multiple_sep: ";"
+  - type: "boolean"
+    name: "--subset_polygons_file"
+    description: "Whether to subset the <dataset_id>_polygons.csv file."
+    info: null
+    default:
+    - true
+    required: false
+    direction: "input"
+    multiple: false
+    multiple_sep: ";"
+  - type: "file"
+    name: "--output"
+    alternatives:
+    - "-o"
+    description: "The directory where the subset data will be stored."
+    info: null
+    example:
+    - "cosmx_data_tiny"
+    must_exist: true
+    create_parent: true
+    required: false
+    direction: "output"
+    multiple: false
+    multiple_sep: ";"
+resources:
+- type: "python_script"
+  path: "script.py"
+  is_executable: true
+- type: "file"
+  path: "setup_logger.py"
+- type: "file"
+  path: "nextflow_labels.config"
+  dest: "nextflow_labels.config"
+description: "Filters the output from NanoString experiment to keep only a subset\
+  \ of the fields of view.\nExpected input folder structure:\npath/to/dataset/\n \
+  \   ├── CellComposite/\n    ├── CellLabels/\n    ├── CellOverlay/\n    ├── CompartmentLabels/\n\
+  \    ├── <dataset_id>_exprMat_file.csv\n    ├── <dataset_id>_fov_positions_file.csv\n\
+  \    ├── <dataset_id>_metadata_file.csv\n    └── <dataset_id>_tx_file.csv  \n"
+test_resources:
+- type: "python_script"
+  path: "test.py"
+  is_executable: true
+- type: "file"
+  path: "Lung5_Rep2_tiny"
+info: null
+status: "enabled"
+scope:
+  image: "private"
+  target: "private"
+repositories:
+- type: "vsh"
+  name: "openpipeline"
+  repo: "openpipeline"
+  tag: "v3.0.0"
+links:
+  repository: "https://github.com/openpipelines-bio/openpipeline_spatial"
+  docker_registry: "ghcr.io"
+runners:
+- type: "executable"
+  id: "executable"
+  docker_setup_strategy: "ifneedbepullelsecachedbuild"
+- type: "nextflow"
+  id: "nextflow"
+  directives:
+    label:
+    - "lowmem"
+    - "singlecpu"
+    tag: "$id"
+  auto:
+    simplifyInput: true
+    simplifyOutput: false
+    transcript: false
+    publish: false
+  config:
+    labels:
+      mem1gb: "memory = 1000000000.B"
+      mem2gb: "memory = 2000000000.B"
+      mem5gb: "memory = 5000000000.B"
+      mem10gb: "memory = 10000000000.B"
+      mem20gb: "memory = 20000000000.B"
+      mem50gb: "memory = 50000000000.B"
+      mem100gb: "memory = 100000000000.B"
+      mem200gb: "memory = 200000000000.B"
+      mem500gb: "memory = 500000000000.B"
+      mem1tb: "memory = 1000000000000.B"
+      mem2tb: "memory = 2000000000000.B"
+      mem5tb: "memory = 5000000000000.B"
+      mem10tb: "memory = 10000000000000.B"
+      mem20tb: "memory = 20000000000000.B"
+      mem50tb: "memory = 50000000000000.B"
+      mem100tb: "memory = 100000000000000.B"
+      mem200tb: "memory = 200000000000000.B"
+      mem500tb: "memory = 500000000000000.B"
+      mem1gib: "memory = 1073741824.B"
+      mem2gib: "memory = 2147483648.B"
+      mem4gib: "memory = 4294967296.B"
+      mem8gib: "memory = 8589934592.B"
+      mem16gib: "memory = 17179869184.B"
+      mem32gib: "memory = 34359738368.B"
+      mem64gib: "memory = 68719476736.B"
+      mem128gib: "memory = 137438953472.B"
+      mem256gib: "memory = 274877906944.B"
+      mem512gib: "memory = 549755813888.B"
+      mem1tib: "memory = 1099511627776.B"
+      mem2tib: "memory = 2199023255552.B"
+      mem4tib: "memory = 4398046511104.B"
+      mem8tib: "memory = 8796093022208.B"
+      mem16tib: "memory = 17592186044416.B"
+      mem32tib: "memory = 35184372088832.B"
+      mem64tib: "memory = 70368744177664.B"
+      mem128tib: "memory = 140737488355328.B"
+      mem256tib: "memory = 281474976710656.B"
+      mem512tib: "memory = 562949953421312.B"
+      cpu1: "cpus = 1"
+      cpu2: "cpus = 2"
+      cpu5: "cpus = 5"
+      cpu10: "cpus = 10"
+      cpu20: "cpus = 20"
+      cpu50: "cpus = 50"
+      cpu100: "cpus = 100"
+      cpu200: "cpus = 200"
+      cpu500: "cpus = 500"
+      cpu1000: "cpus = 1000"
+    script:
+    - "includeConfig(\"nextflow_labels.config\")"
+  debug: false
+  container: "docker"
+engines:
+- type: "docker"
+  id: "docker"
+  image: "python:3.12-slim"
+  target_registry: "images.viash-hub.com"
+  target_tag: "niche-compass"
+  namespace_separator: "/"
+  setup:
+  - type: "apt"
+    packages:
+    - "procps"
+    interactive: false
+  - type: "python"
+    user: false
+    packages:
+    - "spatialdata~=0.5.0"
+    - "pyarrow~=18.0.0"
+    - "squidpy~=1.6.5"
+    upgrade: true
+  test_setup:
+  - type: "apt"
+    packages:
+    - "git"
+    interactive: false
+  - type: "python"
+    user: false
+    packages:
+    - "viashpy==0.9.0"
+    github:
+    - "openpipelines-bio/core#subdirectory=packages/python/openpipeline_testutils"
+    upgrade: true
+  entrypoint: []
+  cmd: null
+- type: "native"
+  id: "native"
+build_info:
+  config: "src/filter/subset_cosmx/config.vsh.yaml"
+  runner: "nextflow"
+  engine: "docker|native"
+  output: "target/_private/nextflow/filter/subset_cosmx"
+  executable: "target/_private/nextflow/filter/subset_cosmx/main.nf"
+  viash_version: "0.9.4"
+  git_commit: "0c1677bb93680d39ec2fb2f6bc68a2fcfae0e831"
+  git_remote: "https://github.com/openpipelines-bio/openpipeline_spatial"
+package_config:
+  name: "openpipeline_spatial"
+  version: "niche-compass"
+  info:
+    test_resources:
+    - type: "s3"
+      path: "s3://openpipelines-bio/openpipeline_spatial/resources_test"
+      dest: "resources_test"
+  repositories:
+  - type: "vsh"
+    name: "openpipeline"
+    repo: "openpipeline"
+    tag: "v3.0.0"
+  viash_version: "0.9.4"
+  source: "src"
+  target: "target"
+  config_mods:
+  - ".resources += {path: '/src/workflows/utils/labels.config', dest: 'nextflow_labels.config'}\n\
+    .runners[.type == 'nextflow'].config.script := 'includeConfig(\"nextflow_labels.config\"\
+    )'"
+  - ".engines += { type: \"native\" }"
+  - ".engines[.type == 'docker'].target_registry := 'images.viash-hub.com'"
+  - ".engines[.type == 'docker'].target_tag := 'niche-compass'"
+  organization: "vsh"
+  links:
+    repository: "https://github.com/openpipelines-bio/openpipeline_spatial"
+    docker_registry: "ghcr.io"

target/_private/nextflow/filter/subset_cosmx/nextflow.config

@@ -0,0 +1,126 @@
+manifest {
+  name = 'filter/subset_cosmx'
+  mainScript = 'main.nf'
+  nextflowVersion = '!>=20.12.1-edge'
+  version = 'niche-compass'
+  description = 'Filters the output from NanoString experiment to keep only a subset of the fields of view.\nExpected input folder structure:\npath/to/dataset/\n    ├── CellComposite/\n    ├── CellLabels/\n    ├── CellOverlay/\n    ├── CompartmentLabels/\n    ├── <dataset_id>_exprMat_file.csv\n    ├── <dataset_id>_fov_positions_file.csv\n    ├── <dataset_id>_metadata_file.csv\n    └── <dataset_id>_tx_file.csv  \n'
+  author = 'Dorien Roosen, Weiwei Schultz'
+}
+
+process.container = 'nextflow/bash:latest'
+
+// detect tempdir
+tempDir = java.nio.file.Paths.get(
+  System.getenv('NXF_TEMP') ?:
+    System.getenv('VIASH_TEMP') ?: 
+    System.getenv('TEMPDIR') ?: 
+    System.getenv('TMPDIR') ?: 
+    '/tmp'
+).toAbsolutePath()
+
+profiles {
+  no_publish {
+    process {
+      withName: '.*' {
+        publishDir = [
+          enabled: false
+        ]
+      }
+    }
+  }
+  mount_temp {
+    docker.temp            = tempDir
+    podman.temp            = tempDir
+    charliecloud.temp      = tempDir
+  }
+  docker {
+    docker.enabled         = true
+    // docker.userEmulation   = true
+    singularity.enabled    = false
+    podman.enabled         = false
+    shifter.enabled        = false
+    charliecloud.enabled   = false
+  }
+  singularity {
+    singularity.enabled    = true
+    singularity.autoMounts = true
+    docker.enabled         = false
+    podman.enabled         = false
+    shifter.enabled        = false
+    charliecloud.enabled   = false
+  }
+  podman {
+    podman.enabled         = true
+    docker.enabled         = false
+    singularity.enabled    = false
+    shifter.enabled        = false
+    charliecloud.enabled   = false
+  }
+  shifter {
+    shifter.enabled        = true
+    docker.enabled         = false
+    singularity.enabled    = false
+    podman.enabled         = false
+    charliecloud.enabled   = false
+  }
+  charliecloud {
+    charliecloud.enabled   = true
+    docker.enabled         = false
+    singularity.enabled    = false
+    podman.enabled         = false
+    shifter.enabled        = false
+  }
+}
+
+process{
+  withLabel: mem1gb { memory = 1000000000.B }
+  withLabel: mem2gb { memory = 2000000000.B }
+  withLabel: mem5gb { memory = 5000000000.B }
+  withLabel: mem10gb { memory = 10000000000.B }
+  withLabel: mem20gb { memory = 20000000000.B }
+  withLabel: mem50gb { memory = 50000000000.B }
+  withLabel: mem100gb { memory = 100000000000.B }
+  withLabel: mem200gb { memory = 200000000000.B }
+  withLabel: mem500gb { memory = 500000000000.B }
+  withLabel: mem1tb { memory = 1000000000000.B }
+  withLabel: mem2tb { memory = 2000000000000.B }
+  withLabel: mem5tb { memory = 5000000000000.B }
+  withLabel: mem10tb { memory = 10000000000000.B }
+  withLabel: mem20tb { memory = 20000000000000.B }
+  withLabel: mem50tb { memory = 50000000000000.B }
+  withLabel: mem100tb { memory = 100000000000000.B }
+  withLabel: mem200tb { memory = 200000000000000.B }
+  withLabel: mem500tb { memory = 500000000000000.B }
+  withLabel: mem1gib { memory = 1073741824.B }
+  withLabel: mem2gib { memory = 2147483648.B }
+  withLabel: mem4gib { memory = 4294967296.B }
+  withLabel: mem8gib { memory = 8589934592.B }
+  withLabel: mem16gib { memory = 17179869184.B }
+  withLabel: mem32gib { memory = 34359738368.B }
+  withLabel: mem64gib { memory = 68719476736.B }
+  withLabel: mem128gib { memory = 137438953472.B }
+  withLabel: mem256gib { memory = 274877906944.B }
+  withLabel: mem512gib { memory = 549755813888.B }
+  withLabel: mem1tib { memory = 1099511627776.B }
+  withLabel: mem2tib { memory = 2199023255552.B }
+  withLabel: mem4tib { memory = 4398046511104.B }
+  withLabel: mem8tib { memory = 8796093022208.B }
+  withLabel: mem16tib { memory = 17592186044416.B }
+  withLabel: mem32tib { memory = 35184372088832.B }
+  withLabel: mem64tib { memory = 70368744177664.B }
+  withLabel: mem128tib { memory = 140737488355328.B }
+  withLabel: mem256tib { memory = 281474976710656.B }
+  withLabel: mem512tib { memory = 562949953421312.B }
+  withLabel: cpu1 { cpus = 1 }
+  withLabel: cpu2 { cpus = 2 }
+  withLabel: cpu5 { cpus = 5 }
+  withLabel: cpu10 { cpus = 10 }
+  withLabel: cpu20 { cpus = 20 }
+  withLabel: cpu50 { cpus = 50 }
+  withLabel: cpu100 { cpus = 100 }
+  withLabel: cpu200 { cpus = 200 }
+  withLabel: cpu500 { cpus = 500 }
+  withLabel: cpu1000 { cpus = 1000 }
+}
+
+includeConfig("nextflow_labels.config")

target/_private/nextflow/filter/subset_cosmx/nextflow_labels.config

@@ -0,0 +1,68 @@
+process {
+  // Default resources for components that hardly do any processing
+  memory = { 2.GB * task.attempt }
+  cpus = 1
+
+  // Retry for exit codes that have something to do with memory issues
+  errorStrategy = { task.exitStatus in 137..140 ? 'retry' : 'terminate' }
+  maxRetries = 3
+  maxMemory = null
+
+  // CPU resources
+  withLabel: singlecpu { cpus = 1 }
+  withLabel: lowcpu { cpus = 4 }
+  withLabel: midcpu { cpus = 10 }
+  withLabel: highcpu { cpus = 20 }
+  
+  // Memory resources
+  withLabel: lowmem { memory = { get_memory( 50.GB * task.attempt ) } }
+  withLabel: midmem { memory = { get_memory( 50.GB * task.attempt ) } }
+  withLabel: highmem { memory = { get_memory( 50.GB * task.attempt ) } }
+  withLabel: veryhighmem { memory = { get_memory( 75.GB * task.attempt ) } }
+
+  // Disk space
+  // Nextflow apparently can't handle empty directives, i.e.
+  // withLabel: lowdisk {}
+  // so for that reason we have to add a dummy directive
+  withLabel: lowdisk {
+    dummyDirective = "dummyValue"
+  }
+  withLabel: middisk {
+    dummyDirective = "dummyValue"
+  }
+  withLabel: highdisk {
+    dummyDirective = "dummyValue"
+  }
+  withLabel: veryhighdisk {
+    dummyDirective = "dummyValue"
+  }
+  // NOTE: The above labels intentionally do not have an effect by default.
+  // The user should set the disk space requirements by adding the following
+  // to the compute environment:
+  //
+  // withLabel: lowdisk { disk = { 20.GB * task.attempt } }
+  // withLabel: middisk { disk = { 100.GB * task.attempt } }
+  // withLabel: highdisk { disk = { 200.GB * task.attempt } }
+  // withLabel: veryhighdisk { disk = { 500.GB * task.attempt } }
+}
+
+def get_memory(to_compare) {
+  if (!process.containsKey("maxMemory") || !process.maxMemory) {
+    return to_compare
+  }
+
+  try {
+    if (process.containsKey("maxRetries") && process.maxRetries && task.attempt == (process.maxRetries as int)) {
+      return process.maxMemory
+    }
+    else if (to_compare.compareTo(process.maxMemory as nextflow.util.MemoryUnit) == 1) {
+      return max_memory as nextflow.util.MemoryUnit
+    }
+    else {
+      return to_compare
+    }  
+  } catch (all) {
+        println "Error processing memory resources. Please check that process.maxMemory '${process.maxMemory}' and process.maxRetries '${process.maxRetries}' are valid!"
+        System.exit(1)
+  }
+}

target/_private/nextflow/filter/subset_cosmx/setup_logger.py

@@ -0,0 +1,12 @@
+def setup_logger():
+    import logging
+    from sys import stdout
+
+    logger = logging.getLogger()
+    logger.setLevel(logging.INFO)
+    console_handler = logging.StreamHandler(stdout)
+    logFormatter = logging.Formatter("%(asctime)s %(levelname)-8s %(message)s")
+    console_handler.setFormatter(logFormatter)
+    logger.addHandler(console_handler)
+
+    return logger

target/dependencies/vsh/vsh/openpipeline/v3.0.0/_private/nextflow/workflows/multiomics/split_modalities/.config.vsh.yaml

@@ -0,0 +1,234 @@
+name: "split_modalities"
+namespace: "workflows/multiomics"
+version: "v3.0.0"
+authors:
+- name: "Dries Schaumont"
+  roles:
+  - "author"
+  - "maintainer"
+  info:
+    role: "Core Team Member"
+    links:
+      email: "dries@data-intuitive.com"
+      github: "DriesSchaumont"
+      orcid: "0000-0002-4389-0440"
+      linkedin: "dries-schaumont"
+    organizations:
+    - name: "Data Intuitive"
+      href: "https://www.data-intuitive.com"
+      role: "Data Scientist"
+argument_groups:
+- name: "Inputs"
+  arguments:
+  - type: "string"
+    name: "--id"
+    description: "ID of the sample."
+    info: null
+    example:
+    - "foo"
+    required: true
+    direction: "input"
+    multiple: false
+    multiple_sep: ";"
+  - type: "file"
+    name: "--input"
+    alternatives:
+    - "-i"
+    description: "Path to the sample."
+    info: null
+    example:
+    - "input.h5mu"
+    must_exist: true
+    create_parent: true
+    required: true
+    direction: "input"
+    multiple: false
+    multiple_sep: ";"
+- name: "Outputs"
+  arguments:
+  - type: "file"
+    name: "--output"
+    alternatives:
+    - "-o"
+    description: "Output directory containing multiple h5mu files."
+    info: null
+    example:
+    - "/path/to/output"
+    must_exist: true
+    create_parent: true
+    required: true
+    direction: "output"
+    multiple: false
+    multiple_sep: ";"
+  - type: "file"
+    name: "--output_types"
+    description: "A csv containing the base filename and modality type per output\
+      \ file."
+    info: null
+    example:
+    - "types.csv"
+    must_exist: true
+    create_parent: true
+    required: true
+    direction: "output"
+    multiple: false
+    multiple_sep: ";"
+resources:
+- type: "nextflow_script"
+  path: "main.nf"
+  is_executable: true
+  entrypoint: "run_wf"
+- type: "file"
+  path: "utils"
+- type: "file"
+  path: "nextflow_labels.config"
+  dest: "nextflow_labels.config"
+description: "A pipeline to split a multimodal mudata files into several unimodal\
+  \ mudata files."
+test_resources:
+- type: "nextflow_script"
+  path: "test.nf"
+  is_executable: true
+  entrypoint: "test_wf"
+- type: "file"
+  path: "pbmc_1k_protein_v3_filtered_feature_bc_matrix.h5mu"
+info:
+  test_dependencies:
+  - name: "split_modalities_test"
+    namespace: "test_workflows/multiomics"
+status: "enabled"
+scope:
+  image: "private"
+  target: "private"
+dependencies:
+- name: "dataflow/split_modalities"
+  alias: "split_modalities_component"
+  repository:
+    type: "local"
+license: "MIT"
+links:
+  repository: "https://github.com/openpipelines-bio/openpipeline"
+  docker_registry: "ghcr.io"
+runners:
+- type: "nextflow"
+  id: "nextflow"
+  directives:
+    tag: "$id"
+  auto:
+    simplifyInput: true
+    simplifyOutput: false
+    transcript: false
+    publish: false
+  config:
+    labels:
+      mem1gb: "memory = 1000000000.B"
+      mem2gb: "memory = 2000000000.B"
+      mem5gb: "memory = 5000000000.B"
+      mem10gb: "memory = 10000000000.B"
+      mem20gb: "memory = 20000000000.B"
+      mem50gb: "memory = 50000000000.B"
+      mem100gb: "memory = 100000000000.B"
+      mem200gb: "memory = 200000000000.B"
+      mem500gb: "memory = 500000000000.B"
+      mem1tb: "memory = 1000000000000.B"
+      mem2tb: "memory = 2000000000000.B"
+      mem5tb: "memory = 5000000000000.B"
+      mem10tb: "memory = 10000000000000.B"
+      mem20tb: "memory = 20000000000000.B"
+      mem50tb: "memory = 50000000000000.B"
+      mem100tb: "memory = 100000000000000.B"
+      mem200tb: "memory = 200000000000000.B"
+      mem500tb: "memory = 500000000000000.B"
+      mem1gib: "memory = 1073741824.B"
+      mem2gib: "memory = 2147483648.B"
+      mem4gib: "memory = 4294967296.B"
+      mem8gib: "memory = 8589934592.B"
+      mem16gib: "memory = 17179869184.B"
+      mem32gib: "memory = 34359738368.B"
+      mem64gib: "memory = 68719476736.B"
+      mem128gib: "memory = 137438953472.B"
+      mem256gib: "memory = 274877906944.B"
+      mem512gib: "memory = 549755813888.B"
+      mem1tib: "memory = 1099511627776.B"
+      mem2tib: "memory = 2199023255552.B"
+      mem4tib: "memory = 4398046511104.B"
+      mem8tib: "memory = 8796093022208.B"
+      mem16tib: "memory = 17592186044416.B"
+      mem32tib: "memory = 35184372088832.B"
+      mem64tib: "memory = 70368744177664.B"
+      mem128tib: "memory = 140737488355328.B"
+      mem256tib: "memory = 281474976710656.B"
+      mem512tib: "memory = 562949953421312.B"
+      cpu1: "cpus = 1"
+      cpu2: "cpus = 2"
+      cpu5: "cpus = 5"
+      cpu10: "cpus = 10"
+      cpu20: "cpus = 20"
+      cpu50: "cpus = 50"
+      cpu100: "cpus = 100"
+      cpu200: "cpus = 200"
+      cpu500: "cpus = 500"
+      cpu1000: "cpus = 1000"
+    script:
+    - "includeConfig(\"nextflow_labels.config\")"
+  debug: false
+  container: "docker"
+engines:
+- type: "native"
+  id: "native"
+build_info:
+  config: "src/workflows/multiomics/split_modalities/config.vsh.yaml"
+  runner: "nextflow"
+  engine: "native"
+  output: "target/_private/nextflow/workflows/multiomics/split_modalities"
+  executable: "target/_private/nextflow/workflows/multiomics/split_modalities/main.nf"
+  viash_version: "0.9.4"
+  git_commit: "e92e56b49125af8ef2ebb11586191a6cbf9a8457"
+  git_remote: "https://github.com/openpipelines-bio/openpipeline"
+  git_tag: "0.2.0-2059-ge92e56b4"
+  dependencies:
+  - "target/nextflow/dataflow/split_modalities"
+package_config:
+  name: "openpipeline"
+  version: "v3.0.0"
+  summary: "Best-practice workflows for single-cell multi-omics analyses.\n"
+  description: "OpenPipelines are extensible single cell analysis pipelines for reproducible\
+    \ and large-scale single cell processing using [Viash](https://viash.io) and [Nextflow](https://www.nextflow.io/).\n\
+    \nIn terms of workflows, the following has been made available, but keep in mind\
+    \ that\nindividual tools and functionality can be executed as standalone components\
+    \ as well.\n\n  * Demultiplexing: conversion of raw sequencing data to FASTQ objects.\n\
+    \  * Ingestion: Read mapping and generating a count matrix.\n  * Single sample\
+    \ processing: cell filtering and doublet detection.\n  * Multisample processing:\
+    \ Count transformation, normalization, QC metric calulations.\n  * Integration:\
+    \ Clustering, integration and batch correction using single and multimodal methods.\n\
+    \  * Downstream analysis workflows\n"
+  info:
+    test_resources:
+    - type: "s3"
+      path: "s3://openpipelines-data"
+      dest: "resources_test"
+    nextflow_labels_ci:
+    - path: "src/workflows/utils/labels_ci.config"
+      description: "Adds the correct memory and CPU labels when running on the Viash\
+        \ Hub CI."
+  viash_version: "0.9.4"
+  source: "src"
+  target: "target"
+  config_mods:
+  - ".resources += {path: '/src/workflows/utils/labels.config', dest: 'nextflow_labels.config'}\n\
+    .runners[.type == 'nextflow'].config.script := 'includeConfig(\"nextflow_labels.config\"\
+    )'"
+  - ".engines += { type: \"native\" }"
+  - ".engines[.type == 'docker'].target_registry := 'images.viash-hub.com'"
+  - ".engines[.type == 'docker'].target_tag := 'v3.0.0'"
+  keywords:
+  - "single-cell"
+  - "multimodal"
+  license: "MIT"
+  organization: "vsh"
+  links:
+    repository: "https://github.com/openpipelines-bio/openpipeline"
+    docker_registry: "ghcr.io"
+    homepage: "https://openpipelines.bio"
+    documentation: "https://openpipelines.bio/fundamentals"
+    issue_tracker: "https://github.com/openpipelines-bio/openpipeline/issues"

target/dependencies/vsh/vsh/openpipeline/v3.0.0/_private/nextflow/workflows/multiomics/split_modalities/nextflow.config

@@ -0,0 +1,126 @@
+manifest {
+  name = 'workflows/multiomics/split_modalities'
+  mainScript = 'main.nf'
+  nextflowVersion = '!>=20.12.1-edge'
+  version = 'v3.0.0'
+  description = 'A pipeline to split a multimodal mudata files into several unimodal mudata files.'
+  author = 'Dries Schaumont'
+}
+
+process.container = 'nextflow/bash:latest'
+
+// detect tempdir
+tempDir = java.nio.file.Paths.get(
+  System.getenv('NXF_TEMP') ?:
+    System.getenv('VIASH_TEMP') ?: 
+    System.getenv('TEMPDIR') ?: 
+    System.getenv('TMPDIR') ?: 
+    '/tmp'
+).toAbsolutePath()
+
+profiles {
+  no_publish {
+    process {
+      withName: '.*' {
+        publishDir = [
+          enabled: false
+        ]
+      }
+    }
+  }
+  mount_temp {
+    docker.temp            = tempDir
+    podman.temp            = tempDir
+    charliecloud.temp      = tempDir
+  }
+  docker {
+    docker.enabled         = true
+    // docker.userEmulation   = true
+    singularity.enabled    = false
+    podman.enabled         = false
+    shifter.enabled        = false
+    charliecloud.enabled   = false
+  }
+  singularity {
+    singularity.enabled    = true
+    singularity.autoMounts = true
+    docker.enabled         = false
+    podman.enabled         = false
+    shifter.enabled        = false
+    charliecloud.enabled   = false
+  }
+  podman {
+    podman.enabled         = true
+    docker.enabled         = false
+    singularity.enabled    = false
+    shifter.enabled        = false
+    charliecloud.enabled   = false
+  }
+  shifter {
+    shifter.enabled        = true
+    docker.enabled         = false
+    singularity.enabled    = false
+    podman.enabled         = false
+    charliecloud.enabled   = false
+  }
+  charliecloud {
+    charliecloud.enabled   = true
+    docker.enabled         = false
+    singularity.enabled    = false
+    podman.enabled         = false
+    shifter.enabled        = false
+  }
+}
+
+process{
+  withLabel: mem1gb { memory = 1000000000.B }
+  withLabel: mem2gb { memory = 2000000000.B }
+  withLabel: mem5gb { memory = 5000000000.B }
+  withLabel: mem10gb { memory = 10000000000.B }
+  withLabel: mem20gb { memory = 20000000000.B }
+  withLabel: mem50gb { memory = 50000000000.B }
+  withLabel: mem100gb { memory = 100000000000.B }
+  withLabel: mem200gb { memory = 200000000000.B }
+  withLabel: mem500gb { memory = 500000000000.B }
+  withLabel: mem1tb { memory = 1000000000000.B }
+  withLabel: mem2tb { memory = 2000000000000.B }
+  withLabel: mem5tb { memory = 5000000000000.B }
+  withLabel: mem10tb { memory = 10000000000000.B }
+  withLabel: mem20tb { memory = 20000000000000.B }
+  withLabel: mem50tb { memory = 50000000000000.B }
+  withLabel: mem100tb { memory = 100000000000000.B }
+  withLabel: mem200tb { memory = 200000000000000.B }
+  withLabel: mem500tb { memory = 500000000000000.B }
+  withLabel: mem1gib { memory = 1073741824.B }
+  withLabel: mem2gib { memory = 2147483648.B }
+  withLabel: mem4gib { memory = 4294967296.B }
+  withLabel: mem8gib { memory = 8589934592.B }
+  withLabel: mem16gib { memory = 17179869184.B }
+  withLabel: mem32gib { memory = 34359738368.B }
+  withLabel: mem64gib { memory = 68719476736.B }
+  withLabel: mem128gib { memory = 137438953472.B }
+  withLabel: mem256gib { memory = 274877906944.B }
+  withLabel: mem512gib { memory = 549755813888.B }
+  withLabel: mem1tib { memory = 1099511627776.B }
+  withLabel: mem2tib { memory = 2199023255552.B }
+  withLabel: mem4tib { memory = 4398046511104.B }
+  withLabel: mem8tib { memory = 8796093022208.B }
+  withLabel: mem16tib { memory = 17592186044416.B }
+  withLabel: mem32tib { memory = 35184372088832.B }
+  withLabel: mem64tib { memory = 70368744177664.B }
+  withLabel: mem128tib { memory = 140737488355328.B }
+  withLabel: mem256tib { memory = 281474976710656.B }
+  withLabel: mem512tib { memory = 562949953421312.B }
+  withLabel: cpu1 { cpus = 1 }
+  withLabel: cpu2 { cpus = 2 }
+  withLabel: cpu5 { cpus = 5 }
+  withLabel: cpu10 { cpus = 10 }
+  withLabel: cpu20 { cpus = 20 }
+  withLabel: cpu50 { cpus = 50 }
+  withLabel: cpu100 { cpus = 100 }
+  withLabel: cpu200 { cpus = 200 }
+  withLabel: cpu500 { cpus = 500 }
+  withLabel: cpu1000 { cpus = 1000 }
+}
+
+includeConfig("nextflow_labels.config")

target/dependencies/vsh/vsh/openpipeline/v3.0.0/_private/nextflow/workflows/multiomics/split_modalities/nextflow_labels.config

@@ -0,0 +1,48 @@
+process {
+  // Default resources for components that hardly do any processing
+  memory = { 2.GB * task.attempt }
+  cpus = 1
+
+  // Retry for exit codes that have something to do with memory issues
+  errorStrategy = { task.exitStatus in 137..140 ? 'retry' : 'terminate' }
+  maxRetries = 3
+
+  // The memory a task is assinged increases with each attempt
+  // uncomment the line below and adjust the value to set a global upper limit on the memory.
+  // resourceLimits = [ memory: 240.Gb ] 
+
+  // CPU resources
+  withLabel: singlecpu { cpus = 1 }
+  withLabel: lowcpu { cpus = 4 }
+  withLabel: midcpu { cpus = 10 }
+  withLabel: highcpu { cpus = 20 }
+  
+  // Memory resources
+  withLabel: lowmem { memory = { task?.resourceLimits?.memory && task?.maxRetries && task.attempt >= task.maxRetries ? task.resourceLimits.memory : 4.GB * task.attempt } }
+  withLabel: midmem { memory = { task?.resourceLimits?.memory && task?.maxRetries && task.attempt >= task.maxRetries ? task.resourceLimits.memory : 25.GB * task.attempt } }
+  withLabel: highmem { memory = { task?.resourceLimits?.memory && task?.maxRetries && task.attempt >= task.maxRetries ? task.resourceLimits.memory : 50.GB * task.attempt } }
+  withLabel: veryhighmem { memory = { task?.resourceLimits?.memory && task?.maxRetries && task.attempt >= task.maxRetries ? task.resourceLimits.memory : 75.GB * task.attempt } }
+
+  // Disk space
+  withLabel: lowdisk {
+    disk = {process.disk ? process.disk : null}
+  }
+  withLabel: middisk {
+    disk = {process.disk ? process.disk : null}
+  }
+  withLabel: highdisk {
+    disk = {process.disk ? process.disk : null}
+  }
+  withLabel: veryhighdisk {
+    disk = {process.disk ? process.disk : null}
+  }
+  
+  // NOTE: The above labels intentionally do not have an effect by default.
+  // The user should set the disk space requirements by adding the following
+  // to the compute environment:
+  //
+  // withLabel: lowdisk { disk = { 20.GB * task.attempt } }
+  // withLabel: middisk { disk = { 100.GB * task.attempt } }
+  // withLabel: highdisk { disk = { 200.GB * task.attempt } }
+  // withLabel: veryhighdisk { disk = { 500.GB * task.attempt } }
+}

target/dependencies/vsh/vsh/openpipeline/v3.0.0/_private/nextflow/workflows/multiomics/split_modalities/utils/errorstrat_ignore.config

@@ -0,0 +1 @@
+process.errorStrategy = 'ignore'