Compare commits
3 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 0cbd5c00cb | |||
| 78f3a8b65e | |||
| 4d7688483b |
26
CHANGELOG.md
26
CHANGELOG.md
@@ -1,3 +1,29 @@
|
||||
# demultiplex v0.3.6
|
||||
|
||||
## Minor updates
|
||||
|
||||
* Allow letter case variants for headers when looking for sample information in run information CSV (PR #38).
|
||||
|
||||
# demultiplex v0.3.5
|
||||
|
||||
## Breaking changes
|
||||
|
||||
* The `demultiplex` workflow now outputs a list of directories
|
||||
for the `output_falco` argument (one for each barcode) instead of one directory
|
||||
for the complete run. The output from the `runner` workflow remained
|
||||
unchanged (PR #33).
|
||||
|
||||
## Minor updates
|
||||
|
||||
* In case Illumina data is detected in the input folder, check for the presence of the 'copyComplete.txt' file.
|
||||
This check can be disabled using `--skip_copycomplete_check` (PR #34).
|
||||
|
||||
# demultiplex v0.3.4
|
||||
|
||||
## Minor updates
|
||||
|
||||
* Resource labels are now automatically included during build (PR #32).
|
||||
|
||||
# demultiplex v0.3.3
|
||||
|
||||
## Breaking change
|
||||
|
||||
21
LICENSE
Normal file
21
LICENSE
Normal file
@@ -0,0 +1,21 @@
|
||||
MIT License
|
||||
|
||||
Copyright (c) 2024 Data Intuitive
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
100
README.md
Normal file
100
README.md
Normal file
@@ -0,0 +1,100 @@
|
||||
# Demultiplex.vsh
|
||||
|
||||
Demultiplex.vsh is a workflow for demultiplexing of raw sequencing data. Currently data from Illumina and Element Biosciences sequencers are supported.
|
||||
|
||||
[](https://web.viash-hub.com/packages/demultiplex)
|
||||
[](https://github.com/viash-hub/demultiplex)
|
||||
[](https://github.com/viash-hub/demultiplex/blob/main/LICENSE)
|
||||
[](https://github.com/viash-hub/demultiplex/issues)
|
||||
[](https://viash.io)
|
||||
|
||||
## Workflow Overview
|
||||
The workflow executes the following steps:
|
||||
1. Unpacking the input data (when a TAR archive is provided)
|
||||
2. Run `bclconvert` or `bases2fastq`
|
||||
3. Run `falco` and convert Illumina InterOp information to csv
|
||||
4. Run `multiqc` to generate a report
|
||||
|
||||
## Usage
|
||||
|
||||
Two variants of the same workflow are provided, depending on the flexibility in the ouput structure required:
|
||||
|
||||
* The `runner` workflow provides a predifined output structure. It requires the minimal amount of parameters to be provided, at the cost of being less flexible. It is located at `target/nextflow/runner/main.nf`
|
||||
* The `demultiplex` workflow (`target/nextflow/demultiplex/main.nf`) allows for more fine-grained tuning, but required more parameters to be provided.
|
||||
|
||||
### Test data
|
||||
|
||||
We have provided test data at `gs://viash-hub-test-data/demultiplex/v3/demultiplex_htrnaseq_meta/SingleCell-RNA_P3_2`, but please feel free to bring your own. The URL of the test data can be provided as-is to the workflow, or you can download everything and specify a local path.
|
||||
|
||||
### Setup
|
||||
|
||||
In order to use the workflows in this package, you'll need to do the following:
|
||||
* Install [nextflow](https://www.nextflow.io/docs/latest/install.html)
|
||||
* Install a nextflow compatible executor. This workflow provides a profile for [docker](https://docs.docker.com/get-started/).
|
||||
|
||||
### Setting up SCM
|
||||
|
||||
In order to let nextflow use the viash-hub workflows, you need to setup a [SCM](https://www.nextflow.io/docs/latest/git.html#git-configuration) file. This can be done once by creating `$HOME/.nextflow/scm` and adding the following:
|
||||
```
|
||||
providers {
|
||||
vsh {
|
||||
platform = 'gitlab'
|
||||
server = "packages.viash-hub.com"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Alternatively, a custom location for the SCM file can be specified using the `NXF_SCM_FILE` environment variable.
|
||||
|
||||
You can check if everything is working by getting the `--help` for a workflow:
|
||||
```bash
|
||||
nextflow run \
|
||||
vsh/demultiplex \
|
||||
-r v0.3.4 \
|
||||
--help
|
||||
```
|
||||
|
||||
### (Optional) Resource usage tuning
|
||||
|
||||
Nextflow's labels can be used to specify the amount of resources a process can use. This workflow uses the following labels for CPU and memory:
|
||||
* `verylowmem`, `lowmem`, `midmem`, `highmem`
|
||||
* `verylowcpu`, `lowcpu`, `midcpu`, `highcpu`
|
||||
|
||||
The defaults for these labels can be found at `src/config/labels.config`. Nextflow checks that the specified resources for a process do not exceed what is available on the machine and will not start if it does. Create your own config file to tune the labels to your needs, for example:
|
||||
|
||||
```
|
||||
// Resource labels
|
||||
withLabel: verylowcpu { cpus = 2 }
|
||||
withLabel: lowcpu { cpus = 8 }
|
||||
withLabel: midcpu { cpus = 16 }
|
||||
withLabel: highcpu { cpus = 16 }
|
||||
|
||||
withLabel: verylowmem { memory = 4.GB }
|
||||
withLabel: lowmem { memory = 8.GB }
|
||||
withLabel: midmem { memory = 8.GB }
|
||||
withLabel: highmem { memory = 8.GB }
|
||||
```
|
||||
|
||||
When starting nextflow using the CLI, you can use `-c` to provide the file to nextflow and overwrite the defaults.
|
||||
|
||||
### Example
|
||||
|
||||
```bash
|
||||
nextflow run vsh/demultiplex \
|
||||
-r v0.3.4 \
|
||||
-main-script target/nextflow/runner/main.nf \
|
||||
--input "gs://viash-hub-test-data/demultiplex/v3/demultiplex_htrnaseq_meta/SingleCell-RNA_P3_2" \
|
||||
--demultiplexer bclconvert \
|
||||
--publish_dir example_output/ \
|
||||
-profile docker \
|
||||
-c labels.config
|
||||
```
|
||||
|
||||
## Acknowledgements
|
||||
|
||||
Developed in collaboration with Data Intuitive and Open Analytics.
|
||||
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
name: demultiplex
|
||||
version: v0.3.3
|
||||
version: v0.3.6
|
||||
description: |
|
||||
Demultiplexing pipeline
|
||||
license: MIT
|
||||
@@ -17,3 +17,5 @@ viash_version: 0.9.0
|
||||
config_mods: |
|
||||
.requirements.commands := ['ps']
|
||||
.runners[.type == 'nextflow'].directives.tag := '$id'
|
||||
.resources += {path: '/src/config/labels.config', dest: 'nextflow_labels.config'}
|
||||
.runners[.type == 'nextflow'].config.script := 'includeConfig("nextflow_labels.config")'
|
||||
|
||||
@@ -16,6 +16,9 @@ argument_groups:
|
||||
type: file
|
||||
required: false
|
||||
multiple: true
|
||||
- name: "--falco_dir"
|
||||
type: file
|
||||
required: true
|
||||
- name: Output arguments
|
||||
arguments:
|
||||
- name: --output_forward
|
||||
@@ -28,6 +31,11 @@ argument_groups:
|
||||
direction: output
|
||||
multiple: true
|
||||
required: false
|
||||
- name: "--output_falco"
|
||||
type: file
|
||||
direction: output
|
||||
required: true
|
||||
multiple: true
|
||||
resources:
|
||||
- type: nextflow_script
|
||||
path: main.nf
|
||||
|
||||
@@ -13,10 +13,12 @@ workflow run_wf {
|
||||
// Gather the following state for all samples
|
||||
def forward_fastqs = states.collect{it.forward_input}.flatten()
|
||||
def reverse_fastqs = states.collect{it.reverse_input}.findAll{it != null}.flatten()
|
||||
def falco_dirs = states.collect{it.falco_dir}
|
||||
|
||||
def resultState = [
|
||||
"output_forward": forward_fastqs,
|
||||
"output_reverse": reverse_fastqs,
|
||||
"output_falco": falco_dirs,
|
||||
// The join ID is the same across all samples from the same run
|
||||
"_meta": ["join_id": states[0]._meta.join_id]
|
||||
]
|
||||
|
||||
@@ -30,8 +30,8 @@ workflow run_wf {
|
||||
return true
|
||||
}
|
||||
// [Data], [BCLConvert_Data] for illumina
|
||||
// [Samples] for Element Biosciences
|
||||
if (header in ["Data", "Samples", "BCLConvert_Data"]) {
|
||||
// [Samples] or sometimes [SAMPLES] for Element Biosciences
|
||||
if (header.toLowerCase() in ["data", "samples", "bclconvert_data"]) {
|
||||
println "Found header [${header}], start parsing."
|
||||
start_parsing = true
|
||||
return
|
||||
|
||||
@@ -41,6 +41,7 @@ argument_groups:
|
||||
type: file
|
||||
direction: output
|
||||
required: false
|
||||
multiple: true
|
||||
default: "$id/qc/fastqc"
|
||||
- name: "--output_multiqc"
|
||||
description: Directory to write falco output to
|
||||
@@ -53,6 +54,14 @@ argument_groups:
|
||||
direction: "output"
|
||||
required: true
|
||||
default: "$id/run_information.csv"
|
||||
- name: "Other arguments"
|
||||
arguments:
|
||||
- name: --skip_copycomplete_check
|
||||
type: boolean_true
|
||||
description: |
|
||||
Disable the check for the presence of a "CopyComplete.txt" file in input
|
||||
directory in case of Illumina data.
|
||||
|
||||
resources:
|
||||
- type: nextflow_script
|
||||
path: main.nf
|
||||
|
||||
@@ -94,6 +94,10 @@ workflow run_wf {
|
||||
// step based on the run dir, not the InterOp dir.
|
||||
def interop_dir = state.input.resolve("InterOp")
|
||||
assert interop_dir.isDirectory(): "Expected InterOp directory to be present."
|
||||
|
||||
def copycomplete_file = state.input.resolve("CopyComplete.txt")
|
||||
assert (copycomplete_file.isFile() || state.skip_copycomplete_check):
|
||||
"'CopyComplete.txt' file was not found!"
|
||||
}
|
||||
|
||||
def resultState = state + newState
|
||||
@@ -169,28 +173,12 @@ workflow run_wf {
|
||||
)
|
||||
|
||||
output_ch = samples_ch
|
||||
|
||||
|
||||
| combine_samples.run(
|
||||
fromState: { id, state ->
|
||||
[
|
||||
"id": state.run_id,
|
||||
"forward_input": state.fastq_forward,
|
||||
"reverse_input": state.fastq_reverse,
|
||||
]
|
||||
},
|
||||
toState: [
|
||||
"forward_fastqs": "output_forward",
|
||||
"reverse_fastqs": "output_reverse",
|
||||
]
|
||||
)
|
||||
| falco.run(
|
||||
directives: [label: ["lowcpu", "lowmem"]],
|
||||
directives: [label: ["verylowcpu", "lowmem"]],
|
||||
fromState: {id, state ->
|
||||
reverse_fastqs_list = state.reverse_fastqs ? state.reverse_fastqs : []
|
||||
[
|
||||
"input": state.forward_fastqs + reverse_fastqs_list,
|
||||
"outdir": "${state.output_falco}",
|
||||
"input": [state.fastq_forward, state.fastq_reverse],
|
||||
"outdir": "$id/qc/falco",
|
||||
"summary_filename": null,
|
||||
"report_filename": null,
|
||||
"data_filename": null,
|
||||
@@ -200,11 +188,28 @@ workflow run_wf {
|
||||
state + [ "output_falco" : result.outdir ]
|
||||
}
|
||||
)
|
||||
|
||||
| combine_samples.run(
|
||||
fromState: { id, state ->
|
||||
[
|
||||
"id": state.run_id,
|
||||
"forward_input": state.fastq_forward,
|
||||
"reverse_input": state.fastq_reverse,
|
||||
"falco_dir": state.output_falco,
|
||||
]
|
||||
},
|
||||
toState: [
|
||||
"forward_fastqs": "output_forward",
|
||||
"reverse_fastqs": "output_reverse",
|
||||
"output_falco": "output_falco",
|
||||
]
|
||||
)
|
||||
|
||||
| multiqc.run(
|
||||
directives: [label: ["midcpu", "midmem"]],
|
||||
fromState: {id, state ->
|
||||
def new_state = [
|
||||
"input": [state.output_falco],
|
||||
"input": state.output_falco,
|
||||
"output_report": state.output_multiqc,
|
||||
"cl_config": 'sp: {fastqc/data: {fn: "*_fastqc_data.txt"}}'
|
||||
]
|
||||
|
||||
@@ -25,7 +25,9 @@ workflow test_illumina {
|
||||
}
|
||||
| map {id, state ->
|
||||
assert state.output.isDirectory(): "Expected bclconvert output to be a directory"
|
||||
assert state.output_falco.isDirectory(): "Expected falco output to be a directory"
|
||||
state.output_falco.each{
|
||||
assert it.isDirectory(): "Expected falco output to be a directory"
|
||||
}
|
||||
assert state.output_multiqc.isFile(): "Expected multiQC output to be a file"
|
||||
fastq_files = state.output.listFiles().collect{it.name}
|
||||
assert ["Undetermined_S0_L001_R1_001.fastq.gz", "Sample23_S3_L001_R1_001.fastq.gz",
|
||||
@@ -76,7 +78,7 @@ workflow test_bases2fastq {
|
||||
}
|
||||
| map {id, state ->
|
||||
assert state.output.isDirectory(): "Expected bases2fastq output to be a directory"
|
||||
assert state.output_falco.isDirectory(): "Expected falco output to be a directory"
|
||||
state.output_falco.each{assert it.isDirectory(): "Expected falco output to be a directory"}
|
||||
assert state.output_multiqc.isFile(): "Expected multiQC output to be a file"
|
||||
}
|
||||
}
|
||||
|
||||
@@ -18,7 +18,7 @@ argument_groups:
|
||||
direction: output
|
||||
required: true
|
||||
requirements:
|
||||
- commands: ["summary", "index-summary"]
|
||||
commands: ["summary", "index-summary"]
|
||||
resources:
|
||||
- type: bash_script
|
||||
path: script.sh
|
||||
@@ -38,4 +38,4 @@ engines:
|
||||
|
||||
runners:
|
||||
- type: executable
|
||||
- type: nextflow
|
||||
- type: nextflow
|
||||
|
||||
@@ -3,7 +3,6 @@
|
||||
set -eo pipefail
|
||||
|
||||
declare -A input_output_mapping=(["par_input"]="par_output"
|
||||
["par_input_falco"]="par_output_falco"
|
||||
["par_input_multiqc"]="par_output_multiqc"
|
||||
["par_input_run_information"]="par_output_run_information"
|
||||
)
|
||||
@@ -23,4 +22,12 @@ do
|
||||
|
||||
echo "Output files for $output_location:"
|
||||
ls "$output_location"
|
||||
done
|
||||
|
||||
echo "Grouping output from $par_input_falco into $par_output_falco"
|
||||
mkdir -p "$par_output_falco"
|
||||
IFS=";" read -ra falco_inputs <<< $par_input_falco
|
||||
for falco_dir in "${falco_inputs[@]}"; do
|
||||
echo "Copying contents of $falco_dir"
|
||||
find -H -D exec "$falco_dir" -type f -maxdepth 1 -exec cp -t "$par_output_falco" {} +
|
||||
done
|
||||
@@ -12,6 +12,7 @@ argument_groups:
|
||||
description: Directory to write falco output to
|
||||
type: file
|
||||
required: true
|
||||
multiple: true
|
||||
- name: "--input_multiqc"
|
||||
description: Location where to write the MultiQC report to.
|
||||
type: file
|
||||
|
||||
@@ -49,7 +49,13 @@ argument_groups:
|
||||
type: file
|
||||
direction: output
|
||||
default: "qc/multiqc_report.html"
|
||||
|
||||
- name: "Other arguments"
|
||||
arguments:
|
||||
- name: --skip_copycomplete_check
|
||||
type: boolean_true
|
||||
description: |
|
||||
Disable the check for the presence of a "CopyComplete.txt" file in input
|
||||
directory in case of Illumina data.
|
||||
resources:
|
||||
- type: nextflow_script
|
||||
path: main.nf
|
||||
|
||||
@@ -24,9 +24,10 @@ workflow run_wf {
|
||||
"input": state.input,
|
||||
"run_information": state.run_information,
|
||||
"demultiplexer": state.demultiplexer,
|
||||
"output": "fastq",
|
||||
"output_falco": "qc/fastqc",
|
||||
"output_multiqc": "qc/multiqc_report.html",
|
||||
"skip_copycomplete_check": state.skip_copycomplete_check,
|
||||
"output": "$id/fastq",
|
||||
"output_falco": "$id/qc/fastqc",
|
||||
"output_multiqc": "$id/qc/multiqc_report.html",
|
||||
]
|
||||
if (state.run_information) {
|
||||
state_to_pass += ["output_run_information": state.run_information.getName()]
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
name: "interop_summary_to_csv"
|
||||
namespace: "io"
|
||||
version: "v0.3.3"
|
||||
version: "v0.3.6"
|
||||
argument_groups:
|
||||
- name: "Input arguments"
|
||||
arguments:
|
||||
@@ -38,6 +38,9 @@ resources:
|
||||
- type: "bash_script"
|
||||
path: "script.sh"
|
||||
is_executable: true
|
||||
- type: "file"
|
||||
path: "nextflow_labels.config"
|
||||
dest: "nextflow_labels.config"
|
||||
info: null
|
||||
status: "enabled"
|
||||
requirements:
|
||||
@@ -109,6 +112,8 @@ runners:
|
||||
cpu200: "cpus = 200"
|
||||
cpu500: "cpus = 500"
|
||||
cpu1000: "cpus = 1000"
|
||||
script:
|
||||
- "includeConfig(\"nextflow_labels.config\")"
|
||||
debug: false
|
||||
container: "docker"
|
||||
engines:
|
||||
@@ -116,7 +121,7 @@ engines:
|
||||
id: "docker"
|
||||
image: "debian:stable-slim"
|
||||
target_registry: "images.viash-hub.com"
|
||||
target_tag: "v0.3.3"
|
||||
target_tag: "v0.3.6"
|
||||
namespace_separator: "/"
|
||||
setup:
|
||||
- type: "apt"
|
||||
@@ -141,12 +146,12 @@ build_info:
|
||||
output: "target/executable/io/interop_summary_to_csv"
|
||||
executable: "target/executable/io/interop_summary_to_csv/interop_summary_to_csv"
|
||||
viash_version: "0.9.0"
|
||||
git_commit: "e297e8ff45464065384476fe521ab2700ec3739c"
|
||||
git_remote: "https://x-access-token:ghs_44flLTQIYGBkc9xcwCZoLYe7mxXzgw1iszFO@github.com/viash-hub/demultiplex"
|
||||
git_tag: "v0.3.2-4-ge297e8f"
|
||||
git_commit: "3a985c83869bcd78ba6d325f1a817514b52a132a"
|
||||
git_remote: "https://github.com/viash-hub/demultiplex"
|
||||
git_tag: "v0.3.5-3-g3a985c8"
|
||||
package_config:
|
||||
name: "demultiplex"
|
||||
version: "v0.3.3"
|
||||
version: "v0.3.6"
|
||||
description: "Demultiplexing pipeline\n"
|
||||
info:
|
||||
test_resources:
|
||||
@@ -157,10 +162,12 @@ package_config:
|
||||
target: "target"
|
||||
config_mods:
|
||||
- ".requirements.commands := ['ps']\n.runners[.type == 'nextflow'].directives.tag\
|
||||
\ := '$id'\n"
|
||||
\ := '$id'\n.resources += {path: '/src/config/labels.config', dest: 'nextflow_labels.config'}\n\
|
||||
.runners[.type == 'nextflow'].config.script := 'includeConfig(\"nextflow_labels.config\"\
|
||||
)'\n"
|
||||
- ".engines += { type: \"native\" }"
|
||||
- ".engines[.type == 'docker'].target_registry := 'images.viash-hub.com'"
|
||||
- ".engines[.type == 'docker'].target_tag := 'v0.3.3'"
|
||||
- ".engines[.type == 'docker'].target_tag := 'v0.3.6'"
|
||||
keywords:
|
||||
- "bioinformatics"
|
||||
- "sequence"
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
#!/usr/bin/env bash
|
||||
|
||||
# interop_summary_to_csv v0.3.3
|
||||
# interop_summary_to_csv v0.3.6
|
||||
#
|
||||
# This wrapper script is auto-generated by viash 0.9.0 and is thus a derivative
|
||||
# work thereof. This software comes with ABSOLUTELY NO WARRANTY from Data
|
||||
@@ -171,7 +171,7 @@ VIASH_META_TEMP_DIR="$VIASH_TEMP"
|
||||
|
||||
# ViashHelp: Display helpful explanation about this executable
|
||||
function ViashHelp {
|
||||
echo "interop_summary_to_csv v0.3.3"
|
||||
echo "interop_summary_to_csv v0.3.6"
|
||||
echo ""
|
||||
echo "Input arguments:"
|
||||
echo " --input"
|
||||
@@ -470,10 +470,10 @@ tar -C /tmp/ --no-same-owner --no-same-permissions -xvf /tmp/interop.tar.gz && \
|
||||
mv /tmp/interop-1.3.1-Linux-GNU/bin/index-summary /tmp/interop-1.3.1-Linux-GNU/bin/summary /usr/local/bin/
|
||||
|
||||
LABEL org.opencontainers.image.description="Companion container for running component io interop_summary_to_csv"
|
||||
LABEL org.opencontainers.image.created="2024-12-19T15:42:23Z"
|
||||
LABEL org.opencontainers.image.created="2025-03-20T19:41:31Z"
|
||||
LABEL org.opencontainers.image.source="https://github.com/viash-hub/demultiplex"
|
||||
LABEL org.opencontainers.image.revision="e297e8ff45464065384476fe521ab2700ec3739c"
|
||||
LABEL org.opencontainers.image.version="v0.3.3"
|
||||
LABEL org.opencontainers.image.revision="3a985c83869bcd78ba6d325f1a817514b52a132a"
|
||||
LABEL org.opencontainers.image.version="v0.3.6"
|
||||
|
||||
VIASHDOCKER
|
||||
fi
|
||||
@@ -609,7 +609,7 @@ while [[ $# -gt 0 ]]; do
|
||||
shift 1
|
||||
;;
|
||||
--version)
|
||||
echo "interop_summary_to_csv v0.3.3"
|
||||
echo "interop_summary_to_csv v0.3.6"
|
||||
exit
|
||||
;;
|
||||
--input)
|
||||
@@ -733,7 +733,7 @@ if [[ "$VIASH_ENGINE_TYPE" == "docker" ]]; then
|
||||
|
||||
# determine docker image id
|
||||
if [[ "$VIASH_ENGINE_ID" == 'docker' ]]; then
|
||||
VIASH_DOCKER_IMAGE_ID='images.viash-hub.com/vsh/demultiplex/io/interop_summary_to_csv:v0.3.3'
|
||||
VIASH_DOCKER_IMAGE_ID='images.viash-hub.com/vsh/demultiplex/io/interop_summary_to_csv:v0.3.6'
|
||||
fi
|
||||
|
||||
# print dockerfile
|
||||
|
||||
@@ -0,0 +1,98 @@
|
||||
process {
|
||||
container = 'nextflow/bash:latest'
|
||||
|
||||
// default resources
|
||||
memory = { 8.Gb * task.attempt }
|
||||
cpus = 8
|
||||
maxForks = 36
|
||||
|
||||
// Retry for exit codes that have something to do with memory issues
|
||||
errorStrategy = { task.exitStatus in 137..140 ? 'retry' : 'terminate' }
|
||||
maxRetries = 3
|
||||
maxMemory = 192.GB
|
||||
|
||||
// Resource labels
|
||||
withLabel: verylowcpu { cpus = 2 }
|
||||
withLabel: lowcpu { cpus = 8 }
|
||||
withLabel: midcpu { cpus = 16 }
|
||||
withLabel: highcpu { cpus = 32 }
|
||||
|
||||
withLabel: verylowmem { memory = { get_memory( 4.GB * task.attempt ) } }
|
||||
withLabel: lowmem { memory = { get_memory( 8.GB * task.attempt ) } }
|
||||
withLabel: midmem { memory = { get_memory( 16.GB * task.attempt ) } }
|
||||
withLabel: highmem { memory = { get_memory( 64.GB * task.attempt ) } }
|
||||
|
||||
}
|
||||
|
||||
profiles {
|
||||
// detect tempdir
|
||||
tempDir = java.nio.file.Paths.get(
|
||||
System.getenv('NXF_TEMP') ?:
|
||||
System.getenv('VIASH_TEMP') ?:
|
||||
System.getenv('TEMPDIR') ?:
|
||||
System.getenv('TMPDIR') ?:
|
||||
'/tmp'
|
||||
).toAbsolutePath()
|
||||
|
||||
mount_temp {
|
||||
docker.temp = tempDir
|
||||
podman.temp = tempDir
|
||||
charliecloud.temp = tempDir
|
||||
}
|
||||
|
||||
no_publish {
|
||||
process {
|
||||
withName: '.*' {
|
||||
publishDir = [
|
||||
enabled: false
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
docker {
|
||||
docker.fixOwnership = true
|
||||
docker.enabled = true
|
||||
// docker.userEmulation = true
|
||||
singularity.enabled = false
|
||||
podman.enabled = false
|
||||
shifter.enabled = false
|
||||
charliecloud.enabled = false
|
||||
}
|
||||
|
||||
local {
|
||||
// This config is for local processing.
|
||||
process {
|
||||
maxMemory = 25.GB
|
||||
withLabel: verylowcpu { cpus = 2 }
|
||||
withLabel: lowcpu { cpus = 4 }
|
||||
withLabel: midcpu { cpus = 6 }
|
||||
withLabel: highcpu { cpus = 12 }
|
||||
|
||||
withLabel: lowmem { memory = { get_memory( 8.GB * task.attempt ) } }
|
||||
withLabel: midmem { memory = { get_memory( 12.GB * task.attempt ) } }
|
||||
withLabel: highmem { memory = { get_memory( 20.GB * task.attempt ) } }
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
def get_memory(to_compare) {
|
||||
if (!process.containsKey("maxMemory") || !process.maxMemory) {
|
||||
return to_compare
|
||||
}
|
||||
|
||||
try {
|
||||
if (process.containsKey("maxRetries") && process.maxRetries && task.attempt == (process.maxRetries as int)) {
|
||||
return process.maxMemory
|
||||
}
|
||||
else if (to_compare.compareTo(process.maxMemory as nextflow.util.MemoryUnit) == 1) {
|
||||
return max_memory as nextflow.util.MemoryUnit
|
||||
}
|
||||
else {
|
||||
return to_compare
|
||||
}
|
||||
} catch (all) {
|
||||
println "Error processing memory resources. Please check that process.maxMemory '${process.maxMemory}' and process.maxRetries '${process.maxRetries}' are valid!"
|
||||
System.exit(1)
|
||||
}
|
||||
}
|
||||
@@ -1,6 +1,6 @@
|
||||
name: "publish"
|
||||
namespace: "io"
|
||||
version: "v0.3.3"
|
||||
version: "v0.3.6"
|
||||
argument_groups:
|
||||
- name: "Input arguments"
|
||||
arguments:
|
||||
@@ -22,7 +22,7 @@ argument_groups:
|
||||
create_parent: true
|
||||
required: true
|
||||
direction: "input"
|
||||
multiple: false
|
||||
multiple: true
|
||||
multiple_sep: ";"
|
||||
- type: "file"
|
||||
name: "--input_multiqc"
|
||||
@@ -94,6 +94,9 @@ resources:
|
||||
- type: "bash_script"
|
||||
path: "code.sh"
|
||||
is_executable: true
|
||||
- type: "file"
|
||||
path: "nextflow_labels.config"
|
||||
dest: "nextflow_labels.config"
|
||||
description: "Publish the processed results of the run"
|
||||
info: null
|
||||
status: "enabled"
|
||||
@@ -166,6 +169,8 @@ runners:
|
||||
cpu200: "cpus = 200"
|
||||
cpu500: "cpus = 500"
|
||||
cpu1000: "cpus = 1000"
|
||||
script:
|
||||
- "includeConfig(\"nextflow_labels.config\")"
|
||||
debug: false
|
||||
container: "docker"
|
||||
engines:
|
||||
@@ -173,7 +178,7 @@ engines:
|
||||
id: "docker"
|
||||
image: "debian:stable-slim"
|
||||
target_registry: "images.viash-hub.com"
|
||||
target_tag: "v0.3.3"
|
||||
target_tag: "v0.3.6"
|
||||
namespace_separator: "/"
|
||||
setup:
|
||||
- type: "apt"
|
||||
@@ -191,12 +196,12 @@ build_info:
|
||||
output: "target/executable/io/publish"
|
||||
executable: "target/executable/io/publish/publish"
|
||||
viash_version: "0.9.0"
|
||||
git_commit: "e297e8ff45464065384476fe521ab2700ec3739c"
|
||||
git_remote: "https://x-access-token:ghs_44flLTQIYGBkc9xcwCZoLYe7mxXzgw1iszFO@github.com/viash-hub/demultiplex"
|
||||
git_tag: "v0.3.2-4-ge297e8f"
|
||||
git_commit: "3a985c83869bcd78ba6d325f1a817514b52a132a"
|
||||
git_remote: "https://github.com/viash-hub/demultiplex"
|
||||
git_tag: "v0.3.5-3-g3a985c8"
|
||||
package_config:
|
||||
name: "demultiplex"
|
||||
version: "v0.3.3"
|
||||
version: "v0.3.6"
|
||||
description: "Demultiplexing pipeline\n"
|
||||
info:
|
||||
test_resources:
|
||||
@@ -207,10 +212,12 @@ package_config:
|
||||
target: "target"
|
||||
config_mods:
|
||||
- ".requirements.commands := ['ps']\n.runners[.type == 'nextflow'].directives.tag\
|
||||
\ := '$id'\n"
|
||||
\ := '$id'\n.resources += {path: '/src/config/labels.config', dest: 'nextflow_labels.config'}\n\
|
||||
.runners[.type == 'nextflow'].config.script := 'includeConfig(\"nextflow_labels.config\"\
|
||||
)'\n"
|
||||
- ".engines += { type: \"native\" }"
|
||||
- ".engines[.type == 'docker'].target_registry := 'images.viash-hub.com'"
|
||||
- ".engines[.type == 'docker'].target_tag := 'v0.3.3'"
|
||||
- ".engines[.type == 'docker'].target_tag := 'v0.3.6'"
|
||||
keywords:
|
||||
- "bioinformatics"
|
||||
- "sequence"
|
||||
|
||||
98
target/executable/io/publish/nextflow_labels.config
Normal file
98
target/executable/io/publish/nextflow_labels.config
Normal file
@@ -0,0 +1,98 @@
|
||||
process {
|
||||
container = 'nextflow/bash:latest'
|
||||
|
||||
// default resources
|
||||
memory = { 8.Gb * task.attempt }
|
||||
cpus = 8
|
||||
maxForks = 36
|
||||
|
||||
// Retry for exit codes that have something to do with memory issues
|
||||
errorStrategy = { task.exitStatus in 137..140 ? 'retry' : 'terminate' }
|
||||
maxRetries = 3
|
||||
maxMemory = 192.GB
|
||||
|
||||
// Resource labels
|
||||
withLabel: verylowcpu { cpus = 2 }
|
||||
withLabel: lowcpu { cpus = 8 }
|
||||
withLabel: midcpu { cpus = 16 }
|
||||
withLabel: highcpu { cpus = 32 }
|
||||
|
||||
withLabel: verylowmem { memory = { get_memory( 4.GB * task.attempt ) } }
|
||||
withLabel: lowmem { memory = { get_memory( 8.GB * task.attempt ) } }
|
||||
withLabel: midmem { memory = { get_memory( 16.GB * task.attempt ) } }
|
||||
withLabel: highmem { memory = { get_memory( 64.GB * task.attempt ) } }
|
||||
|
||||
}
|
||||
|
||||
profiles {
|
||||
// detect tempdir
|
||||
tempDir = java.nio.file.Paths.get(
|
||||
System.getenv('NXF_TEMP') ?:
|
||||
System.getenv('VIASH_TEMP') ?:
|
||||
System.getenv('TEMPDIR') ?:
|
||||
System.getenv('TMPDIR') ?:
|
||||
'/tmp'
|
||||
).toAbsolutePath()
|
||||
|
||||
mount_temp {
|
||||
docker.temp = tempDir
|
||||
podman.temp = tempDir
|
||||
charliecloud.temp = tempDir
|
||||
}
|
||||
|
||||
no_publish {
|
||||
process {
|
||||
withName: '.*' {
|
||||
publishDir = [
|
||||
enabled: false
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
docker {
|
||||
docker.fixOwnership = true
|
||||
docker.enabled = true
|
||||
// docker.userEmulation = true
|
||||
singularity.enabled = false
|
||||
podman.enabled = false
|
||||
shifter.enabled = false
|
||||
charliecloud.enabled = false
|
||||
}
|
||||
|
||||
local {
|
||||
// This config is for local processing.
|
||||
process {
|
||||
maxMemory = 25.GB
|
||||
withLabel: verylowcpu { cpus = 2 }
|
||||
withLabel: lowcpu { cpus = 4 }
|
||||
withLabel: midcpu { cpus = 6 }
|
||||
withLabel: highcpu { cpus = 12 }
|
||||
|
||||
withLabel: lowmem { memory = { get_memory( 8.GB * task.attempt ) } }
|
||||
withLabel: midmem { memory = { get_memory( 12.GB * task.attempt ) } }
|
||||
withLabel: highmem { memory = { get_memory( 20.GB * task.attempt ) } }
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
def get_memory(to_compare) {
|
||||
if (!process.containsKey("maxMemory") || !process.maxMemory) {
|
||||
return to_compare
|
||||
}
|
||||
|
||||
try {
|
||||
if (process.containsKey("maxRetries") && process.maxRetries && task.attempt == (process.maxRetries as int)) {
|
||||
return process.maxMemory
|
||||
}
|
||||
else if (to_compare.compareTo(process.maxMemory as nextflow.util.MemoryUnit) == 1) {
|
||||
return max_memory as nextflow.util.MemoryUnit
|
||||
}
|
||||
else {
|
||||
return to_compare
|
||||
}
|
||||
} catch (all) {
|
||||
println "Error processing memory resources. Please check that process.maxMemory '${process.maxMemory}' and process.maxRetries '${process.maxRetries}' are valid!"
|
||||
System.exit(1)
|
||||
}
|
||||
}
|
||||
@@ -1,6 +1,6 @@
|
||||
#!/usr/bin/env bash
|
||||
|
||||
# publish v0.3.3
|
||||
# publish v0.3.6
|
||||
#
|
||||
# This wrapper script is auto-generated by viash 0.9.0 and is thus a derivative
|
||||
# work thereof. This software comes with ABSOLUTELY NO WARRANTY from Data
|
||||
@@ -171,7 +171,7 @@ VIASH_META_TEMP_DIR="$VIASH_TEMP"
|
||||
|
||||
# ViashHelp: Display helpful explanation about this executable
|
||||
function ViashHelp {
|
||||
echo "publish v0.3.3"
|
||||
echo "publish v0.3.6"
|
||||
echo ""
|
||||
echo "Publish the processed results of the run"
|
||||
echo ""
|
||||
@@ -181,7 +181,7 @@ function ViashHelp {
|
||||
echo " Directory to write fastq data to"
|
||||
echo ""
|
||||
echo " --input_falco"
|
||||
echo " type: file, required parameter, file must exist"
|
||||
echo " type: file, required parameter, multiple values allowed, file must exist"
|
||||
echo " Directory to write falco output to"
|
||||
echo ""
|
||||
echo " --input_multiqc"
|
||||
@@ -490,10 +490,10 @@ RUN apt-get update && \
|
||||
rm -rf /var/lib/apt/lists/*
|
||||
|
||||
LABEL org.opencontainers.image.description="Companion container for running component io publish"
|
||||
LABEL org.opencontainers.image.created="2024-12-19T15:42:24Z"
|
||||
LABEL org.opencontainers.image.created="2025-03-20T19:41:30Z"
|
||||
LABEL org.opencontainers.image.source="https://github.com/viash-hub/demultiplex"
|
||||
LABEL org.opencontainers.image.revision="e297e8ff45464065384476fe521ab2700ec3739c"
|
||||
LABEL org.opencontainers.image.version="v0.3.3"
|
||||
LABEL org.opencontainers.image.revision="3a985c83869bcd78ba6d325f1a817514b52a132a"
|
||||
LABEL org.opencontainers.image.version="v0.3.6"
|
||||
|
||||
VIASHDOCKER
|
||||
fi
|
||||
@@ -629,7 +629,7 @@ while [[ $# -gt 0 ]]; do
|
||||
shift 1
|
||||
;;
|
||||
--version)
|
||||
echo "publish v0.3.3"
|
||||
echo "publish v0.3.6"
|
||||
exit
|
||||
;;
|
||||
--input)
|
||||
@@ -644,14 +644,20 @@ while [[ $# -gt 0 ]]; do
|
||||
shift 1
|
||||
;;
|
||||
--input_falco)
|
||||
[ -n "$VIASH_PAR_INPUT_FALCO" ] && ViashError Bad arguments for option \'--input_falco\': \'$VIASH_PAR_INPUT_FALCO\' \& \'$2\' - you should provide exactly one argument for this option. && exit 1
|
||||
VIASH_PAR_INPUT_FALCO="$2"
|
||||
if [ -z "$VIASH_PAR_INPUT_FALCO" ]; then
|
||||
VIASH_PAR_INPUT_FALCO="$2"
|
||||
else
|
||||
VIASH_PAR_INPUT_FALCO="$VIASH_PAR_INPUT_FALCO;""$2"
|
||||
fi
|
||||
[ $# -lt 2 ] && ViashError Not enough arguments passed to --input_falco. Use "--help" to get more information on the parameters. && exit 1
|
||||
shift 2
|
||||
;;
|
||||
--input_falco=*)
|
||||
[ -n "$VIASH_PAR_INPUT_FALCO" ] && ViashError Bad arguments for option \'--input_falco=*\': \'$VIASH_PAR_INPUT_FALCO\' \& \'$2\' - you should provide exactly one argument for this option. && exit 1
|
||||
VIASH_PAR_INPUT_FALCO=$(ViashRemoveFlags "$1")
|
||||
if [ -z "$VIASH_PAR_INPUT_FALCO" ]; then
|
||||
VIASH_PAR_INPUT_FALCO=$(ViashRemoveFlags "$1")
|
||||
else
|
||||
VIASH_PAR_INPUT_FALCO="$VIASH_PAR_INPUT_FALCO;"$(ViashRemoveFlags "$1")
|
||||
fi
|
||||
shift 1
|
||||
;;
|
||||
--input_multiqc)
|
||||
@@ -808,7 +814,7 @@ if [[ "$VIASH_ENGINE_TYPE" == "docker" ]]; then
|
||||
|
||||
# determine docker image id
|
||||
if [[ "$VIASH_ENGINE_ID" == 'docker' ]]; then
|
||||
VIASH_DOCKER_IMAGE_ID='images.viash-hub.com/vsh/demultiplex/io/publish:v0.3.3'
|
||||
VIASH_DOCKER_IMAGE_ID='images.viash-hub.com/vsh/demultiplex/io/publish:v0.3.6'
|
||||
fi
|
||||
|
||||
# print dockerfile
|
||||
@@ -952,9 +958,17 @@ if [ ! -z "$VIASH_PAR_INPUT" ] && [ ! -e "$VIASH_PAR_INPUT" ]; then
|
||||
ViashError "Input file '$VIASH_PAR_INPUT' does not exist."
|
||||
exit 1
|
||||
fi
|
||||
if [ ! -z "$VIASH_PAR_INPUT_FALCO" ] && [ ! -e "$VIASH_PAR_INPUT_FALCO" ]; then
|
||||
ViashError "Input file '$VIASH_PAR_INPUT_FALCO' does not exist."
|
||||
exit 1
|
||||
if [ ! -z "$VIASH_PAR_INPUT_FALCO" ]; then
|
||||
IFS=';'
|
||||
set -f
|
||||
for file in $VIASH_PAR_INPUT_FALCO; do
|
||||
unset IFS
|
||||
if [ ! -e "$file" ]; then
|
||||
ViashError "Input file '$file' does not exist."
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
set +f
|
||||
fi
|
||||
if [ ! -z "$VIASH_PAR_INPUT_MULTIQC" ] && [ ! -e "$VIASH_PAR_INPUT_MULTIQC" ]; then
|
||||
ViashError "Input file '$VIASH_PAR_INPUT_MULTIQC' does not exist."
|
||||
@@ -1070,8 +1084,15 @@ if [ ! -z "$VIASH_PAR_INPUT" ]; then
|
||||
VIASH_PAR_INPUT=$(ViashDockerAutodetectMount "$VIASH_PAR_INPUT")
|
||||
fi
|
||||
if [ ! -z "$VIASH_PAR_INPUT_FALCO" ]; then
|
||||
VIASH_DIRECTORY_MOUNTS+=( "$(ViashDockerAutodetectMountArg "$VIASH_PAR_INPUT_FALCO")" )
|
||||
VIASH_PAR_INPUT_FALCO=$(ViashDockerAutodetectMount "$VIASH_PAR_INPUT_FALCO")
|
||||
VIASH_TEST_INPUT_FALCO=()
|
||||
IFS=';'
|
||||
for var in $VIASH_PAR_INPUT_FALCO; do
|
||||
unset IFS
|
||||
VIASH_DIRECTORY_MOUNTS+=( "$(ViashDockerAutodetectMountArg "$var")" )
|
||||
var=$(ViashDockerAutodetectMount "$var")
|
||||
VIASH_TEST_INPUT_FALCO+=( "$var" )
|
||||
done
|
||||
VIASH_PAR_INPUT_FALCO=$(IFS=';' ; echo "${VIASH_TEST_INPUT_FALCO[*]}")
|
||||
fi
|
||||
if [ ! -z "$VIASH_PAR_INPUT_MULTIQC" ]; then
|
||||
VIASH_DIRECTORY_MOUNTS+=( "$(ViashDockerAutodetectMountArg "$VIASH_PAR_INPUT_MULTIQC")" )
|
||||
@@ -1203,7 +1224,6 @@ $( if [ ! -z ${VIASH_META_MEMORY_PIB+x} ]; then echo "${VIASH_META_MEMORY_PIB}"
|
||||
set -eo pipefail
|
||||
|
||||
declare -A input_output_mapping=(["par_input"]="par_output"
|
||||
["par_input_falco"]="par_output_falco"
|
||||
["par_input_multiqc"]="par_output_multiqc"
|
||||
["par_input_run_information"]="par_output_run_information"
|
||||
)
|
||||
@@ -1224,6 +1244,14 @@ do
|
||||
echo "Output files for \$output_location:"
|
||||
ls "\$output_location"
|
||||
done
|
||||
|
||||
echo "Grouping output from \$par_input_falco into \$par_output_falco"
|
||||
mkdir -p "\$par_output_falco"
|
||||
IFS=";" read -ra falco_inputs <<< \$par_input_falco
|
||||
for falco_dir in "\${falco_inputs[@]}"; do
|
||||
echo "Copying contents of \$falco_dir"
|
||||
find -H -D exec "\$falco_dir" -type f -maxdepth 1 -exec cp -t "\$par_output_falco" {} +
|
||||
done
|
||||
VIASHMAIN
|
||||
bash "\$tempscript" &
|
||||
wait "\$!"
|
||||
@@ -1238,7 +1266,17 @@ if [[ "$VIASH_ENGINE_TYPE" == "docker" ]]; then
|
||||
VIASH_PAR_INPUT=$(ViashDockerStripAutomount "$VIASH_PAR_INPUT")
|
||||
fi
|
||||
if [ ! -z "$VIASH_PAR_INPUT_FALCO" ]; then
|
||||
VIASH_PAR_INPUT_FALCO=$(ViashDockerStripAutomount "$VIASH_PAR_INPUT_FALCO")
|
||||
unset VIASH_TEST_INPUT_FALCO
|
||||
IFS=';'
|
||||
for var in $VIASH_PAR_INPUT_FALCO; do
|
||||
unset IFS
|
||||
if [ -z "$VIASH_TEST_INPUT_FALCO" ]; then
|
||||
VIASH_TEST_INPUT_FALCO="$(ViashDockerStripAutomount "$var")"
|
||||
else
|
||||
VIASH_TEST_INPUT_FALCO="$VIASH_TEST_INPUT_FALCO;""$(ViashDockerStripAutomount "$var")"
|
||||
fi
|
||||
done
|
||||
VIASH_PAR_INPUT_FALCO="$VIASH_TEST_INPUT_FALCO"
|
||||
fi
|
||||
if [ ! -z "$VIASH_PAR_INPUT_MULTIQC" ]; then
|
||||
VIASH_PAR_INPUT_MULTIQC=$(ViashDockerStripAutomount "$VIASH_PAR_INPUT_MULTIQC")
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
name: "untar"
|
||||
namespace: "io"
|
||||
version: "v0.3.3"
|
||||
version: "v0.3.6"
|
||||
argument_groups:
|
||||
- name: "Input arguments"
|
||||
arguments:
|
||||
@@ -45,6 +45,9 @@ resources:
|
||||
- type: "bash_script"
|
||||
path: "script.sh"
|
||||
is_executable: true
|
||||
- type: "file"
|
||||
path: "nextflow_labels.config"
|
||||
dest: "nextflow_labels.config"
|
||||
description: "Unpack a .tar file. When the contents of the .tar file is just a single\
|
||||
\ directory,\nput the contents of the directory into the output folder instead of\
|
||||
\ that directory.\n"
|
||||
@@ -123,6 +126,8 @@ runners:
|
||||
cpu200: "cpus = 200"
|
||||
cpu500: "cpus = 500"
|
||||
cpu1000: "cpus = 1000"
|
||||
script:
|
||||
- "includeConfig(\"nextflow_labels.config\")"
|
||||
debug: false
|
||||
container: "docker"
|
||||
engines:
|
||||
@@ -130,7 +135,7 @@ engines:
|
||||
id: "docker"
|
||||
image: "debian:stable-slim"
|
||||
target_registry: "images.viash-hub.com"
|
||||
target_tag: "v0.3.3"
|
||||
target_tag: "v0.3.6"
|
||||
namespace_separator: "/"
|
||||
setup:
|
||||
- type: "apt"
|
||||
@@ -148,12 +153,12 @@ build_info:
|
||||
output: "target/executable/io/untar"
|
||||
executable: "target/executable/io/untar/untar"
|
||||
viash_version: "0.9.0"
|
||||
git_commit: "e297e8ff45464065384476fe521ab2700ec3739c"
|
||||
git_remote: "https://x-access-token:ghs_44flLTQIYGBkc9xcwCZoLYe7mxXzgw1iszFO@github.com/viash-hub/demultiplex"
|
||||
git_tag: "v0.3.2-4-ge297e8f"
|
||||
git_commit: "3a985c83869bcd78ba6d325f1a817514b52a132a"
|
||||
git_remote: "https://github.com/viash-hub/demultiplex"
|
||||
git_tag: "v0.3.5-3-g3a985c8"
|
||||
package_config:
|
||||
name: "demultiplex"
|
||||
version: "v0.3.3"
|
||||
version: "v0.3.6"
|
||||
description: "Demultiplexing pipeline\n"
|
||||
info:
|
||||
test_resources:
|
||||
@@ -164,10 +169,12 @@ package_config:
|
||||
target: "target"
|
||||
config_mods:
|
||||
- ".requirements.commands := ['ps']\n.runners[.type == 'nextflow'].directives.tag\
|
||||
\ := '$id'\n"
|
||||
\ := '$id'\n.resources += {path: '/src/config/labels.config', dest: 'nextflow_labels.config'}\n\
|
||||
.runners[.type == 'nextflow'].config.script := 'includeConfig(\"nextflow_labels.config\"\
|
||||
)'\n"
|
||||
- ".engines += { type: \"native\" }"
|
||||
- ".engines[.type == 'docker'].target_registry := 'images.viash-hub.com'"
|
||||
- ".engines[.type == 'docker'].target_tag := 'v0.3.3'"
|
||||
- ".engines[.type == 'docker'].target_tag := 'v0.3.6'"
|
||||
keywords:
|
||||
- "bioinformatics"
|
||||
- "sequence"
|
||||
|
||||
98
target/executable/io/untar/nextflow_labels.config
Normal file
98
target/executable/io/untar/nextflow_labels.config
Normal file
@@ -0,0 +1,98 @@
|
||||
process {
|
||||
container = 'nextflow/bash:latest'
|
||||
|
||||
// default resources
|
||||
memory = { 8.Gb * task.attempt }
|
||||
cpus = 8
|
||||
maxForks = 36
|
||||
|
||||
// Retry for exit codes that have something to do with memory issues
|
||||
errorStrategy = { task.exitStatus in 137..140 ? 'retry' : 'terminate' }
|
||||
maxRetries = 3
|
||||
maxMemory = 192.GB
|
||||
|
||||
// Resource labels
|
||||
withLabel: verylowcpu { cpus = 2 }
|
||||
withLabel: lowcpu { cpus = 8 }
|
||||
withLabel: midcpu { cpus = 16 }
|
||||
withLabel: highcpu { cpus = 32 }
|
||||
|
||||
withLabel: verylowmem { memory = { get_memory( 4.GB * task.attempt ) } }
|
||||
withLabel: lowmem { memory = { get_memory( 8.GB * task.attempt ) } }
|
||||
withLabel: midmem { memory = { get_memory( 16.GB * task.attempt ) } }
|
||||
withLabel: highmem { memory = { get_memory( 64.GB * task.attempt ) } }
|
||||
|
||||
}
|
||||
|
||||
profiles {
|
||||
// detect tempdir
|
||||
tempDir = java.nio.file.Paths.get(
|
||||
System.getenv('NXF_TEMP') ?:
|
||||
System.getenv('VIASH_TEMP') ?:
|
||||
System.getenv('TEMPDIR') ?:
|
||||
System.getenv('TMPDIR') ?:
|
||||
'/tmp'
|
||||
).toAbsolutePath()
|
||||
|
||||
mount_temp {
|
||||
docker.temp = tempDir
|
||||
podman.temp = tempDir
|
||||
charliecloud.temp = tempDir
|
||||
}
|
||||
|
||||
no_publish {
|
||||
process {
|
||||
withName: '.*' {
|
||||
publishDir = [
|
||||
enabled: false
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
docker {
|
||||
docker.fixOwnership = true
|
||||
docker.enabled = true
|
||||
// docker.userEmulation = true
|
||||
singularity.enabled = false
|
||||
podman.enabled = false
|
||||
shifter.enabled = false
|
||||
charliecloud.enabled = false
|
||||
}
|
||||
|
||||
local {
|
||||
// This config is for local processing.
|
||||
process {
|
||||
maxMemory = 25.GB
|
||||
withLabel: verylowcpu { cpus = 2 }
|
||||
withLabel: lowcpu { cpus = 4 }
|
||||
withLabel: midcpu { cpus = 6 }
|
||||
withLabel: highcpu { cpus = 12 }
|
||||
|
||||
withLabel: lowmem { memory = { get_memory( 8.GB * task.attempt ) } }
|
||||
withLabel: midmem { memory = { get_memory( 12.GB * task.attempt ) } }
|
||||
withLabel: highmem { memory = { get_memory( 20.GB * task.attempt ) } }
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
def get_memory(to_compare) {
|
||||
if (!process.containsKey("maxMemory") || !process.maxMemory) {
|
||||
return to_compare
|
||||
}
|
||||
|
||||
try {
|
||||
if (process.containsKey("maxRetries") && process.maxRetries && task.attempt == (process.maxRetries as int)) {
|
||||
return process.maxMemory
|
||||
}
|
||||
else if (to_compare.compareTo(process.maxMemory as nextflow.util.MemoryUnit) == 1) {
|
||||
return max_memory as nextflow.util.MemoryUnit
|
||||
}
|
||||
else {
|
||||
return to_compare
|
||||
}
|
||||
} catch (all) {
|
||||
println "Error processing memory resources. Please check that process.maxMemory '${process.maxMemory}' and process.maxRetries '${process.maxRetries}' are valid!"
|
||||
System.exit(1)
|
||||
}
|
||||
}
|
||||
@@ -1,6 +1,6 @@
|
||||
#!/usr/bin/env bash
|
||||
|
||||
# untar v0.3.3
|
||||
# untar v0.3.6
|
||||
#
|
||||
# This wrapper script is auto-generated by viash 0.9.0 and is thus a derivative
|
||||
# work thereof. This software comes with ABSOLUTELY NO WARRANTY from Data
|
||||
@@ -171,7 +171,7 @@ VIASH_META_TEMP_DIR="$VIASH_TEMP"
|
||||
|
||||
# ViashHelp: Display helpful explanation about this executable
|
||||
function ViashHelp {
|
||||
echo "untar v0.3.3"
|
||||
echo "untar v0.3.6"
|
||||
echo ""
|
||||
echo "Unpack a .tar file. When the contents of the .tar file is just a single"
|
||||
echo "directory,"
|
||||
@@ -476,10 +476,10 @@ RUN apt-get update && \
|
||||
rm -rf /var/lib/apt/lists/*
|
||||
|
||||
LABEL org.opencontainers.image.description="Companion container for running component io untar"
|
||||
LABEL org.opencontainers.image.created="2024-12-19T15:42:24Z"
|
||||
LABEL org.opencontainers.image.created="2025-03-20T19:41:30Z"
|
||||
LABEL org.opencontainers.image.source="https://github.com/viash-hub/demultiplex"
|
||||
LABEL org.opencontainers.image.revision="e297e8ff45464065384476fe521ab2700ec3739c"
|
||||
LABEL org.opencontainers.image.version="v0.3.3"
|
||||
LABEL org.opencontainers.image.revision="3a985c83869bcd78ba6d325f1a817514b52a132a"
|
||||
LABEL org.opencontainers.image.version="v0.3.6"
|
||||
|
||||
VIASHDOCKER
|
||||
fi
|
||||
@@ -615,7 +615,7 @@ while [[ $# -gt 0 ]]; do
|
||||
shift 1
|
||||
;;
|
||||
--version)
|
||||
echo "untar v0.3.3"
|
||||
echo "untar v0.3.6"
|
||||
exit
|
||||
;;
|
||||
--input)
|
||||
@@ -745,7 +745,7 @@ if [[ "$VIASH_ENGINE_TYPE" == "docker" ]]; then
|
||||
|
||||
# determine docker image id
|
||||
if [[ "$VIASH_ENGINE_ID" == 'docker' ]]; then
|
||||
VIASH_DOCKER_IMAGE_ID='images.viash-hub.com/vsh/demultiplex/io/untar:v0.3.3'
|
||||
VIASH_DOCKER_IMAGE_ID='images.viash-hub.com/vsh/demultiplex/io/untar:v0.3.6'
|
||||
fi
|
||||
|
||||
# print dockerfile
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
name: "combine_samples"
|
||||
namespace: "dataflow"
|
||||
version: "v0.3.3"
|
||||
version: "v0.3.6"
|
||||
argument_groups:
|
||||
- name: "Input arguments"
|
||||
arguments:
|
||||
@@ -30,6 +30,15 @@ argument_groups:
|
||||
direction: "input"
|
||||
multiple: true
|
||||
multiple_sep: ";"
|
||||
- type: "file"
|
||||
name: "--falco_dir"
|
||||
info: null
|
||||
must_exist: true
|
||||
create_parent: true
|
||||
required: true
|
||||
direction: "input"
|
||||
multiple: false
|
||||
multiple_sep: ";"
|
||||
- name: "Output arguments"
|
||||
arguments:
|
||||
- type: "file"
|
||||
@@ -50,11 +59,23 @@ argument_groups:
|
||||
direction: "output"
|
||||
multiple: true
|
||||
multiple_sep: ";"
|
||||
- type: "file"
|
||||
name: "--output_falco"
|
||||
info: null
|
||||
must_exist: true
|
||||
create_parent: true
|
||||
required: true
|
||||
direction: "output"
|
||||
multiple: true
|
||||
multiple_sep: ";"
|
||||
resources:
|
||||
- type: "nextflow_script"
|
||||
path: "main.nf"
|
||||
is_executable: true
|
||||
entrypoint: "run_wf"
|
||||
- type: "file"
|
||||
path: "nextflow_labels.config"
|
||||
dest: "nextflow_labels.config"
|
||||
description: "Combine fastq files from across samples into one event with a list of\
|
||||
\ fastq files per orientation."
|
||||
info: null
|
||||
@@ -125,6 +146,8 @@ runners:
|
||||
cpu200: "cpus = 200"
|
||||
cpu500: "cpus = 500"
|
||||
cpu1000: "cpus = 1000"
|
||||
script:
|
||||
- "includeConfig(\"nextflow_labels.config\")"
|
||||
debug: false
|
||||
container: "docker"
|
||||
engines:
|
||||
@@ -139,12 +162,12 @@ build_info:
|
||||
output: "target/nextflow/dataflow/combine_samples"
|
||||
executable: "target/nextflow/dataflow/combine_samples/main.nf"
|
||||
viash_version: "0.9.0"
|
||||
git_commit: "e297e8ff45464065384476fe521ab2700ec3739c"
|
||||
git_remote: "https://x-access-token:ghs_44flLTQIYGBkc9xcwCZoLYe7mxXzgw1iszFO@github.com/viash-hub/demultiplex"
|
||||
git_tag: "v0.3.2-4-ge297e8f"
|
||||
git_commit: "3a985c83869bcd78ba6d325f1a817514b52a132a"
|
||||
git_remote: "https://github.com/viash-hub/demultiplex"
|
||||
git_tag: "v0.3.5-3-g3a985c8"
|
||||
package_config:
|
||||
name: "demultiplex"
|
||||
version: "v0.3.3"
|
||||
version: "v0.3.6"
|
||||
description: "Demultiplexing pipeline\n"
|
||||
info:
|
||||
test_resources:
|
||||
@@ -155,10 +178,12 @@ package_config:
|
||||
target: "target"
|
||||
config_mods:
|
||||
- ".requirements.commands := ['ps']\n.runners[.type == 'nextflow'].directives.tag\
|
||||
\ := '$id'\n"
|
||||
\ := '$id'\n.resources += {path: '/src/config/labels.config', dest: 'nextflow_labels.config'}\n\
|
||||
.runners[.type == 'nextflow'].config.script := 'includeConfig(\"nextflow_labels.config\"\
|
||||
)'\n"
|
||||
- ".engines += { type: \"native\" }"
|
||||
- ".engines[.type == 'docker'].target_registry := 'images.viash-hub.com'"
|
||||
- ".engines[.type == 'docker'].target_tag := 'v0.3.3'"
|
||||
- ".engines[.type == 'docker'].target_tag := 'v0.3.6'"
|
||||
keywords:
|
||||
- "bioinformatics"
|
||||
- "sequence"
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
// combine_samples v0.3.3
|
||||
// combine_samples v0.3.6
|
||||
//
|
||||
// This wrapper script is auto-generated by viash 0.9.0 and is thus a derivative
|
||||
// work thereof. This software comes with ABSOLUTELY NO WARRANTY from Data
|
||||
@@ -2806,7 +2806,7 @@ meta = [
|
||||
"config": processConfig(readJsonBlob('''{
|
||||
"name" : "combine_samples",
|
||||
"namespace" : "dataflow",
|
||||
"version" : "v0.3.3",
|
||||
"version" : "v0.3.6",
|
||||
"argument_groups" : [
|
||||
{
|
||||
"name" : "Input arguments",
|
||||
@@ -2839,6 +2839,16 @@ meta = [
|
||||
"direction" : "input",
|
||||
"multiple" : true,
|
||||
"multiple_sep" : ";"
|
||||
},
|
||||
{
|
||||
"type" : "file",
|
||||
"name" : "--falco_dir",
|
||||
"must_exist" : true,
|
||||
"create_parent" : true,
|
||||
"required" : true,
|
||||
"direction" : "input",
|
||||
"multiple" : false,
|
||||
"multiple_sep" : ";"
|
||||
}
|
||||
]
|
||||
},
|
||||
@@ -2864,6 +2874,16 @@ meta = [
|
||||
"direction" : "output",
|
||||
"multiple" : true,
|
||||
"multiple_sep" : ";"
|
||||
},
|
||||
{
|
||||
"type" : "file",
|
||||
"name" : "--output_falco",
|
||||
"must_exist" : true,
|
||||
"create_parent" : true,
|
||||
"required" : true,
|
||||
"direction" : "output",
|
||||
"multiple" : true,
|
||||
"multiple_sep" : ";"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -2874,6 +2894,11 @@ meta = [
|
||||
"path" : "main.nf",
|
||||
"is_executable" : true,
|
||||
"entrypoint" : "run_wf"
|
||||
},
|
||||
{
|
||||
"type" : "file",
|
||||
"path" : "/src/config/labels.config",
|
||||
"dest" : "nextflow_labels.config"
|
||||
}
|
||||
],
|
||||
"description" : "Combine fastq files from across samples into one event with a list of fastq files per orientation.",
|
||||
@@ -2950,7 +2975,10 @@ meta = [
|
||||
"cpu200" : "cpus = 200",
|
||||
"cpu500" : "cpus = 500",
|
||||
"cpu1000" : "cpus = 1000"
|
||||
}
|
||||
},
|
||||
"script" : [
|
||||
"includeConfig(\\"nextflow_labels.config\\")"
|
||||
]
|
||||
},
|
||||
"debug" : false,
|
||||
"container" : "docker"
|
||||
@@ -2972,13 +3000,13 @@ meta = [
|
||||
"engine" : "native|native",
|
||||
"output" : "target/nextflow/dataflow/combine_samples",
|
||||
"viash_version" : "0.9.0",
|
||||
"git_commit" : "e297e8ff45464065384476fe521ab2700ec3739c",
|
||||
"git_remote" : "https://x-access-token:ghs_44flLTQIYGBkc9xcwCZoLYe7mxXzgw1iszFO@github.com/viash-hub/demultiplex",
|
||||
"git_tag" : "v0.3.2-4-ge297e8f"
|
||||
"git_commit" : "3a985c83869bcd78ba6d325f1a817514b52a132a",
|
||||
"git_remote" : "https://github.com/viash-hub/demultiplex",
|
||||
"git_tag" : "v0.3.5-3-g3a985c8"
|
||||
},
|
||||
"package_config" : {
|
||||
"name" : "demultiplex",
|
||||
"version" : "v0.3.3",
|
||||
"version" : "v0.3.6",
|
||||
"description" : "Demultiplexing pipeline\n",
|
||||
"info" : {
|
||||
"test_resources" : [
|
||||
@@ -2992,10 +3020,10 @@ meta = [
|
||||
"source" : "src",
|
||||
"target" : "target",
|
||||
"config_mods" : [
|
||||
".requirements.commands := ['ps']\n.runners[.type == 'nextflow'].directives.tag := '$id'\n",
|
||||
".requirements.commands := ['ps']\n.runners[.type == 'nextflow'].directives.tag := '$id'\n.resources += {path: '/src/config/labels.config', dest: 'nextflow_labels.config'}\n.runners[.type == 'nextflow'].config.script := 'includeConfig(\\"nextflow_labels.config\\")'\n",
|
||||
".engines += { type: \\"native\\" }",
|
||||
".engines[.type == 'docker'].target_registry := 'images.viash-hub.com'",
|
||||
".engines[.type == 'docker'].target_tag := 'v0.3.3'"
|
||||
".engines[.type == 'docker'].target_tag := 'v0.3.6'"
|
||||
],
|
||||
"keywords" : [
|
||||
"bioinformatics",
|
||||
@@ -3033,10 +3061,12 @@ workflow run_wf {
|
||||
// Gather the following state for all samples
|
||||
def forward_fastqs = states.collect{it.forward_input}.flatten()
|
||||
def reverse_fastqs = states.collect{it.reverse_input}.findAll{it != null}.flatten()
|
||||
def falco_dirs = states.collect{it.falco_dir}
|
||||
|
||||
def resultState = [
|
||||
"output_forward": forward_fastqs,
|
||||
"output_reverse": reverse_fastqs,
|
||||
"output_falco": falco_dirs,
|
||||
// The join ID is the same across all samples from the same run
|
||||
"_meta": ["join_id": states[0]._meta.join_id]
|
||||
]
|
||||
|
||||
@@ -2,7 +2,7 @@ manifest {
|
||||
name = 'dataflow/combine_samples'
|
||||
mainScript = 'main.nf'
|
||||
nextflowVersion = '!>=20.12.1-edge'
|
||||
version = 'v0.3.3'
|
||||
version = 'v0.3.6'
|
||||
description = 'Combine fastq files from across samples into one event with a list of fastq files per orientation.'
|
||||
}
|
||||
|
||||
@@ -122,4 +122,4 @@ process{
|
||||
withLabel: cpu1000 { cpus = 1000 }
|
||||
}
|
||||
|
||||
|
||||
includeConfig("nextflow_labels.config")
|
||||
|
||||
@@ -0,0 +1,98 @@
|
||||
process {
|
||||
container = 'nextflow/bash:latest'
|
||||
|
||||
// default resources
|
||||
memory = { 8.Gb * task.attempt }
|
||||
cpus = 8
|
||||
maxForks = 36
|
||||
|
||||
// Retry for exit codes that have something to do with memory issues
|
||||
errorStrategy = { task.exitStatus in 137..140 ? 'retry' : 'terminate' }
|
||||
maxRetries = 3
|
||||
maxMemory = 192.GB
|
||||
|
||||
// Resource labels
|
||||
withLabel: verylowcpu { cpus = 2 }
|
||||
withLabel: lowcpu { cpus = 8 }
|
||||
withLabel: midcpu { cpus = 16 }
|
||||
withLabel: highcpu { cpus = 32 }
|
||||
|
||||
withLabel: verylowmem { memory = { get_memory( 4.GB * task.attempt ) } }
|
||||
withLabel: lowmem { memory = { get_memory( 8.GB * task.attempt ) } }
|
||||
withLabel: midmem { memory = { get_memory( 16.GB * task.attempt ) } }
|
||||
withLabel: highmem { memory = { get_memory( 64.GB * task.attempt ) } }
|
||||
|
||||
}
|
||||
|
||||
profiles {
|
||||
// detect tempdir
|
||||
tempDir = java.nio.file.Paths.get(
|
||||
System.getenv('NXF_TEMP') ?:
|
||||
System.getenv('VIASH_TEMP') ?:
|
||||
System.getenv('TEMPDIR') ?:
|
||||
System.getenv('TMPDIR') ?:
|
||||
'/tmp'
|
||||
).toAbsolutePath()
|
||||
|
||||
mount_temp {
|
||||
docker.temp = tempDir
|
||||
podman.temp = tempDir
|
||||
charliecloud.temp = tempDir
|
||||
}
|
||||
|
||||
no_publish {
|
||||
process {
|
||||
withName: '.*' {
|
||||
publishDir = [
|
||||
enabled: false
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
docker {
|
||||
docker.fixOwnership = true
|
||||
docker.enabled = true
|
||||
// docker.userEmulation = true
|
||||
singularity.enabled = false
|
||||
podman.enabled = false
|
||||
shifter.enabled = false
|
||||
charliecloud.enabled = false
|
||||
}
|
||||
|
||||
local {
|
||||
// This config is for local processing.
|
||||
process {
|
||||
maxMemory = 25.GB
|
||||
withLabel: verylowcpu { cpus = 2 }
|
||||
withLabel: lowcpu { cpus = 4 }
|
||||
withLabel: midcpu { cpus = 6 }
|
||||
withLabel: highcpu { cpus = 12 }
|
||||
|
||||
withLabel: lowmem { memory = { get_memory( 8.GB * task.attempt ) } }
|
||||
withLabel: midmem { memory = { get_memory( 12.GB * task.attempt ) } }
|
||||
withLabel: highmem { memory = { get_memory( 20.GB * task.attempt ) } }
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
def get_memory(to_compare) {
|
||||
if (!process.containsKey("maxMemory") || !process.maxMemory) {
|
||||
return to_compare
|
||||
}
|
||||
|
||||
try {
|
||||
if (process.containsKey("maxRetries") && process.maxRetries && task.attempt == (process.maxRetries as int)) {
|
||||
return process.maxMemory
|
||||
}
|
||||
else if (to_compare.compareTo(process.maxMemory as nextflow.util.MemoryUnit) == 1) {
|
||||
return max_memory as nextflow.util.MemoryUnit
|
||||
}
|
||||
else {
|
||||
return to_compare
|
||||
}
|
||||
} catch (all) {
|
||||
println "Error processing memory resources. Please check that process.maxMemory '${process.maxMemory}' and process.maxRetries '${process.maxRetries}' are valid!"
|
||||
System.exit(1)
|
||||
}
|
||||
}
|
||||
@@ -43,6 +43,16 @@
|
||||
}
|
||||
|
||||
|
||||
,
|
||||
"falco_dir": {
|
||||
"type":
|
||||
"string",
|
||||
"description": "Type: `file`, required. ",
|
||||
"help_text": "Type: `file`, required. "
|
||||
|
||||
}
|
||||
|
||||
|
||||
}
|
||||
},
|
||||
|
||||
@@ -75,6 +85,17 @@
|
||||
}
|
||||
|
||||
|
||||
,
|
||||
"output_falco": {
|
||||
"type":
|
||||
"string",
|
||||
"description": "Type: List of `file`, required, default: `$id.$key.output_falco_*.output_falco_*`, multiple_sep: `\";\"`. ",
|
||||
"help_text": "Type: List of `file`, required, default: `$id.$key.output_falco_*.output_falco_*`, multiple_sep: `\";\"`. "
|
||||
,
|
||||
"default":"$id.$key.output_falco_*.output_falco_*"
|
||||
}
|
||||
|
||||
|
||||
}
|
||||
},
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
name: "gather_fastqs_and_validate"
|
||||
namespace: "dataflow"
|
||||
version: "v0.3.3"
|
||||
version: "v0.3.6"
|
||||
argument_groups:
|
||||
- name: "Input arguments"
|
||||
arguments:
|
||||
@@ -49,6 +49,9 @@ resources:
|
||||
path: "main.nf"
|
||||
is_executable: true
|
||||
entrypoint: "run_wf"
|
||||
- type: "file"
|
||||
path: "nextflow_labels.config"
|
||||
dest: "nextflow_labels.config"
|
||||
description: "From a directory containing fastq files, gather the files per sample\
|
||||
\ \nand validate according to the contents of the sample sheet.\n"
|
||||
info: null
|
||||
@@ -119,6 +122,8 @@ runners:
|
||||
cpu200: "cpus = 200"
|
||||
cpu500: "cpus = 500"
|
||||
cpu1000: "cpus = 1000"
|
||||
script:
|
||||
- "includeConfig(\"nextflow_labels.config\")"
|
||||
debug: false
|
||||
container: "docker"
|
||||
engines:
|
||||
@@ -133,12 +138,12 @@ build_info:
|
||||
output: "target/nextflow/dataflow/gather_fastqs_and_validate"
|
||||
executable: "target/nextflow/dataflow/gather_fastqs_and_validate/main.nf"
|
||||
viash_version: "0.9.0"
|
||||
git_commit: "e297e8ff45464065384476fe521ab2700ec3739c"
|
||||
git_remote: "https://x-access-token:ghs_44flLTQIYGBkc9xcwCZoLYe7mxXzgw1iszFO@github.com/viash-hub/demultiplex"
|
||||
git_tag: "v0.3.2-4-ge297e8f"
|
||||
git_commit: "3a985c83869bcd78ba6d325f1a817514b52a132a"
|
||||
git_remote: "https://github.com/viash-hub/demultiplex"
|
||||
git_tag: "v0.3.5-3-g3a985c8"
|
||||
package_config:
|
||||
name: "demultiplex"
|
||||
version: "v0.3.3"
|
||||
version: "v0.3.6"
|
||||
description: "Demultiplexing pipeline\n"
|
||||
info:
|
||||
test_resources:
|
||||
@@ -149,10 +154,12 @@ package_config:
|
||||
target: "target"
|
||||
config_mods:
|
||||
- ".requirements.commands := ['ps']\n.runners[.type == 'nextflow'].directives.tag\
|
||||
\ := '$id'\n"
|
||||
\ := '$id'\n.resources += {path: '/src/config/labels.config', dest: 'nextflow_labels.config'}\n\
|
||||
.runners[.type == 'nextflow'].config.script := 'includeConfig(\"nextflow_labels.config\"\
|
||||
)'\n"
|
||||
- ".engines += { type: \"native\" }"
|
||||
- ".engines[.type == 'docker'].target_registry := 'images.viash-hub.com'"
|
||||
- ".engines[.type == 'docker'].target_tag := 'v0.3.3'"
|
||||
- ".engines[.type == 'docker'].target_tag := 'v0.3.6'"
|
||||
keywords:
|
||||
- "bioinformatics"
|
||||
- "sequence"
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
// gather_fastqs_and_validate v0.3.3
|
||||
// gather_fastqs_and_validate v0.3.6
|
||||
//
|
||||
// This wrapper script is auto-generated by viash 0.9.0 and is thus a derivative
|
||||
// work thereof. This software comes with ABSOLUTELY NO WARRANTY from Data
|
||||
@@ -2806,7 +2806,7 @@ meta = [
|
||||
"config": processConfig(readJsonBlob('''{
|
||||
"name" : "gather_fastqs_and_validate",
|
||||
"namespace" : "dataflow",
|
||||
"version" : "v0.3.3",
|
||||
"version" : "v0.3.6",
|
||||
"argument_groups" : [
|
||||
{
|
||||
"name" : "Input arguments",
|
||||
@@ -2867,6 +2867,11 @@ meta = [
|
||||
"path" : "main.nf",
|
||||
"is_executable" : true,
|
||||
"entrypoint" : "run_wf"
|
||||
},
|
||||
{
|
||||
"type" : "file",
|
||||
"path" : "/src/config/labels.config",
|
||||
"dest" : "nextflow_labels.config"
|
||||
}
|
||||
],
|
||||
"description" : "From a directory containing fastq files, gather the files per sample \nand validate according to the contents of the sample sheet.\n",
|
||||
@@ -2943,7 +2948,10 @@ meta = [
|
||||
"cpu200" : "cpus = 200",
|
||||
"cpu500" : "cpus = 500",
|
||||
"cpu1000" : "cpus = 1000"
|
||||
}
|
||||
},
|
||||
"script" : [
|
||||
"includeConfig(\\"nextflow_labels.config\\")"
|
||||
]
|
||||
},
|
||||
"debug" : false,
|
||||
"container" : "docker"
|
||||
@@ -2965,13 +2973,13 @@ meta = [
|
||||
"engine" : "native|native",
|
||||
"output" : "target/nextflow/dataflow/gather_fastqs_and_validate",
|
||||
"viash_version" : "0.9.0",
|
||||
"git_commit" : "e297e8ff45464065384476fe521ab2700ec3739c",
|
||||
"git_remote" : "https://x-access-token:ghs_44flLTQIYGBkc9xcwCZoLYe7mxXzgw1iszFO@github.com/viash-hub/demultiplex",
|
||||
"git_tag" : "v0.3.2-4-ge297e8f"
|
||||
"git_commit" : "3a985c83869bcd78ba6d325f1a817514b52a132a",
|
||||
"git_remote" : "https://github.com/viash-hub/demultiplex",
|
||||
"git_tag" : "v0.3.5-3-g3a985c8"
|
||||
},
|
||||
"package_config" : {
|
||||
"name" : "demultiplex",
|
||||
"version" : "v0.3.3",
|
||||
"version" : "v0.3.6",
|
||||
"description" : "Demultiplexing pipeline\n",
|
||||
"info" : {
|
||||
"test_resources" : [
|
||||
@@ -2985,10 +2993,10 @@ meta = [
|
||||
"source" : "src",
|
||||
"target" : "target",
|
||||
"config_mods" : [
|
||||
".requirements.commands := ['ps']\n.runners[.type == 'nextflow'].directives.tag := '$id'\n",
|
||||
".requirements.commands := ['ps']\n.runners[.type == 'nextflow'].directives.tag := '$id'\n.resources += {path: '/src/config/labels.config', dest: 'nextflow_labels.config'}\n.runners[.type == 'nextflow'].config.script := 'includeConfig(\\"nextflow_labels.config\\")'\n",
|
||||
".engines += { type: \\"native\\" }",
|
||||
".engines[.type == 'docker'].target_registry := 'images.viash-hub.com'",
|
||||
".engines[.type == 'docker'].target_tag := 'v0.3.3'"
|
||||
".engines[.type == 'docker'].target_tag := 'v0.3.6'"
|
||||
],
|
||||
"keywords" : [
|
||||
"bioinformatics",
|
||||
@@ -3043,8 +3051,8 @@ workflow run_wf {
|
||||
return true
|
||||
}
|
||||
// [Data], [BCLConvert_Data] for illumina
|
||||
// [Samples] for Element Biosciences
|
||||
if (header in ["Data", "Samples", "BCLConvert_Data"]) {
|
||||
// [Samples] or sometimes [SAMPLES] for Element Biosciences
|
||||
if (header.toLowerCase() in ["data", "samples", "bclconvert_data"]) {
|
||||
println "Found header [${header}], start parsing."
|
||||
start_parsing = true
|
||||
return
|
||||
|
||||
@@ -2,7 +2,7 @@ manifest {
|
||||
name = 'dataflow/gather_fastqs_and_validate'
|
||||
mainScript = 'main.nf'
|
||||
nextflowVersion = '!>=20.12.1-edge'
|
||||
version = 'v0.3.3'
|
||||
version = 'v0.3.6'
|
||||
description = 'From a directory containing fastq files, gather the files per sample \nand validate according to the contents of the sample sheet.\n'
|
||||
}
|
||||
|
||||
@@ -122,4 +122,4 @@ process{
|
||||
withLabel: cpu1000 { cpus = 1000 }
|
||||
}
|
||||
|
||||
|
||||
includeConfig("nextflow_labels.config")
|
||||
|
||||
@@ -0,0 +1,98 @@
|
||||
process {
|
||||
container = 'nextflow/bash:latest'
|
||||
|
||||
// default resources
|
||||
memory = { 8.Gb * task.attempt }
|
||||
cpus = 8
|
||||
maxForks = 36
|
||||
|
||||
// Retry for exit codes that have something to do with memory issues
|
||||
errorStrategy = { task.exitStatus in 137..140 ? 'retry' : 'terminate' }
|
||||
maxRetries = 3
|
||||
maxMemory = 192.GB
|
||||
|
||||
// Resource labels
|
||||
withLabel: verylowcpu { cpus = 2 }
|
||||
withLabel: lowcpu { cpus = 8 }
|
||||
withLabel: midcpu { cpus = 16 }
|
||||
withLabel: highcpu { cpus = 32 }
|
||||
|
||||
withLabel: verylowmem { memory = { get_memory( 4.GB * task.attempt ) } }
|
||||
withLabel: lowmem { memory = { get_memory( 8.GB * task.attempt ) } }
|
||||
withLabel: midmem { memory = { get_memory( 16.GB * task.attempt ) } }
|
||||
withLabel: highmem { memory = { get_memory( 64.GB * task.attempt ) } }
|
||||
|
||||
}
|
||||
|
||||
profiles {
|
||||
// detect tempdir
|
||||
tempDir = java.nio.file.Paths.get(
|
||||
System.getenv('NXF_TEMP') ?:
|
||||
System.getenv('VIASH_TEMP') ?:
|
||||
System.getenv('TEMPDIR') ?:
|
||||
System.getenv('TMPDIR') ?:
|
||||
'/tmp'
|
||||
).toAbsolutePath()
|
||||
|
||||
mount_temp {
|
||||
docker.temp = tempDir
|
||||
podman.temp = tempDir
|
||||
charliecloud.temp = tempDir
|
||||
}
|
||||
|
||||
no_publish {
|
||||
process {
|
||||
withName: '.*' {
|
||||
publishDir = [
|
||||
enabled: false
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
docker {
|
||||
docker.fixOwnership = true
|
||||
docker.enabled = true
|
||||
// docker.userEmulation = true
|
||||
singularity.enabled = false
|
||||
podman.enabled = false
|
||||
shifter.enabled = false
|
||||
charliecloud.enabled = false
|
||||
}
|
||||
|
||||
local {
|
||||
// This config is for local processing.
|
||||
process {
|
||||
maxMemory = 25.GB
|
||||
withLabel: verylowcpu { cpus = 2 }
|
||||
withLabel: lowcpu { cpus = 4 }
|
||||
withLabel: midcpu { cpus = 6 }
|
||||
withLabel: highcpu { cpus = 12 }
|
||||
|
||||
withLabel: lowmem { memory = { get_memory( 8.GB * task.attempt ) } }
|
||||
withLabel: midmem { memory = { get_memory( 12.GB * task.attempt ) } }
|
||||
withLabel: highmem { memory = { get_memory( 20.GB * task.attempt ) } }
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
def get_memory(to_compare) {
|
||||
if (!process.containsKey("maxMemory") || !process.maxMemory) {
|
||||
return to_compare
|
||||
}
|
||||
|
||||
try {
|
||||
if (process.containsKey("maxRetries") && process.maxRetries && task.attempt == (process.maxRetries as int)) {
|
||||
return process.maxMemory
|
||||
}
|
||||
else if (to_compare.compareTo(process.maxMemory as nextflow.util.MemoryUnit) == 1) {
|
||||
return max_memory as nextflow.util.MemoryUnit
|
||||
}
|
||||
else {
|
||||
return to_compare
|
||||
}
|
||||
} catch (all) {
|
||||
println "Error processing memory resources. Please check that process.maxMemory '${process.maxMemory}' and process.maxRetries '${process.maxRetries}' are valid!"
|
||||
System.exit(1)
|
||||
}
|
||||
}
|
||||
@@ -1,5 +1,5 @@
|
||||
name: "demultiplex"
|
||||
version: "v0.3.3"
|
||||
version: "v0.3.6"
|
||||
argument_groups:
|
||||
- name: "Input arguments"
|
||||
arguments:
|
||||
@@ -71,7 +71,7 @@ argument_groups:
|
||||
create_parent: true
|
||||
required: false
|
||||
direction: "output"
|
||||
multiple: false
|
||||
multiple: true
|
||||
multiple_sep: ";"
|
||||
- type: "file"
|
||||
name: "--output_multiqc"
|
||||
@@ -96,11 +96,22 @@ argument_groups:
|
||||
direction: "output"
|
||||
multiple: false
|
||||
multiple_sep: ";"
|
||||
- name: "Other arguments"
|
||||
arguments:
|
||||
- type: "boolean_true"
|
||||
name: "--skip_copycomplete_check"
|
||||
description: "Disable the check for the presence of a \"CopyComplete.txt\" file\
|
||||
\ in input\ndirectory in case of Illumina data.\n"
|
||||
info: null
|
||||
direction: "input"
|
||||
resources:
|
||||
- type: "nextflow_script"
|
||||
path: "main.nf"
|
||||
is_executable: true
|
||||
entrypoint: "run_wf"
|
||||
- type: "file"
|
||||
path: "nextflow_labels.config"
|
||||
dest: "nextflow_labels.config"
|
||||
description: "Demultiplexing of raw sequencing data"
|
||||
test_resources:
|
||||
- type: "nextflow_script"
|
||||
@@ -217,6 +228,8 @@ runners:
|
||||
cpu200: "cpus = 200"
|
||||
cpu500: "cpus = 500"
|
||||
cpu1000: "cpus = 1000"
|
||||
script:
|
||||
- "includeConfig(\"nextflow_labels.config\")"
|
||||
debug: false
|
||||
container: "docker"
|
||||
engines:
|
||||
@@ -231,9 +244,9 @@ build_info:
|
||||
output: "target/nextflow/demultiplex"
|
||||
executable: "target/nextflow/demultiplex/main.nf"
|
||||
viash_version: "0.9.0"
|
||||
git_commit: "e297e8ff45464065384476fe521ab2700ec3739c"
|
||||
git_remote: "https://x-access-token:ghs_44flLTQIYGBkc9xcwCZoLYe7mxXzgw1iszFO@github.com/viash-hub/demultiplex"
|
||||
git_tag: "v0.3.2-4-ge297e8f"
|
||||
git_commit: "3a985c83869bcd78ba6d325f1a817514b52a132a"
|
||||
git_remote: "https://github.com/viash-hub/demultiplex"
|
||||
git_tag: "v0.3.5-3-g3a985c8"
|
||||
dependencies:
|
||||
- "target/nextflow/io/untar"
|
||||
- "target/nextflow/dataflow/gather_fastqs_and_validate"
|
||||
@@ -245,7 +258,7 @@ build_info:
|
||||
- "target/dependencies/vsh/vsh/biobox/v0.3.0/nextflow/multiqc"
|
||||
package_config:
|
||||
name: "demultiplex"
|
||||
version: "v0.3.3"
|
||||
version: "v0.3.6"
|
||||
description: "Demultiplexing pipeline\n"
|
||||
info:
|
||||
test_resources:
|
||||
@@ -256,10 +269,12 @@ package_config:
|
||||
target: "target"
|
||||
config_mods:
|
||||
- ".requirements.commands := ['ps']\n.runners[.type == 'nextflow'].directives.tag\
|
||||
\ := '$id'\n"
|
||||
\ := '$id'\n.resources += {path: '/src/config/labels.config', dest: 'nextflow_labels.config'}\n\
|
||||
.runners[.type == 'nextflow'].config.script := 'includeConfig(\"nextflow_labels.config\"\
|
||||
)'\n"
|
||||
- ".engines += { type: \"native\" }"
|
||||
- ".engines[.type == 'docker'].target_registry := 'images.viash-hub.com'"
|
||||
- ".engines[.type == 'docker'].target_tag := 'v0.3.3'"
|
||||
- ".engines[.type == 'docker'].target_tag := 'v0.3.6'"
|
||||
keywords:
|
||||
- "bioinformatics"
|
||||
- "sequence"
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
// demultiplex v0.3.3
|
||||
// demultiplex v0.3.6
|
||||
//
|
||||
// This wrapper script is auto-generated by viash 0.9.0 and is thus a derivative
|
||||
// work thereof. This software comes with ABSOLUTELY NO WARRANTY from Data
|
||||
@@ -2805,7 +2805,7 @@ meta = [
|
||||
"resources_dir": moduleDir.toRealPath().normalize(),
|
||||
"config": processConfig(readJsonBlob('''{
|
||||
"name" : "demultiplex",
|
||||
"version" : "v0.3.3",
|
||||
"version" : "v0.3.6",
|
||||
"argument_groups" : [
|
||||
{
|
||||
"name" : "Input arguments",
|
||||
@@ -2884,7 +2884,7 @@ meta = [
|
||||
"create_parent" : true,
|
||||
"required" : false,
|
||||
"direction" : "output",
|
||||
"multiple" : false,
|
||||
"multiple" : true,
|
||||
"multiple_sep" : ";"
|
||||
},
|
||||
{
|
||||
@@ -2915,6 +2915,17 @@ meta = [
|
||||
"multiple_sep" : ";"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"name" : "Other arguments",
|
||||
"arguments" : [
|
||||
{
|
||||
"type" : "boolean_true",
|
||||
"name" : "--skip_copycomplete_check",
|
||||
"description" : "Disable the check for the presence of a \\"CopyComplete.txt\\" file in input\ndirectory in case of Illumina data.\n",
|
||||
"direction" : "input"
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"resources" : [
|
||||
@@ -2923,6 +2934,11 @@ meta = [
|
||||
"path" : "main.nf",
|
||||
"is_executable" : true,
|
||||
"entrypoint" : "run_wf"
|
||||
},
|
||||
{
|
||||
"type" : "file",
|
||||
"path" : "/src/config/labels.config",
|
||||
"dest" : "nextflow_labels.config"
|
||||
}
|
||||
],
|
||||
"description" : "Demultiplexing of raw sequencing data",
|
||||
@@ -3079,7 +3095,10 @@ meta = [
|
||||
"cpu200" : "cpus = 200",
|
||||
"cpu500" : "cpus = 500",
|
||||
"cpu1000" : "cpus = 1000"
|
||||
}
|
||||
},
|
||||
"script" : [
|
||||
"includeConfig(\\"nextflow_labels.config\\")"
|
||||
]
|
||||
},
|
||||
"debug" : false,
|
||||
"container" : "docker"
|
||||
@@ -3101,13 +3120,13 @@ meta = [
|
||||
"engine" : "native|native",
|
||||
"output" : "target/nextflow/demultiplex",
|
||||
"viash_version" : "0.9.0",
|
||||
"git_commit" : "e297e8ff45464065384476fe521ab2700ec3739c",
|
||||
"git_remote" : "https://x-access-token:ghs_44flLTQIYGBkc9xcwCZoLYe7mxXzgw1iszFO@github.com/viash-hub/demultiplex",
|
||||
"git_tag" : "v0.3.2-4-ge297e8f"
|
||||
"git_commit" : "3a985c83869bcd78ba6d325f1a817514b52a132a",
|
||||
"git_remote" : "https://github.com/viash-hub/demultiplex",
|
||||
"git_tag" : "v0.3.5-3-g3a985c8"
|
||||
},
|
||||
"package_config" : {
|
||||
"name" : "demultiplex",
|
||||
"version" : "v0.3.3",
|
||||
"version" : "v0.3.6",
|
||||
"description" : "Demultiplexing pipeline\n",
|
||||
"info" : {
|
||||
"test_resources" : [
|
||||
@@ -3121,10 +3140,10 @@ meta = [
|
||||
"source" : "src",
|
||||
"target" : "target",
|
||||
"config_mods" : [
|
||||
".requirements.commands := ['ps']\n.runners[.type == 'nextflow'].directives.tag := '$id'\n",
|
||||
".requirements.commands := ['ps']\n.runners[.type == 'nextflow'].directives.tag := '$id'\n.resources += {path: '/src/config/labels.config', dest: 'nextflow_labels.config'}\n.runners[.type == 'nextflow'].config.script := 'includeConfig(\\"nextflow_labels.config\\")'\n",
|
||||
".engines += { type: \\"native\\" }",
|
||||
".engines[.type == 'docker'].target_registry := 'images.viash-hub.com'",
|
||||
".engines[.type == 'docker'].target_tag := 'v0.3.3'"
|
||||
".engines[.type == 'docker'].target_tag := 'v0.3.6'"
|
||||
],
|
||||
"keywords" : [
|
||||
"bioinformatics",
|
||||
@@ -3251,6 +3270,10 @@ workflow run_wf {
|
||||
// step based on the run dir, not the InterOp dir.
|
||||
def interop_dir = state.input.resolve("InterOp")
|
||||
assert interop_dir.isDirectory(): "Expected InterOp directory to be present."
|
||||
|
||||
def copycomplete_file = state.input.resolve("CopyComplete.txt")
|
||||
assert (copycomplete_file.isFile() || state.skip_copycomplete_check):
|
||||
"'CopyComplete.txt' file was not found!"
|
||||
}
|
||||
|
||||
def resultState = state + newState
|
||||
@@ -3326,28 +3349,12 @@ workflow run_wf {
|
||||
)
|
||||
|
||||
output_ch = samples_ch
|
||||
|
||||
|
||||
| combine_samples.run(
|
||||
fromState: { id, state ->
|
||||
[
|
||||
"id": state.run_id,
|
||||
"forward_input": state.fastq_forward,
|
||||
"reverse_input": state.fastq_reverse,
|
||||
]
|
||||
},
|
||||
toState: [
|
||||
"forward_fastqs": "output_forward",
|
||||
"reverse_fastqs": "output_reverse",
|
||||
]
|
||||
)
|
||||
| falco.run(
|
||||
directives: [label: ["lowcpu", "lowmem"]],
|
||||
directives: [label: ["verylowcpu", "lowmem"]],
|
||||
fromState: {id, state ->
|
||||
reverse_fastqs_list = state.reverse_fastqs ? state.reverse_fastqs : []
|
||||
[
|
||||
"input": state.forward_fastqs + reverse_fastqs_list,
|
||||
"outdir": "${state.output_falco}",
|
||||
"input": [state.fastq_forward, state.fastq_reverse],
|
||||
"outdir": "$id/qc/falco",
|
||||
"summary_filename": null,
|
||||
"report_filename": null,
|
||||
"data_filename": null,
|
||||
@@ -3357,11 +3364,28 @@ workflow run_wf {
|
||||
state + [ "output_falco" : result.outdir ]
|
||||
}
|
||||
)
|
||||
|
||||
| combine_samples.run(
|
||||
fromState: { id, state ->
|
||||
[
|
||||
"id": state.run_id,
|
||||
"forward_input": state.fastq_forward,
|
||||
"reverse_input": state.fastq_reverse,
|
||||
"falco_dir": state.output_falco,
|
||||
]
|
||||
},
|
||||
toState: [
|
||||
"forward_fastqs": "output_forward",
|
||||
"reverse_fastqs": "output_reverse",
|
||||
"output_falco": "output_falco",
|
||||
]
|
||||
)
|
||||
|
||||
| multiqc.run(
|
||||
directives: [label: ["midcpu", "midmem"]],
|
||||
fromState: {id, state ->
|
||||
def new_state = [
|
||||
"input": [state.output_falco],
|
||||
"input": state.output_falco,
|
||||
"output_report": state.output_multiqc,
|
||||
"cl_config": 'sp: {fastqc/data: {fn: "*_fastqc_data.txt"}}'
|
||||
]
|
||||
|
||||
@@ -2,7 +2,7 @@ manifest {
|
||||
name = 'demultiplex'
|
||||
mainScript = 'main.nf'
|
||||
nextflowVersion = '!>=20.12.1-edge'
|
||||
version = 'v0.3.3'
|
||||
version = 'v0.3.6'
|
||||
description = 'Demultiplexing of raw sequencing data'
|
||||
}
|
||||
|
||||
@@ -122,4 +122,4 @@ process{
|
||||
withLabel: cpu1000 { cpus = 1000 }
|
||||
}
|
||||
|
||||
|
||||
includeConfig("nextflow_labels.config")
|
||||
|
||||
98
target/nextflow/demultiplex/nextflow_labels.config
Normal file
98
target/nextflow/demultiplex/nextflow_labels.config
Normal file
@@ -0,0 +1,98 @@
|
||||
process {
|
||||
container = 'nextflow/bash:latest'
|
||||
|
||||
// default resources
|
||||
memory = { 8.Gb * task.attempt }
|
||||
cpus = 8
|
||||
maxForks = 36
|
||||
|
||||
// Retry for exit codes that have something to do with memory issues
|
||||
errorStrategy = { task.exitStatus in 137..140 ? 'retry' : 'terminate' }
|
||||
maxRetries = 3
|
||||
maxMemory = 192.GB
|
||||
|
||||
// Resource labels
|
||||
withLabel: verylowcpu { cpus = 2 }
|
||||
withLabel: lowcpu { cpus = 8 }
|
||||
withLabel: midcpu { cpus = 16 }
|
||||
withLabel: highcpu { cpus = 32 }
|
||||
|
||||
withLabel: verylowmem { memory = { get_memory( 4.GB * task.attempt ) } }
|
||||
withLabel: lowmem { memory = { get_memory( 8.GB * task.attempt ) } }
|
||||
withLabel: midmem { memory = { get_memory( 16.GB * task.attempt ) } }
|
||||
withLabel: highmem { memory = { get_memory( 64.GB * task.attempt ) } }
|
||||
|
||||
}
|
||||
|
||||
profiles {
|
||||
// detect tempdir
|
||||
tempDir = java.nio.file.Paths.get(
|
||||
System.getenv('NXF_TEMP') ?:
|
||||
System.getenv('VIASH_TEMP') ?:
|
||||
System.getenv('TEMPDIR') ?:
|
||||
System.getenv('TMPDIR') ?:
|
||||
'/tmp'
|
||||
).toAbsolutePath()
|
||||
|
||||
mount_temp {
|
||||
docker.temp = tempDir
|
||||
podman.temp = tempDir
|
||||
charliecloud.temp = tempDir
|
||||
}
|
||||
|
||||
no_publish {
|
||||
process {
|
||||
withName: '.*' {
|
||||
publishDir = [
|
||||
enabled: false
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
docker {
|
||||
docker.fixOwnership = true
|
||||
docker.enabled = true
|
||||
// docker.userEmulation = true
|
||||
singularity.enabled = false
|
||||
podman.enabled = false
|
||||
shifter.enabled = false
|
||||
charliecloud.enabled = false
|
||||
}
|
||||
|
||||
local {
|
||||
// This config is for local processing.
|
||||
process {
|
||||
maxMemory = 25.GB
|
||||
withLabel: verylowcpu { cpus = 2 }
|
||||
withLabel: lowcpu { cpus = 4 }
|
||||
withLabel: midcpu { cpus = 6 }
|
||||
withLabel: highcpu { cpus = 12 }
|
||||
|
||||
withLabel: lowmem { memory = { get_memory( 8.GB * task.attempt ) } }
|
||||
withLabel: midmem { memory = { get_memory( 12.GB * task.attempt ) } }
|
||||
withLabel: highmem { memory = { get_memory( 20.GB * task.attempt ) } }
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
def get_memory(to_compare) {
|
||||
if (!process.containsKey("maxMemory") || !process.maxMemory) {
|
||||
return to_compare
|
||||
}
|
||||
|
||||
try {
|
||||
if (process.containsKey("maxRetries") && process.maxRetries && task.attempt == (process.maxRetries as int)) {
|
||||
return process.maxMemory
|
||||
}
|
||||
else if (to_compare.compareTo(process.maxMemory as nextflow.util.MemoryUnit) == 1) {
|
||||
return max_memory as nextflow.util.MemoryUnit
|
||||
}
|
||||
else {
|
||||
return to_compare
|
||||
}
|
||||
} catch (all) {
|
||||
println "Error processing memory resources. Please check that process.maxMemory '${process.maxMemory}' and process.maxRetries '${process.maxRetries}' are valid!"
|
||||
System.exit(1)
|
||||
}
|
||||
}
|
||||
@@ -80,10 +80,10 @@
|
||||
"output_falco": {
|
||||
"type":
|
||||
"string",
|
||||
"description": "Type: `file`, default: `$id.$key.output_falco.output_falco`. Directory to write falco output to",
|
||||
"help_text": "Type: `file`, default: `$id.$key.output_falco.output_falco`. Directory to write falco output to"
|
||||
"description": "Type: List of `file`, default: `$id.$key.output_falco_*.output_falco_*`, multiple_sep: `\";\"`. Directory to write falco output to",
|
||||
"help_text": "Type: List of `file`, default: `$id.$key.output_falco_*.output_falco_*`, multiple_sep: `\";\"`. Directory to write falco output to"
|
||||
,
|
||||
"default":"$id.$key.output_falco.output_falco"
|
||||
"default":"$id.$key.output_falco_*.output_falco_*"
|
||||
}
|
||||
|
||||
|
||||
@@ -109,6 +109,27 @@
|
||||
}
|
||||
|
||||
|
||||
}
|
||||
},
|
||||
|
||||
|
||||
"other arguments" : {
|
||||
"title": "Other arguments",
|
||||
"type": "object",
|
||||
"description": "No description",
|
||||
"properties": {
|
||||
|
||||
|
||||
"skip_copycomplete_check": {
|
||||
"type":
|
||||
"boolean",
|
||||
"description": "Type: `boolean_true`, default: `false`. Disable the check for the presence of a \"CopyComplete",
|
||||
"help_text": "Type: `boolean_true`, default: `false`. Disable the check for the presence of a \"CopyComplete.txt\" file in input\ndirectory in case of Illumina data.\n"
|
||||
,
|
||||
"default":false
|
||||
}
|
||||
|
||||
|
||||
}
|
||||
},
|
||||
|
||||
@@ -153,6 +174,10 @@
|
||||
"$ref": "#/definitions/output arguments"
|
||||
},
|
||||
|
||||
{
|
||||
"$ref": "#/definitions/other arguments"
|
||||
},
|
||||
|
||||
{
|
||||
"$ref": "#/definitions/nextflow input-output arguments"
|
||||
}
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
name: "interop_summary_to_csv"
|
||||
namespace: "io"
|
||||
version: "v0.3.3"
|
||||
version: "v0.3.6"
|
||||
argument_groups:
|
||||
- name: "Input arguments"
|
||||
arguments:
|
||||
@@ -38,6 +38,9 @@ resources:
|
||||
- type: "bash_script"
|
||||
path: "script.sh"
|
||||
is_executable: true
|
||||
- type: "file"
|
||||
path: "nextflow_labels.config"
|
||||
dest: "nextflow_labels.config"
|
||||
info: null
|
||||
status: "enabled"
|
||||
requirements:
|
||||
@@ -109,6 +112,8 @@ runners:
|
||||
cpu200: "cpus = 200"
|
||||
cpu500: "cpus = 500"
|
||||
cpu1000: "cpus = 1000"
|
||||
script:
|
||||
- "includeConfig(\"nextflow_labels.config\")"
|
||||
debug: false
|
||||
container: "docker"
|
||||
engines:
|
||||
@@ -116,7 +121,7 @@ engines:
|
||||
id: "docker"
|
||||
image: "debian:stable-slim"
|
||||
target_registry: "images.viash-hub.com"
|
||||
target_tag: "v0.3.3"
|
||||
target_tag: "v0.3.6"
|
||||
namespace_separator: "/"
|
||||
setup:
|
||||
- type: "apt"
|
||||
@@ -141,12 +146,12 @@ build_info:
|
||||
output: "target/nextflow/io/interop_summary_to_csv"
|
||||
executable: "target/nextflow/io/interop_summary_to_csv/main.nf"
|
||||
viash_version: "0.9.0"
|
||||
git_commit: "e297e8ff45464065384476fe521ab2700ec3739c"
|
||||
git_remote: "https://x-access-token:ghs_44flLTQIYGBkc9xcwCZoLYe7mxXzgw1iszFO@github.com/viash-hub/demultiplex"
|
||||
git_tag: "v0.3.2-4-ge297e8f"
|
||||
git_commit: "3a985c83869bcd78ba6d325f1a817514b52a132a"
|
||||
git_remote: "https://github.com/viash-hub/demultiplex"
|
||||
git_tag: "v0.3.5-3-g3a985c8"
|
||||
package_config:
|
||||
name: "demultiplex"
|
||||
version: "v0.3.3"
|
||||
version: "v0.3.6"
|
||||
description: "Demultiplexing pipeline\n"
|
||||
info:
|
||||
test_resources:
|
||||
@@ -157,10 +162,12 @@ package_config:
|
||||
target: "target"
|
||||
config_mods:
|
||||
- ".requirements.commands := ['ps']\n.runners[.type == 'nextflow'].directives.tag\
|
||||
\ := '$id'\n"
|
||||
\ := '$id'\n.resources += {path: '/src/config/labels.config', dest: 'nextflow_labels.config'}\n\
|
||||
.runners[.type == 'nextflow'].config.script := 'includeConfig(\"nextflow_labels.config\"\
|
||||
)'\n"
|
||||
- ".engines += { type: \"native\" }"
|
||||
- ".engines[.type == 'docker'].target_registry := 'images.viash-hub.com'"
|
||||
- ".engines[.type == 'docker'].target_tag := 'v0.3.3'"
|
||||
- ".engines[.type == 'docker'].target_tag := 'v0.3.6'"
|
||||
keywords:
|
||||
- "bioinformatics"
|
||||
- "sequence"
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
// interop_summary_to_csv v0.3.3
|
||||
// interop_summary_to_csv v0.3.6
|
||||
//
|
||||
// This wrapper script is auto-generated by viash 0.9.0 and is thus a derivative
|
||||
// work thereof. This software comes with ABSOLUTELY NO WARRANTY from Data
|
||||
@@ -2806,7 +2806,7 @@ meta = [
|
||||
"config": processConfig(readJsonBlob('''{
|
||||
"name" : "interop_summary_to_csv",
|
||||
"namespace" : "io",
|
||||
"version" : "v0.3.3",
|
||||
"version" : "v0.3.6",
|
||||
"argument_groups" : [
|
||||
{
|
||||
"name" : "Input arguments",
|
||||
@@ -2855,6 +2855,11 @@ meta = [
|
||||
"type" : "bash_script",
|
||||
"path" : "script.sh",
|
||||
"is_executable" : true
|
||||
},
|
||||
{
|
||||
"type" : "file",
|
||||
"path" : "/src/config/labels.config",
|
||||
"dest" : "nextflow_labels.config"
|
||||
}
|
||||
],
|
||||
"status" : "enabled",
|
||||
@@ -2935,7 +2940,10 @@ meta = [
|
||||
"cpu200" : "cpus = 200",
|
||||
"cpu500" : "cpus = 500",
|
||||
"cpu1000" : "cpus = 1000"
|
||||
}
|
||||
},
|
||||
"script" : [
|
||||
"includeConfig(\\"nextflow_labels.config\\")"
|
||||
]
|
||||
},
|
||||
"debug" : false,
|
||||
"container" : "docker"
|
||||
@@ -2947,7 +2955,7 @@ meta = [
|
||||
"id" : "docker",
|
||||
"image" : "debian:stable-slim",
|
||||
"target_registry" : "images.viash-hub.com",
|
||||
"target_tag" : "v0.3.3",
|
||||
"target_tag" : "v0.3.6",
|
||||
"namespace_separator" : "/",
|
||||
"setup" : [
|
||||
{
|
||||
@@ -2977,13 +2985,13 @@ meta = [
|
||||
"engine" : "docker|native",
|
||||
"output" : "target/nextflow/io/interop_summary_to_csv",
|
||||
"viash_version" : "0.9.0",
|
||||
"git_commit" : "e297e8ff45464065384476fe521ab2700ec3739c",
|
||||
"git_remote" : "https://x-access-token:ghs_44flLTQIYGBkc9xcwCZoLYe7mxXzgw1iszFO@github.com/viash-hub/demultiplex",
|
||||
"git_tag" : "v0.3.2-4-ge297e8f"
|
||||
"git_commit" : "3a985c83869bcd78ba6d325f1a817514b52a132a",
|
||||
"git_remote" : "https://github.com/viash-hub/demultiplex",
|
||||
"git_tag" : "v0.3.5-3-g3a985c8"
|
||||
},
|
||||
"package_config" : {
|
||||
"name" : "demultiplex",
|
||||
"version" : "v0.3.3",
|
||||
"version" : "v0.3.6",
|
||||
"description" : "Demultiplexing pipeline\n",
|
||||
"info" : {
|
||||
"test_resources" : [
|
||||
@@ -2997,10 +3005,10 @@ meta = [
|
||||
"source" : "src",
|
||||
"target" : "target",
|
||||
"config_mods" : [
|
||||
".requirements.commands := ['ps']\n.runners[.type == 'nextflow'].directives.tag := '$id'\n",
|
||||
".requirements.commands := ['ps']\n.runners[.type == 'nextflow'].directives.tag := '$id'\n.resources += {path: '/src/config/labels.config', dest: 'nextflow_labels.config'}\n.runners[.type == 'nextflow'].config.script := 'includeConfig(\\"nextflow_labels.config\\")'\n",
|
||||
".engines += { type: \\"native\\" }",
|
||||
".engines[.type == 'docker'].target_registry := 'images.viash-hub.com'",
|
||||
".engines[.type == 'docker'].target_tag := 'v0.3.3'"
|
||||
".engines[.type == 'docker'].target_tag := 'v0.3.6'"
|
||||
],
|
||||
"keywords" : [
|
||||
"bioinformatics",
|
||||
@@ -3423,7 +3431,7 @@ meta["defaults"] = [
|
||||
"container" : {
|
||||
"registry" : "images.viash-hub.com",
|
||||
"image" : "vsh/demultiplex/io/interop_summary_to_csv",
|
||||
"tag" : "v0.3.3"
|
||||
"tag" : "v0.3.6"
|
||||
},
|
||||
"tag" : "$id"
|
||||
}'''),
|
||||
|
||||
@@ -2,7 +2,7 @@ manifest {
|
||||
name = 'io/interop_summary_to_csv'
|
||||
mainScript = 'main.nf'
|
||||
nextflowVersion = '!>=20.12.1-edge'
|
||||
version = 'v0.3.3'
|
||||
version = 'v0.3.6'
|
||||
}
|
||||
|
||||
process.container = 'nextflow/bash:latest'
|
||||
@@ -121,4 +121,4 @@ process{
|
||||
withLabel: cpu1000 { cpus = 1000 }
|
||||
}
|
||||
|
||||
|
||||
includeConfig("nextflow_labels.config")
|
||||
|
||||
@@ -0,0 +1,98 @@
|
||||
process {
|
||||
container = 'nextflow/bash:latest'
|
||||
|
||||
// default resources
|
||||
memory = { 8.Gb * task.attempt }
|
||||
cpus = 8
|
||||
maxForks = 36
|
||||
|
||||
// Retry for exit codes that have something to do with memory issues
|
||||
errorStrategy = { task.exitStatus in 137..140 ? 'retry' : 'terminate' }
|
||||
maxRetries = 3
|
||||
maxMemory = 192.GB
|
||||
|
||||
// Resource labels
|
||||
withLabel: verylowcpu { cpus = 2 }
|
||||
withLabel: lowcpu { cpus = 8 }
|
||||
withLabel: midcpu { cpus = 16 }
|
||||
withLabel: highcpu { cpus = 32 }
|
||||
|
||||
withLabel: verylowmem { memory = { get_memory( 4.GB * task.attempt ) } }
|
||||
withLabel: lowmem { memory = { get_memory( 8.GB * task.attempt ) } }
|
||||
withLabel: midmem { memory = { get_memory( 16.GB * task.attempt ) } }
|
||||
withLabel: highmem { memory = { get_memory( 64.GB * task.attempt ) } }
|
||||
|
||||
}
|
||||
|
||||
profiles {
|
||||
// detect tempdir
|
||||
tempDir = java.nio.file.Paths.get(
|
||||
System.getenv('NXF_TEMP') ?:
|
||||
System.getenv('VIASH_TEMP') ?:
|
||||
System.getenv('TEMPDIR') ?:
|
||||
System.getenv('TMPDIR') ?:
|
||||
'/tmp'
|
||||
).toAbsolutePath()
|
||||
|
||||
mount_temp {
|
||||
docker.temp = tempDir
|
||||
podman.temp = tempDir
|
||||
charliecloud.temp = tempDir
|
||||
}
|
||||
|
||||
no_publish {
|
||||
process {
|
||||
withName: '.*' {
|
||||
publishDir = [
|
||||
enabled: false
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
docker {
|
||||
docker.fixOwnership = true
|
||||
docker.enabled = true
|
||||
// docker.userEmulation = true
|
||||
singularity.enabled = false
|
||||
podman.enabled = false
|
||||
shifter.enabled = false
|
||||
charliecloud.enabled = false
|
||||
}
|
||||
|
||||
local {
|
||||
// This config is for local processing.
|
||||
process {
|
||||
maxMemory = 25.GB
|
||||
withLabel: verylowcpu { cpus = 2 }
|
||||
withLabel: lowcpu { cpus = 4 }
|
||||
withLabel: midcpu { cpus = 6 }
|
||||
withLabel: highcpu { cpus = 12 }
|
||||
|
||||
withLabel: lowmem { memory = { get_memory( 8.GB * task.attempt ) } }
|
||||
withLabel: midmem { memory = { get_memory( 12.GB * task.attempt ) } }
|
||||
withLabel: highmem { memory = { get_memory( 20.GB * task.attempt ) } }
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
def get_memory(to_compare) {
|
||||
if (!process.containsKey("maxMemory") || !process.maxMemory) {
|
||||
return to_compare
|
||||
}
|
||||
|
||||
try {
|
||||
if (process.containsKey("maxRetries") && process.maxRetries && task.attempt == (process.maxRetries as int)) {
|
||||
return process.maxMemory
|
||||
}
|
||||
else if (to_compare.compareTo(process.maxMemory as nextflow.util.MemoryUnit) == 1) {
|
||||
return max_memory as nextflow.util.MemoryUnit
|
||||
}
|
||||
else {
|
||||
return to_compare
|
||||
}
|
||||
} catch (all) {
|
||||
println "Error processing memory resources. Please check that process.maxMemory '${process.maxMemory}' and process.maxRetries '${process.maxRetries}' are valid!"
|
||||
System.exit(1)
|
||||
}
|
||||
}
|
||||
@@ -1,6 +1,6 @@
|
||||
name: "publish"
|
||||
namespace: "io"
|
||||
version: "v0.3.3"
|
||||
version: "v0.3.6"
|
||||
argument_groups:
|
||||
- name: "Input arguments"
|
||||
arguments:
|
||||
@@ -22,7 +22,7 @@ argument_groups:
|
||||
create_parent: true
|
||||
required: true
|
||||
direction: "input"
|
||||
multiple: false
|
||||
multiple: true
|
||||
multiple_sep: ";"
|
||||
- type: "file"
|
||||
name: "--input_multiqc"
|
||||
@@ -94,6 +94,9 @@ resources:
|
||||
- type: "bash_script"
|
||||
path: "code.sh"
|
||||
is_executable: true
|
||||
- type: "file"
|
||||
path: "nextflow_labels.config"
|
||||
dest: "nextflow_labels.config"
|
||||
description: "Publish the processed results of the run"
|
||||
info: null
|
||||
status: "enabled"
|
||||
@@ -166,6 +169,8 @@ runners:
|
||||
cpu200: "cpus = 200"
|
||||
cpu500: "cpus = 500"
|
||||
cpu1000: "cpus = 1000"
|
||||
script:
|
||||
- "includeConfig(\"nextflow_labels.config\")"
|
||||
debug: false
|
||||
container: "docker"
|
||||
engines:
|
||||
@@ -173,7 +178,7 @@ engines:
|
||||
id: "docker"
|
||||
image: "debian:stable-slim"
|
||||
target_registry: "images.viash-hub.com"
|
||||
target_tag: "v0.3.3"
|
||||
target_tag: "v0.3.6"
|
||||
namespace_separator: "/"
|
||||
setup:
|
||||
- type: "apt"
|
||||
@@ -191,12 +196,12 @@ build_info:
|
||||
output: "target/nextflow/io/publish"
|
||||
executable: "target/nextflow/io/publish/main.nf"
|
||||
viash_version: "0.9.0"
|
||||
git_commit: "e297e8ff45464065384476fe521ab2700ec3739c"
|
||||
git_remote: "https://x-access-token:ghs_44flLTQIYGBkc9xcwCZoLYe7mxXzgw1iszFO@github.com/viash-hub/demultiplex"
|
||||
git_tag: "v0.3.2-4-ge297e8f"
|
||||
git_commit: "3a985c83869bcd78ba6d325f1a817514b52a132a"
|
||||
git_remote: "https://github.com/viash-hub/demultiplex"
|
||||
git_tag: "v0.3.5-3-g3a985c8"
|
||||
package_config:
|
||||
name: "demultiplex"
|
||||
version: "v0.3.3"
|
||||
version: "v0.3.6"
|
||||
description: "Demultiplexing pipeline\n"
|
||||
info:
|
||||
test_resources:
|
||||
@@ -207,10 +212,12 @@ package_config:
|
||||
target: "target"
|
||||
config_mods:
|
||||
- ".requirements.commands := ['ps']\n.runners[.type == 'nextflow'].directives.tag\
|
||||
\ := '$id'\n"
|
||||
\ := '$id'\n.resources += {path: '/src/config/labels.config', dest: 'nextflow_labels.config'}\n\
|
||||
.runners[.type == 'nextflow'].config.script := 'includeConfig(\"nextflow_labels.config\"\
|
||||
)'\n"
|
||||
- ".engines += { type: \"native\" }"
|
||||
- ".engines[.type == 'docker'].target_registry := 'images.viash-hub.com'"
|
||||
- ".engines[.type == 'docker'].target_tag := 'v0.3.3'"
|
||||
- ".engines[.type == 'docker'].target_tag := 'v0.3.6'"
|
||||
keywords:
|
||||
- "bioinformatics"
|
||||
- "sequence"
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
// publish v0.3.3
|
||||
// publish v0.3.6
|
||||
//
|
||||
// This wrapper script is auto-generated by viash 0.9.0 and is thus a derivative
|
||||
// work thereof. This software comes with ABSOLUTELY NO WARRANTY from Data
|
||||
@@ -2806,7 +2806,7 @@ meta = [
|
||||
"config": processConfig(readJsonBlob('''{
|
||||
"name" : "publish",
|
||||
"namespace" : "io",
|
||||
"version" : "v0.3.3",
|
||||
"version" : "v0.3.6",
|
||||
"argument_groups" : [
|
||||
{
|
||||
"name" : "Input arguments",
|
||||
@@ -2830,7 +2830,7 @@ meta = [
|
||||
"create_parent" : true,
|
||||
"required" : true,
|
||||
"direction" : "input",
|
||||
"multiple" : false,
|
||||
"multiple" : true,
|
||||
"multiple_sep" : ";"
|
||||
},
|
||||
{
|
||||
@@ -2920,6 +2920,11 @@ meta = [
|
||||
"type" : "bash_script",
|
||||
"path" : "./code.sh",
|
||||
"is_executable" : true
|
||||
},
|
||||
{
|
||||
"type" : "file",
|
||||
"path" : "/src/config/labels.config",
|
||||
"dest" : "nextflow_labels.config"
|
||||
}
|
||||
],
|
||||
"description" : "Publish the processed results of the run",
|
||||
@@ -3001,7 +3006,10 @@ meta = [
|
||||
"cpu200" : "cpus = 200",
|
||||
"cpu500" : "cpus = 500",
|
||||
"cpu1000" : "cpus = 1000"
|
||||
}
|
||||
},
|
||||
"script" : [
|
||||
"includeConfig(\\"nextflow_labels.config\\")"
|
||||
]
|
||||
},
|
||||
"debug" : false,
|
||||
"container" : "docker"
|
||||
@@ -3013,7 +3021,7 @@ meta = [
|
||||
"id" : "docker",
|
||||
"image" : "debian:stable-slim",
|
||||
"target_registry" : "images.viash-hub.com",
|
||||
"target_tag" : "v0.3.3",
|
||||
"target_tag" : "v0.3.6",
|
||||
"namespace_separator" : "/",
|
||||
"setup" : [
|
||||
{
|
||||
@@ -3036,13 +3044,13 @@ meta = [
|
||||
"engine" : "docker|native",
|
||||
"output" : "target/nextflow/io/publish",
|
||||
"viash_version" : "0.9.0",
|
||||
"git_commit" : "e297e8ff45464065384476fe521ab2700ec3739c",
|
||||
"git_remote" : "https://x-access-token:ghs_44flLTQIYGBkc9xcwCZoLYe7mxXzgw1iszFO@github.com/viash-hub/demultiplex",
|
||||
"git_tag" : "v0.3.2-4-ge297e8f"
|
||||
"git_commit" : "3a985c83869bcd78ba6d325f1a817514b52a132a",
|
||||
"git_remote" : "https://github.com/viash-hub/demultiplex",
|
||||
"git_tag" : "v0.3.5-3-g3a985c8"
|
||||
},
|
||||
"package_config" : {
|
||||
"name" : "demultiplex",
|
||||
"version" : "v0.3.3",
|
||||
"version" : "v0.3.6",
|
||||
"description" : "Demultiplexing pipeline\n",
|
||||
"info" : {
|
||||
"test_resources" : [
|
||||
@@ -3056,10 +3064,10 @@ meta = [
|
||||
"source" : "src",
|
||||
"target" : "target",
|
||||
"config_mods" : [
|
||||
".requirements.commands := ['ps']\n.runners[.type == 'nextflow'].directives.tag := '$id'\n",
|
||||
".requirements.commands := ['ps']\n.runners[.type == 'nextflow'].directives.tag := '$id'\n.resources += {path: '/src/config/labels.config', dest: 'nextflow_labels.config'}\n.runners[.type == 'nextflow'].config.script := 'includeConfig(\\"nextflow_labels.config\\")'\n",
|
||||
".engines += { type: \\"native\\" }",
|
||||
".engines[.type == 'docker'].target_registry := 'images.viash-hub.com'",
|
||||
".engines[.type == 'docker'].target_tag := 'v0.3.3'"
|
||||
".engines[.type == 'docker'].target_tag := 'v0.3.6'"
|
||||
],
|
||||
"keywords" : [
|
||||
"bioinformatics",
|
||||
@@ -3121,7 +3129,6 @@ $( if [ ! -z ${VIASH_META_MEMORY_PIB+x} ]; then echo "${VIASH_META_MEMORY_PIB}"
|
||||
set -eo pipefail
|
||||
|
||||
declare -A input_output_mapping=(["par_input"]="par_output"
|
||||
["par_input_falco"]="par_output_falco"
|
||||
["par_input_multiqc"]="par_output_multiqc"
|
||||
["par_input_run_information"]="par_output_run_information"
|
||||
)
|
||||
@@ -3142,6 +3149,14 @@ do
|
||||
echo "Output files for \\$output_location:"
|
||||
ls "\\$output_location"
|
||||
done
|
||||
|
||||
echo "Grouping output from \\$par_input_falco into \\$par_output_falco"
|
||||
mkdir -p "\\$par_output_falco"
|
||||
IFS=";" read -ra falco_inputs <<< \\$par_input_falco
|
||||
for falco_dir in "\\${falco_inputs[@]}"; do
|
||||
echo "Copying contents of \\$falco_dir"
|
||||
find -H -D exec "\\$falco_dir" -type f -maxdepth 1 -exec cp -t "\\$par_output_falco" {} +
|
||||
done
|
||||
VIASHMAIN
|
||||
bash "$tempscript"
|
||||
'''
|
||||
@@ -3503,7 +3518,7 @@ meta["defaults"] = [
|
||||
"container" : {
|
||||
"registry" : "images.viash-hub.com",
|
||||
"image" : "vsh/demultiplex/io/publish",
|
||||
"tag" : "v0.3.3"
|
||||
"tag" : "v0.3.6"
|
||||
},
|
||||
"tag" : "$id"
|
||||
}'''),
|
||||
|
||||
@@ -2,7 +2,7 @@ manifest {
|
||||
name = 'io/publish'
|
||||
mainScript = 'main.nf'
|
||||
nextflowVersion = '!>=20.12.1-edge'
|
||||
version = 'v0.3.3'
|
||||
version = 'v0.3.6'
|
||||
description = 'Publish the processed results of the run'
|
||||
}
|
||||
|
||||
@@ -122,4 +122,4 @@ process{
|
||||
withLabel: cpu1000 { cpus = 1000 }
|
||||
}
|
||||
|
||||
|
||||
includeConfig("nextflow_labels.config")
|
||||
|
||||
98
target/nextflow/io/publish/nextflow_labels.config
Normal file
98
target/nextflow/io/publish/nextflow_labels.config
Normal file
@@ -0,0 +1,98 @@
|
||||
process {
|
||||
container = 'nextflow/bash:latest'
|
||||
|
||||
// default resources
|
||||
memory = { 8.Gb * task.attempt }
|
||||
cpus = 8
|
||||
maxForks = 36
|
||||
|
||||
// Retry for exit codes that have something to do with memory issues
|
||||
errorStrategy = { task.exitStatus in 137..140 ? 'retry' : 'terminate' }
|
||||
maxRetries = 3
|
||||
maxMemory = 192.GB
|
||||
|
||||
// Resource labels
|
||||
withLabel: verylowcpu { cpus = 2 }
|
||||
withLabel: lowcpu { cpus = 8 }
|
||||
withLabel: midcpu { cpus = 16 }
|
||||
withLabel: highcpu { cpus = 32 }
|
||||
|
||||
withLabel: verylowmem { memory = { get_memory( 4.GB * task.attempt ) } }
|
||||
withLabel: lowmem { memory = { get_memory( 8.GB * task.attempt ) } }
|
||||
withLabel: midmem { memory = { get_memory( 16.GB * task.attempt ) } }
|
||||
withLabel: highmem { memory = { get_memory( 64.GB * task.attempt ) } }
|
||||
|
||||
}
|
||||
|
||||
profiles {
|
||||
// detect tempdir
|
||||
tempDir = java.nio.file.Paths.get(
|
||||
System.getenv('NXF_TEMP') ?:
|
||||
System.getenv('VIASH_TEMP') ?:
|
||||
System.getenv('TEMPDIR') ?:
|
||||
System.getenv('TMPDIR') ?:
|
||||
'/tmp'
|
||||
).toAbsolutePath()
|
||||
|
||||
mount_temp {
|
||||
docker.temp = tempDir
|
||||
podman.temp = tempDir
|
||||
charliecloud.temp = tempDir
|
||||
}
|
||||
|
||||
no_publish {
|
||||
process {
|
||||
withName: '.*' {
|
||||
publishDir = [
|
||||
enabled: false
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
docker {
|
||||
docker.fixOwnership = true
|
||||
docker.enabled = true
|
||||
// docker.userEmulation = true
|
||||
singularity.enabled = false
|
||||
podman.enabled = false
|
||||
shifter.enabled = false
|
||||
charliecloud.enabled = false
|
||||
}
|
||||
|
||||
local {
|
||||
// This config is for local processing.
|
||||
process {
|
||||
maxMemory = 25.GB
|
||||
withLabel: verylowcpu { cpus = 2 }
|
||||
withLabel: lowcpu { cpus = 4 }
|
||||
withLabel: midcpu { cpus = 6 }
|
||||
withLabel: highcpu { cpus = 12 }
|
||||
|
||||
withLabel: lowmem { memory = { get_memory( 8.GB * task.attempt ) } }
|
||||
withLabel: midmem { memory = { get_memory( 12.GB * task.attempt ) } }
|
||||
withLabel: highmem { memory = { get_memory( 20.GB * task.attempt ) } }
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
def get_memory(to_compare) {
|
||||
if (!process.containsKey("maxMemory") || !process.maxMemory) {
|
||||
return to_compare
|
||||
}
|
||||
|
||||
try {
|
||||
if (process.containsKey("maxRetries") && process.maxRetries && task.attempt == (process.maxRetries as int)) {
|
||||
return process.maxMemory
|
||||
}
|
||||
else if (to_compare.compareTo(process.maxMemory as nextflow.util.MemoryUnit) == 1) {
|
||||
return max_memory as nextflow.util.MemoryUnit
|
||||
}
|
||||
else {
|
||||
return to_compare
|
||||
}
|
||||
} catch (all) {
|
||||
println "Error processing memory resources. Please check that process.maxMemory '${process.maxMemory}' and process.maxRetries '${process.maxRetries}' are valid!"
|
||||
System.exit(1)
|
||||
}
|
||||
}
|
||||
@@ -27,8 +27,8 @@
|
||||
"input_falco": {
|
||||
"type":
|
||||
"string",
|
||||
"description": "Type: `file`, required. Directory to write falco output to",
|
||||
"help_text": "Type: `file`, required. Directory to write falco output to"
|
||||
"description": "Type: List of `file`, required, multiple_sep: `\";\"`. Directory to write falco output to",
|
||||
"help_text": "Type: List of `file`, required, multiple_sep: `\";\"`. Directory to write falco output to"
|
||||
|
||||
}
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
name: "untar"
|
||||
namespace: "io"
|
||||
version: "v0.3.3"
|
||||
version: "v0.3.6"
|
||||
argument_groups:
|
||||
- name: "Input arguments"
|
||||
arguments:
|
||||
@@ -45,6 +45,9 @@ resources:
|
||||
- type: "bash_script"
|
||||
path: "script.sh"
|
||||
is_executable: true
|
||||
- type: "file"
|
||||
path: "nextflow_labels.config"
|
||||
dest: "nextflow_labels.config"
|
||||
description: "Unpack a .tar file. When the contents of the .tar file is just a single\
|
||||
\ directory,\nput the contents of the directory into the output folder instead of\
|
||||
\ that directory.\n"
|
||||
@@ -123,6 +126,8 @@ runners:
|
||||
cpu200: "cpus = 200"
|
||||
cpu500: "cpus = 500"
|
||||
cpu1000: "cpus = 1000"
|
||||
script:
|
||||
- "includeConfig(\"nextflow_labels.config\")"
|
||||
debug: false
|
||||
container: "docker"
|
||||
engines:
|
||||
@@ -130,7 +135,7 @@ engines:
|
||||
id: "docker"
|
||||
image: "debian:stable-slim"
|
||||
target_registry: "images.viash-hub.com"
|
||||
target_tag: "v0.3.3"
|
||||
target_tag: "v0.3.6"
|
||||
namespace_separator: "/"
|
||||
setup:
|
||||
- type: "apt"
|
||||
@@ -148,12 +153,12 @@ build_info:
|
||||
output: "target/nextflow/io/untar"
|
||||
executable: "target/nextflow/io/untar/main.nf"
|
||||
viash_version: "0.9.0"
|
||||
git_commit: "e297e8ff45464065384476fe521ab2700ec3739c"
|
||||
git_remote: "https://x-access-token:ghs_44flLTQIYGBkc9xcwCZoLYe7mxXzgw1iszFO@github.com/viash-hub/demultiplex"
|
||||
git_tag: "v0.3.2-4-ge297e8f"
|
||||
git_commit: "3a985c83869bcd78ba6d325f1a817514b52a132a"
|
||||
git_remote: "https://github.com/viash-hub/demultiplex"
|
||||
git_tag: "v0.3.5-3-g3a985c8"
|
||||
package_config:
|
||||
name: "demultiplex"
|
||||
version: "v0.3.3"
|
||||
version: "v0.3.6"
|
||||
description: "Demultiplexing pipeline\n"
|
||||
info:
|
||||
test_resources:
|
||||
@@ -164,10 +169,12 @@ package_config:
|
||||
target: "target"
|
||||
config_mods:
|
||||
- ".requirements.commands := ['ps']\n.runners[.type == 'nextflow'].directives.tag\
|
||||
\ := '$id'\n"
|
||||
\ := '$id'\n.resources += {path: '/src/config/labels.config', dest: 'nextflow_labels.config'}\n\
|
||||
.runners[.type == 'nextflow'].config.script := 'includeConfig(\"nextflow_labels.config\"\
|
||||
)'\n"
|
||||
- ".engines += { type: \"native\" }"
|
||||
- ".engines[.type == 'docker'].target_registry := 'images.viash-hub.com'"
|
||||
- ".engines[.type == 'docker'].target_tag := 'v0.3.3'"
|
||||
- ".engines[.type == 'docker'].target_tag := 'v0.3.6'"
|
||||
keywords:
|
||||
- "bioinformatics"
|
||||
- "sequence"
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
// untar v0.3.3
|
||||
// untar v0.3.6
|
||||
//
|
||||
// This wrapper script is auto-generated by viash 0.9.0 and is thus a derivative
|
||||
// work thereof. This software comes with ABSOLUTELY NO WARRANTY from Data
|
||||
@@ -2806,7 +2806,7 @@ meta = [
|
||||
"config": processConfig(readJsonBlob('''{
|
||||
"name" : "untar",
|
||||
"namespace" : "io",
|
||||
"version" : "v0.3.3",
|
||||
"version" : "v0.3.6",
|
||||
"argument_groups" : [
|
||||
{
|
||||
"name" : "Input arguments",
|
||||
@@ -2866,6 +2866,11 @@ meta = [
|
||||
"type" : "bash_script",
|
||||
"path" : "script.sh",
|
||||
"is_executable" : true
|
||||
},
|
||||
{
|
||||
"type" : "file",
|
||||
"path" : "/src/config/labels.config",
|
||||
"dest" : "nextflow_labels.config"
|
||||
}
|
||||
],
|
||||
"description" : "Unpack a .tar file. When the contents of the .tar file is just a single directory,\nput the contents of the directory into the output folder instead of that directory.\n",
|
||||
@@ -2954,7 +2959,10 @@ meta = [
|
||||
"cpu200" : "cpus = 200",
|
||||
"cpu500" : "cpus = 500",
|
||||
"cpu1000" : "cpus = 1000"
|
||||
}
|
||||
},
|
||||
"script" : [
|
||||
"includeConfig(\\"nextflow_labels.config\\")"
|
||||
]
|
||||
},
|
||||
"debug" : false,
|
||||
"container" : "docker"
|
||||
@@ -2966,7 +2974,7 @@ meta = [
|
||||
"id" : "docker",
|
||||
"image" : "debian:stable-slim",
|
||||
"target_registry" : "images.viash-hub.com",
|
||||
"target_tag" : "v0.3.3",
|
||||
"target_tag" : "v0.3.6",
|
||||
"namespace_separator" : "/",
|
||||
"setup" : [
|
||||
{
|
||||
@@ -2989,13 +2997,13 @@ meta = [
|
||||
"engine" : "docker|native",
|
||||
"output" : "target/nextflow/io/untar",
|
||||
"viash_version" : "0.9.0",
|
||||
"git_commit" : "e297e8ff45464065384476fe521ab2700ec3739c",
|
||||
"git_remote" : "https://x-access-token:ghs_44flLTQIYGBkc9xcwCZoLYe7mxXzgw1iszFO@github.com/viash-hub/demultiplex",
|
||||
"git_tag" : "v0.3.2-4-ge297e8f"
|
||||
"git_commit" : "3a985c83869bcd78ba6d325f1a817514b52a132a",
|
||||
"git_remote" : "https://github.com/viash-hub/demultiplex",
|
||||
"git_tag" : "v0.3.5-3-g3a985c8"
|
||||
},
|
||||
"package_config" : {
|
||||
"name" : "demultiplex",
|
||||
"version" : "v0.3.3",
|
||||
"version" : "v0.3.6",
|
||||
"description" : "Demultiplexing pipeline\n",
|
||||
"info" : {
|
||||
"test_resources" : [
|
||||
@@ -3009,10 +3017,10 @@ meta = [
|
||||
"source" : "src",
|
||||
"target" : "target",
|
||||
"config_mods" : [
|
||||
".requirements.commands := ['ps']\n.runners[.type == 'nextflow'].directives.tag := '$id'\n",
|
||||
".requirements.commands := ['ps']\n.runners[.type == 'nextflow'].directives.tag := '$id'\n.resources += {path: '/src/config/labels.config', dest: 'nextflow_labels.config'}\n.runners[.type == 'nextflow'].config.script := 'includeConfig(\\"nextflow_labels.config\\")'\n",
|
||||
".engines += { type: \\"native\\" }",
|
||||
".engines[.type == 'docker'].target_registry := 'images.viash-hub.com'",
|
||||
".engines[.type == 'docker'].target_tag := 'v0.3.3'"
|
||||
".engines[.type == 'docker'].target_tag := 'v0.3.6'"
|
||||
],
|
||||
"keywords" : [
|
||||
"bioinformatics",
|
||||
@@ -3465,7 +3473,7 @@ meta["defaults"] = [
|
||||
"container" : {
|
||||
"registry" : "images.viash-hub.com",
|
||||
"image" : "vsh/demultiplex/io/untar",
|
||||
"tag" : "v0.3.3"
|
||||
"tag" : "v0.3.6"
|
||||
},
|
||||
"tag" : "$id"
|
||||
}'''),
|
||||
|
||||
@@ -2,7 +2,7 @@ manifest {
|
||||
name = 'io/untar'
|
||||
mainScript = 'main.nf'
|
||||
nextflowVersion = '!>=20.12.1-edge'
|
||||
version = 'v0.3.3'
|
||||
version = 'v0.3.6'
|
||||
description = 'Unpack a .tar file. When the contents of the .tar file is just a single directory,\nput the contents of the directory into the output folder instead of that directory.\n'
|
||||
}
|
||||
|
||||
@@ -122,4 +122,4 @@ process{
|
||||
withLabel: cpu1000 { cpus = 1000 }
|
||||
}
|
||||
|
||||
|
||||
includeConfig("nextflow_labels.config")
|
||||
|
||||
98
target/nextflow/io/untar/nextflow_labels.config
Normal file
98
target/nextflow/io/untar/nextflow_labels.config
Normal file
@@ -0,0 +1,98 @@
|
||||
process {
|
||||
container = 'nextflow/bash:latest'
|
||||
|
||||
// default resources
|
||||
memory = { 8.Gb * task.attempt }
|
||||
cpus = 8
|
||||
maxForks = 36
|
||||
|
||||
// Retry for exit codes that have something to do with memory issues
|
||||
errorStrategy = { task.exitStatus in 137..140 ? 'retry' : 'terminate' }
|
||||
maxRetries = 3
|
||||
maxMemory = 192.GB
|
||||
|
||||
// Resource labels
|
||||
withLabel: verylowcpu { cpus = 2 }
|
||||
withLabel: lowcpu { cpus = 8 }
|
||||
withLabel: midcpu { cpus = 16 }
|
||||
withLabel: highcpu { cpus = 32 }
|
||||
|
||||
withLabel: verylowmem { memory = { get_memory( 4.GB * task.attempt ) } }
|
||||
withLabel: lowmem { memory = { get_memory( 8.GB * task.attempt ) } }
|
||||
withLabel: midmem { memory = { get_memory( 16.GB * task.attempt ) } }
|
||||
withLabel: highmem { memory = { get_memory( 64.GB * task.attempt ) } }
|
||||
|
||||
}
|
||||
|
||||
profiles {
|
||||
// detect tempdir
|
||||
tempDir = java.nio.file.Paths.get(
|
||||
System.getenv('NXF_TEMP') ?:
|
||||
System.getenv('VIASH_TEMP') ?:
|
||||
System.getenv('TEMPDIR') ?:
|
||||
System.getenv('TMPDIR') ?:
|
||||
'/tmp'
|
||||
).toAbsolutePath()
|
||||
|
||||
mount_temp {
|
||||
docker.temp = tempDir
|
||||
podman.temp = tempDir
|
||||
charliecloud.temp = tempDir
|
||||
}
|
||||
|
||||
no_publish {
|
||||
process {
|
||||
withName: '.*' {
|
||||
publishDir = [
|
||||
enabled: false
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
docker {
|
||||
docker.fixOwnership = true
|
||||
docker.enabled = true
|
||||
// docker.userEmulation = true
|
||||
singularity.enabled = false
|
||||
podman.enabled = false
|
||||
shifter.enabled = false
|
||||
charliecloud.enabled = false
|
||||
}
|
||||
|
||||
local {
|
||||
// This config is for local processing.
|
||||
process {
|
||||
maxMemory = 25.GB
|
||||
withLabel: verylowcpu { cpus = 2 }
|
||||
withLabel: lowcpu { cpus = 4 }
|
||||
withLabel: midcpu { cpus = 6 }
|
||||
withLabel: highcpu { cpus = 12 }
|
||||
|
||||
withLabel: lowmem { memory = { get_memory( 8.GB * task.attempt ) } }
|
||||
withLabel: midmem { memory = { get_memory( 12.GB * task.attempt ) } }
|
||||
withLabel: highmem { memory = { get_memory( 20.GB * task.attempt ) } }
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
def get_memory(to_compare) {
|
||||
if (!process.containsKey("maxMemory") || !process.maxMemory) {
|
||||
return to_compare
|
||||
}
|
||||
|
||||
try {
|
||||
if (process.containsKey("maxRetries") && process.maxRetries && task.attempt == (process.maxRetries as int)) {
|
||||
return process.maxMemory
|
||||
}
|
||||
else if (to_compare.compareTo(process.maxMemory as nextflow.util.MemoryUnit) == 1) {
|
||||
return max_memory as nextflow.util.MemoryUnit
|
||||
}
|
||||
else {
|
||||
return to_compare
|
||||
}
|
||||
} catch (all) {
|
||||
println "Error processing memory resources. Please check that process.maxMemory '${process.maxMemory}' and process.maxRetries '${process.maxRetries}' are valid!"
|
||||
System.exit(1)
|
||||
}
|
||||
}
|
||||
@@ -1,5 +1,5 @@
|
||||
name: "runner"
|
||||
version: "v0.3.3"
|
||||
version: "v0.3.6"
|
||||
argument_groups:
|
||||
- name: "Input arguments"
|
||||
arguments:
|
||||
@@ -84,11 +84,22 @@ argument_groups:
|
||||
direction: "output"
|
||||
multiple: false
|
||||
multiple_sep: ";"
|
||||
- name: "Other arguments"
|
||||
arguments:
|
||||
- type: "boolean_true"
|
||||
name: "--skip_copycomplete_check"
|
||||
description: "Disable the check for the presence of a \"CopyComplete.txt\" file\
|
||||
\ in input\ndirectory in case of Illumina data.\n"
|
||||
info: null
|
||||
direction: "input"
|
||||
resources:
|
||||
- type: "nextflow_script"
|
||||
path: "main.nf"
|
||||
is_executable: true
|
||||
entrypoint: "run_wf"
|
||||
- type: "file"
|
||||
path: "nextflow_labels.config"
|
||||
dest: "nextflow_labels.config"
|
||||
description: "Runner for demultiplexing of raw sequencing data"
|
||||
info: null
|
||||
status: "enabled"
|
||||
@@ -165,6 +176,8 @@ runners:
|
||||
cpu200: "cpus = 200"
|
||||
cpu500: "cpus = 500"
|
||||
cpu1000: "cpus = 1000"
|
||||
script:
|
||||
- "includeConfig(\"nextflow_labels.config\")"
|
||||
debug: false
|
||||
container: "docker"
|
||||
engines:
|
||||
@@ -179,15 +192,15 @@ build_info:
|
||||
output: "target/nextflow/runner"
|
||||
executable: "target/nextflow/runner/main.nf"
|
||||
viash_version: "0.9.0"
|
||||
git_commit: "e297e8ff45464065384476fe521ab2700ec3739c"
|
||||
git_remote: "https://x-access-token:ghs_44flLTQIYGBkc9xcwCZoLYe7mxXzgw1iszFO@github.com/viash-hub/demultiplex"
|
||||
git_tag: "v0.3.2-4-ge297e8f"
|
||||
git_commit: "3a985c83869bcd78ba6d325f1a817514b52a132a"
|
||||
git_remote: "https://github.com/viash-hub/demultiplex"
|
||||
git_tag: "v0.3.5-3-g3a985c8"
|
||||
dependencies:
|
||||
- "target/nextflow/demultiplex"
|
||||
- "target/nextflow/io/publish"
|
||||
package_config:
|
||||
name: "demultiplex"
|
||||
version: "v0.3.3"
|
||||
version: "v0.3.6"
|
||||
description: "Demultiplexing pipeline\n"
|
||||
info:
|
||||
test_resources:
|
||||
@@ -198,10 +211,12 @@ package_config:
|
||||
target: "target"
|
||||
config_mods:
|
||||
- ".requirements.commands := ['ps']\n.runners[.type == 'nextflow'].directives.tag\
|
||||
\ := '$id'\n"
|
||||
\ := '$id'\n.resources += {path: '/src/config/labels.config', dest: 'nextflow_labels.config'}\n\
|
||||
.runners[.type == 'nextflow'].config.script := 'includeConfig(\"nextflow_labels.config\"\
|
||||
)'\n"
|
||||
- ".engines += { type: \"native\" }"
|
||||
- ".engines[.type == 'docker'].target_registry := 'images.viash-hub.com'"
|
||||
- ".engines[.type == 'docker'].target_tag := 'v0.3.3'"
|
||||
- ".engines[.type == 'docker'].target_tag := 'v0.3.6'"
|
||||
keywords:
|
||||
- "bioinformatics"
|
||||
- "sequence"
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
// runner v0.3.3
|
||||
// runner v0.3.6
|
||||
//
|
||||
// This wrapper script is auto-generated by viash 0.9.0 and is thus a derivative
|
||||
// work thereof. This software comes with ABSOLUTELY NO WARRANTY from Data
|
||||
@@ -2805,7 +2805,7 @@ meta = [
|
||||
"resources_dir": moduleDir.toRealPath().normalize(),
|
||||
"config": processConfig(readJsonBlob('''{
|
||||
"name" : "runner",
|
||||
"version" : "v0.3.3",
|
||||
"version" : "v0.3.6",
|
||||
"argument_groups" : [
|
||||
{
|
||||
"name" : "Input arguments",
|
||||
@@ -2901,6 +2901,17 @@ meta = [
|
||||
"multiple_sep" : ";"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"name" : "Other arguments",
|
||||
"arguments" : [
|
||||
{
|
||||
"type" : "boolean_true",
|
||||
"name" : "--skip_copycomplete_check",
|
||||
"description" : "Disable the check for the presence of a \\"CopyComplete.txt\\" file in input\ndirectory in case of Illumina data.\n",
|
||||
"direction" : "input"
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"resources" : [
|
||||
@@ -2909,6 +2920,11 @@ meta = [
|
||||
"path" : "main.nf",
|
||||
"is_executable" : true,
|
||||
"entrypoint" : "run_wf"
|
||||
},
|
||||
{
|
||||
"type" : "file",
|
||||
"path" : "/src/config/labels.config",
|
||||
"dest" : "nextflow_labels.config"
|
||||
}
|
||||
],
|
||||
"description" : "Runner for demultiplexing of raw sequencing data",
|
||||
@@ -2999,7 +3015,10 @@ meta = [
|
||||
"cpu200" : "cpus = 200",
|
||||
"cpu500" : "cpus = 500",
|
||||
"cpu1000" : "cpus = 1000"
|
||||
}
|
||||
},
|
||||
"script" : [
|
||||
"includeConfig(\\"nextflow_labels.config\\")"
|
||||
]
|
||||
},
|
||||
"debug" : false,
|
||||
"container" : "docker"
|
||||
@@ -3021,13 +3040,13 @@ meta = [
|
||||
"engine" : "native|native",
|
||||
"output" : "target/nextflow/runner",
|
||||
"viash_version" : "0.9.0",
|
||||
"git_commit" : "e297e8ff45464065384476fe521ab2700ec3739c",
|
||||
"git_remote" : "https://x-access-token:ghs_44flLTQIYGBkc9xcwCZoLYe7mxXzgw1iszFO@github.com/viash-hub/demultiplex",
|
||||
"git_tag" : "v0.3.2-4-ge297e8f"
|
||||
"git_commit" : "3a985c83869bcd78ba6d325f1a817514b52a132a",
|
||||
"git_remote" : "https://github.com/viash-hub/demultiplex",
|
||||
"git_tag" : "v0.3.5-3-g3a985c8"
|
||||
},
|
||||
"package_config" : {
|
||||
"name" : "demultiplex",
|
||||
"version" : "v0.3.3",
|
||||
"version" : "v0.3.6",
|
||||
"description" : "Demultiplexing pipeline\n",
|
||||
"info" : {
|
||||
"test_resources" : [
|
||||
@@ -3041,10 +3060,10 @@ meta = [
|
||||
"source" : "src",
|
||||
"target" : "target",
|
||||
"config_mods" : [
|
||||
".requirements.commands := ['ps']\n.runners[.type == 'nextflow'].directives.tag := '$id'\n",
|
||||
".requirements.commands := ['ps']\n.runners[.type == 'nextflow'].directives.tag := '$id'\n.resources += {path: '/src/config/labels.config', dest: 'nextflow_labels.config'}\n.runners[.type == 'nextflow'].config.script := 'includeConfig(\\"nextflow_labels.config\\")'\n",
|
||||
".engines += { type: \\"native\\" }",
|
||||
".engines[.type == 'docker'].target_registry := 'images.viash-hub.com'",
|
||||
".engines[.type == 'docker'].target_tag := 'v0.3.3'"
|
||||
".engines[.type == 'docker'].target_tag := 'v0.3.6'"
|
||||
],
|
||||
"keywords" : [
|
||||
"bioinformatics",
|
||||
@@ -3095,9 +3114,10 @@ workflow run_wf {
|
||||
"input": state.input,
|
||||
"run_information": state.run_information,
|
||||
"demultiplexer": state.demultiplexer,
|
||||
"output": "fastq",
|
||||
"output_falco": "qc/fastqc",
|
||||
"output_multiqc": "qc/multiqc_report.html",
|
||||
"skip_copycomplete_check": state.skip_copycomplete_check,
|
||||
"output": "$id/fastq",
|
||||
"output_falco": "$id/qc/fastqc",
|
||||
"output_multiqc": "$id/qc/multiqc_report.html",
|
||||
]
|
||||
if (state.run_information) {
|
||||
state_to_pass += ["output_run_information": state.run_information.getName()]
|
||||
|
||||
@@ -2,7 +2,7 @@ manifest {
|
||||
name = 'runner'
|
||||
mainScript = 'main.nf'
|
||||
nextflowVersion = '!>=20.12.1-edge'
|
||||
version = 'v0.3.3'
|
||||
version = 'v0.3.6'
|
||||
description = 'Runner for demultiplexing of raw sequencing data'
|
||||
}
|
||||
|
||||
@@ -122,4 +122,4 @@ process{
|
||||
withLabel: cpu1000 { cpus = 1000 }
|
||||
}
|
||||
|
||||
|
||||
includeConfig("nextflow_labels.config")
|
||||
|
||||
98
target/nextflow/runner/nextflow_labels.config
Normal file
98
target/nextflow/runner/nextflow_labels.config
Normal file
@@ -0,0 +1,98 @@
|
||||
process {
|
||||
container = 'nextflow/bash:latest'
|
||||
|
||||
// default resources
|
||||
memory = { 8.Gb * task.attempt }
|
||||
cpus = 8
|
||||
maxForks = 36
|
||||
|
||||
// Retry for exit codes that have something to do with memory issues
|
||||
errorStrategy = { task.exitStatus in 137..140 ? 'retry' : 'terminate' }
|
||||
maxRetries = 3
|
||||
maxMemory = 192.GB
|
||||
|
||||
// Resource labels
|
||||
withLabel: verylowcpu { cpus = 2 }
|
||||
withLabel: lowcpu { cpus = 8 }
|
||||
withLabel: midcpu { cpus = 16 }
|
||||
withLabel: highcpu { cpus = 32 }
|
||||
|
||||
withLabel: verylowmem { memory = { get_memory( 4.GB * task.attempt ) } }
|
||||
withLabel: lowmem { memory = { get_memory( 8.GB * task.attempt ) } }
|
||||
withLabel: midmem { memory = { get_memory( 16.GB * task.attempt ) } }
|
||||
withLabel: highmem { memory = { get_memory( 64.GB * task.attempt ) } }
|
||||
|
||||
}
|
||||
|
||||
profiles {
|
||||
// detect tempdir
|
||||
tempDir = java.nio.file.Paths.get(
|
||||
System.getenv('NXF_TEMP') ?:
|
||||
System.getenv('VIASH_TEMP') ?:
|
||||
System.getenv('TEMPDIR') ?:
|
||||
System.getenv('TMPDIR') ?:
|
||||
'/tmp'
|
||||
).toAbsolutePath()
|
||||
|
||||
mount_temp {
|
||||
docker.temp = tempDir
|
||||
podman.temp = tempDir
|
||||
charliecloud.temp = tempDir
|
||||
}
|
||||
|
||||
no_publish {
|
||||
process {
|
||||
withName: '.*' {
|
||||
publishDir = [
|
||||
enabled: false
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
docker {
|
||||
docker.fixOwnership = true
|
||||
docker.enabled = true
|
||||
// docker.userEmulation = true
|
||||
singularity.enabled = false
|
||||
podman.enabled = false
|
||||
shifter.enabled = false
|
||||
charliecloud.enabled = false
|
||||
}
|
||||
|
||||
local {
|
||||
// This config is for local processing.
|
||||
process {
|
||||
maxMemory = 25.GB
|
||||
withLabel: verylowcpu { cpus = 2 }
|
||||
withLabel: lowcpu { cpus = 4 }
|
||||
withLabel: midcpu { cpus = 6 }
|
||||
withLabel: highcpu { cpus = 12 }
|
||||
|
||||
withLabel: lowmem { memory = { get_memory( 8.GB * task.attempt ) } }
|
||||
withLabel: midmem { memory = { get_memory( 12.GB * task.attempt ) } }
|
||||
withLabel: highmem { memory = { get_memory( 20.GB * task.attempt ) } }
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
def get_memory(to_compare) {
|
||||
if (!process.containsKey("maxMemory") || !process.maxMemory) {
|
||||
return to_compare
|
||||
}
|
||||
|
||||
try {
|
||||
if (process.containsKey("maxRetries") && process.maxRetries && task.attempt == (process.maxRetries as int)) {
|
||||
return process.maxMemory
|
||||
}
|
||||
else if (to_compare.compareTo(process.maxMemory as nextflow.util.MemoryUnit) == 1) {
|
||||
return max_memory as nextflow.util.MemoryUnit
|
||||
}
|
||||
else {
|
||||
return to_compare
|
||||
}
|
||||
} catch (all) {
|
||||
println "Error processing memory resources. Please check that process.maxMemory '${process.maxMemory}' and process.maxRetries '${process.maxRetries}' are valid!"
|
||||
System.exit(1)
|
||||
}
|
||||
}
|
||||
@@ -109,6 +109,27 @@
|
||||
}
|
||||
|
||||
|
||||
}
|
||||
},
|
||||
|
||||
|
||||
"other arguments" : {
|
||||
"title": "Other arguments",
|
||||
"type": "object",
|
||||
"description": "No description",
|
||||
"properties": {
|
||||
|
||||
|
||||
"skip_copycomplete_check": {
|
||||
"type":
|
||||
"boolean",
|
||||
"description": "Type: `boolean_true`, default: `false`. Disable the check for the presence of a \"CopyComplete",
|
||||
"help_text": "Type: `boolean_true`, default: `false`. Disable the check for the presence of a \"CopyComplete.txt\" file in input\ndirectory in case of Illumina data.\n"
|
||||
,
|
||||
"default":false
|
||||
}
|
||||
|
||||
|
||||
}
|
||||
},
|
||||
|
||||
@@ -157,6 +178,10 @@
|
||||
"$ref": "#/definitions/output arguments"
|
||||
},
|
||||
|
||||
{
|
||||
"$ref": "#/definitions/other arguments"
|
||||
},
|
||||
|
||||
{
|
||||
"$ref": "#/definitions/nextflow input-output arguments"
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user