Recent searches

No recent searches

Search options

Only available when logged in.
eupolicy.social is one of the many independent Mastodon servers you can use to participate in the fediverse.
This Mastodon server is a friendly and respectful discussion space for people working in areas related to EU policy. When you request to create an account, please tell us something about you.

Administered by:

Server stats:

196
active users

eupolicy.social: AboutProfiles directoryPrivacy policy

Mastodon: AboutGet the appKeyboard shortcutsView source codev4.4.1

#reproduciblecomputing

1 post1 participant0 posts today
*
Christian Meesters

Interested in HPC compliant data analysis workflows?

I am offering an NHR (German association for HPC resources) course for building and using Snakemake workflows on HPC clusters in Mainz, Germany! Two days: 9. & 10. December 2025 - on-site.

To find out more and perhaps enrol, visit the course page: indico.zdv.uni-mainz.de/event/

IndicoBuilding HPC-Compliant Snakemake Data Analysis WorkflowsOverview This tutorial equips participants with the essential skills to design and implement High-Performance Computing (HPC) compliant data analysis workflows using the Snakemake workflow management system. Through hands-on exercises and practical demonstrations, attendees will learn how to harness the power of this workflow manager to utilize HPC resources effectively and ensure reproducibility in their data analysis workflows. The Snakemake workflow system is widely used in...
#Snakemake#HPC#ReproducibleResearch
Continued thread
*
Christian Meesters

This morning, I am travelling to the #isc25 and hit a minor bug on #researchequals. Hence, no updates in the collection.

But still a few to describe without adding the latest contributions:

For instance, this one (zenodo.org/records/15490064) by Filipe G. Vieira: a helper function to extract checksums from files to compare with checksums Snakemake was already able to calculate. Really handy!

ZenodoExtra Input Helper Functions for SnakemakeThis Snakemake Hackathon Contribution adds several helper functions to infer file sizes and adds functions to calculate hash sums or return a file's content through a callable.
#Snakemake#ReproducibleComputing#OpenScience
Continued thread
Christian Meesters

Before I continue uploading - and I do have a couple of more contributions to add to the #ResearchEquals collection - first another contribution by Johanna Elena Schmitz and Jens Zentgraf made at the #SnakemakeHackathon2025

One difficulty when dealing with a different scientific question: Do I need to re-invent the wheel (read: write a Workflow from scratch?) just to address my slightly different question?

Snakemake already allowed to incorporate "alien" workflows, even #Nextflow workflows, into desired workflows. The new contribution allows for a more dynamic contribution - with very little changes.

Check it out: zenodo.org/records/15489694

ZenodoAllowing Dynamic Load of Modules for SnakemakeSnakemake modules had to be explicitly defined and loaded at the beginning of a workflow. This limited the flexibility of workflows, particularly when dealing with complex dependency structures or when modules needed to be loaded conditionally based on runtime parameters. This contribution eases the procedure of dynamically adding 3rd party modules.
#Snakemake#ReproducibleComputing#OpenScience
Continued thread
Christian Meesters

Let's take a look at another contribution of Johanna Elena Schmitz and Jens Zentgraf from the #SnakemakeHackathon2025

Snakemake users probably know that

`$ snakemake [args] --report`

will generate a self-contained HTML report. Including all plots and #metadata a researcher's heart longs for.

Now, why trigger this manually? If the workflow runs successfully, now we can write (or configure):

`$ snakemake [args] --report-after-run`

and Snakemake will autogenerate the same report.

For details see doi.org/10.5281/zenodo.1548976

#Snakemake #ReproducibleComputing
#OpenScience

ZenodoCreate Report After Running a Snakemake WorkflowThis contribution adds a flag to Snakemake to allow for immediate report creation after a workflow finished.
Continued thread
Christian Meesters

One important feature implemented in the #SnakemakeHackathon2025 : Snakemake will calculate file checksums to detect changes. If a file changes, the rule producing it needs to be re-executed when a workflow it re-triggered. But what if a file is too big for reasonable checksum calculation? You do not what to wait forever, after all.

This contribution describes the implementation of a threshold users may set: doi.org/10.5281/zenodo.1548940

ZenodoAdjusting Snakemake's Maximum File Size for ChecksumsSnakemake calculates checksums for input and output files during DAG building to determine if a rule needs to be re-executed. For huge files, computing these checksums can be a significant performance bottleneck, slowing down the DAG generation process. This paper describes the limiting of checksum calculation upon file sizes.
#Snakemake#ReproducibleComputing#OpenScience
Continued thread
Christian Meesters

One important bug fix during the #SnakemakeHackathon2025 : the config replacement. Now, users can overwrite existing configurations entirely with `--replace-workflow-config`.

Details: zenodo.org/records/15479268

More at researchequals.com/collections

ZenodoConfig Replacement in SnakemakeThis work was completed as part of the Snakemake Hackathon 2025.  Previously, when a configuration file was specified via the Snakemake CLI (e.g., `snakemake --config config.yaml`), the contents of that file were *merged* with any configuration variables already defined within the Snakefile. This meant that the CLI config would only extend or update existing values, rather than completely replacing the workflow's default configuration. This work item introduces the `--replace-workflow-config` parameter to the Snakemake CLI.
#Snakemake#ReproducibleComputing#openscience
Christian Meesters

Did you know? During the #SnakemakeHackathon2025 we had a staggering 194 work items!

It took a while, but now we are gathering contribution reports and present them online as a ResearchEquals (fediscience.org/@ResearchEqual) collection:

researchequals.com/collections

The first 10 are online and I will post some highlights in the coming weeks.

fediscience.orgFediScience.org
#Snakemake#ReproducibleComputing#ReproducibleResearch
*
Christian Meesters

We have a new release for the #SLURM support plugin of #Snakemake !

It's a minor feature release which enables custom log directories and auto-deletion of successful job logs (such that not zillions of zero-meaning files accumulate!).

Check it out at: github.com/snakemake/snakemake

It took a while to get to this release (stress, family, and sickness took a toll). Hopefully, a future release will not take that long to be realized — the feature request list is considerable. 😉

GitHubRelease v0.12.0 · snakemake/snakemake-executor-plugin-slurm0.12.0 (2025-01-08) Features custom log file behaviour (#159) (cc3d21b) experimenting with automated release post to Mastodon (#166) (c06325d) Documentation fix headings in further.md (#168) (53...
#HPC#ReproducibleResearch#ReproducibleComputing
TrendingLive feeds
About