2.13. Using Containers

2.13.1. Running Tools Inside Docker

Docker containers simplify software installation by providing a complete known-good runtime for software and its dependencies. However, containers are also purposefully isolated from the host system, so in order to run a tool inside a Docker container there is additional work to ensure that input files are available inside the container and output files can be recovered from the container. A CWL runner can perform this work automatically, allowing you to use Docker to simplify your software management while avoiding the complexity of invoking and managing Docker containers.

One of the responsibilities of the CWL runner is to adjust the paths of input files to reflect the location where they appear inside the container.

This example runs a simple Node.js script inside a Docker container which will then print “Hello World” to the standard output.

docker.cwl

#!/usr/bin/env cwl-runner

cwlVersion: v1.0
class: CommandLineTool
baseCommand: node
hints:
  DockerRequirement:
    dockerPull: node:slim
inputs:
  src:
    type: File
    inputBinding:
      position: 1
outputs:
  example_out:
    type: stdout
stdout: output.txt

docker-job.yml

src:
  class: File
  path: hello.js

Before we run this, let’s just break it down and see what some bits do. Most of this has been explained in previous sections, the only part that is really new is the dockerRequirement section.

baseCommand: node
hints:
  DockerRequirement:
    dockerPull: node:slim

baseCommand: node tells CWL that we will be running this command in a container. We then need to specify some hints for how to find the container we want. In this case we list just our requirements for the docker container in DockerRequirements. The dockerPull: parameter takes the same value that you would pass to a docker pull command. That is, the name of the container image (you can even specify the tag, which is good idea for best practices when using containers for reproducible research). In this case we have used a container called node:slim.

Provide a “hello.js” and invoke cwltool providing the tool description and the input object on the command line:

hello.js

console.log("Hello World");

$ cwltool docker.cwl docker-job.yml
INFO /home/docs/checkouts/readthedocs.org/user_builds/common-workflow-languageuser-guide/envs/stable/bin/cwltool 3.1.20221008225030
INFO Resolved 'docker.cwl' to 'file:///home/docs/checkouts/readthedocs.org/user_builds/common-workflow-languageuser-guide/checkouts/stable/src/_includes/cwl/using-containers/docker.cwl'
INFO ['udocker', 'pull', 'node:slim']
Info: downloading layer sha256:bd159e379b3b1bc0134341e4ffdeab5f966ec422ae04818bb69ecef08a823b05
Info: downloading layer sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
Info: downloading layer sha256:154e3758048b273104a377666acd12ac09b66f2a05e695b323bc777cc4540822
Info: downloading layer sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
Info: downloading layer sha256:2db009280b0aed42bb7981dadd09e0a8e8e35d5e7e0114f29cdf034d37d0b1f2
Info: downloading layer sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
Info: downloading layer sha256:988f01a0dbb4a6807f2da2f2690d8ca99314e469f60dfeb6fd53679d160f1c14
Info: downloading layer sha256:f48f6e2be91a28eefcaa370e7933299bb48ec596d4207f9d406ca093afcfe898
Info: downloading layer sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
Info: downloading layer sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
INFO [job docker.cwl] /tmp/lz9a0uqi$ udocker \
    --quiet \
    run \
    --volume=/tmp/lz9a0uqi:/tLjOoY \
    --volume=/tmp/bctpdmr9:/tmp \
    --volume=/home/docs/checkouts/readthedocs.org/user_builds/common-workflow-languageuser-guide/checkouts/stable/src/_includes/cwl/using-containers/hello.js:/var/lib/cwl/stg42645f51-2747-4185-b96c-3962412ae9ef/hello.js \
    --workdir=/tLjOoY \
    --rm \
    --env=TMPDIR=/tmp \
    --env=HOME=/tLjOoY \
    node:slim \
    node \
    /var/lib/cwl/stg42645f51-2747-4185-b96c-3962412ae9ef/hello.js > /tmp/lz9a0uqi/output.txt
INFO [job docker.cwl] Max memory used: 21MiB
INFO [job docker.cwl] completed success
{
    "example_out": {
        "location": "file:///home/docs/checkouts/readthedocs.org/user_builds/common-workflow-languageuser-guide/checkouts/stable/src/_includes/cwl/using-containers/output.txt",
        "basename": "output.txt",
        "class": "File",
        "checksum": "sha1$648a6a6ffffdaa0badb23b8baf90b6168dd16b3a",
        "size": 12,
        "path": "/home/docs/checkouts/readthedocs.org/user_builds/common-workflow-languageuser-guide/checkouts/stable/src/_includes/cwl/using-containers/output.txt"
    }
}
INFO Final process status is success

$ cat output.txt
Message is: Hello world!

Notice the CWL runner has constructed a Docker command line to run the script.

In this example, the path to the script hello.js is /home/me/cwl/user_guide/hello.js outside the container but /var/lib/cwl/job369354770_examples/hello.js inside the container, as reflected in the invocation of the node command.