Secrets and libraries

A job that only reads public files is a demonstration job. A real job needs its code, its dependencies and a credential to reach a database or a storage. Those three things arrive by three different paths, and confusing them is the most frequent cause of a run failing on a denied access.

Libraries: what lands in /workspace

A library is a file the scheduler drops into the run’s container before starting it. It is the mechanism by which the code itself arrives, not only the dependencies.

# 1. upload the file — the API returns a key
curl -s -X POST "$GRAAL_API/libraries" \
  -H "Authorization: Bearer $TOKEN" -H "X-Tenant: $TENANT" \
  -F file=@processing.py
# → [{"key":"…","name":"processing.py",…}]

# 2. reference the key in the job
#    "libraries": [{"type": "file", "key": "<the returned key>"}]
#    "options":   {"type": "python", "file": "processing.py", …}

options.file names the file as it will appear in /workspace, that is, its original name — not the key.

Python dependencies

The python runtime image installs at start-up whatever it finds in /workspace: a requirements.txt, or archives (.whl, .tar.gz). Upload them as libraries, just like the code.

The install runs as the container’s unprivileged user, not as root: the image is prepared for it. You have nothing to configure.

Why not a custom image?

You can: each runtime type resolves its image through a configuration table, and you can point it at yours (see runtime images). Building an image makes sense when the dependencies are heavy and stable; a library makes sense when they change along with the code.

Secrets: what must not lie around

A project secret is sealed in the database: encrypted with AES-256-GCM, with a key derived per tenant, and the key name is used as authenticated data — a secret moved from one entry to another will not decrypt. Detail on the security page.

When a run starts, an init container calls the API back with the job’s identity, fetches the secrets that identity is entitled to, and drops them into a file in the container, under /workspace/secrets/. The job reads that file.

The detour is not ceremony. It buys three things passing through the environment does not:

1. The secret does not appear in the job definition, so not in an export, not in a copied configuration, not in a screenshot.
2. It does not appear in the process environment, where any sub-process, stack trace or diagnostic dump would pick it up.
3. Rights apply: it is the job’s identity that asks, not the person who created the job.

options.env: what belongs there, and what does not

Put there	Do not put there
A bucket name, a path, a processing date	A password, an API key, a token
A log level, a feature flag	A full connection string
The address of an internal service	Anything whose leak would force you to rotate

The short rule: the environment describes, it does not authenticate.

The root key and existing vaults

• The secrets root key is generated once, at installation, and backed up: the secrets sealed with it only open with it (see installation).
• No third-party vault is required. A tenant that already has one may designate it for its secrets.

Go further

• Security and governance — the encryption, in detail
• Roles and permissions — granting an identity the right to read a secret
• Runtime images — when to build an image rather than upload a library