Skip to content
Docs
FRBook a demo

Jobs and workflows

A job runs while nobody sits in front of the screen: it can be scheduled, triggered by an event and followed afterwards. This page covers the job types, the runs and the workflows of graal Data Platform.

Job types

Section titled “Job types”

Every job has a type, which decides what it executes and which image it runs in.

Type What it executes Image
python a Python script or package, dependencies installed at startup python
bash a shell command, for operations tasks toolbox
spark a Spark workload (spark-submit), sized by instance type spark-py
lowcode a graph from the low-code editor, translated to code python

The contract also carries specialised types — docker, dbt, dask, flink, ray —, each served by its own runtime image. A type whose image is not configured on your installation answers 501 feature_disabled (see runtime images).

Creating a job, step by step and as commands, is in the quickstart.

A run, and what you see of it

Section titled “A run, and what you see of it”

A job that is started produces a run. While it executes, four things are readable:

  • the logs, cursor-paginated — the console follows them live, an API client replays them from any point;
  • the events, timestamped: queued, started, step, finished;
  • the container metrics;
  • the files produced, in the project’s bucket.

A run carries a status (queued, pending, running, succeeded, failed, cancelled, retrying, skipped) and a duration. That is what the job list shows. The API returns the end status of a run as complete, failed or killed; the console shows it as “succeeded”, “failed” or “cancelled”.

A job also carries its maximum duration (timeout_seconds) and its number of retries (max_retries): a run that fails is restarted automatically up to that number.

Workflows and scheduling

Section titled “Workflows and scheduling”

A workflow chains steps in a graph: each step is a job, dependencies are edges. A failed step stops the branch that depends on it without taking the others down. Each step carries its own retries: retries (0 to 10 new attempts) and retry_delay_seconds (the delay before a new attempt, 10 seconds by default, 24 hours at most). Each attempt is a new run of the job, which receives its number in the GRAAL_ATTEMPT variable.

A workflow or a job is triggered:

  • by hand, from the console, the API or an agent;
  • by a standard cron schedule — on a job, it is set by JSON Patch on the schedule field (example on the REST API page);
  • on an event: a file arriving, another workflow finishing, or a signed webhook call.

Backfills replay a scheduled job or workflow over a range of past dates, 1 to 5 runs at a time; each run receives the logical date it processes in the GRAAL_LOGICAL_DATE variable, and a running backfill can be cancelled.

Alerts notify by email or by webhook when a run fails.

Go further

Section titled “Go further”