Command line and Terraform

Two tools drive graal Data Platform outside the console, through the same REST API: graalctl, the command line, and the graalsystems Terraform provider, to describe projects, identities, jobs and workflows in a Git repository. Both are open source, under the Apache 2.0 license.

graalctl

A single binary for Linux, macOS and Windows (github.com/graalsystems/graalctl).

curl -fsSL https://raw.githubusercontent.com/graalsystems/graalctl/main/install.sh | sh

The script downloads the latest version for your system, checks its SHA-256 sum and installs it in /usr/local/bin (or ~/.local/bin). On Windows, take graalctl-windows-x86_64.exe from the releases page.

Logging in

A context names an installation and a tenant:

graalctl config set-context prod \
  --api-url https://graal.example.com/api/v1 \
  --auth-url https://sso.example.com \
  --tenant acme
graalctl login            # username and password
graalctl login --device   # SSO: a URL and a code to enter in any browser

--auth-url is the address of the installation’s OpenID Connect server, without /realms/…: the realm is derived from the tenant. For a script, an integration or an agent, prefer the credentials of an application to those of a person:

echo "$SECRET" | graalctl login --client-id my-application --client-secret-stdin

Commands

graalctl projects list
graalctl libraries upload first_job.py           # prints the library key
graalctl jobs create --project <PROJECT> -f job.json
graalctl jobs run <JOB> --follow                 # starts a run and follows its logs
graalctl jobs schedule <JOB> "0 6 * * *"         # JSON Patch on /schedule
graalctl runs list --job <JOB>
graalctl api /instance-types                     # any API operation

-o json prints the API objects, -o id one identifier per line. The job.json file has the shape described in the quickstart.

In a continuous integration pipeline

Nothing needs to be stored on the CI runner: the API address, the tenant and the application credentials come from the environment.

export GRAAL_API_URL=https://graal.example.com/api/v1
export GRAAL_KEYCLOAK_URL=https://sso.example.com
export GRAAL_TENANT=acme
export GRAAL_CLIENT_ID=ci GRAAL_CLIENT_SECRET=…    # CI secrets
graalctl jobs run <JOB> --follow --timeout 3600

The exit code follows the run: 0 when it succeeds, 1 when it fails or is cancelled, 124 when --timeout is reached.

Terraform provider

The graalsystems provider (github.com/graalsystems/terraform-provider-graalsystems) manages projects, identities, jobs, workflows, workspaces, users and groups.

terraform {
  required_providers {
    graalsystems = {
      source  = "graalsystems/graalsystems"
      version = "~> 1.4.0"
    }
  }
}

# Credentials: GRAAL_CLIENT_ID and GRAAL_CLIENT_SECRET.
provider "graalsystems" {
  api_url  = "https://graal.example.com/api/v1"
  auth_url = "https://sso.example.com"
  tenant   = "acme"
}

resource "graalsystems_project" "analytics" {
  name = "Analytics"
}

A change becomes a JSON Patch on the resource: the API has no full PUT.

Examples

github.com/graalsystems/examples gathers ready-to-run jobs, one per type (Python, bash, Spark, Dask, Ray, Flink, dbt…), each with its job.json, and a GitHub Actions CI example with graalctl.

Its projects/ directory adds whole projects: several jobs that hand data to each other through Iceberg tables, the identity that runs them with only the rights it needs, and the workflow that chains them. A daily data pipeline (ingestion, aggregation, quality checks) and a demand forecast (preparation, a training notebook tracked in MLflow, publication) each deploy in one command:

projects/deploy-project.sh projects/energy-consumption --run