Skip to content

fix(airflow): Update manifests #431

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
NickLarsenNZ wants to merge 3 commits into
base: main
Choose a base branch
from
Draft

fix(airflow): Update manifests #431

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
bcd65ff
fix(airflow): Update manifests
NickLarsenNZ Jul 10, 2026
d01808c
fix(airflow): Use correct postgres service name
NickLarsenNZ Jul 10, 2026
9363205
docs(demo/airflow-scheduled-job): Add missing namespace to command
NickLarsenNZ Jul 10, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions demos/airflow-scheduled-job/03-enable-and-run-spark-dag.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,16 +23,16 @@ spec:
- |
kubectl rollout status --watch statefulset/airflow-webserver-default
kubectl rollout status --watch statefulset/airflow-scheduler-default
AIRFLOW_ADMIN_PASSWORD=$(cat /airflow-credentials/adminUser.password)
AIRFLOW_ADMIN_PASSWORD=$(cat /airflow-admin-credentials/adminUser.password)
ACCESS_TOKEN=$(curl -XPOST http://airflow-webserver-default-headless:8080/auth/token -H 'Content-Type: application/json' -d '{"username": "admin", "password": "'$AIRFLOW_ADMIN_PASSWORD'"}' | jq -r .access_token)
curl -H "Authorization: Bearer $ACCESS_TOKEN" -H 'Content-Type: application/json' -XPATCH http://airflow-webserver-default-headless:8080/api/v2/dags/sparkapp_dag -d '{"is_paused": false}' | jq
curl -H "Authorization: Bearer $ACCESS_TOKEN" -H 'Content-Type: application/json' -XPOST http://airflow-webserver-default-headless:8080/api/v2/dags/sparkapp_dag/dagRuns -d '{"logical_date": null}' | jq
volumeMounts:
- name: airflow-credentials
mountPath: /airflow-credentials
- name: airflow-admin-credentials
mountPath: /airflow-admin-credentials
volumes:
- name: airflow-credentials
- name: airflow-admin-credentials
secret:
secretName: airflow-credentials
secretName: airflow-admin-credentials
restartPolicy: OnFailure
backoffLimit: 20 # give some time for the Airflow cluster to be available
10 changes: 5 additions & 5 deletions demos/airflow-scheduled-job/04-enable-and-run-date-dag.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,16 +23,16 @@ spec:
- |
kubectl rollout status --watch statefulset/airflow-webserver-default
kubectl rollout status --watch statefulset/airflow-scheduler-default
AIRFLOW_ADMIN_PASSWORD=$(cat /airflow-credentials/adminUser.password)
AIRFLOW_ADMIN_PASSWORD=$(cat /airflow-admin-credentials/adminUser.password)
ACCESS_TOKEN=$(curl -XPOST http://airflow-webserver-default-headless:8080/auth/token -H 'Content-Type: application/json' -d '{"username": "admin", "password": "'$AIRFLOW_ADMIN_PASSWORD'"}' | jq -r .access_token)
curl -H "Authorization: Bearer $ACCESS_TOKEN" -H 'Content-Type: application/json' -XPATCH http://airflow-webserver-default-headless:8080/api/v2/dags/date_demo -d '{"is_paused": false}' | jq
curl -H "Authorization: Bearer $ACCESS_TOKEN" -H 'Content-Type: application/json' -XPOST http://airflow-webserver-default-headless:8080/api/v2/dags/date_demo/dagRuns -d '{"logical_date": null}' | jq
volumeMounts:
- name: airflow-credentials
mountPath: /airflow-credentials
- name: airflow-admin-credentials
mountPath: /airflow-admin-credentials
volumes:
- name: airflow-credentials
- name: airflow-admin-credentials
secret:
secretName: airflow-credentials
secretName: airflow-admin-credentials
restartPolicy: OnFailure
backoffLimit: 20 # give some time for the Airflow cluster to be available
10 changes: 5 additions & 5 deletions demos/airflow-scheduled-job/05-enable-and-run-kafka-dag.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ spec:
kubectl rollout status --watch statefulset/airflow-scheduler-default

# Airflow: activate DAG
AIRFLOW_ADMIN_PASSWORD=$(cat /airflow-credentials/adminUser.password)
AIRFLOW_ADMIN_PASSWORD=$(cat /airflow-admin-credentials/adminUser.password)
ACCESS_TOKEN=$(curl -XPOST http://airflow-webserver-default-headless:8080/auth/token -H 'Content-Type: application/json' -d '{"username": "admin", "password": "'$AIRFLOW_ADMIN_PASSWORD'"}' | jq -r .access_token)
curl -H "Authorization: Bearer $ACCESS_TOKEN" -H 'Content-Type: application/json' -XPATCH http://airflow-webserver-default-headless:8080/api/v2/dags/kafka_watcher -d '{"is_paused": false}' | jq

Expand All @@ -58,11 +58,11 @@ spec:
--producer.config /stackable/config/client.properties \
--topic test-topic'
volumeMounts:
- name: airflow-credentials
mountPath: /airflow-credentials
- name: airflow-admin-credentials
mountPath: /airflow-admin-credentials
volumes:
- name: airflow-credentials
- name: airflow-admin-credentials
secret:
secretName: airflow-credentials
secretName: airflow-admin-credentials
restartPolicy: OnFailure
backoffLimit: 20 # give some time for the Airflow cluster to be available
8 changes: 4 additions & 4 deletions demos/airflow-scheduled-job/06-create-opa-users.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,11 +42,11 @@ spec:
--password "richard.roe" \
--role "User"
volumeMounts:
- name: airflow-credentials
mountPath: /airflow-credentials
- name: airflow-admin-credentials
mountPath: /airflow-admin-credentials
volumes:
- name: airflow-credentials
- name: airflow-admin-credentials
secret:
secretName: airflow-credentials
secretName: airflow-admin-credentials
restartPolicy: OnFailure
backoffLimit: 20 # give some time for the Airflow cluster to be available
2 changes: 1 addition & 1 deletion docs/modules/demos/pages/airflow-scheduled-job.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ To list the installed Stackable services run the following command:

[source,console]
----
$ stackablectl stacklet list
$ stackablectl stacklet list -n airflow-demo

┌─────────┬───────────────┬───────────┬───────────────────────────────────────────────────────────────────────────────────────────────────────────┬─────────────────────────────────┐
│ PRODUCT ┆ NAME ┆ NAMESPACE ┆ ENDPOINTS ┆ CONDITIONS │
Expand Down
20 changes: 16 additions & 4 deletions stacks/airflow/airflow.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,11 @@ spec:
productVersion: 3.2.2
pullPolicy: IfNotPresent
clusterConfig:
metadataDatabase:
postgresql:
host: postgresql-airflow
database: airflow
credentialsSecretName: airflow-postgresql-credentials
authorization:
opa:
configMapName: opa
Expand All @@ -17,7 +22,7 @@ spec:
entryTimeToLive: 5s
maxEntries: 10
loadExamples: false
credentialsSecret: airflow-credentials
credentialsSecretName: airflow-admin-credentials
volumes:
- name: airflow-dags
configMap:
Expand Down Expand Up @@ -571,13 +576,20 @@ data:
apiVersion: v1
kind: Secret
metadata:
name: airflow-credentials
name: airflow-admin-credentials
type: Opaque
stringData:
adminUser.username: admin
adminUser.firstname: Airflow
adminUser.lastname: Admin
adminUser.email: airflow@airflow.com
adminUser.password: "{{ airflowAdminPassword }}"
connections.secretKey: "{{ airflowSecretKey }}"
connections.sqlalchemyDatabaseUri: postgresql+psycopg2://airflow:airflow@postgresql-airflow/airflow
---
apiVersion: v1
kind: Secret
metadata:
name: airflow-postgresql-credentials
type: Opaque
stringData:
username: airflow
password: airflow
Loading