Skip to content

fix: Airflow demo improvements - #457

Merged
adwk67 merged 3 commits into
mainfrom
fix/airflow-demo-improvements
Oct 2, 2026
Merged

adwk67 merged 3 commits into
mainfrom
fix/airflow-demo-improvements

Conversation

@adwk67

@adwk67 adwk67 commented Sep 30, 2026 •

Copy link
Copy Markdown
Member

This PR fixes:

  • replaces cluster-wide permissions with namespace-scoped ones
  • replaces minio and mc images from quay.io/minio with ones from docker.io/pgsty/mc
  • the airflow versions so it uses 3.3.1 (and not 3.1.6 as determined previously by the podOverrides)

And changes:

  • the spark sensor such that logs are streamed from the driver and executor pods to the Airflow UI
  • N.B. this does not necessitate granting Airflow any extra permissions that it has already anyway to access Spark logs (at whatever level)
image

@adwk67 adwk67 self-assigned this Sep 30, 2026
@sweb
sweb self-requested a review September 30, 2026 09:19
sweb
sweb previously approved these changes Oct 2, 2026

@sweb sweb left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor nit, lgtm

Comment thread demos/airflow-scheduled-job/01-airflow-demo-clusterrole.yaml
Comment thread stacks/airflow/airflow.yaml Outdated

@sweb sweb left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@adwk67
adwk67 merged commit 43005a8 into main Oct 2, 2026
2 checks passed
@adwk67
adwk67 deleted the fix/airflow-demo-improvements branch October 2, 2026 14:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Development: Done

Development

Successfully merging this pull request may close these issues.

2 participants