#107
B+
Rank #75
Dagster
A data orchestration platform for building, scheduling, and observing data pipelines with strong typing and asset-based lineage. Standout trait is its software-defined-assets model, which treats pipeline outputs as first-class, testable artifacts.
73.5
/ 100
⚙ Full Stack
Python webserver + daemon architecture backed by Postgres for run/event storage, with pluggable executors (multiprocess, Docker, Kubernetes Jobs, Celery) for actually running pipeline code. Ships official Helm charts and Docker images for each component.
📈 Scaling Analysis
Scales well for pipeline volume via its distributed executor model and asset partitioning, and Postgres-backed run storage means state survives worker restarts cleanly. The webserver/daemon split is a reasonable stateless-vs-stateful separation, though heavy pipelines still need careful executor tuning to avoid single-node bottlenecks.
🚀 Running on Nexlayer
Deploy as three pods — webserver, daemon, and Postgres (`postgres.pod:5432`) — with `DAGSTER_POSTGRES_HOST` pointed at the Postgres pod's `.pod` DNS name and all component images pulled via mirror.gcr.io. The k8s-executor mode expects to create its own Jobs, which conflicts with Nexlayer's no-runner-creates-resources policy, so stick to the multiprocess or Docker executor inside the pod.