Airflow vs. Dagster: Lessons From Running Both in Production

What Airflow Gets Right

Airflow is battle tested. That matters when you are on call and the scheduler is the only thing standing between you and a broken report in the morning. The core scheduling model is simple, the UI is familiar, and nearly every data engineer has already run or debugged an Airflow DAG in production. That shared mental model is real leverage when you are hiring or rotating ownership across teams.

The ecosystem is huge. There is an operator for almost everything. If you need to hit an API, move files, trigger a Spark job, or kick off a dbt run, there is a plugin or operator that gets you ninety percent of the way there. Most cloud vendors now offer managed Airflow, which means you can avoid running the scheduler and webserver yourself. That is a big deal for small teams that want to focus on data, not infrastructure.

The DAG based model is also a good fit for scheduled batch jobs. When the problem is straightforward, run these tasks at 2 AM, Airflow is clean and reliable. It excels at orchestrating pipelines where the tasks are the primary unit of work and you are mostly concerned with timing. I still use it for simple cron style workloads because it is proven and boring, and boring is a feature in production.

Where Airflow Shows Its Age

Airflow is task centric, not data centric. That is not just a semantic critique, it changes how you think about pipelines. The system asks you to model tasks and dependencies, but it does not make datasets first class objects. If you want lineage, freshness, or asset awareness, you bolt it on with plugins or external tooling. That works, but it is always an extra layer rather than the default.

The testing story is also painful. You can unit test operators, and you can run DAGs in a test environment, but neither feels natural. Most teams end up with a mix of brittle integration tests and manual validation in the UI. It is possible to do well, but Airflow does not guide you there. For pipelines that are essentially software projects, that gap starts to hurt.

The asset vs task mental model mismatch is the root of many rough edges. When your real concern is whether a dataset is fresh, Airflow makes you infer that from task runs. When you want to reason about data contracts between layers, you end up documenting it in Confluence instead of in the tool itself. Observability is solid once you invest in logs, metrics, and plugins, but you are building that stack yourself. Airflow is powerful, but it shows its age in the way it treats data as a side effect rather than a product.

The Dagster Mental Shift

Dagster's key insight is software defined assets. Instead of describing a set of tasks that run, you define the assets that get materialized. Each asset knows its upstream dependencies. That means lineage is built in, not layered on later. The UI shows you the graph of data products, not just a list of scheduled jobs. When you think in assets, the system reflects how stakeholders actually consume data.

IOManagers push you to formalize data contracts. You decide how assets are persisted, how partitions are structured, and how data is versioned. The system makes those choices explicit instead of burying them in an operator or a random script. Partitioned assets are also first class. You can declare daily partitions, backfill a range, and see the state of each partition in the UI. That clarity is rare in task based orchestrators.

The difference is easiest to see in code. In Airflow, you describe a task that produces data. In Dagster, you declare the data itself. Here is a short comparison from a pipeline that builds a curated users table.

# Airflow task
@dag.task
def build_users():
    df = extract_users()
    df = transform_users(df)
    load_users(df)

# Dagster asset
@asset
def users():
    df = extract_users()
    df = transform_users(df)
    return df

The Dagster version tells the system what asset exists and lets the IOManager decide how it is persisted. That unlocks lineage, backfills, and data catalog features with almost no extra work. It is a different mindset, and once it clicks, it is hard to unsee.

Real World Migration Lessons

The biggest surprise during migration was how much logic was not actually tied to Airflow. A lot of our operators just called Python functions. When I reframed those functions as assets, the move was less painful than I expected. The hard part was not rewriting code, it was deciding which pieces should be assets and which should remain tasks. Anything that produced a durable dataset became an asset. Anything that was purely operational, like sending notifications or triggering downstream systems, stayed as a task or a sensor.

Testing improved quickly. With Dagster, I could run assets in isolation and use plain pytest to validate their outputs. I wrote tests that materialized assets against a small local dataset, then asserted schema and row level expectations. The key shift was treating assets like functions with return values instead of tasks with side effects. It aligned naturally with pytest fixtures and made our tests faster and less flaky. We still kept some integration tests for end to end runs, but the unit tests gave us confidence without spinning up the scheduler.

Observability also improved, but only after we adjusted our habits. In Airflow, we were used to task logs and retries as the primary signal. In Dagster, we leaned into asset checks and explicit metadata on materialize events. That meant adding lightweight quality checks and structured metrics, like row counts and min or max timestamps, directly in the asset code. Those signals became part of the asset history, which made it easier to understand when a dataset drifted or when a backfill produced unexpected shape changes.

What was easier than expected was the developer experience. Dagster's local UI and reload flow reduced friction. New engineers could run a subset of assets, see the lineage, and understand the pipeline without reading an entire DAG file. What was harder was retraining how we think about ownership. Airflow trained the team to think in terms of jobs. With Dagster, you are responsible for data products, and that changes how you document, test, and monitor.

We also underestimated the migration effort around backfills and partitions. Airflow backfills are well understood, and our operators had custom logic built around execution dates. Dagster handles partitions cleanly, but you need to be explicit about partition definitions and how your IOManagers store them. Once we did that, backfills became cleaner, but there was a learning curve. The best lesson was to migrate in slices, keep Airflow running for legacy DAGs, and move critical assets first.

When to Stick With Airflow

If you have a mature Airflow codebase with years of operational knowledge, you do not need to rip it out. Airflow is stable, it scales, and your team already knows how to keep it healthy. For simple scheduled jobs and cron style pipelines, it is still a strong choice. The overhead of a migration might outweigh any benefit, especially if the DAGs are small and the data products are not complex.

Managed Airflow investment is another reason to stay. If your platform is built around a cloud managed Airflow service with tight IAM integration, logging, and monitoring, you would need a compelling reason to move. I still use Airflow in environments where the managed service is a core part of our reliability story. It is not outdated in that context. It is a proven scheduler with a large support surface area.

The Bottom Line

Dagster is my preference for greenfield platforms where I want assets to be first class, lineage to be visible, and tests to feel like real software engineering. Airflow remains the right call when you have deep investment, a large library of existing DAGs, and a team that already knows how to operate it confidently. Neither tool is wrong. The right choice depends on how much you value asset centric design versus stability and familiarity.