TL;DR / Key Takeaways
- A new orchestrator called Kestra raised $25 million with a simple promise: replace Airflow's Python code with simple YAML files.
- This declarative, language-agnostic approach is changing how developers build and manage data pipelines.
Why Your Python Pipelines Are Broken
Modern data workflows demand intricate chains of jobs: pulling, cleaning, loading, and API calls. Relying on basic schedulers like cron for these critical sequences is a recipe for disaster. A single step failure means no retries, no logs, and no clear path to understanding what broke, leaving your entire pipeline derailed.
For years, Apache Airflow reigned as the king of orchestration, designed to tame this complexity. However, its fundamental design presents a significant hurdle. Every single pipeline in Apache Airflow is a Python program, making workflows heavy to run, cumbersome to maintain, and a substantial barrier for any team member not proficient in Python development.
Now, Kestra, the Tool Trying to Kill Apache Airflow, has emerged, having recently raised $25 million on its promise. Its premise is deceptively simple, challenging the very notion of pipeline definition. Kestra posits that workflows shouldn't be programs at all; instead, You stop writing your data pipelines in Python, and You start writing them in YAML, creating simple, readable configuration files.
Kestra's Declarative Revolution
Workflows in Kestra are defined as 'flows,' simple declarative YAML files that separate orchestration logic from business logic. This declarative approach means You stop writing complex data pipelines in Python, and instead specify a list of tasks and a trigger within a configuration file. This core design choice simplifies pipeline creation and makes them readable by non-Python users, a significant departure from Apache Airflow's Python-centric model.
This shift enables a critical advantage: language agnosticism. A single flow can seamlessly execute diverse tasks, eliminating the need for language-specific operators. One flow might run Python scripts, then Node.js apps, followed by Bash commands, and conclude with SQL queries or even fire up a container, all within the same sequence. Kestra truly doesn't care what each step is written in.
Kestra’s platform boasts a remarkably clean UI, where the visual editor and underlying YAML code remain constantly synchronized. This integration allows for live execution monitoring, with diagrams lighting up as tasks run. Users also gain immediate access to timeline views for performance analysis and one-click access to detailed logs for any step, providing comprehensive visibility without ever writing a single line of orchestration code.
Kestra vs. The World
Kestra's YAML pipelines offer a stark contrast to Apache Airflow's Python programs. Its declarative nature means readable configs anyone can review and approve pull requests, simplifying collaboration. Reviewers also report Kestra's engine handles parallel work with greater efficiency than Apache Airflow's scheduling, reducing bottlenecks in complex data flows.
Moving beyond traditional orchestrators, Kestra carves a distinct niche against SaaS platforms like Zapier or Make. It positions itself as developer-first and self-hosted, granting You complete control over your infrastructure. This eliminates the unpredictable per-task billing models common with SaaS solutions, providing cost predictability and autonomy for enterprise-grade operations.
Kestra's market adoption shows significant momentum. The company claims it ran 2 billion workflows in 2025—a twenty-fold increase from the prior year. This rapid growth is underscored by a robust customer base, including industry giants such as: - Apple - JPMorgan - Toyota - Bloomberg A recent $25 million funding round further solidifies investor confidence in this declarative, config-first approach to orchestration. While these growth numbers are company-reported, they signal a strong shift towards Kestra's methodology.
The Catches: Is Kestra for You?
Kestra presents specific considerations for adoption. As a Java application, it demands substantial system resources, typically requiring around 4GB of RAM and multiple CPU cores. This allocation is necessary just to run the server peacefully, which can be a significant factor for resource-constrained environments or local development.
Enjoying this? Get one like it in your inbox each morning.
one email a day · unsubscribe in two clicks · no third-party tracking
While YAML excels at defining clear, linear workflows, its declarative structure becomes cumbersome for complex, dynamic branching logic. In scenarios demanding intricate conditional paths or runtime adaptations, Python-native tools like Apache Airflow still maintain a clear advantage, offering the flexibility of programmatic control. Evaluate your workflow complexity before committing.
Kestra also operates on an open-core model, which impacts enterprise readiness. Crucial features reside behind a paywall: - Single Sign-On (SSO) - Role-Based Access Control (RBAC) - Audit logs The free version restricts access to a single shared login, posing a significant hurdle for teams requiring granular user management or compliance without an enterprise subscription.
So, is Kestra for You? If your primary goal is readable, configuration-driven orchestration for straightforward pipelines, and you can accommodate its resource footprint, this Tool Trying to Kill Apache Airflow is a strong contender. However, if your workflows demand extensive programmatic control, dynamic decisions, or enterprise-grade authentication and authorization features without a paid tier, You should Stay with Python-centric solutions.
Frequently Asked Questions
What is Kestra?
Kestra is an open-source orchestration platform that uses YAML configuration files to define and manage complex workflows, positioning itself as a modern, language-agnostic alternative to tools like Apache Airflow.
How is Kestra different from Apache Airflow?
The primary difference is Kestra's 'configuration-as-code' approach using YAML, whereas Airflow requires pipelines to be written as Python programs. Kestra is also language-agnostic, allowing tasks in Python, Node, SQL, and Bash within a single workflow, while Airflow is heavily Python-centric.
What are the main drawbacks of Kestra?
Kestra's drawbacks include being a resource-intensive Java application, YAML's limitations for complex dynamic branching logic compared to Python, and an open-core model that places features like SSO and RBAC behind a paywall.
Is Kestra free to use?
Yes, Kestra has a free, open-source version with a full-featured engine. However, enterprise features like single sign-on (SSO), role-based access control (RBAC), and audit logs are part of a paid tier.