The Evolution of Open-Source Orchestration: Inside Kestra’s Four-Year Architecture Rewrite and the Launch of Version 2.0

Executive Overview

Four years after introducing the open-source community to Kestra—an ambitious data orchestration and scheduling platform originally built around Kafka and Elasticsearch—the project’s maintainers have reached a monumental milestone. In February 2022, the platform’s debut announcement proudly cited 350,000 monthly executions running at enterprise scale inside Leroy Merlin. Yet, as the creators are quick to admit, that initial architecture could not survive contact with reality.

The word "infinitely" scalable, once attached to the platform’s moniker, has been permanently retired. More importantly, the heavy, opinionated technology stack required to run early iterations of the software drew immediate and justified criticism from prospective users.

This month marks the official release of Kestra 2.0, a Long-Term Support (LTS) release representing the culmination of a massive, multi-year engine rewrite. By decoupling the core architecture into a distinct control plane and data plane, eliminating legacy technical debt, and offering flexible, independent queue and repository backends, Kestra 2.0 addresses the primary deployment hurdles that stymied adoption in constrained environments.

Crucially, this sweeping architectural transformation has been achieved without compromising the core tenets that defined the project from day one: declarative YAML workflows, containerized execution, language agnosticism, and a steadfast commitment to the Apache 2.0 open-source license. As the platform crosses the threshold of four billion total executions worldwide, version 2.0 positions Kestra not merely as an alternative scheduler, but as a robust, production-ready orchestration standard designed to weather the complex realities of modern distributed infrastructure.

Detailed Chronology: From Heavy Stacks to the Control/Data Plane Divide

The 2022 Reality Check: Listening to the Community

When Kestra first launched publicly, its original architecture relied on Apache Kafka to serve as both the message queue and the primary database, paired with Elasticsearch to back the user interface. While performant for large enterprises like Leroy Merlin, the feedback from the wider developer community was swift, unanimous, and uncompromising: requiring a dedicated Kafka cluster and an Elasticsearch cluster merely to schedule basic enterprise jobs was wildly disproportionate and practically absurd.

The project’s maintainers chose a pragmatic path. Just four months after the initial launch, the team shipped a JDBC backend, proving that a single, lightweight Postgres or MySQL instance could comfortably handle everything, including queues and state management. This event established a guiding philosophy for the project: when users highlight friction in the deployment story, the engineering team fixes the deployment story.

Over the subsequent years, iterative improvements accumulated behind the scenes, paving the way for a much larger structural reckoning. Version 1.x had pushed architectural design as far as it could go, but it suffered from an immutable structural flaw.

The Architectural Bottleneck of Version 1.x

In the 1.x architecture, every worker instance required a direct, active connection to the central database. This single design decision dictated rigid boundaries regarding where users were allowed to execute workloads.

Consider a typical enterprise security posture: stringent network policies often prevent outbound connections from isolated data enclaves to a central Postgres database hosted elsewhere. Under version 1.x, systems administrators faced two undesirable options:

1. Deploy a complete, standalone instance of Kestra within every isolated network site, multiplying operational overhead.
2. Abandon orchestration entirely, resorting to legacy shell scripts executed locally via cron with zero visibility, monitoring, or dependency management.

Faced with this dilemma, security-conscious teams routinely chose the second option, leaving critical business logic trapped in unmonitored silos. Kestra’s maintainers watched this happen repeatedly, recognizing that the engine itself had become the barrier to entry.

Breaking the Monolith: The Genesis of Kestra 2.0

To solve this fundamental networking challenge, version 2.0 introduces a clean architectural separation: a control plane and a data plane.

• The Control Plane: This layer houses the core orchestrating components, including the executor, the scheduler, the web server, the indexer, and a newly minted worker controller. Critically, none of these components execute user code. They manage state, coordinate schedules, and serve the user interface.
• The Data Plane: This layer consists exclusively of the workers. This is where user code, scripts, and containerized tasks actually execute.

The gRPC Streaming Bridge

In Kestra 2.0, workers no longer need direct access to the central database. Instead, each worker opens a single, persistent gRPC stream connecting outbound to the worker controller.

Crucially, this connection is always initiated by the worker; inbound connections to the data plane are entirely eliminated. Workloads travel outward from the control plane along this established stream, while execution results, logs, and system metrics flow back along the exact same channel.

Security teams can lock down this channel using Transport Layer Security (TLS), while requiring each worker to present valid client certificates or JSON Web Tokens (JWTs) before any job is dispatched.

Practical Implications for Enterprise Deployments

The operational impact of this decoupled architecture is profound. A Kestra 2.0 worker:

• Holds zero database credentials.
• Requires no inbound network rules penetrating its perimeter.
• Can be deployed seamlessly across different cloud providers, isolated geographic regions, on-premises environments adjacent to sensitive data stores, or within restricted networks that exclusively permit outbound HTTP/gRPC traffic.

The control plane remains safely tucked inside the management zone of your choosing, while the data plane lives precisely where your regulatory and security frameworks demand.

Eliminating Internal Technical Debt

Beyond network topology, version 2.0 targets deep-seated internal debt. In version 1.x, the queue and repository layers were bound in a rigid pairing: users were forced to choose either JDBC for both or Kafka paired with Elasticsearch.

This coupling meant the engineering team had to maintain two distinct engine implementations, fix every bug twice, and continuously troubleshoot behavioral inconsistencies between the two execution paths.

Kestra 2.0 unifies these mechanics. The codebase now features a single, cohesive executor, scheduler, and worker design. Users are free to pick their queue and repository backends independently. Notably, the legacy Kafka Streams engine has been entirely deprecated and removed—a simplification that the core engineering team welcomes with open arms.

Supporting Context & Metrics: Scaling to Billions of Executions

When the platform was first documented in early 2022, boasting 350,000 monthly executions was viewed as a significant badge of honor. Fast forward four years, and the scale of adoption has exploded past expectations.

Today, more than four billion executions have run on Kestra worldwide. The vast majority of these workflows execute silently on open-source installations deployed across local servers, edge devices, and hybrid cloud hardware that the core maintainers will never physically see.

This massive accumulation of unseen production usage places a profound responsibility on the project team. When an open-source tool achieves this level of ubiquity, stability becomes paramount. Version 2.0 has been designated as a Long-Term Support (LTS) release, specifically engineered to respect the operational stability of existing deployments while delivering monumental performance and architectural gains.

To demonstrate how frictionless adoption remains for newcomers, a fully functional production-grade server running an embedded database can be spun up locally with a single Docker command:

docker run --pull=always --rm -it -p 8080:8080 --user=root 
  --name kestra 
  -v kestra_data:/app/storage 
  -v kestra_db:/app/data 
  -v /var/run/docker.sock:/var/run/docker.sock 
  -v /tmp:/tmp 
  -e KESTRA_PLUGINS_AUTO_INSTALL_ENABLED=true 
  kestra/kestra:latest-slim server local

This single command encapsulates the project’s ongoing dedication to out-of-the-box usability: no complex configuration spirals, no mandatory microservice meshes, and immediate access to a complete orchestration engine right out of the box.

Official Statements and Core Principles

Major version upgrades in the open-source software ecosystem frequently spark community anxiety. Historically, major version boundaries serve as the natural inflection point where projects alter their licensing models, erect paywalls around core features, or pivot away from community-first development.

The maintainers of Kestra have addressed these concerns directly and unequivocally.

The License Remains Unchanged

"The engine, the UI, the no-code editor and all the plugins are Apache 2.0, in the public repository… We know a major version is where projects usually change it. We did not, and we do not plan to."

The core engineering team understands that trust is difficult to build and easy to shatter. By keeping the entire engine, user interface, and all community plugins strictly under the permissive Apache 2.0 license, Kestra reinforces its open-core business model transparently.

The boundaries of the open-source edition versus enterprise offerings are clearly demarcated:

• Paid/Enterprise Features: Advanced capabilities such as Role-Based Access Control (RBAC), Single Sign-On (SSO), detailed audit logs, multi-tenancy, worker groups, organizational policies, case management, and cloud virtual machine task runners are gated behind commercial tiers.
• Open Source Edition: Everything required to write, schedule, monitor, and execute production-grade workflows at scale is entirely free and open-source. The community edition is treated as a fully featured, enterprise-grade orchestrator rather than a restricted trial or demo.

What Remained Untouched: Declarative YAML and Portability

Amidst a complete engine overhaul, the core abstraction layer of Kestra was deliberately preserved.

• Declarative YAML: Workflows continue to be defined cleanly in declarative YAML.
• Language Agnosticism: Developers can write tasks in any programming language. Each script executes safely inside its own dedicated container, keeping business logic entirely decoupled from orchestrator-specific syntax.
• No Vendor Lock-in: Nothing from Kestra is imported directly into user codebases. If an enterprise eventually decides to migrate to an alternative orchestration platform, engineers rewrite the declarative flow configurations rather than refactoring the underlying business logic.

Future Outlook: The Road Ahead for Kestra 2.0

The release of Kestra 2.0 represents the most ambitious engineering undertaking in the project’s history. While rigorous internal testing and release candidate cycles successfully identified and resolved standard edge cases, an architectural shift of this magnitude inevitably meets the unpredictable friction of diverse production environments.

The project leads have issued an open call to the community: break it and report back.

As administrators and platform engineers begin rolling out Kestra 2.0 across global infrastructures, the feedback loop established in 2022 remains active. The maintainers view community scrutiny not as a risk, but as the primary mechanism to ensure that the LTS release delivers a rock-solid, uneventful upgrade path for the billions of executions yet to come.

By successfully bridging the gap between uncompromising security requirements and developer-friendly workflows, Kestra 2.0 cements open-source data orchestration as a mature, highly adaptable discipline capable of meeting the demands of modern cloud-native enterprises.