Kedro is a Python framework designed for creating reproducible, maintainable, and modular data science and machine learning pipelines.
Why it matters
overview
Kedro is a data science framework tool developed by QuantumBlack (McKinsey's AI division) that enables data scientists and engineers to create reproducible, maintainable, and modular data science and machine learning pipelines. It focuses on structuring code within a data science project and building modular pipelines in a single execution environment, emphasizing code modularity and best practices for data engineering and data science.
features
Kedro provides a comprehensive set of features designed to enhance the development and deployment of data science and machine learning projects, focusing on structure, reproducibility, and collaboration.
use cases
Kedro is primarily designed for data scientists and engineers who require a structured and reproducible approach to developing and deploying machine learning and data processing pipelines. It is particularly beneficial for teams and projects that aim to move beyond exploratory Jupyter notebooks into production-grade systems.
how to use
To begin using Kedro, users typically install the framework via pip and then initialize a new project using the Kedro CLI. This sets up a standardized project structure, allowing for the definition of data pipelines through nodes and a data catalog.
pip install kedro.kedro new and follow the prompts.conf/base/catalog.yml file using the Data Catalog.kedro run command from the project root, or visualize them with kedro viz run.pricing
Kedro is an open-source Python framework and is available for free. There are no paid tiers or subscription plans for the core framework.
Similar Tools
Kedro is positioned as a framework that applies software engineering best practices to data and machine learning pipelines, differentiating itself from other tools in the MLOps and data engineering ecosystem through its opinionated project structure and emphasis on reproducibility.
It focuses on building modular and maintainable data pipelines by defining tasks and their dependencies in a YAML file or Python script.
Ploomber offers a similar approach to Kedro in structuring pipelines and managing dependencies, but it might have a slightly different philosophy on project layout and configuration, potentially requiring some adaptation to its specific conventions.
It emphasizes data assets, providing a unified programming model for defining, observing, and orchestrating data pipelines.
While both focus on data pipelines, Dagster's core concept revolves around data assets and their lineage, offering more built-in observability and testing capabilities compared to Kedro's primary focus on code structure and modularity.
It provides a flexible and robust framework for building, observing, and reacting to data workflows, with a strong emphasis on resilience and dynamic execution.
Prefect offers more advanced features for workflow orchestration, retries, and dynamic task mapping than Kedro, which is primarily focused on structuring the project and defining static pipelines.
It helps build complex pipelines of batch jobs by handling dependency resolution, workflow management, and visualization.
Luigi is a more mature and battle-tested framework for defining task dependencies and running batch jobs, but it might feel less opinionated about overall project structure and best practices compared to Kedro's comprehensive project template.
More on Stork
Other tools in this category, matched by shared tags