Skip to content

Kedro

Kedro is a Python framework designed for creating reproducible, maintainable, and modular data science and machine learning pipelines.

Free · Python · API

stork
I am a flat drawing.
One flat drawing and one typed sentence.
This drawing was never animated.
A photo, a cartoon, a painting or a 3D character. From 10 cents a minute. Free to try.
Make a photo talkDevelopers: there is an API

What is Kedro?

Kedro is a data science framework tool developed by QuantumBlack (McKinsey's AI division) that enables data scientists and engineers to create reproducible, maintainable, and modular data science and machine learning pipelines. It focuses on structuring code within a data science project and building modular pipelines in a single execution environment, emphasizing code modularity and best practices for data engineering and data science.

Key Features of Kedro

Kedro provides a comprehensive set of features designed to enhance the development and deployment of data science and machine learning projects, focusing on structure, reproducibility, and collaboration.

  • Pipeline Visualisation (Kedro-Viz) for interactive flowcharts of pipelines.
  • Data Catalog supporting over 100 dataset types including S3, GCP, Azure, Pandas, and Spark.
  • Integrations with platforms like Amazon SageMaker, Apache Airflow, Apache Spark, Azure ML, and Databricks.
  • Standardized Project Template for configuration, source code, tests, documentation, and notebooks.
  • Dedicated IDE support through a Visual Studio Code extension.
  • Pipeline Abstraction with dataset-driven workflow and automatic dependency resolution.
  • Enforcement of Coding Standards using tools like pytest, Sphinx, ruff, and Python logging.
  • Flexible Deployment options for single/distributed-machine environments, including Argo, Prefect, Kubeflow, AWS Batch, AWS Sagemaker, Databricks, and Dask.

Who Should Use Kedro?

Kedro is primarily designed for data scientists and engineers who require a structured and reproducible approach to developing and deploying machine learning and data processing pipelines. It is particularly beneficial for teams and projects that aim to move beyond exploratory Jupyter notebooks into production-grade systems.

  • Data scientists building reproducible ML pipelines to prevent 'notebook chaos' and ensure consistent results.
  • Data engineers consolidating data sources, cleaning data, and creating features for machine learning models.
  • Teams requiring standardized project layouts and workflows for effective collaboration across varied experience levels.
  • Developers needing deployment flexibility for pipelines to targets such as Airflow, Kubeflow, Vertex AI, and Databricks.
  • Organizations transitioning exploratory data science code into maintainable and modular experiments for production environments.

How to Use Kedro

To begin using Kedro, users typically install the framework via pip and then initialize a new project using the Kedro CLI. This sets up a standardized project structure, allowing for the definition of data pipelines through nodes and a data catalog.

  1. Install Kedro using pip install kedro.
  2. Create a new Kedro project with kedro new and follow the prompts.
  3. Define data processing steps as 'nodes' (Python functions) within the project's source code.
  4. Configure data sources and destinations in the conf/base/catalog.yml file using the Data Catalog.
  5. Assemble nodes into 'pipelines' by defining their dependencies and execution order.
  6. Run pipelines using the kedro run command from the project root, or visualize them with kedro viz run.

Kedro Pricing & Plans

Kedro is an open-source Python framework and is available for free. There are no paid tiers or subscription plans for the core framework.

  • Kedro: Free

Pros

  • Enforces reproducibility in data science projects through dependency management and structured execution.
  • Promotes a structured approach, moving projects from disorganized Jupyter notebooks to production-ready code.
  • Features a YAML-based Data Catalog that abstracts data sources and supports over 100 dataset types.
  • Provides interactive pipeline visualization with Kedro-Viz, aiding understanding and communication.
  • Enhances modularity and reusability of code through isolated nodes and support for modular pipelines.
  • Maintains stability with continuous development and a focus on avoiding breaking changes.

Cons

  • Users may experience a learning curve due to the need to understand Kedro's specific project structure and object creation.
  • The data catalog can be less effective for unstructured data setups involving flat files and manual file movement.
  • Some data scientists report a temporary drop in productivity when initially transitioning from notebooks to Kedro for new code.
  • Kedro is not a drop-in replacement for managed data ingestion tools; users may need to write custom Python code for data pulling and schema drift.

Kedro vs Competitors

Kedro is positioned as a framework that applies software engineering best practices to data and machine learning pipelines, differentiating itself from other tools in the MLOps and data engineering ecosystem through its opinionated project structure and emphasis on reproducibility.

See every Kedro alternative, compared

Kedro

Free · Python · API

Enjoying this? Get one like it in your inbox each morning.

one email a day · unsubscribe in two clicks · no third-party tracking