Airflow with python install#
Airflow with python upgrade#
Airflow with python pro#
Airflow with python Offline#

To get past this, we updated the virtual environment’s files with the following command (basically replacing the shebang in the files with the final location of the virtual environment). However, in Python 3, this option is deprecated. Enable virtual environment relocation: We build virtual environments, ship them to Airflow boxes and we make use of the “–relocatable” option in virtualenv to make them “shippable”.These dependencies can be defined in one file (pyproject.toml).

Airflow with python install#

Use Poetry for dependency management of Python 2 and 3: We use Poetry to help install the right versions of dependencies for Python 2 and 3 needed for Airflow to work.

Pyenv comes with a virtual environment plugin and we used that for creating virtual environments. We resolved this by installing pyenv to manage Python installations and decouple from system installations.

Decouple Python system installation using pyenv : This is not a necessary step but we noticed that we were depending on system Python on Airflow boxes which had version 3.5.2 installed, and which don’t work with the more recent versions of Airflow.

To set up a working python3 virtual environment, we did the following:

Airflow with python pro#

Pro tip: The above order of migration is important migrating scheduler or webserver to Python 3 before fixing DAG incompatibilities will result in errors because the scheduler in Python 3 will be unable to parse the DAG. Let’s delve into each step in the following sections (except for the clean-up steps, which are self-explanatory). We did these steps in a dev environment first and then in prod. Clean up phase 2: Cleaned up Python 2 references and virtual environments, and terminated Python 2 celery workers.Migrate the Airflow services to Python 3: Switched web server, scheduler, and flower to Python 3.If DAGs failed, we fixed forward or rolled back to Python 2 workers. We started with a small set and kept increasing depending on success rate. Move the DAGs to Python 3 workers: Tested and switched DAGs from Python 2 to Python 3 workers.Fix Python 3 incompatibilities in DAGs: Updated DAGs to be Python 2/3 compatible but let them continue to run on Python 2 workers.Clean up phase 1: Removed stale or unused DAGs to make the migration easier.

Set up Python 3 workers: Launched Python 3 celery workers running the virtual environment from step 1, and make them pick tasks from a “python3” queue on Redis.

Set up a virtual environment: Created a working Python 3 virtual environment with Airflow and all the dependencies.

Here are the list of things we did for the migration:

High-level steps for Airflow Python 3 migration We run hundreds of workflows and tens of thousands of airflow tasks every day.

Airflow with python Offline#

At Slack, we use Airflow to schedule our data warehouse workflows, which includes product metrics, business metrics, and also other use-cases like search and offline indexing. This post describes how we moved our Apache Airflow infrastructure, and hundreds of DAGs, from Python 2 to Python 3 in a more reliable “Red-black” deployment using Celery queues, resulting in the migration being completely transparent to our users.Īpache Airflow is a tool for describing, executing, and monitoring workflows. This year, due to Python 2 reaching end of life, we again had a major migration of Airflow from Python 2 to 3 and we wanted to put our learnings from 2019 into practice.

Airflow with python upgrade#

Last year, we migrated Airflow from 1.8 to 1.10 at Slack (see here) and we did a “Big bang” upgrade because of the constraints we had.