Running airflow with docker

Hello,

I have deployed airflow with docker and it is running locally fine. What I don’t understand is how will I make it run scheduled dags when my laptop is off?

Also is it possible to let my colleague use the same scheduler as I do?

I find the docker concept a bit confusing…

1 Like

Hi @LuukvanVliet:

I personally heard of docker but haven’t really got the chance to experiment with it. Perhaps this article may help you.

1 Like

Hi @masterryan.prof,

In the mean time I figured it’s best practise to deploy airflow on a linux VM using docker.
It is actually quite interesting since you can deploy databases as a docker image and control the database from inside the container (ssh into running container). Docker-compose lets you connect containers like a multi-container application using a yaml (infrastructure as code) config file which is not too hard to understand.

1 Like

I see @LuukvanVliet. Sounds interesting that you can use VM software to run docker containers… Never tried it before but will read up on it during my free time. Do you have any links you find useful and mind sharing them here?

Thanks!

Actually it was a course on udemy that I followed: https://www.udemy.com/course/apache-airflow/
The course is hands-on and by working through it I learned the concept of docker and Airflow. I now have many of my scripts run on airflow which is free and offers so much!

1 Like