How to make pipelines of pipelines elegantly

In the Multiple Dependencies course in the Data Engineering track, I learned a ton about how to implement pipelines functionally, which was amazing!

I am an entry level Machine Learning Engineer and I regularly test lots of models on data, maybe with normalizing the data or not, and plenty of variations to see what’s best.

My question is: If I want to run lots of different types of pipelines (I imagine it like a pipeline of pipelines) would I just run them in a function I write, still using the decorators within the function?

If someone could post an example of what a good pythonic implementation might look like given the simple sample functions in the course, that would be incredible!


Hi, Scott

Just saw your post. One tool that was incredibly helpful for me at work was dagster.

In a way, the framework is still being developed, so somethings don’t work in a 100% clear understandable way, but the core functionalities are awesome.

Maybe this will help you see an interesting implementation of DAGs - or, as it was on my case - take of the burden of coming up with a framework to run my pipelines.