Introduction to Workflow Management Platform (Airflow)

Start Course
level
Intermediate
Subject
Data Science
Course Length
50 minutes
Subscription
Data Academy

Course Description

In this course, you will look at creating a pipeline by breaking down a job into multiple executable stages. You will implement a simple linear pipeline and then move further by implementing a multi-stage data pipeline, then automate the multi-stage pipeline using Bash. Further to this you will improve the efficiency by running the pipeline as an asynchronous process using the ETL workflow and then create DAG for the pipeline and implement it using Airflow. This course is part of the Artificial Intelligence Infrastructure learning path - complete this path to learn how to build AI systems for enterprise.

Requirements

  • To complete the Artificial Intelligence Infrastructure learning path you should have intermediate knowledge of Python and Google Collab. You should also understand data preparation work as well as knowledge of common ML and AI algorithms, different databases such as MySQL, MongoDB, and Cassandra and data formats for large collections of data. Knowledge of Spark is also useful, along with AI system design principles.