Prefect as a Data Engineering Project Workflow Tool (Video)

Choosing a workflow management tool that suits a team’s current needs and projected future projects can be a difficult decision. Excellent options abound (NiFi, Airflow, etc), and every team’s use case is unique.

In this talk, Mary Clair Thompson - Data Engineer with Duke University's Office of Information Technology - discusses the decisions that led her team choose Prefect as their workflow management tool. She gives an overview of their system’s architecture and integration with CI/CD and version control tools. Finally, she demonstrates some of the features of the Prefect UI.