Apache Airflow on AWS EKS: The Hands-On Guide

Apache Airflow on AWS EKS: The Hands-On Guide

Struggling to set up Airflow on AWS EKS?

You are at the right place!

With more than 15,000 students, I got many feedbacks about how difficult it is to configure Airflow on AWS with the official Helm chart.

Guess what? You are about to learn everything you need to set up a production-ready architecture for Apache Airflow on AWS EKS

This course is designed to guide you through the different steps of creating a real world architecture:

1. Configuring the EKS cluster following best practices

2. Deploying automatically changes with GitOps

3. Using Helm to configure and set up Airflow on Kubernetes

4. Configuring the official Helm chart of Airflow to use the Kubernetes Executor and many different features

5. Deploying DAGs in Airflow with Git-Sync and AWS EFS

6. Deploying DAGs/Airflow through CI/CD pipelines with AWS CodePipeline

7. Testing your DAGs automatically

8. Securing your credentials and sensitive data in a Secret Backend

9. Enabling remote logging with AWS S3

10. Creating 3 different environments dev/staging and prod

11. Making the production environment scalable and highly available

12. and more!

WARNING:

• The course is not meant to learn the basic of Airflow, you must be already familiar with it.

• If you already know Kubernetes/Docker/AWS your learning will be easier, but no worries I explain everything you need.

• YOU WON'T learn how to interact with AWS in your DAGs. This course is about designing an architecture not about DAGs.

• The course is NOT free-tier eligible as we are going to use many AWS services and set up a real world architecture.

How to set up Apache Airflow on AWS EKS with the Kubernetes Executor and be production-ready!

Url: View Details

What you will learn

• How to Set Up a Production Ready Architecture for Airflow on AWS EKS From A-Z
• How to deploy DAGs from Git (public and private)
• How to Create CI/CD Pipelines with AWS CodePipeline Deploy DAGs

Rating: 4.24194

Level: All Levels

Duration: 9.5 hours

Instructor: Marc Lamberti