Building Batch Data Pipelines on GCP
- Offered byCoursera
- Public/Government Institute
Building Batch Data Pipelines on GCP at Coursera Overview
Duration | 13 hours |
Total fee | Free |
Mode of learning | Online |
Difficulty level | Intermediate |
Official Website | Explore Free Course  |
Credential | Certificate |
- Overview
- Highlights
- Course Details
- Curriculum
Building Batch Data Pipelines on GCP at Coursera Highlights
- Taught by top companies and universities.
- Affordable programs and 7 day free trial.
- Shareable Certificate upon completion.
Building Batch Data Pipelines on GCP at Coursera Course details
- Data pipelines typically fall under one of the Extra-Load, Extract-Load-Transform or Extract-Transform-Load paradigms. This course describes which paradigm should be used and when for batch data. Furthermore, this course covers several technologies on Google Cloud Platform for data transformation including BigQuery, executing Spark on Cloud Dataproc, pipeline graphs in Cloud Data Fusion and serverless data processing with Cloud Dataflow. Learners will get hands-on experience building data pipeline components on Google Cloud Platform using QwikLabs.
- New! CERTIFICATE COMPLETION CHALLENGE to unlock benefits from Coursera and Google Cloud
- Enroll and complete Cloud Engineering with Google Cloud or Cloud Architecture with Google Cloud Professional Certificate or Data Engineering with Google Cloud Professional Certificate before November 8, 2020 to receive the following benefits;
- => Google Cloud t-shirt, for the first 1,000 eligible learners to complete. While supplies last. > Exclusive access to Big => Interview ($950 value) and career coaching
- => 30 days free access to Qwiklabs ($50 value) to earn Google Cloud recognized skill badges by completing challenge quests
Building Batch Data Pipelines on GCP at Coursera Curriculum
Introduction
Course Introduction
Getting Started with Google Cloud and Qwiklabs
EL, ELT, ETL
Quality considerations
How to carry out operations in BigQuery
Shortcomings
ETL to solve data quality issues
EL, ELT, ETL
The Hadoop ecosystem
Running Hadoop on Cloud Dataproc
GCS instead of HDFS
Optimizing Dataproc
Optimizing Dataproc Storage
Optimizing Dataproc Templates and Autoscaling
Optimizing Dataproc Monitoring
Lab Intro: Running Apache Spark jobs on Cloud Dataproc
Summary
Executing Spark on Cloud Dataproc
Manage Data Pipelines with Cloud Data Fusion and Cloud Composer
Introduction
Components of Data Fusion
Building a Pipeline
Exploring Data using Wrangler
Lab: Building and executing a pipeline graph in Cloud Data Fusion
Orchestrating work between GCP services with Cloud Composer
Apache Airflow Environment
DAGs and Operators
Workflow scheduling
Monitoring and Logging
Lab: An Introduction to Cloud Composer
Cloud Data Fusion and Cloud Composer
Cloud Dataflow
Why customers value Dataflow
Building Cloud Dataflow Pipelines in code
Key considerations with designing pipelines
Transforming data with PTransforms
Lab: Building a Simple Dataflow Pipeline
Aggregating with GroupByKey and Combine
Lab: MapReduce in Cloud Dataflow
Side Inputs and Windows of data
Lab: Practicing Pipeline Side Inputs
Creating and re-using Pipeline Templates
Cloud Dataflow SQL pipelines
Data Processing with Cloud Dataflow
Course Summary
Other courses offered by Coursera
Student Forum
Anything you would want to ask experts?