Introduction to Data Engineering on Google Cloud links

Module 00: Course Introduction

Full name: Introduction to Data Engineering on Google Cloud

Product code: T-IDEGC-B

These links are based on the current course version.

Online course options:

• Introduction to Data Engineering on Google Cloud - Google Cloud Skills Boost
• Introduction to Data Engineering on Google Cloud - Coursera

Module 01: Data Engineering Tasks and Components

Websites

• BigQuery overview https://cloud.google.com/bigquery/docs/introduction
• BigQuery documentation https://cloud.google.com/bigquery/docs
• Cloud Storage documentation https://cloud.google.com/storage/docs
• Cloud Storage classes https://cloud.google.com/storage/docs/storage-classes
• Database options on Google Cloud https://cloud.google.com/products/databases
• Knowledge Catalog (Dataplex) https://cloud.google.com/dataplex/docs/introduction
• Analytics Hub overview https://cloud.google.com/bigquery/docs/analytics-hub-introduction
• Data formats overview https://cloud.google.com/bigquery/docs/loading-data#data_formats

Videos

• What is BigQuery? https://www.youtube.com/watch?v=m8WqxLd1jSc
• Analytics Hub in a minute https://www.youtube.com/watch?v=h8_xCbTYtTU

Module 02: Data Replication and Migration

Websites

• Storage Transfer Service https://cloud.google.com/storage-transfer-service
• Transfer Appliance documentation https://cloud.google.com/transfer-appliance/docs/4.0/overview
• Datastream overview https://cloud.google.com/datastream/docs/overview
• Database Migration Service https://cloud.google.com/database-migration
• gcloud storage command reference https://cloud.google.com/sdk/gcloud/reference/storage
• Change data capture with Datastream https://cloud.google.com/datastream/docs/concepts

Videos

• Introduction to Datastream for BigQuery https://www.youtube.com/watch?v=vMo6Zgkvt40
• What is Storage Transfer Service? https://www.youtube.com/watch?v=1oIw2mT_4-M

Module 03: The Extract and Load Data Pipeline Pattern

Websites

• Loading data into BigQuery https://cloud.google.com/bigquery/docs/loading-data
• BigQuery Data Transfer Service https://cloud.google.com/bigquery/docs/dts-introduction
• BigLake overview https://cloud.google.com/biglake/docs/introduction
• External tables in BigQuery https://cloud.google.com/bigquery/docs/external-tables
• Lakehouse on Google Cloud https://cloud.google.com/solutions/lakehouse
• bq command-line tool reference https://cloud.google.com/bigquery/docs/bq-command-line-tool
• LOAD DATA statement https://cloud.google.com/bigquery/docs/reference/standard-sql/other-statements#load_data_statement
• Querying Google Sheets data https://cloud.google.com/bigquery/docs/external-data-drive

Videos

• BigLake in a minute https://www.youtube.com/watch?v=UdpNYk0EP0g
• BigQuery Migration Service: SQL and data transfer https://www.youtube.com/watch?v=vY_m9dM6WxE
• Open Analytics Lakehouse on Google Cloud https://www.youtube.com/watch?v=_HmbwjODKgo

Module 04: The Extract, Load, and Transform Data Pipeline Pattern

Websites

• Dataform overview https://cloud.google.com/dataform/docs/overview
• Dataform quickstart https://cloud.google.com/dataform/docs/quickstart
• BigQuery scripting (procedural language) https://cloud.google.com/bigquery/docs/reference/standard-sql/scripting
• User-defined functions (UDFs) https://cloud.google.com/bigquery/docs/reference/standard-sql/user-defined-functions
• Stored procedures https://cloud.google.com/bigquery/docs/procedures
• Scheduled queries in BigQuery https://cloud.google.com/bigquery/docs/scheduling-queries
• Remote functions https://cloud.google.com/bigquery/docs/remote-functions
• BigQuery DataFrames (bigframes) https://cloud.google.com/bigquery/docs/bigquery-dataframes-introduction
• Apache Spark procedures in BigQuery https://cloud.google.com/bigquery/docs/spark-procedures

Videos

• Streamline Your SQL Workflows with Dataform https://www.youtube.com/watch?v=8pVM6iXq4Jw

Module 05: The Extract, Transform, and Load Data Pipeline Pattern

Websites

• Dataflow documentation https://cloud.google.com/dataflow/docs
• Apache Beam overview https://beam.apache.org/documentation/
• Dataflow templates https://cloud.google.com/dataflow/docs/concepts/dataflow-templates
• Google-provided Dataflow templates https://cloud.google.com/dataflow/docs/guides/templates/provided-templates
• Pub/Sub documentation https://cloud.google.com/pubsub/docs/overview
• Managed Service for Apache Spark (Dataproc) https://cloud.google.com/dataproc/docs
• Dataproc Serverless https://cloud.google.com/dataproc-serverless/docs
• Cloud Data Fusion https://cloud.google.com/data-fusion/docs
• Dataprep by Trifacta https://cloud.google.com/dataprep/docs
• Bigtable documentation https://cloud.google.com/bigtable/docs

Videos

• What is Dataflow? https://www.youtube.com/watch?v=KalJ0VuEM7s
• What is Pub/Sub? https://www.youtube.com/watch?v=cvu53CnZmGI
• Dataproc in a minute https://www.youtube.com/watch?v=Jj6mp7Sam10
• Data Science Workloads on Dataproc Serverless https://www.youtube.com/watch?v=dLcDBs9ilf0
• What is Cloud Bigtable? https://www.youtube.com/watch?v=P4q4nqJAamo
• Data Fusion in a minute https://www.youtube.com/watch?v=xjsNWh1TLKo

Module 06: Automation Techniques

Websites

• Cloud Scheduler https://cloud.google.com/scheduler/docs
• Cloud Workflows https://cloud.google.com/workflows/docs
• Cloud Composer (managed Airflow) https://cloud.google.com/composer/docs
• Cloud Run functions https://cloud.google.com/functions/docs
• Eventarc overview https://cloud.google.com/eventarc/docs/overview
• Dataproc Workflow Templates https://cloud.google.com/dataproc/docs/concepts/workflows/overview

Videos

• Cloud Composer in a minute https://www.youtube.com/watch?v=pSjD5nHkXLU
• Cloud Run in a minute https://www.youtube.com/watch?v=AL2rAmWFZjM
• Event-driven orchestration with Eventarc and Workflows https://www.youtube.com/watch?v=2SI12QE-2DU
• Getting started with Cloud Tasks and Cloud Scheduler https://www.youtube.com/watch?v=P9MCC9KmM_8
• Cloud Workflows in a minute https://www.youtube.com/watch?v=K_Gn0pZysLg

Module 07: Course Summary

Websites

• Google Cloud Data Engineering Learning Path https://cloud.google.com/learn/training/data-engineering-and-analytics
• Google Cloud Certification - Professional Data Engineer https://cloud.google.com/learn/certification/data-engineer
• Google Cloud Architecture Center https://cloud.google.com/architecture
• Google Cloud blog - Data Analytics https://cloud.google.com/blog/products/data-analytics