Data Engineering Program

At Scholars Venue, we'll help you connect what you learn to what you do in real-world projects, illustrating how data flows through systems step-by-step with clear and engaging visuals.

Download

Course Syllabus

Trail Videos Free

Video 1

Video 2

Video 3

Video 4

SQL for Data Engineering

Introduction to Database

Why RDBMS?
Importance of RDBMS?

Postgres Setup & Environment

PostgreSQL Installation

Database Fundamentals

Database Operations
Table Operations
Data Types & Inserting Data

Data Manipulation (CRUD)

CRUD Operations

Filtering Data

WHERE Clause

Aggregation & Grouping

Aggregate Functions
GROUP BY

Sorting Data

ORDER BY

SQL Joins

INNER JOIN
LEFT JOIN
RIGHT JOIN
FULL JOIN
Joins Summary

Database Relationships

Primary & Foreign Keys

Advanced SQL

Window Functions
PARTITION BY
Advanced SQL – Part 1
Advanced SQL – Part 2

AI for SQL

AI-Assisted SQL Development

Industry Updates & New Features

SQL Trends, Tools, and New Features

Python for Data Engineering

Introduction to Python

Why Python for Data Engineering?
Python Setup and VS Code

Python Fundamentals

Variables and Data Types
Operators and Expressions
Conditions
Loops

Python Collections

Lists
Tuples and Sets
Dictionaries

Functions and Modules

Functions
Modules and Packages
Python Virtual Environments

Files and Error Handling

File Handling
Exception Handling
Logging Basics

Data Formats

Working with CSV Files
Working with JSON Files
Working with Excel Files

Python Database Integration

Connecting Python with PostgreSQL
Reading Data from PostgreSQL
Writing Data to PostgreSQL

Data Processing with Python

Introduction to Pandas
Data Cleaning with Pandas
Data Transformation with Pandas
Data Validation with Pandas

ETL Development

Understanding ETL Pipelines
Building Python ETL Scripts
Configuration Files
Reusable ETL Framework

API and Automation

Introduction to APIs
Reading API Data Using Python
Scheduling Python Scripts
Automation Best Practices

AI for Python

AI-Assisted Python Development
AI for Debugging and Documentation
Prompt Engineering for Python

Industry Updates & New Features

Python Tools and Data Engineering Trends

Big Data on Cloudera

Introduction to Big Data

What is Big Data?
Big Data Challenges
Hadoop Ecosystem Overview

Cloudera Environment

Introduction to Cloudera
Cloudera Architecture
Cloudera Setup and Services

Linux for Big Data

Linux Commands
File System and Permissions
Shell Script Basics

HDFS

HDFS Architecture
HDFS Commands
Loading Data into HDFS

MapReduce

MapReduce Basics
MapReduce Architecture
MapReduce Processing

Hive

Hive Introduction
Hive Databases and Tables
Managed and External Tables
HiveQL Queries and Joins
Partitioning and Bucketing

Sqoop

Sqoop Introduction
PostgreSQL to HDFS
PostgreSQL to Hive
Hadoop to PostgreSQL

AI for Big Data

AI for Hive, Sqoop, and Hadoop

Industry Updates & New Features

Modern Big Data Architecture

Apache Spark and PySpark

Introduction to Apache Spark

Why Apache Spark?
Spark Architecture
Spark Components

Spark Fundamentals

RDDs
DataFrames
Spark SQL

PySpark Setup

PySpark Environment
SparkSession
Reading Data Files

PySpark Transformations

Select, Filter, and Rename
Columns and Expressions
Aggregations and GROUP BY
Joins and Window Functions

PySpark Data Processing

Handling Null Values
Data Cleaning
Data Transformation
Writing Output Data

Spark Performance

Partitioning and Shuffle
Cache and Persist
Spark Optimization
Debugging Spark Jobs

Spark Streaming

Structured Streaming Basics
Streaming Data Pipeline

AI for PySpark

AI-Assisted PySpark Development

Industry Updates & New Features

Spark Trends and Best Practices

Databricks Data Engineering

Introduction to Databricks

What is Databricks?
Databricks Workspace
Clusters and Compute

Databricks Development

Notebooks
Databricks File System
Running PySpark Notebooks

Delta Lake

Delta Lake Introduction
Delta Tables and ACID
Delta MERGE and Upserts
Time Travel and Optimization

Lakehouse Architecture

Bronze, Silver, and Gold Layers
Building Medallion Pipelines
Data Quality Checks

Databricks Workflows

Jobs and Workflows
Parameterized Notebooks
Scheduling Pipelines
Monitoring and Troubleshooting

Databricks on AWS

AWS and Databricks Setup
Amazon S3 Integration
Delta Lake on S3
Cloud ETL Pipeline

AI for Databricks

AI for Databricks Development
AI for Pipeline Monitoring

Industry Updates & New Features

Databricks and Lakehouse Trends

Project Phase 1: On-Premise Cloudera Data Engineering

Project Foundation

Project Roadmap and Business Problem
On-Premise Architecture Design

Source Data Preparation

PostgreSQL Source Database Setup
Source Tables and Data Loading

Data Ingestion

Sqoop Import to HDFS
Sqoop Import to Hive

Data Processing

Hive Transformations
PySpark Data Cleaning
PySpark Business Transformations

Data Warehouse Layer

Building Fact and Dimension Tables
Creating Final Data Mart

Project Delivery

Data Validation and Testing
Git Project Version Control
Final Project Presentation

Project Phase 2: AWS Databricks Data Engineering

Cloud Project Foundation

Cloud Business Problem and Architecture
AWS and Databricks Project Setup

Cloud Data Lake Setup

Creating Amazon S3 Buckets
Raw Data Landing Zone

Bronze Layer

Loading Raw Data into Bronze
Bronze Delta Tables

Silver Layer

Data Cleaning and Transformation
Silver Delta Tables

Gold Layer

Business Aggregations
Gold Data Mart Tables

Pipeline Orchestration

Databricks Jobs and Workflows
Pipeline Scheduling
Monitoring and Error Handling

Project Delivery

Data Quality and Optimization
GitHub Project Documentation
Final Cloud Project Presentation

Projects

Apply what you learn by building real-world projects from this specialization.

ETL Pipeline with Apache Spark

Build a scalable ETL pipeline using Apache Spark to process large-scale datasets on AWS.

Real-Time Data Streaming

Implement a real-time streaming architecture using Kafka and Azure Event Hubs.

Cloud Data Warehouse

Design and deploy a cloud data warehouse with automated data ingestion and transformation.