Data Processing Fundamentals

Processing data
D ATA E N G I N E E R I N G F O R E V E R YO N E
Hadrien Lacroix
Content Developer at DataCamp
DATA ENGINEERING FOR EVERYONE
A general de nition
Data processing: converting raw data into meaningful information

Data processing value
Conceptually At Spot ix
Remove unwanted data No long term need for testing feature data
Optimize memory, process and network Can't afford to store and stream les this
costs big
Convert data from one type to another

Data processing value
Conceptually At Spot ix
Remove unwanted data No need for lossless format
To save memory Can't afford to store les this big
Convert data from one type to another Convert songs from .flac to .ogg
Organize data Reorganize data from the data lake to data

warehouses
To t into a schema/structure
Employee table example
Increase productivity
Enable data scientists

How data engineers process data
Data manipulation, cleaning, and tidying tasks Rejecting corrupt song les
that can be automated Deciding what happens with missing metadata
that will always need to be done Separate artists and albums tables...
Store data in a sanely structured database ...but provide view combining them
Create views on top of the database tables Indexing
Optimizing the performance of the database

1 The difference between batch and stream will be explained in the next lesson!

Summary
What data processing is
Why it's necessary
What it consists in
How we process data at Spot ix

Let's practice!
Scheduling data
Hadrien Lacroix
Scheduling
Can apply to any task listed in data processing
Scheduling is the glue of your system
Holds each piece and organize how they work together
Runs tasks in a speci c order and resolves all dependencies

Manual, time and sensor scheduling
Manually Manually update the employee table

Manual, time and sensor scheduling
Automatically run at a speci c time Update the employee table at 6 AM
Automatically run if a speci c condition is met

Sensor scheduling

Manual, time, and sensor scheduling
Automatically run at a speci c time Update the employee table at 6 AM
Automatically run if a speci c condition is met Update the department tables if a new
Sensor scheduling employee was added

Batches and streams
Batches Songs uploaded by artists
Group records at intervals Employee table
Often cheaper Revenue table
Streams New users signing in
Send individual records right away
Another example: online vs. of ine listening

Scheduling tools

Summary
What scheduling is
Different ways to set it up
Difference between batches and streams
How scheduling is implemented at Spot ix
Air ow, Luigi

Let's practice!
Parallel computing
Hadrien Lacroix
Parallel computing
Basis of modern data processing tools
Necessary:
Mainly because of memory
Also for processing power
How it works:
Split tasks up into several smaller subtasks
Distribute these subtasks over several computers

1 Emojis by Mohamed Hassan

Bene ts and risks of parallel computing
Employees = processing units
Advantages
Extra processing power
Reduced memory footprint
Disadvantages
Moving data incurs a cost
Communication time

Summary
Bene ts and risks
How it's implemented at Spot ix

Let's practice!
Cloud computing
Hadrien Lacroix
Content Developer
Cloud computing for data processing
Servers on premises Servers on the cloud
Bought Rented
Need space Don't need space
Electrical and maintenance cost Use just the resources we need
Enough power for peak moments When we need them
Processing power unused at quieter times The closer to the user the better

Cloud computing for data storage
Database reliability: data replication
Risk with sensitive data

Multicloud
Pros Cons
Reducing reliance on a single vendor Cloud providers try to lock in consumers
Cost-ef ciencies Incompatibility
Local laws requiring certain data to be Security and governance

physically present within the country
Militating against disasters

Summary
Bene ts and risks of cloud computing
How it is implemented at Spot ix
Can cite the main cloud providers and their services

Let's practice!
We are the
champions
Hadrien Lacroix
Actually, YOU are the champion!

What you learned - chapter 1
What Data Engineering is
How important it is
How data engineers differ from data scientists
What a data pipeline is and how it works

The different structures data can take
How fundamentals SQL is
The differences between data lakes, data warehouses and databases

How data is processed
How scheduling holds it all together
Parallel computing
Cloud computing

And some more
What SQL code actually looks like
Main tools and technologies used in data engineering
And some more

Lexicon

A promise is a promise, DataChamps!
All the exercises are song titles
Search for "DataChamps" on Spotify

Congratulations!

Data Processing Fundamentals

Hochgeladen von

Dokumentinformationen

Originaltitel

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

Data Processing Fundamentals

Hochgeladen von

Copyright:

Verfügbare Formate

Processing data

DATA ENGINEERING FOR EVERYONE

Convert data from one type to another

DATA ENGINEERING FOR EVERYONE

Remove unwanted data No need for lossless format

To save memory Can't afford to store les this big

Organize data Reorganize data from the data lake to data

DATA ENGINEERING FOR EVERYONE

DATA ENGINEERING FOR EVERYONE

DATA ENGINEERING FOR EVERYONE

Why it's necessary

How we process data at Spot ix

DATA ENGINEERING FOR EVERYONE

Scheduling is the glue of your system

Holds each piece and organize how they work together

Runs tasks in a speci c order and resolves all dependencies

DATA ENGINEERING FOR EVERYONE

DATA ENGINEERING FOR EVERYONE

Automatically run at a speci c time Update the employee table at 6 AM

Automatically run if a speci c condition is met

DATA ENGINEERING FOR EVERYONE

Automatically run at a speci c time Update the employee table at 6 AM

DATA ENGINEERING FOR EVERYONE

DATA ENGINEERING FOR EVERYONE

DATA ENGINEERING FOR EVERYONE

Different ways to set it up

Difference between batches and streams

How scheduling is implemented at Spot ix

Air ow, Luigi

DATA ENGINEERING FOR EVERYONE

Also for processing power

Distribute these subtasks over several computers

DATA ENGINEERING FOR EVERYONE

DATA ENGINEERING FOR EVERYONE

Reduced memory footprint

DATA ENGINEERING FOR EVERYONE

How it's implemented at Spot ix

DATA ENGINEERING FOR EVERYONE

Need space Don't need space

Electrical and maintenance cost Use just the resources we need

Enough power for peak moments When we need them

DATA ENGINEERING FOR EVERYONE

Risk with sensitive data

DATA ENGINEERING FOR EVERYONE

Reducing reliance on a single vendor Cloud providers try to lock in consumers

Cost-ef ciencies Incompatibility

Local laws requiring certain data to be Security and governance

Militating against disasters

DATA ENGINEERING FOR EVERYONE

How it is implemented at Spot ix

Can cite the main cloud providers and their services

DATA ENGINEERING FOR EVERYONE

DATA ENGINEERING FOR EVERYONE

How data engineers differ from data scientists

What a data pipeline is and how it works

DATA ENGINEERING FOR EVERYONE

How fundamentals SQL is

The differences between data lakes, data warehouses and databases

DATA ENGINEERING FOR EVERYONE

How scheduling holds it all together

DATA ENGINEERING FOR EVERYONE

Main tools and technologies used in data engineering

And some more