
Build Your Own Data Pipeline - Andreas Kretz
We talked about:
Andreas’s background
Why data engineering is becoming more popular
Who to hire first – a data engineer or a data scientist?
How can I, as a data scientist, learn to build pipelines?
Don’t use too many tools
What is a data pipeline and why do we need it?
What is ingestion?
Can just one person build a data pipeline?
Approaches to building data pipelines for data scientists
Processing frameworks
Common setup for data pipelines — car price prediction
Productionizing the model with the help of a data pipeline
Scheduling
Orchestration
Start simple
Learning DevOps to implement data…
The skinny
The skinny isn't ready yet — notes appear once the transcript is processed.