Building ETL Pipelines with Python - Create and deploy enterprise-ready ETL pipelines by employing modern methods

Brij Kishore Pandey, Emily Ro Schoof

Summary

Modern extract, transform, and load (ETL) pipelines for data engineering have favored the Python language for its broad range of uses and a large assortment of tools, applications, and open source components. With its simplicity and extensive library support, Python has emerged as the undisputed choice for data processing.
In this book, you’ll walk through the end-to-end process of ETL data pipeline development, starting with an introduction to the fundamentals of data pipelines and establishing a Python development environment to create pipelines. Once you've explored the ETL pipeline design principles and ET development process, you'll be equipped to design custom ETL pipelines. Next, you'll get to grips with the steps in the ETL process, which involves extracting valuable data; performing transformations, through cleaning, manipulation, and ensuring data integrity; and ultimately loading the processed data into storage systems. You’ll also review several ETL modules in Python, comparing their pros and cons when building data pipelines and leveraging cloud tools, such as AWS, to create scalable data pipelines. Lastly, you’ll learn about the concept of test-driven development for ETL pipelines to ensure safe deployments.
By the end of this book, you’ll have worked on several hands-on examples to create high-performance ETL pipelines to develop robust, scalable, and resilient environments using Python.

Available since: 09/29/2023.

Print length: 246 pages.

Science & Technology

Other books that might interest you

The Cloud Computing Revolution -...

Lisa Carter

" The Cloud Computing Revolution: From Virtualization to Automation: Unveiling the Cloud Computing Revolution." vividly explains how cloud computing has changed the whole scenario of technology, reformed whole industries, and changed the course of information technology. It gives an overview of how cloud computing has been evolving from its initial days of virtualization to the automation era, wherein innovative technologies at the edge, AI, and serverless architecture are driving the change at a pace that has never been seen before. 
The comprehensive book is dedicated to explaining, in five elaborated chapters, major ideas, methods, and technologies relating to cloud computing. Learn about the history and development of cloud computing; study core cloud service models: SaaS-Software as a Service, PaaS-Platform as a Service, and IaaS-Infrastructure as a Service; and learn new paradigms that continue to push the envelope regarding what can be done with the cloud. 
It pinpoints the ways for cloud computing adoption, such as techniques of migration, optimization, and governance. It puts into view practical case studies that spot how effective cloud deployments have been across industries. 
This book, Cloud Computing Revolution, is important for the IT professional, the corporate officer, and anyone interested in technology because it introduces various trends that are emerging and the role of cloud computing in this process of digital transformation. Whether one is a novice or highly experienced in cloud computing, this book provides the competencies and skills to approach, with confidence, the ever-evolving world of cloud computing.

Show book

Cryptocurrency Trading - The...

Owen Hill

This Audiobook will focus strictly on the best cryptocurrencies that exist today on the market. The aim, is to learn and understand why specific cryptocurrencies are way more improved and advantageous than others. In order to identify an outstanding cryptocurrency, the best coins have been handpicked for your reference, and examined in full depth. This book will teach you, how to analyze every nitty-gritty details, and provide you with fundamental knowledge on what is most important to consider, when thinking about investing in cryptocurrency. First, this book will begin with old school coins, those that are strongest by market capitalisation, and already dominating the market. Next, it moves on, explaining enhanced technologies, such as privacy based coins. Privacy based cryptocurrencies are already dominating a large percentage of the market, and this book will show you how they diverge from one another. With this book you will learn about the power of Cryptocurrency, and where it originated. You will be shown the huge potential of digital currency, and how to invest in it. And yes, you will also learn why the slow death of our most recent iteration of money will result in the boom of the virtual dollar.

Show book

Love Sex & Mushrooms -...

Cardy Raper

When a young girl, Cardy Raper told her mother, "When I grow up I want to be a scientist and make grand discoveries!" Her mother responded, "You could become a nurse." Science was a man's world then. Cardy refused to take "no" for an answer.   Her dream seemed attainable when she met her mentor, Professor John "Red" Raper at the University of Chicago who said "Yes, you can be a scientist!" They became soul mates, fell in love, married, parented children, moved to Harvard, and did research together on the versatile sex life of fungi.  Red's untimely death left Cardy alone in the competitive world of cuttingedge science. She carried on, obtained a doctoral degree, learned the techniques of molecular genetics, and established her own laboratory where she conducted pioneering research on the genetic and molecular determinants of sexual reproduction in a mushroombearing fungus with 20,000 different sexes. This fungus has served as a model organism for exploring the way in which sensing molecules, such as pheromones, function to communicate in more complex organisms.

Show book

The Enigma Story - The Truth...

Dermot Turing

The Enigma cipher was supposed to be the German's impenetrable defence for its military communications against prying eyes during World War II. All manner of secrets were entrusted to it. When the Allies finally managed to crack the code, it heralded a turning point in the war. 
 
This is the fascinating story of how the code was created, adopted by the Nazis, and finally broken. Dermot Turing, the nephew of the great codebreaker Alan Turing, explores the twists and turns of German encryption efforts from the end of World War I through to Hitler's demise and the great lengths to which the Allies went to break it. 
 
The Enigma Story reveals the efforts of the codebreakers at Bletchley Park, the machines called 'bombes' specially designed to break it, and the vast resources devoted in America to decrypting German messages. From the cloak-and-dagger heroics of men like Hans-Thilo Schmidt and Gustave Bertrand to the brilliant mathematical discoveries of men like Henryk Zygalski and Dilly Knox to the fraught decision-making of Allied High Command, the battle for the code was at the heart of the Allied victory in World War II. 
 
This extraordinary tale of intrigue, ingenuity and courage brings to life the complete story of the Enigma in a lively and entertaining narrative.

Show book

Data Science for Beginners - A...

Simon Winston

This book is an accessible and comprehensive guide that demystifies the world of data science and empowers readers to embark on their journey of understanding and utilizing the power of data. Whether you're a complete novice or have a basic understanding of the subject, this book will equip you with the foundational knowledge and practical skills needed to navigate the ever-evolving landscape of data science.
As you delve deeper into the book, you'll discover the essential components of the data science process, including data collection, data cleaning, data exploration, and data visualization. You'll learn how to extract valuable insights from data using statistical analysis and machine learning algorithms. The author provides step-by-step instructions and practical exercises that reinforce your understanding of these concepts, allowing you to apply them to real-world problems.
What sets this book apart is its emphasis on hands-on learning. Through engaging projects and case studies, you'll have the opportunity to tackle real-world datasets and develop your data science skills in a practical context. You'll learn how to clean messy data, perform exploratory data analysis, build predictive models, and effectively communicate your findings to non-technical stakeholders.
Whether you're a student, a professional transitioning into data science, or simply curious about the field, tis book equips you with the knowledge and tools necessary to kick-start your data science journey. By the end of this book, you'll have a solid foundation in data science and be well-prepared to take on more advanced topics in the field.

Show book

Adventures of Old Mr Toad

Thornton W. Burgess

The Adventures of Old Mr. Toad is another in the long line of children's books by conservationist Thornton W. Burgess. In this book, we follow the adventures of Old Mr. Toad as he joins the Spring Chorus at the Smiling Pool, shows off his babies, displays his special tongue to Peter Rabbit and has a very special encounter with Buster Bear. We also learn little lessons about life such as pride can burst like a great big bubble, your friends' hidden talents may surprise you and it is considered impolite to watch someone change his clothes - especially when he eats them. (Summary by John Lieder)

Show book