How to Read This Book

This book is different from usual IT books.

The Philosophy

The Idea, instead of being a reference handbook, this book is a story from start to finish for the most part. Convergent evolution (CE) and patterns can be explored on their own. Still, each chapter builds up to the actual data engineering design patterns (DEDP). If you skip, you might not understand the underlying history and concepts of certain concepts lived ahead of them.

The Narrative

My 20-year journey in the field—transitioning from business intelligence to big data, and eventually, data engineering. Along the way, I've explored the convergent evolution of design patterns, detailing my experiences as a data engineer and technical author.

The Approach

This book provides a bottom-up approach from standard data engineering terms established over the last two decades, analyzing beyond the hype. Analyze different convergent evolution and their definitions, history, and core concepts, finding foundational patterns they follow despite the various evolutions they were invented. Based on the several CEs and found data engineering patterns, I tried to convert them into best practices and applicable data engineering design patterns.

Navigation

After a short intro to the book and the relevant terms, you can choose to explore the convergent evolution and the design patterns interesting to you and read through how I discovered them through relevant CE and arrived at a recurring pattern and ultimately, the design patterns.

I will add personal anecdotes from my 20+ years of experience where I find it appropriate and to be adding value.

Good to know:

1. All definitions of data engineering that are outdated quickly are referenced to the Data Engineering Vault where they get updated on a more regular basis. Sometimes, I skip over a deeper explanation, as you'll find more details in the referenced Vault article. As this is the online version of the book, it's easy to look up each term you do not know and skip the one you do know.
2. The definitions of terms in the field are often outdated faster than this book can cope with, that's why the focus of this book is on the patterns that are long-term and updated less frequently.

And lastly, the book is designed to be written short and succinctly. This hopefully leads to a more interesting book by cutting out anything non-relevant.

Website

Navigation on the website is trivial, but here are some hints:

• Hit s anywhere for a search within the whole book
• Hit the burger icon for toggle the left panel with chapter
• hit the brush for changing the theme (I recommend Rose Pink). Light theme is the default, as Diagrams will be shown best with a white background.

Terminologies

To align on terminologies used in this book, here are the common abbreviations for this book in the glossary and explainer for callouts that highlight certain parts of this book.

Hints and Callouts

The call-outs (also known as admonitions) are hints to highlight certain things. It's a way to include less relevant side content without significantly interrupting the reading flow.

Below are the most used callouts shortly, explaining what you can expect from them.

Glossary

Hopefully, most terms are explained with the first use in this book, but in any case here are some abbreviations and common terms I use in this book as a reference.

For more details and terms on my Data Engineering Vault, simply hit Cmd+k to search for any term.