There is a large volume of literature (1, 2, 3) about scaling data pipelines. “Use Kafka! Build a lake house! Don’t build a lake house, use Snowflake! Don’t use Snowflake, use XYZ!” However, with advances in hardware and the rapid maturation of data software, there is a simpler approach. This article will light up the path to highly performant single node analytics with an MDS-in-a-box open source stack: Meltano, DuckDB, dbt, & Apache Superset on Windows using Windows Subsystem for Linux (WSL). There are many options within the MDS, so if you are using another stack to build an MDS-in-a-box, please share it with the community on the DuckDB Twitter, GitHub, or Discord, or the dbt slack! Or just stop by for a friendly debate about our choice of tools
https://duckdb.org/2022/10/12/modern-data-stack-in-a-box.html
https://duckdb.org/2022/10/12/modern-data-stack-in-a-box.html
The Modern Data Stack: Past, Present, and Future
https://www.getdbt.com/blog/future-of-the-modern-data-stack
Learn how some of the most amazing companies in the world are organising their data stack. Learn more about the tools that they are using and why.