Data Engineering Weekly

Semantic layers have been with us for decades—sometimes buried inside BI tools, living in analysts’ heads. But as data complexity grows and AI pushes its way into the stack, the conversation is shifting. In a recent conversation with David Jayatillake , a long-time data leader with experience at Cube, Delphi Labs, and multiple startups, we explored how semantic layers move from BI lock-in to invis...

Data Engineering Weekly recently hosted Jacopo Tagliabue , CTO of Bauplan , for an insightful podcast exploring innovative solutions in data engineering. Jacopo shared valuable perspectives drawn from his entrepreneurial journey, his experience building multiple companies, and his deep understanding of data engineering challenges. This extensive conversation spanned the complexities of data engine...

In our latest episode of Data Engineering Weekly, co-hosted by Aswin, we explored the practical realities of AI deployment and data readiness with our distinguished guest, Avinash Narasimha, AI Solutions Leader at Koch Industries. This discussion shed significant light on the maturity, challenges, and potential that generative AI and data preparedness present in contemporary enterprises. Introduci...

The modern data stack constantly evolves, with new technologies promising to solve age-old problems like scalability, cost, and data silos. Apache Iceberg, an open table format, has recently generated significant buzz. But is it truly revolutionary, or is it destined to repeat the pitfalls of past solutions like Hadoop? In a recent episode of the Data Engineering Weekly podcast, we delved into thi...

Lakehouse architecture represents a major evolution in data engineering. It combines data lakes' flexibility with data warehouses' structured reliability, providing a unified platform for diverse data workloads ranging from traditional business intelligence to advanced analytics and machine learning. Roy Hassan , a product leader at Upsolver, now Qlik , offers a comprehensive reality check on Lake...

Fluss is a compelling new project in the realm of real-time data processing. I spoke with Jark Wu , who leads the Fluss and Flink SQL team at Alibaba Cloud, to understand its origins and potential. Jark is a key figure in the Apache Flink community, known for his work in building Flink SQL from the ground up and creating Flink CDC and Fluss. You can read the Q&A version of the conversation here, a...

Exploring the Evolution of Lakehouse Technology: A Conversation with Vinoth Chandar and Onehouse CEO In this episode, Ananth, author of Data Engineering Weekly and CEO of Onehouse, discusses the latest developments in the Lakehouse technology space, particularly focusing on Apache Hudi, Iceberg, and Delta Lake. They discuss the intricacies of building high-scale data ecosystems, the impact of tabl...

Agents of Change: Navigating 2025 with AI and Data Innovation In this episode of Dew, the hosts and guests discuss their predictions for 2025, focusing on the rise and impact of agentic AI. The conversation covers three main categories: 1. The role of agent AI 2. The future workforce dynamic involving human and AI agent 3. Innovations in data platforms heading into 2025. Highlights include insight...

Welcome to another insightful edition of Data Engineering Weekly. As we approach the end of 2023, it's an opportune time to reflect on the key trends and developments that have shaped the field of data engineering this year. In this article, we'll summarize the crucial points from a recent podcast featuring Ananth and Ashwin, two prominent voices in the data engineering community. Understanding th...

Welcome to another episode of Data Engineering Weekly. Aswin and I select 3 to 4 articles from each edition of Data Engineering Weekly and discuss them from the author’s and our perspectives. On DEW #133, we selected the following article LakeFs: How to Implement Write-Audit-Publish (WAP) I wrote extensively about the WAP pattern in my latest article, An Engineering Guide to Data Quality - A Data...

Welcome to another episode of Data Engineering Weekly. Aswin and I select 3 to 4 articles from each edition of Data Engineering Weekly and discuss them from the author’s and our perspectives. On DEW #132, we selected the following article Cowboy Ventures: The New Generative AI Infra Stack Generative AI has taken the tech industry by storm. In Q1 2023, a whopping $1.7B was invested into gen AI star...

Welcome to another episode of Data Engineering Weekly. Aswin and I select 3 to 4 articles from each edition of Data Engineering Weekly and discuss them from the author’s and our perspectives. On DEW #131, we selected the following article Ramon Marrero: DBT Model Contracts - Importance and Pitfalls dbt introduces model contract with 1.5 release. There were a few critics of the dbt model implementa...

Welcome to another episode of Data Engineering Weekly. Aswin and I select 3 to 4 articles from each edition of Data Engineering Weekly and discuss them from the author’s and our perspectives. On DEW #129, we selected the following article DoorDash identifies Five big areas for using Generative AI. Generative AI took the industry by storm, and every company is trying to figure out what it means to...

Welcome to another episode of Data Engineering Weekly. Aswin and I select 3 to 4 articles from each edition of Data Engineering Weekly and discuss them from the author’s and our perspectives. On DEW #124, we selected the following article dbt: State of Analytics Engineering dbt publishes the state of analytical [data???🤔] engineering. If you follow Data Engineering Weekly, We actively talk about...

Welcome to another episode of Data Engineering Weekly Radio. Ananth and Aswin discussed a blog from BuzzFeed that shares lessons learned from building products powered by generative AI. The blog highlights how generative AI can be integrated into a company's work culture and workflow to enhance creativity rather than replace jobs. BuzzFeed provided their employees with intuitive access to APIs and...

DBT Reimagined by Pedram Navid The challenge with this, having the Jinja templating, I found out two things. One is like; it is on runtime. So you have to build it and then run some simulations to understand whether you did it correctly or not. Jinja Templates also add cognitive load. The developers have to know how the Jinja template will work; how SQL will work, and it becomes a bit difficult to...

Hey folks, have you heard about the Data Council conference in Austin? The three-day event was jam-packed with exciting discussions and innovative ideas on data engineering and infrastructure, data science and algorithms, MLOps, generative AI, streaming infrastructure, analytics, and data culture and community.  "People are so nice in the data community. Meeting them and brainstorming with many id...

In this episode of Data Engineering Weekly Radio, we delve into modern data stacks under pressure and the potential consolidation of the data industry. We refer to a four-part article series that explores the data infrastructure landscape and the Software as a Service (SaaS) products available in data engineering, machine learning, and artificial intelligence. We discussed that the siloed nature o...

We are back in our Data Engineering Weekly Radio for edition #121 . We will take 2 or 3 articles from each week's Data Engineering Weekly edition and go through an in-depth analysis.  Please subscribe to our Podcast on your favorite apps. From editor #121, we took the following articles Oda: Data as a product at Oda Oda writes an exciting blog about “Data as a Product,” describing why we must trea...

We are back in our Data Engineering Weekly Radio for edition #120 . We will take 2 or 3 articles from each week's Data Engineering Weekly edition and go through an in-depth analysis.  From editor #120, we took the following articles Topic 1: Colin Campbell: The Case for Data Contracts - Preventative data quality rather than reactive data quality In this episode, we focus on the importance of data...

We are super excited to be back to discussing Data Engineering Weekly Newsletter articles every week. We will take 2 or 3 articles from each week's Data Engineering Weekly edition and go through an in-depth analysis.  On Data Engineering Weekly edition #119, We are taking three articles. #1 Netflix's article about Scaling Media Machine Learning at Netflix https://netflixtechblog.com/scaling-media-...