Data Archives - Software Engineering Daily

Data integration infrastructure is not easy to build. Moving large amounts of data from one place to another has historically required developers to build ad hoc integration points to move data between SaaS services, data lakes, and data warehouses. Today, there are dedicated systems and services for moving these large batches of data. Airbyte builds The post Airbyte Engineering with Michel Tricot...

Modern organizations eventually face data governance challenges.  Keeping track of where data came from, what systems update it, in what ways updates can be made are just some of the issues to be tackled.  Large organizations face additional challenges like training, onboarding, and capturing the institutional knowledge that leaves with the departure of key team The post Select Star with Shinji Ki...

The solution many turn to for capturing their streaming data is InfluxDB.  In this episode, I interview Brian Gilmore, Director of Product Management at InfluxData, about how real time applications achieve success built on top of InfluxDB. When most people hear the phrase Internet of Things, it typically evokes an image of connected devices we The post Time Series IoT on InfluxDB with Brian Gilmor...

 Lior Gavish James Densmore Data infrastructure is a fast-moving sector of the software market. As the volume of data has increased, so too has the quality of tooling to support data management and data engineering. In today’s show, we have a guest from a data intensive company as well as a company that builds a The post Data Engineering Trends with Lior Gavish and James Densmore appeared first on...

Running a database company requires expertise in both technical and managerial skills. There are deeply technical engineering questions around query paths, scalability, and distributed systems. And there are complex managerial questions around developer productivity and task allocation. Sam Lambert is the CEO of PlanetScale, which is building modern relational database infrastructure. Before Plane...

SingleStore is a multi-use, multi-model database designed for transactional and analytic workloads, as well as search and other domain specific applications. SingleStore is the evolution of the database company MemSQL, which sought to bring fast, in-memory SQL database technology to market. Jordan Tigani is Chief Product Officer of SingleStore and joins the show to talk The post SingleStore with J...

DuckDB is a relational database management system with no external dependencies, with a simple system for deployment and integration into build processes. It enables complex queries in SQL with a large function library, and provides transactional guarantees through multi-version concurrency control. Hannes Mühleisen works on DuckDB and joins the show to talk about query engines The post DuckDB wit...

Customer data pipelines power the backend of many successful web platforms. In a customer data pipeline, data is collected from sources such as mobile apps and cloud SaaS tools, transformed and munged using data engineering, stored in data warehouses, and piped to analytics, advertising platforms, and data infrastructure. RudderStack is an open source customer data The post RudderStack Engineering...

The data lake architecture has become broadly adopted in a relatively short period of time.  In a nutshell, that means data in it’s raw format stored in cloud object storage.  Modern software and data engineers have no shortage of options for accessing their data lake, but that list shrinks quickly if you care about features The post Apache Hudi with Vinoth Chandar appeared first on Software Engin...

A data catalog provides an index into the data sets and schemas of a company. Data teams are growing in size, and more companies than ever have a data team, so the market for data catalog is larger than ever. Mark is the CEO of Stemma and the co-creator of Amundsen, a data catalog that came out of The post Data Catalog in Practice with Mark Grover appeared first on Software Engineering Daily .

Splunk is a monitoring and logging platform that has evolved over its 18 years of existence. In its modern focus on observability it is focused on open source and AIOps. Observability has evolved with the growth of Kubernetes, and Splunk’s work around OpenTelemetry has kept parity with the open source community of Kubernetes. Spiros Xanthos The post Splunk Platform with Spiros Xanthos appeared fir...

Barry McCardel Co-Founder and CEO at Hex Caitlin Colgrove Co-Founder and CTO at Hex In contrast to other IDEs, the notebook interface offers software developers a unique environment idealized for data professionals.  Despite the growth in popularity, a surprising learning curve still exists for setup and configuration.  A siloed notebook offers no native collaboration tools.  The post Hex Collabor...

When writing code, test driven development is a common accepted methodology to ensure the development of high quality software.  Your organization’s data, on the other hand, is an entirely different challenge.  Data can be missing due to human error, a failure with a 3rd party provider, a botched release, or dozens of other issues.  When The post Data Quality Using Anomalo with Jeremy Stanley appe...

Database product companies typically have a few phases. First, the company will develop a technology with some kind of innovation such as speed, scalability, or durability. The company will offer support contracts around that technology for a period of time, before eventually building a managed, hosted offering. PlanetScale is a database company built around the The post Scaling PlanetScale with S...

Couchbase is a distributed NoSQL cloud database. Since its creation, Couchbase has expanded into edge computing, application services, and most recently a database-as-a-service called Capella.  Couchbase started as an in-memory cache and needed to be rearchitected to be a persistent storage system. In this episode, I interview Ravi Mayuram, SVP Products and Engineering at Couchbase The post Couchb...

If you haven’t encountered a data quality problem, then you haven’t yet worked on a large enough project.  Invariably, a gap exists between the state of raw data and what an analyst or machine learning engineer needs to solve their problem.  Many organizations needing to automate data preparation workflows look to Trifacta as a solution.  The post Trifacta with Joe Hellerstein appeared first on So...

Relational databases have been a fixture of software applications for decades.  They are highly tuned for performance and typically offer explicit guarantees like transactional consistency.  More recently, there’s been a figurative cambrian explosion of other-than-relational databases.  Simple key value stores or counters were an early win in this space. Managing a graph data structure is The post...

The lifeblood of most companies is their sales departments.  When you’re selling something other than a commodity, it’s typically necessary to carefully groom the onboarding experience for inbound future customers.  Historically, companies approached this in a one-size-fits-all manner, giving all customers a common experience. In today’s data-driven age, a better experience can be provided that Th...

Application observability is a fairly mature area.  Engineering teams have a wide selection of tools they can choose to adopt and a significant amount of thought leadership and philosophy already exists giving guidance for managing your application.  That application is going to persist data.  As you scale up, your system is invariably going to experience The post Metaplane with Kevin Hu appeared...

Consumers are increasingly becoming aware of how detrimental it can be when companies mismanage data.  This demand has fueled regulations, defined standards, and applied pressure to companies.  Modern enterprises need to consider corporate risk management and regulatory compliance. In this interview, I speak with Terry O’Daniel, Director of Engineering (Risk & Compliance) at Instacart. Sponsor...

The internet is a layer cake of technologies and protocols.  At a fundamental level, the internet runs on the TCP/IP protocol.  It’s a packet based system.  When your browser requests a file from a web server, that server chops up the file into tiny pieces known as packets and puts them on the network labeled The post Scalable Streaming Video with Amit Mishra appeared first on Software Engineering...

It does not matter if it runs on your machine.  Your code must run in the production environment and it must do so performantly.  For that, you need tooling to better understand your application’s behavior under different circumstances.  In the earliest days of software development, all we had were logs, which are still around and The post Observability Using Honeycomb.io with Christine Yen...

The manner in which users interact with technology has rapidly switched to mobile consumption.  The devices almost all of us carry with us at all times open endless opportunities for developers to create location-based experiences. Foursquare became a household name when the introduced social check-ins.  Today they’re a location data platform.  Ankit Patel is the The post Location-Based Experience...

Modern business applications are complex.  It’s not enough to have raw logs or some basic telemetry.  Today’s enterprise organizations require an application performance monitoring solution or APM.  Today’s applications are complex distributed systems whose performance depends on a wide variety of factors.  Every single line of code can affect production and teams need insights i...

Infrastructure as Code is an approach to machine provisioning and setup in which a programmer describes the underlying services they need for their projects.  However, this infrastructure code doesn’t compile a binary artifact like traditional source code.  The successful completion of running the code signals that the servers and other components described in the configuration The post Infrastruc...