Software Engineering Daily

Running a database company requires expertise in both technical and managerial skills. There are deeply technical engineering questions around query paths, scalability, and distributed systems. And there are complex managerial questions around developer productivity and task allocation. Sam Lambert is the CEO of PlanetScale, which is building modern relational database infrastructure. Before Plane...

GitOps is a deployment and infrastructure management strategy based around continuous delivery and Kubernetes, with Git at the center of deployment workflows. Policy management can be used to define permissions and rules around who can deploy and what constitutes safe deployment. There is a synergy between GitOps tooling and policy management tooling. Alexis Richardson is the co-founder and CEO of...

A company’s response to an incident defines how that company responds to an adverse, unexpected scenario. Kintaba automates the incident management process so teams can quickly report, respond, resolve, and reflect on major incidents collaboratively. Kintaba's incident response platforms gives you instant access to incident management processes based on the practices of companies like Facebook, Go...

SingleStore is a multi-use, multi-model database designed for transactional and analytic workloads, as well as search and other domain specific applications. SingleStore is the evolution of the database company MemSQL, which sought to bring fast, in-memory SQL database technology to market. Jordan Tigani is Chief Product Officer of SingleStore and joins the show to talk through the architecture an...

Compute resources continue to trend towards being cheaper, easier to use, and faster. Despite these positives, more compute demands more energy and therefore an increasing carbon footprint. With many companies committing to controlling their net carbon emissions, tools are required for engineers to not only measure their cloud infrastructure, but to make informed choices about how to set up the in...

Vantage is a system for optimizing cloud costs. It provides tools and interfaces for developers to analyze how they are spending on AWS resources, and has recently expanded into GCP as well. Vantage users gain an easy interface into their costs that would otherwise be hard to analyze via the raw AWS console.  Ben Schaechter is the founder of Vantage. In a previous show he gave an overview of Vanta...

At Lyft, Ketan Umare worked on Flyte, an orchestration system for machine learning. Flyte provides reliability and APIs for machine learning workflows, and is used at companies outside of Lyft such as Spotify. Since leaving Lyft, Ketan founded Union.ai, a company focused on productionizing Flyte as a service. He joins the show to talk about the architecture and usage of Flyte, as well as how he is...

DuckDB is a relational database management system with no external dependencies, with a simple system for deployment and integration into build processes. It enables complex queries in SQL with a large function library, and provides transactional guarantees through multi-version concurrency control. Hannes Mühleisen works on DuckDB and joins the show to talk about query engines and OLAP system Spo...

Substack is a platform for subscription-based content, covering newsletters, podcasts, and soon video. The design and speed of the platform are notable–with a minimalistic format that simply gets creators paid for its work, Substack has gained massive popularity. Chris Best is the CEO of Substack and joins the show to talk about the platform, architecture, and engineering of Substack. Sponsorship...

Customer data pipelines power the backend of many successful web platforms. In a customer data pipeline, data is collected from sources such as mobile apps and cloud SaaS tools, transformed and munged using data engineering, stored in data warehouses, and piped to analytics, advertising platforms, and data infrastructure. RudderStack is an open source customer data pipeline system that pulls toget...

Managing Kubernetes nodes leads to operational complexity, security issues, and nodes that are perhaps more expensive to run than necessary. Deferring the node management to an underlying platform abstracts away these problems and can improve operations. Madhuri Yechuri runs Elotl, a nodeless Kubernetes platform. She joins the show to talk about the architecture and purpose of her company. Sponsor...

Retool is a company that allows customers to build complex internal tools using a high level GUI. Users configure the relationships between these different tools, giving them the ability to build applications even without much background in engineering. Of course, having some engineering expertise helps, as users can write JavaScript to interact with the higher level components. Snir Kodesh is the...

The software supply chain consists of packages, imports, dependencies, containers, and APIs. These different components each have unique security risks. To ensure the security of their software supply chain, many developers use tools to analyze and scan their infrastructure for vulnerabilities. Barak Schoster works at Bridgecrew, a DevSecOps cloud security platform. He joins the show to talk about...

The data lake architecture has become broadly adopted in a relatively short period of time. In a nutshell, that means data in it’s raw format stored in cloud object storage. Modern software and data engineers have no shortage of options for accessing their data lake, but that list shrinks quickly if you care about features like transactions. Apache Hudi is a platform for building streaming data la...

National Instruments develops software and hardware for engineering in a wide variety of domains, from aerospace to government technology to application testing. The interface between hardware and software presents a variety of difficult engineering challenges. Luke Schreier is a Senior Vice President at National Instruments and joins the show to discuss the engineering and management of the compa...

Automating video search requires a data pipeline that extracts metadata from videos and allows users to annotate the video with information that correlates with that metadata. The video needs to be segmented into intervals that define the search space, and the search space needs to be queryable by a user. Sieve Data is an automated video search platform that enables developers to add video search...

Increasingly, technology groups need to be strategic about the cloud services they adopt to ensure their vendor’s pricing is both fair and measured exactly right for the application’s unique access pattern. Definitely answering questions like these can take a significant amount of time and energy from your most valuable engineers. Taloflow is a company trying to make this decision making process e...

A data catalog provides an index into the data sets and schemas of a company. Data teams are growing in size, and more companies than ever have a data team, so the market for data catalog is larger than ever. Amundsen is a data catalog that came out of Lyft. We have previously explored the basics of Amundsen. In today’s episode, Mark Grover returns to the show to talk about the art and science of...

Splunk is a monitoring and logging platform that has evolved over its 18 years of existence. In its modern focus on observability, it is focused on open source and AIOps. Observability has evolved with the growth of Kubernetes, and Splunk’s work around OpenTelemetry has kept parity with the open-source community of Kubernetes. Spiros Xanthos is the general manager of observability at Splunk. He jo...

Francesco Cesarini founded Erlang Solutions in 1999 with a mission to help companies adopt Erlang. In this interview, I speak with Francesco and Gabor Olah from Erlang Solutions. We discuss the Erlang language, it's ecosystem, and features like concurrency, resilience, and scalability that motivate adoption. We use Java and the Java Virtual Machine as a comparison point for Erlang and it's virtual...

Serious software projects require several environments. Your production environment is obviously mission critical. A staging environment is also necessary to perform validation and regression testing before taking the risk of pushing an update to production. Best practices and approaches for managing these and other environments vary from organization to organization. In some sense, different soft...

In contrast to other IDEs, the notebook interface offers software developers a unique environment idealized for data professionals. Despite the growth in popularity, a surprising learning curve still exists for setup and configuration. A siloed notebook offers no native collaboration tools. While one can connect to a SQL database programmatically, if you’re looking for an ideal ergonomic environme...

When writing code, test driven development is a common accepted methodology to ensure the development of high quality software. Your organization’s data, on the other hand, is an entirely different challenge. Data can be missing due to human error, a failure with a 3rd party provider, a botched release, or dozens of other issues. When not missing, data can still become corrupted or start exhibitin...

When you visit a web page, the creator’s intent is to present you a seamless experience that fills your browser window. That web page or web application is generally divided up in some meaningful way across navigation elements, content, ads, header, footer, and other components. Those components may represent the work of independent teams. Typically a web app is built in a single code base, pullin...

Lee Atchison spent seven years at Amazon working in retail, software distribution, and Amazon Web Services. He then moved to New Relic, where he has spent four years scaling the company’s internal architecture. From his decade of experience at fast-growing web technology companies, Lee has written the book Architecting for Scale, from O’Reilly. As an application scales, it becomes significantly mo...