Software Engineering Daily

Unbounded data streams create difficult challenges for our application architectures. The data never stops coming, and we are forced to assume that we will never know if or when we have seen all of our data. Some streaming systems give us the tools to deal partially with unbounded data streams, but we have to complement those streaming systems with batch processing, in a technique known as the Lam...

TensorFlow is Google’s open source machine learning library. Rajat Monga is the engineering director for TensorFlow. In this episode, we cover how to use TensorFlow, including an example of how to build a machine learning model to identify whether a picture contains a cat or not. TensorFlow was built with the mission of simplifying the process of deploying a machine learning model from research to...

Data Validation is the process of ensuring that data is accurate. In many software domains, an application is pulling in large quantities of data from external sources. That data will eventually be exposed to users, and it needs to be correct. Radius Intelligence is a company that aggregates data on small businesses. In order to ensure that business addresses and phone numbers are correct, Radius...

Machine learning has become simplified. Similar to how Ruby on Rails made web development approachable, scikit-learn takes away much of the frustrating aspects of machine learning, and lets the developer focus on building functionality with high-level APIs. Per Harald Borgen is a developer at Xeneta. He started programming fairly recently, but has already built a machine learning application that...

Flexport is a technology company that makes logistics, supply chain management, and freight forwarding software. Shipping freight across the world requires container ships, airplanes, trains, warehouses, and trucks. Flexport’s software integrates with many of these different shipping companies, and provides a dashboard for the end user to understand how their products are being shipped around the...

When you hear about massive data breaches like the recent ones from LinkedIn, MySpace, or Ashley Madison, how can you find out whether your own data was compromised? Troy Hunt created the website HaveIBeenPwned.com to answer this question. When a major data breach occurs, Troy acquires a copy of the stolen data and provides a safe way for individuals to check if their credentials have been stolen....

The Linux kernel of many popular operating system distributions contains 200-500 million lines of code. The average user never touches many of the libraries that are contained in these operating system distributions. For example, if you spin up a virtual machine on a cloud service provider, the virtual machine will have a USB driver. This is wasted space, because you can’t even interact with the U...

Prometheus is a tool for monitoring our distributed applications. It allows us to focus on the services we are deploying rather than the individual machines that make up instances of that service. The monitoring service itself is a portion of a distributed system that is treated differently than the services we are monitoring. We don’t want to use a consensus-based tool like Zookeeper or Consul be...

The technology underlying podcasts is simple–a podcaster publishes mp3 files to an RSS feed, and the listener subscribes to that feed, receiving mp3s whenever the feed is updated. Unfortunately, the simplicity of podcasts makes it difficult to build automated advertising infrastructure on top of that simple RSS model. This lack of rich automated advertising has kept podcasting from flourishing. aC...

GraphQL was open sourced out of Facebook, and gave developers a way to unify their different data sources into a single endpoint. Although the promise of GraphQL is appealing, the process of setting up a GraphQL server that can communicate with each disparate data source can prove to be complex. Scaphold.io provides GraphQL as a service, and today’s guests are the creators of Scaphold, Vince Ning...

Most episodes of Software Engineering Daily are interviews with an expert about a technical software concept. Occasionally I write editorials, and also record them as a podcast. The first editorial was about 10 Philosophies for Engineers, the second was about how poker relates to software engineering, and the third was about music and software engineering. Today’s episode is called “You Are Not A...

Alec Ross worked in the White House as a Senior Policy Advisor to Hillary Clinton. His book Industries of the Future explores the biggest technological opportunities and threats to our society. The industries addressed in his book include robotics, genetics, and cybersecurity. Technological familiarity is increasingly correlated with an individual’s optimism. Cyberwarfare presents attack vectors t...

Clojure is a dynamically typed functional language that runs on the JVM. Today’s guest Alex Miller gives us an overview of Clojure’s core functionality. Alex is a developer of Cognitect, and a founder of the the Strange Loop conference. We discuss the data structures, garbage collection, and concurrency support. How does Clojure compare to other JVM languages like Scala and Groovy? How does Clojur...

Odd Networks is building a platform for anyone to launch their own over-the-top streaming video service. With Odd Networks, you can deploy your own video channel using a Roku, Amazon Fire TV, Apple TV, and other services. Creating a streaming video service with interoperability between these different platforms presents numerous technical challenges, and today’s guest Kris Walker explains how Odd...

MobyCraft is a client-side Minecraft mod to manage and visualize Docker containers. MobyCraft was created by Aditya Gupta. I met him at DockerCon, where he gave a presentation about his project. He also discussed his interaction with the Netflix team, who integrated MobyCraft with their container management tool called Titus. You can watch a video online of Titus managing hundreds of 3-D container...

Prometheus is an open-source monitoring tool built at SoundCloud. It can be used to produce detailed time-series data about a distributed architecture. Prometheus is based on the monitoring system inside Google’s infrastructure, called Borgmon. Julius Volz is the creator of Prometheus, and he joins the show to explain why he built Prometheus and how it differs from previous monitoring tools. Prome...

Monitoring translates machine data into actionable business metrics, and is a key component of a modern software company. James Turnbull’s new book “The Art of Monitoring” describes how organizations can build their monitoring infrastructure. James joins the show today to outline the strategies that a company can use to proactively monitor their systems. We talk about pull- vs. push-based monitori...

Financial technology has changed significantly in the last decade, and companies both new and old are adapting to that change. Newer companies like TransferWise and Stripe are often called “fintech” companies–short for “financial technology”. Established companies like banks may not refer to themselves as fintech companies, but the way that they do business is changing, due to technological advanc...

github has grown to have 10 million users and 30 million repositories. Getting to this scale has required innovation in many places–github has significantly altered the code for git itself, and has built unique infrastructure and written low level code to architect for git repository management at scale. Despite the need for cutting-edge technologies to support github, the development culture at g...

Pixar has made some of the most successful movies of all time: Toy Story, WALL-E, Monsters Inc, and many others. These movies are made with cutting-edge computer animation techniques that Pixar often has to invent in order to tell the story it wants to tell. Pixar has teamed up with Khan Academy to teach anyone who wants to learn the basics of computer animation–Pixar-style. The collaboration with...

Open source software has become the rule for how software is written rather than the exception. OSCON is O’Reilly’s open source conference, where companies and individuals talk about where the open-source world is going. Rachel Roumeliotis is the chair of OSCON, and she joins the show today to talk about the state of open source, and how the conference has developed since she started working at O’...

Finally–the Android operating system has been put on an iPhone, and today’s guest is Nick Lee, who accomplished that feat. Nick works at Tendigi, a design and engineering firm. In the past, Nick has put Windows 95 on an Apple Watch. Why would you do something like this? In today’s interview with Nick, we talk about the technical challenges of bringing Nick’s bizarre sense of technological irony in...

The container management system Kubernetes was open sourced by Google with the intention of creating a cloud service based on the project. Today, the Kubernetes ecosystem is looking similar to the Android ecosystem, with different vendors providing different ways to use Kubernetes, from RedHat’s OpenShift to Google Container Engine. Craig Mcluckie was a member of the team who originally devised Ku...

Container management systems like Kubernetes and Docker Swarm give us a higher level management tool for architectures built out of distributed containers. Container platforms like Rancher provide a higher layer of usability, and today’s guest Darren Shepherd of Rancher Labs takes us through what a container platform is. This interview is part of our continued coverage of Kubernetes, Docker, and t...

Breakthroughs in modern data research tend to come from companies like Google, Facebook, and Amazon, with projects like MapReduce, Cassandra, and Dynamo. Twenty years ago, this types of breakthroughs would be happening in academia, which causes today’s guest Peter Bailis to ask: is the academic data community having an identity crisis? Peter is an assistant professor at Stanford University, where...