Software Engineering Daily

The Linux kernel of many popular operating system distributions contains 200-500 million lines of code. The average user never touches many of the libraries that are contained in these operating system distributions. For example, if you spin up a virtual machine on a cloud service provider, the virtual machine will have a USB driver. This is wasted space, because you can’t even interact with the U...

Prometheus is a tool for monitoring our distributed applications. It allows us to focus on the services we are deploying rather than the individual machines that make up instances of that service. The monitoring service itself is a portion of a distributed system that is treated differently than the services we are monitoring. We don’t want to use a consensus-based tool like Zookeeper or Consul be...

The technology underlying podcasts is simple–a podcaster publishes mp3 files to an RSS feed, and the listener subscribes to that feed, receiving mp3s whenever the feed is updated. Unfortunately, the simplicity of podcasts makes it difficult to build automated advertising infrastructure on top of that simple RSS model. This lack of rich automated advertising has kept podcasting from flourishing. aC...

GraphQL was open sourced out of Facebook, and gave developers a way to unify their different data sources into a single endpoint. Although the promise of GraphQL is appealing, the process of setting up a GraphQL server that can communicate with each disparate data source can prove to be complex. Scaphold.io provides GraphQL as a service, and today’s guests are the creators of Scaphold, Vince Ning...

Alec Ross worked in the White House as a Senior Policy Advisor to Hillary Clinton. His book Industries of the Future explores the biggest technological opportunities and threats to our society. The industries addressed in his book include robotics, genetics, and cybersecurity. Technological familiarity is increasingly correlated with an individual’s optimism. Cyberwarfare presents attack vectors t...

Clojure is a dynamically typed functional language that runs on the JVM. Today’s guest Alex Miller gives us an overview of Clojure’s core functionality. Alex is a developer of Cognitect, and a founder of the the Strange Loop conference. We discuss the data structures, garbage collection, and concurrency support. How does Clojure compare to other JVM languages like Scala and Groovy? How does Clojur...

Odd Networks is building a platform for anyone to launch their own over-the-top streaming video service. With Odd Networks, you can deploy your own video channel using a Roku, Amazon Fire TV, Apple TV, and other services. Creating a streaming video service with interoperability between these different platforms presents numerous technical challenges, and today’s guest Kris Walker explains how Odd...

MobyCraft is a client-side Minecraft mod to manage and visualize Docker containers. MobyCraft was created by Aditya Gupta. I met him at DockerCon, where he gave a presentation about his project. He also discussed his interaction with the Netflix team, who integrated MobyCraft with their container management tool called Titus. You can watch a video online of Titus managing hundreds of 3-D container...

Prometheus is an open-source monitoring tool built at SoundCloud. It can be used to produce detailed time-series data about a distributed architecture. Prometheus is based on the monitoring system inside Google’s infrastructure, called Borgmon. Julius Volz is the creator of Prometheus, and he joins the show to explain why he built Prometheus and how it differs from previous monitoring tools. Prome...

Financial technology has changed significantly in the last decade, and companies both new and old are adapting to that change. Newer companies like TransferWise and Stripe are often called “fintech” companies–short for “financial technology”. Established companies like banks may not refer to themselves as fintech companies, but the way that they do business is changing, due to technological advanc...

github has grown to have 10 million users and 30 million repositories. Getting to this scale has required innovation in many places–github has significantly altered the code for git itself, and has built unique infrastructure and written low level code to architect for git repository management at scale. Despite the need for cutting-edge technologies to support github, the development culture at g...

Pixar has made some of the most successful movies of all time: Toy Story, WALL-E, Monsters Inc, and many others. These movies are made with cutting-edge computer animation techniques that Pixar often has to invent in order to tell the story it wants to tell. Pixar has teamed up with Khan Academy to teach anyone who wants to learn the basics of computer animation–Pixar-style. The collaboration with...

Open source software has become the rule for how software is written rather than the exception. OSCON is O’Reilly’s open source conference, where companies and individuals talk about where the open-source world is going. Rachel Roumeliotis is the chair of OSCON, and she joins the show today to talk about the state of open source, and how the conference has developed since she started working at O’...

Finally–the Android operating system has been put on an iPhone, and today’s guest is Nick Lee, who accomplished that feat. Nick works at Tendigi, a design and engineering firm. In the past, Nick has put Windows 95 on an Apple Watch. Why would you do something like this? In today’s interview with Nick, we talk about the technical challenges of bringing Nick’s bizarre sense of technological irony in...

Container management systems like Kubernetes and Docker Swarm give us a higher level management tool for architectures built out of distributed containers. Container platforms like Rancher provide a higher layer of usability, and today’s guest Darren Shepherd of Rancher Labs takes us through what a container platform is. This interview is part of our continued coverage of Kubernetes, Docker, and t...

In a typical data analytics system, there are a variety of technologies interacting. HDFS for storing files, Spark for distributed machine learning, pandas for data analysis in Python–each of these different technologies has a different format for how data is represented. Serialization and deserialization between these different formats causes significant latency across the overall system. Apache...

EconTalk is a weekly economics podcast that has been going for a decade. On EconTalk, Russ Roberts brings on writers, intellectuals, and entrepreneurs for engaging conversations about the world as seen through the lens of economics. Russ Roberts is today’s guest, and it is a treat because I have been listening to EconTalk since 2006 and it was a central point of inspiration for what Software Engin...

On smart thermostats, sensor-driven assembly lines, and electronically monitored farms, the internet of things is producing huge volumes of data. To take advantage of that data, an application needs tools for storing and analyzing that data. Today’s guest is Jean-Cristophe Cimono, the CTO of mnubo, a cloud platform for connected objects. Today we walk through the architecture of mnubo and the use...

At software conferences, I like to walk around the vendor booths and talk to the representatives from different companies. By talking to the vendors about their marketing pitches, I get an idea of how those companies are positioning themselves for the future, and the complex business landscape of software becomes slightly easier to understand. At recent conferences, many of the big vendors have be...

Lee Atchison spent seven years at Amazon working in retail, software distribution, and Amazon Web Services. He then moved to New Relic, where he has spent four years scaling the company’s internal architecture. From his decade of experience at fast growing web technology companies, Lee has written the book Architecting for Scale, from O’Reilly. As an application scales, it becomes significantly mo...

Scheduling is the method by which work is assigned to resources to complete that work. At the operating system level, this can mean scheduling of threads and processes. At the data center level, this can mean scheduling Hadoop jobs or other workflows that require the orchestration of a network of computers. Adrian Cockcroft worked on scheduling at Sun Microsystems, eBay, and Netflix. In each of th...

Call centers are a vulnerable point of attack for large enterprises. Fraud accounts for more than $20 billion in lost money every year, and a significant portion of that fraud is due to customer service representatives being fraudulent social engineering attacks. Chris Halaschek joins the show today to discuss how Pindrop Security is addressing this attack vector. Every phone call that gets made t...

At most universities, there is not a course titled “cloud computing”. Most students leave college without an understanding of distributed systems, cloud service providers, and the fundamentals of how a data center works. Kasper Nissen and Martin Jensen are changing that with KubeCloud, a small tangible cloud computing cluster that runs on Raspberry Pis. Kasper and Martin started KubeCloud as a mas...

Transferring money from one country to another is expensive, and the banks that facilitate money transfer have tricked us into believing that it should be expensive. On today’s show, Harsh Sinha explains the peer-to-peer system of transferring money with TransferWise, where he works as VP of engineering. Harsh also discusses the larger picture of FinTech companies. The emergence of so many compani...

Apache Cassandra is a distributed database that can handle large amounts of data with no single point of failure. Since 2008, Cassandra has been widely adopted and the software and the community around it have grown steadily. A software developer interacting with Cassandra uses CQL, the Cassandra Query Language. ScyllaDB is another open-source database that has been created to be totally compatibl...