Software Engineering Daily

The code in the Linux kernel changes all the time–11k lines are added, 5.8k lines are removed, and 2k lines are modified DAILY. Linux is an open source operating system that has been worked on for 25 years, and one reason the project is able to move so fast is its governance and release structure. Greg Kroah-Hartman is a fellow at the Linux Foundation, where he takes part in many of the developmen...

Modern software applications are often built out of loosely coupled microservices. These services can be written in different languages, by different people, but communication between services needs to be standardized. For this reason, a service proxy is useful. A service proxy is a sidecar container that sits next to a service and facilitates communications with other services. Once every service...

Containers make it easier for engineers to deploy software. Orchestration systems like Kubernetes make it easier to manage and scale the different containers that contain services. The popular container infrastructure powered by Kubernetes is often called “cloud native.” On Software Engineering Daily, we have been exploring cloud native software to get a complete picture of the problems in the spa...

Search engines run our lives. The path we take to information is dictated by Google, Facebook, Amazon, and other forms of search. Search engines feel objective and truthful, but are built through ongoing experimentation and subjective decision making. That’s what has kept Danny Sullivan writing about search engines for twenty years. The content Google prioritizes, the ads that we see, the way that...

Professional hackathon programmers travel around the hackathon circuit, winning merchandise and small cash prizes. There are enough hackathons that some programmers actually do this as a full-time job. For example, Peter Ma, a programmer who describes himself as a “rapid prototype specialist.” Peter is a great programmer, and he has received lots of offers to work at big tech companies. What drive...

Software Engineering Daily has been around for almost two years. In this episode Pranay Mohan and Erika Hokanson join me for a reflection on where we have been and where we are going. Pranay was the producer of Software Engineering Daily for the first year, after which he left and went to work at Snapchat. Erika joined the show 9 months ago to work on operations, ad sales, and expansion plans. The...

Software architecture address the challenge of communicating and navigating large, complex systems to stakeholders, both technical and non-technical. Over the years software architecture has gone in and out of fashion. Today we discuss why software architecture is important, what it means to have software architecture, and how to properly structure teams and incorporate architecture. Today’s show...

A self-driving car needs to be able to quickly respond to changes in driving conditions. A factory needs to be able to quickly respond to changes in workplace safety. For these kinds of applications, we need processing power closer to the user of the application. If we put all of our application logic in the cloud, we will have to make a network round trip for every request. Servers in the cloud a...

John Looney spent more than 10 years at Google. He started with infrastructure, and was part of the team that migrated Google File System to Colossus, the successor to GFS. Imagine migrating every piece of data on Google from one distributed file system to another. In this episode, John sheds light on the engineering culture that has made Google so successful. He has very entertaining stories abou...

A data-driven organization is more efficient because the company can learn what to focus on. In this episode, Edaena Salinas from The Women in Tech Show interviews Rya Sciban, Product Manager at Periscope Data, who explains the needs of data teams in an organization. We talked about what data analysis is and how this changes as the amount of data grows. Rya explained what analytics clusters are an...

Deep learning allows engineers to build models that can make decisions based on training data. These models improve over time using stochastic gradient descent. When a model gets big enough, the training must be broken up across multiple machines. Two strategies for doing this are “model parallelism” which divides the model across machines and “data parallelism” which divides the data across multi...

Modern architectures often consist of containers that run services. Those containers scale up and down depending on the demand for the services. These large software systems often use a technique known as event sourcing, where every change to the system is kept in an event log. When an event on the log is processed, several different data stores might be updated in response. In these architectures...

Kubernetes is an orchestration system for managing containers. Since it was open sourced by Google, Kubernetes has created a wave of innovation in the infrastructure technology space. Another recent innovation has been the “serverless” execution tools–such as AWS Lambda and Google Cloud Functions. Serverless execution, otherwise known as functions-as-a-service, allows a developer to execute code a...

Microsoft’s past is full of stories. It’s early period of corporate domination in the 1990s was followed by a period of government antitrust scrutiny, and a period of unsure product direction. Today, Microsoft’s focus on cloud has allowed the company to regain its footing with a clear trajectory for growth. Since 2002, Richard Campbell has chronicled the Microsoft developer community as co-host of...

Kubernetes makes it easier for engineering teams to manage their distributed systems architecture. But it’s still not simple to deploy and operate a Kubernetes cluster. Google Container Engine (GKE) is a managed control plane for Kubernetes. Just as developers can use Google App Engine to easily deploy monolithic apps against a platform as a service, we can use Google Container Engine to deploy mi...

Engineers have plenty to be skeptical about. We look to data sets to give us something resembling objective truth. Some areas of research have so many variables that it is hard to isolate facts. Kyle Polich hosts the popular data science show Data Skeptic, where he examines problems and solutions around data, and he is one of the guests today in our round table discussion. There are some big unans...

DNS stands for domain name system. This is the naming system that maps the entire internet. It associates information with domain names. More specifically, DNS specifies mappings between numerical IP addresses and domain names. Most engineers know these basic facts about DNS, but they may not know how much engineering a complex company like Etsy or Zappos puts into their DNS configuration. Dynamic...

Video object segmentation allows computer vision to identify objects as they move through space in a video. The DAVIS challenge is a contest among machine learning researchers working off of a shared dataset of annotated videos. The organizers of the DAVIS challenge join the show today to explain how video object segmentation models are trained and how different competitors take part in the DAVIS...

On January 31st 2017, GitLab experienced a major outage of their online repository hosting service. The primary database server experienced data loss due to a combination of malicious spam attacks and engineering mistakes that occurred while trying to respond to those spam attacks. GitLab responded to the event transparently. The company put up a postmortem describing the event in detail. In subse...

Different databases have different access patterns. Key-value, document, graph, and columnar databases are useful under different circumstances. For example, if you are a bank, and you have a database of customers and the transactions they have performed, the ideal access pattern for aggregating the total amount of all transactions might be a columnar store. If the transaction amounts are all in o...

With a fast-growing field like data science, it is important to keep some amount of skepticism. Tools can be overhyped, buzzwords can be overemphasized, and people can forget the fundamentals. If you have bad data, you will get bad results in your experimentation. If you don’t know what statistical approach you want to take to your data, it doesn’t matter how well you know Spark or TensorFlow. And...

Apple controls the iOS ecosystem. As an accident of history, Apple also controls the podcasting ecosystem. Unlike most ecosystems within Apple’s dominion, podcasts remain open. A podcaster merely has to record an mp3, distribute it via RSS feed, and submit that RSS feed to the iTunes podcast portal. Podcasting has thrived in recent years, but very few technology companies have managed to take adva...

Social networks like Facebook and Twitter facilitate interactions between individuals. Every message I send to you on Facebook goes through Facebook’s servers before reaching you. This is known as the client-server model. Since the early days of the internet, engineers have always envisioned a peer-to-peer model, where I could communicate to you directly, without a company brokering that relations...

Everyone in the world should have some basic level of guaranteed healthcare. This is not controversial. But what should that basic level of healthcare be? Should it extend into the later years of your life, when the majority of your health costs are incurred? And how much has modern technology driven down the cost of what it should cost to treat a patient? Healthcare today has lots of problems wit...

Relay is a JavaScript framework for building data-driven React applications. Facebook open sourced Relay around the same time they open sourced GraphQL, and Facebook expected Relay to be the more popular of the two projects. However, the reality was reversed. Open source companies like Meteor quickly began to build GraphQL tools and a few businesses were started around GraphQL. One year later, the...