Software Engineering Daily

Cloud computing lowered the cost and improved accessibility to tools for storing large volumes of data. In the early 2000s, Hadoop caused a revolution in large scale batch processing. Since then, companies have been building ways to store and access their data faster and more efficiently. At the same time, the sheer volume of data has increased and machine learning has given rise to methods of ext...

Starting an Internet business is harder than it should be. You need to incorporate, create an operating agreement, set up a system to accept payments, and many other straightforward tasks. In the 1990s, this was how it felt to set up anything on the Internet. You always had to stand up a web server on your own infrastructure, before you could get to the interesting part–which was building an actua...

When I buy a mattress online, I pay for it with my credit card. Behind the scenes, a complex series of transactions occur between a payment gateway, the credit card company, and a few banks. There are problems with this process–it is slow, complex, and involves the synchronization of several different parties. Some consumers will not want to purchase the mattress because they do not have cash, and...

Superpedestrian is a robotic bicycle wheel that learns how you pedal and personalizes your bicycle ride. The engineering challenges of Superpedestrian are at the intersection of robotics, software, and real-time analytics. The first half of today’s show is about Superpedestrian. Goss Nuzzo Jones and Matt Cole are engineers at Superpedestrian. The slides for their presentation are also in the show...

Phones are constantly tracking the location of a user in space. Devices like cars, smart watches, and drones are also picking up high volumes of location data. This location data is also called “geospatial data.” The amount of geospatial data is rapidly increasing, and there is a growing demand for software to perform operations over that data. Geospatial data sets are often massive–so it is non-t...

Last year, the WannaCry ransomware attack shut down hospitals, public transportation systems, and governments, demanding payment to unlock key computer systems. A programmer named Marcus Hutchins was able to stop WannaCry by registering a DNS entry buried in the WannaCry code. Not long after he stopped the WannaCry attack, Marcus Hutchins was arrested at a security conference in Las Vegas. Marcus’...

Alexis Le-Quoc started Datadog in 2010, after living through the Internet boom and bust cycle of the late 90s and early 2000s. In 2010, cloud was just starting to become popular. There was a gap in the market for infrastructure monitoring tools, which Alexis helped fill with the first version of Datadog. Since 2010, the number of different cloud infrastructure products has proliferated. There were...

Technology is pushing us rapidly toward a future that is impossible to forecast. We try to imagine what that future might look like, and we can’t help having our predictions shaped by the media we have consumed. 1984, Terminator, Gattaca, Ex Machina, Black Mirror–all of these stories present a dystopian future. But if you look around the world, the most successful technologists are mostly guided b...

Podcasting about cryptocurrencies is a strange occupation. You get emails all the time from companies doing a token sale that you would never want to be affiliated with. You get angry tweets from anonymous Twitter accounts that are on one side of the Bitcoin scaling debate. You get to interview extreme personalities, and the technical discussions can be highly educational. Brian Fabian Crain start...

Keybase is a platform for managing public key infrastructure. Keybase’s products simplify the complicated process of associating your identity with a public key. Keybase is the subject of the first half of today’s show. Michael Maxim, an engineer from Keybase gives an overview for how the technology works and what kinds of applications Keybase unlocks. The second half of today’s show is about Clar...

Google’s central system for managing compute resources is called Borg. On Borg, millions of Linux containers process a wide variety of workloads. When a new application is spun up, Borg provides that application with the resources it needs. Workloads at Google usually fall into one of two distinct categories: long-running application workloads (such as Gmail) and batch workloads (such as a MapRedu...

Rajat Monga is a director of engineering at Google where he works on TensorFlow. TensorFlow is a framework for numerical computation developed at Google. The majority of TensorFlow users are building machine learning applications such as image recognition, recommendation systems, and natural language processing–but TensorFlow is actually applicable to a broader range of scientific computation than...

There are hundreds of different databases. There are tens of continuous delivery products. There is an ocean of cloud providers and CRM systems and monitoring platforms and sales prospecting tools. The range of available software products is so diverse that it can be overwhelming to figure out which products to buy. Siftery is a company that was started to index software products and help buyers m...

A message broker is an architectural component that sends messages between different nodes in a distributed system. Message brokers are useful because the sender of a message does not always know who might want to receive that message. Message brokers can be used to implement the “publish/subscribe” pattern, and by centralizing the message workloads within the pub/sub system, it lets system operat...

Stripe processes payments for thousands of businesses. A single payment could involve 10 different networked services. If a payment fails, engineers need to be able to diagnose what happened. The root cause could lie in any of those services. Distributed tracing is used to find the causes of failures and latency within networked services. In a distributed trace, each period of time associated with...

Bitcoin and Bitcoin Cash are two cryptocurrencies with similar properties. But the supporters of each of these Bitcoin versions have strongly divergent opinions on the direction of the Bitcoin project. At the center of this debate is the subject of block size. Bitcoin’s block size determines how many transactions fit into each block that is mined. A larger block size leads to faster transactions a...

Most new frontend webapps today use ReactJS. An increasing number of mobile apps are created using the cross-platform components of React Native. GraphQL, Facebook’s open source data-fetching middleware tool is being used by more and more companies, who are finding that it simplifies their development. Facebook’s open source suite of technologies created a new developer ecosystem. There is an incr...

Machine learning tools are rapidly maturing. TensorFlow gave developers an open source version of Google’s internal machine learning framework. Cloud computing provides a cost effective, accessible way of training models. Edge computing allows for low latency deployments of models. But even if you are a kid with a laptop who has learned all the machine learning algorithms, read all of the deep lea...

Let’s Talk Bitcoin is one of the most popular podcasts about cryptocurrencies. Adam B. Levine started it after three other podcasts he started did not get the traction he had hoped for. Adam parlayed the success of Let’s Talk Bitcoin into a network of podcasts–the Let’s Talk Bitcoin Network–which also includes one of my favorite shows, Epicenter. Adam joins me on today’s episode for a discussion o...

Monitoring a Kubernetes cluster allows operators to track the resource utilization of the containers within that cluster. In today’s episode, Ilan Rabinovitch joins the show to explore the different options for setting up monitoring, and some common design patterns around Kubernetes logging and metrics gathering. Ilan is the VP of product and community at Datadog. Earlier in his career, Ilan spent...

Laura Shin is the host of Unchained, a podcast about cryptocurrencies and decentralized technology. For every episode, Laura does significant research and preparation, so the content turns out polished and high quality. Her enthusiasm for the subject of cryptocurrencies comes through in her reporting. Podcasting about cryptocurrencies requires walking a fine line. Cryptocurrencies have a mixture o...

Social networks can make you feel connected to a global society. But those social networks are controlled by a corporate entity. The profit motivations of the corporation are not directly aligned with the experience of the users. Mastodon is an open source, decentralized social network. Eugen Rochko started building Mastodon in response to his dissatisfaction with centralized social networks like...

Go is a language designed to improve systems programming. Go includes abstractions that simplify aspects of low level engineering that are historically difficult—concurrency, resource allocation, and dependency management. In that light, it makes sense that the Kubernetes container orchestration system was written in Go. Erik St. Martin is a cloud developer advocate at Microsoft, where he focuses...

Tammy Butow has worked at Digital Ocean and Dropbox, where she built out infrastructure and managed engineering teams. At both of these companies, the customer base was at a massive scale. At Dropbox, Tammy worked on the database that holds metadata used by Dropbox users to access their files. To call this metadata system simply a “database” is an understatement–it is actually a multi-tiered syste...

Software engineers have interacted with operations teams since software was being written. In the 1990s, most operations teams worked with physical infrastructure. They made sure that servers were provisioned correctly and installed with the proper software. When software engineers shipped bad code that took down a software company, the operations teams had to help recover the system—which often m...