Software Engineering Daily

Google Compute Engine is the public cloud built by Google. It provides infrastructure- and platform-as-a-service capabilities that rival Amazon Web Services. Today’s guest Joe Beda was there from the beginning of GCE, and he was also one of the early engineers on the Kubernetes project. Google’s internal systems have made it easy for employees to spin up compute resources, but it was not a simple...

Cloud computing was something much different in 2011, when Brian Gracely and Aaron Delp started The Cloudcast, a podcast I listen to on a regular basis. The Cloudcast features technical discussions about cloud infrastructure technology, and one of the most recent shows was a monologue by Brian Gracely where he explained his perspective on the industry rumblings about a Docker fork. The utility of...

The Simpsons is a classic, beloved television show. The scripts of The Simpsons have been made publicly available, and include dialogue, location, and character information. Todd Schneider used these scripts and other information sources as a corpus to analyze The Simpsons and find interesting statistics–such as who the most important supporting characters were, and how the ratings of the show hav...

Data engineering is the software engineering that enables data scientists to work effectively. In today’s episode, we explore the different sides of data engineering–the data science algorithms that need to be processed and the implementation of software architectures that enable those algorithms to run smoothly. The PANCAKE STACK is a 12-letter acronym that Chris Fregly gave to a collection of da...

When a user of a social network updates her profile, that profile update needs to propagate to several databases that want to know about such an update–search indexes, user databases, caches, and other services. When Neha Narkhede was at LinkedIn, she helped develop Kafka, which was deployed at LinkedIn to help solve this very problem. Using Kafka as an event queue, LinkedIn adopted the CQRS archi...

The intent of the DevOps movement is to get organizations moving faster and more effectively by breaking down siloes, and improving communication. Gene Kim’s book The Phoenix Project illustrated this by telling the fictional story of a company adopting a DevOps mentality. Although that book was fiction, Gene is an experienced engineer, having worked as founder and CTO of Tripwire, a software compa...

At Netflix, developers write applications with a variety of requirements–from simple requests for a list of movies to more resource-intensive requests like a complex machine learning workflow. Netflix wants developers to be able to request the resources they need from a compute cluster and receive those resources on-demand, without thinking too much about the state of that pool of resources they a...

Building a monitoring system is a complex distributed systems problem. Events are produced from different points in an application and must be aggregated in order to form metrics. These events are often ingested by a time series database, which forms the backbone of our monitoring system. Theo Schlossnagle is the CEO of Circonus, where he has been working on architecting the company’s monitoring s...

In order to move software updates from the development team to production, companies do a variety of things. Some teams might email files to each other or use FTP or even floppy disks. Most companies today at least use version control systems like Git together with separate servers for development and production. When code is ready to move to production, a build that is on the development server g...

Kafka Streams is a library for building streaming applications that transform input Kafka topics into output Kafka topics. In a time when there are numerous streaming frameworks already out there, why do we need yet another? To quote today’s guest Jay Kreps “the gap we see Kafka Streams filling is less the analytics-focused domain these frameworks focus on and more building core applications and m...

Scala is a functional programming language built on the JVM. For more than a decade, this didn’t mean anything to front end web developers. More recently, ScalaJS has brought Scala to the front end. ScalaJS is a project that compiles any Scala program down to JavaScript–so that all of your Scala programs can run on the browser. Haoyi Li has worked on ScalaJS extensively and has written an online b...

Software teams are traditionally composed of roles such as project manager, developer, QA, and manager. What happens if you throw out all of those titles, hire mostly engineers, and ask them to do whatever they think is best? That is the core idea behind Fred George’s idea of Developer Anarchy. In today’s episode, David Curry guest hosts an interview with Fred George. They talk about the structure...

Platform as a service can mean different things to different people. The most prominent feature of a PaaS is the ability to abstract away issues that every developer within an organization has to deal with. As an example, developers today don’t need to fear scalability and load balancing issues as much as engineers of the 90’s and early 2000s. We can develop our applications without worrying about...

Docker containers changed how engineers think about building software, and the company most responsible for the widespread adoption of containers is Docker itself. Since containerization has caught on in the mainstream, companies like RedHat, Google, Huawei, and many other big players have built platform products that utilize Docker containers. Docker containers are the unit that many engineers us...

Africa has a huge reservoir of untapped technical talent. Tech companies have a huge amount of work they need skilled employees for. Andela recruits the most talented technologists across Africa, shapes them into world-class software developers, and places them with tech companies worldwide such as Google and Microsoft. Brice Nkengsa is the director of engineering at Andela, and he joins the show...

After nine years at Microsoft, Scott Berkun left to become an author. One of his books on project management was read by Matt Mullenweg, the creator of the WordPress blogging tool that runs a large percentage of the internet (including Software Engineering Daily). Scott became friends with the WordPress founder, who is also the CEO of Automattic, a company that sells WordPress hosting and support....

Mark Zuckerberg may be the most powerful person in the world. At no other time in history has a single human had such fine-grained control over the most influential tool for media. Today’s guests are Michael Zimmer and Nick Proferes, the creators of The Zuckerberg Files, an index of every recorded word that Mark Zuckerberg has said in text, video, or audio. Why would someone create this? I was par...

Scikit-learn is a set of machine learning tools in Python that provides easy-to-use interfaces for building predictive models. In a previous episode with Per Harald Borgen about Machine Learning For Sales, he illustrated how easy it is to get up and running and productive with scikit-learn, even if you are not a machine learning expert. Srini Kadamati hosts today’s show and interviews Andreas Muel...

Containers have become the unit of infrastructure that many technology stacks deploy to. With the shift to containers, the attack surface of an application has changed, and we need to reconsider our security models; the resource allocation of our containers, the interactions between different containers on a single machine, and the big picture–how the external web may interact with our containers....

Security for the popular chat application Slack is a major focus for the company. A corporate Slack account is as valuable to a hacker as a corporate email account. In today’s episode, Ryan Huber and I talk through Slack’s approach to security–from philosophical discussions of how to company approaches security to the technical practices of logging and monitoring, and why Slack has a separate Amaz...

The role of “tech lead” is a combination of a software engineer, a project manager, and an architect. A tech lead might spend 30% of her time coding and 70% managing a project, resolving conflicts, and planning. In today’s episode, we discuss the structure of software teams, and when it might make sense to have a tech lead on your team, in contrast to the more common team structure of project mana...

Devoxx4Kids is an organization that is inspiring children through robotics, programming, and engineering. Kids who attend a Devoxx4Kids workshop are exposed to entertaining and educational activities, such as writing a Minecraft mod or programming an Arduino. Arun Gupta is a software engineer and the president of the board for Devoxx4Kids USA. In this episode we discuss coding education for kids,...

Google builds cloud services for developers, such as PubSub, Cloud Storage, BigQuery, and Cloud DataStore. On Software Engineering Daily, we’ve done lots of shows about how these types of services are built. In this episode, we are zooming in on the interaction between the developer using a cloud service and the design and engineering of the client APIs. To build a useful cloud service, Google has...

When the US government hacks its own citizens, The Electronic Frontier Foundation is often the best source of reporting to find out what laws the government has broken. When a change to the privacy policy of Google or Facebook is made, the Electronic Frontier Foundation is the best place to find out how that change in privacy exploits users. The Electronic Frontier Foundation provides legal defens...

Batch and stream processing systems have been evolving for the past decade. From MapReduce to Apache Storm to Dataflow, the best practices for large volume data processing have become more sophisticated as the industry and open source communities have iterated on them. Dataflow and Apache Beam are projects that present a unified batch and stream processing system. A previous episode with Frances P...