Software Engineering Daily

Advertising fraud takes billions of dollars out of the economy every year. We don’t know exactly how much money is being lost, because we don’t know what percentage of Internet users are bots. Are You A Human is a company designed to solve that exact problem, and provide a service for verifying whether a user is real or automated. Ben Trenda is the CEO of Are You A Human. In this episode, we discu...

Blockchain technologies like Bitcoin and Ethereum have not impacted the lives of most consumers today. The theoretical breakthroughs that blockchain enables will eventually happen–I will be able to pay 1 cent to a knowledge worker in Africa without having to pay a 5 cent transaction fee. My servers will be able to pay other servers for small compute jobs. We will have decentralized versions of sha...

Kafka is a distributed log for producers and consumers to publish messages to each other. We’ve done many shows about Kafka as a key building block for distributed systems, but we often leave out the discussion of the complexities of setting up Kafka and monitoring it. Kafka deployments can be a complex piece of software to manage. Tom Crayford is an engineer at Heroku, where he helped engineer th...

Netflix has a highly interactive user interface. As I move my mouse around the page, hovering over titles and inspecting movie descriptions, there is a lot going on under the hood. One component of this UI is RxJS, a library for building reactive JavaScript. Reactive programming uses the observer pattern to create objects that emit streams of events. We can compose these streams together to create...

When Uber’s engineering team published a blog post about moving to MySQL from Postgres, Markus Winand started receiving lots of email. Markus writes about databases on his blog “Use The Index, Luke,” a guide to database performance for developers. The people emailing Markus wanted to know–if Postgres doesn’t work well for Uber, is it safe to use for anyone? Markus wrote a detailed blog post that w...

Bot fraud, the New York tech scene, RethinkDB and open source; these topics and more are discussed in today’s episode. Two of the most popular guests return to the show to explore a variety of topics. Ben Halpern is the creator of The Practical Dev, a massively popular Twitter account and blog that you may recognize from its parody O’Reilly book covers. Haseeb Qureshi is an engineer at Airbnb, and...

Google Compute Engine is the public cloud built by Google. It provides infrastructure- and platform-as-a-service capabilities that rival Amazon Web Services. Today’s guest Joe Beda was there from the beginning of GCE, and he was also one of the early engineers on the Kubernetes project. Google’s internal systems have made it easy for employees to spin up compute resources, but it was not a simple...

Cloud computing was something much different in 2011, when Brian Gracely and Aaron Delp started The Cloudcast, a podcast I listen to on a regular basis. The Cloudcast features technical discussions about cloud infrastructure technology, and one of the most recent shows was a monologue by Brian Gracely where he explained his perspective on the industry rumblings about a Docker fork. The utility of...

The Simpsons is a classic, beloved television show. The scripts of The Simpsons have been made publicly available, and include dialogue, location, and character information. Todd Schneider used these scripts and other information sources as a corpus to analyze The Simpsons and find interesting statistics–such as who the most important supporting characters were, and how the ratings of the show hav...

Data engineering is the software engineering that enables data scientists to work effectively. In today’s episode, we explore the different sides of data engineering–the data science algorithms that need to be processed and the implementation of software architectures that enable those algorithms to run smoothly. The PANCAKE STACK is a 12-letter acronym that Chris Fregly gave to a collection of da...

When a user of a social network updates her profile, that profile update needs to propagate to several databases that want to know about such an update–search indexes, user databases, caches, and other services. When Neha Narkhede was at LinkedIn, she helped develop Kafka, which was deployed at LinkedIn to help solve this very problem. Using Kafka as an event queue, LinkedIn adopted the CQRS archi...

The intent of the DevOps movement is to get organizations moving faster and more effectively by breaking down siloes, and improving communication. Gene Kim’s book The Phoenix Project illustrated this by telling the fictional story of a company adopting a DevOps mentality. Although that book was fiction, Gene is an experienced engineer, having worked as founder and CTO of Tripwire, a software compa...

At Netflix, developers write applications with a variety of requirements–from simple requests for a list of movies to more resource-intensive requests like a complex machine learning workflow. Netflix wants developers to be able to request the resources they need from a compute cluster and receive those resources on-demand, without thinking too much about the state of that pool of resources they a...

Building a monitoring system is a complex distributed systems problem. Events are produced from different points in an application and must be aggregated in order to form metrics. These events are often ingested by a time series database, which forms the backbone of our monitoring system. Theo Schlossnagle is the CEO of Circonus, where he has been working on architecting the company’s monitoring s...

In order to move software updates from the development team to production, companies do a variety of things. Some teams might email files to each other or use FTP or even floppy disks. Most companies today at least use version control systems like Git together with separate servers for development and production. When code is ready to move to production, a build that is on the development server g...

Kafka Streams is a library for building streaming applications that transform input Kafka topics into output Kafka topics. In a time when there are numerous streaming frameworks already out there, why do we need yet another? To quote today’s guest Jay Kreps “the gap we see Kafka Streams filling is less the analytics-focused domain these frameworks focus on and more building core applications and m...

Scala is a functional programming language built on the JVM. For more than a decade, this didn’t mean anything to front end web developers. More recently, ScalaJS has brought Scala to the front end. ScalaJS is a project that compiles any Scala program down to JavaScript–so that all of your Scala programs can run on the browser. Haoyi Li has worked on ScalaJS extensively and has written an online b...

Software teams are traditionally composed of roles such as project manager, developer, QA, and manager. What happens if you throw out all of those titles, hire mostly engineers, and ask them to do whatever they think is best? That is the core idea behind Fred George’s idea of Developer Anarchy. In today’s episode, David Curry guest hosts an interview with Fred George. They talk about the structure...

Platform as a service can mean different things to different people. The most prominent feature of a PaaS is the ability to abstract away issues that every developer within an organization has to deal with. As an example, developers today don’t need to fear scalability and load balancing issues as much as engineers of the 90’s and early 2000s. We can develop our applications without worrying about...

Docker containers changed how engineers think about building software, and the company most responsible for the widespread adoption of containers is Docker itself. Since containerization has caught on in the mainstream, companies like RedHat, Google, Huawei, and many other big players have built platform products that utilize Docker containers. Docker containers are the unit that many engineers us...

Africa has a huge reservoir of untapped technical talent. Tech companies have a huge amount of work they need skilled employees for. Andela recruits the most talented technologists across Africa, shapes them into world-class software developers, and places them with tech companies worldwide such as Google and Microsoft. Brice Nkengsa is the director of engineering at Andela, and he joins the show...

After nine years at Microsoft, Scott Berkun left to become an author. One of his books on project management was read by Matt Mullenweg, the creator of the WordPress blogging tool that runs a large percentage of the internet (including Software Engineering Daily). Scott became friends with the WordPress founder, who is also the CEO of Automattic, a company that sells WordPress hosting and support....

Mark Zuckerberg may be the most powerful person in the world. At no other time in history has a single human had such fine-grained control over the most influential tool for media. Today’s guests are Michael Zimmer and Nick Proferes, the creators of The Zuckerberg Files, an index of every recorded word that Mark Zuckerberg has said in text, video, or audio. Why would someone create this? I was par...

Scikit-learn is a set of machine learning tools in Python that provides easy-to-use interfaces for building predictive models. In a previous episode with Per Harald Borgen about Machine Learning For Sales, he illustrated how easy it is to get up and running and productive with scikit-learn, even if you are not a machine learning expert. Srini Kadamati hosts today’s show and interviews Andreas Muel...

Containers have become the unit of infrastructure that many technology stacks deploy to. With the shift to containers, the attack surface of an application has changed, and we need to reconsider our security models; the resource allocation of our containers, the interactions between different containers on a single machine, and the big picture–how the external web may interact with our containers....