Software Engineering Daily

The vision of code-free programming has existed for decades. Software engineers have always dreamed of empowering non-technical users with the same creative tools that programmers have access to.  For many years, the underlying technology of the web was not powerful enough to make this dream a reality. Platforms such as WordPress, Squarespace, and Wix have allowed for some of the functionality of...

“Serverless” is an execution model where applications are scheduled and deployed to servers that are not directly managed by the application developer.  In serverless execution, an application only loads and operates when a user actually needs to get a response from that application. This saves on resources, because many applications do not need to run at all times–they only need to be available f...

Data engineering is difficult.  Companies want to be able to maximize the value they get from their large data sets, but there are so many steps required for deriving that value that most companies feel like they are always far behind the ideal state of where they could be.  The cloud makes it cheap to save data. Tools like Spark and Snowflake give us usable APIs for simplifying the data processin...

Java programs run in a different environment than they did ten years ago. Modern infrastructure runs on containers sitting in a Kubernetes cluster. The optimal configuration for a Java program in that context is different than it was for an environment dominated by virtual machines and bare metal. When you are co-scheduling your services with each other, those services could be fighting for resour...

Crash monitoring emerged as a software category over the last decade. Crash monitoring software allows developers to understand when their applications are crashing on client devices. For example, we have an app for Software Engineering Daily that people download on Android or iOS. Users download the app to their smartphone. When the user is playing an episode, and the app crashes, the details of...

Software bugs cause unexpected problems at every company.  Some problems are small. A website goes down in the middle of the night, and the outage triggers a phone call to an engineer who has to wake up and fix the problem. Other problems can be significantly larger. When a major problem occurs, it can cause millions of dollars in losses and requires hours of work to fix. When software unexpectedl...

Frontend software development has become as complex as backend development. There was a time when frontend web development was simple. There was a small number of JavaScript frameworks and templating systems. Your CSS was simple configuration for the colors on your webpage. Today, there is a giant ecosystem of frontend tools, APIs, and middleware delivering data to the user. Gatsby is a framework...

Drones will deliver food to us. Drones will be able to extinguish fires. Drones will be used to relay Internet signals and make the world more connected.  These all sound like great ideas, so why aren’t there more drones in the sky today? There are many answers to that question, some of which relate to engineering and some of which are about the regulatory barriers.  Chris Anderson is the CEO of 3...

A large social network needs to develop systems for ingesting, storing, and processing large volumes of data. Data engineering at scale requires multiple engineering teams that are responsible for different areas of the infrastructure. Data needs to be structured coherently in order to minimize the data cleaning process. Machine learning models need to be developed, deployed, and iterated on at sc...

Every company has a software supply chain. A company builds its products from custom code, paid APIs, paid proprietary binaries, and open source software libraries. As the types of software available have increased, the management of the software supply chain has become complex.  Large software companies have always needed to ensure the security of their software. With the growing variety of open...

Machine learning is widely understood by the software community. But it is still hard to build a company around machine learning, because there is not easy access to large, unique data sets. Scale is a platform for training and validating data that is used for machine learning. Most machine learning models are built with supervised learning. Labeled examples are analyzed to understand the mathemat...

Photo credit: Elisabeth Fall Ben Horowitz started Loudcloud with Marc Andreessen in 1999. He ran the company for eight years and chronicled his experience in his first book The Hard Thing About Hard Things. In his time running Loudcloud, the dotcom bubble burst, but Loudcloud needed cash so badly that he took the company public in 2001. Loudcloud went through layoffs, downsizing, and a difficult s...

Facebook leadership has a significant amount of engineers in its ranks, and engineers understand how to create an environment that appeals to other engineers.  Engineers do not like working on projects that they are not interested in, so Facebook optimizes for matching engineers to enjoyable work. Engineers do not like taking orders from managers, so Facebook optimizes for a collaborative relation...

Chaos Monkeys is a book about Silicon Valley startups and Facebook. It is one of the most accurate books written about the modern technology industry, and captures both the negatives and the positives of software companies. Antonio Garcia Martinez is the author of Chaos Monkeys. He wrote the book after going through a gauntlet of prototypical Silicon Valley experiences. Antonio founded an adtech c...

Facebook engineering is unique. Software is built at Facebook in a way that is distinctly different than any other company. In our series of shows about Facebook engineering, we have mostly covered the positive side of Facebook’s practices. In today’s show, we explore the downsides.  Facebook moves fast. Engineers within the company must move fast, which can reduce the time spent on formal process...

Peter Deng has worked on most of Facebook’s major products: Newsfeed, Instagram, Oculus, and Messenger. These different products have different requirements, but are all part of the same ethos of connecting people through social networks.  Facebook is a consumer product company that is powered by a strong engineering workforce. The relationship between product managers and engineering are two part...

Jeff Rothschild was one of the earliest engineers to join Facebook. In the 1990s, Jeff had co-founded Veritas Software and helped it to its IPO in 2004. After Veritas, Jeff worked on several other of his own companies. He was working with Accel Partners on investments when he started to learn about Facebook. Accel was an investor in Facebook, having made a large investment in 2005. After the inves...

In 2009, the Berkeley AMPLab was a center of innovation.  Three projects from AMPLab have turned into successful open source projects and companies: Spark, Mesos, and Alluxio. Haoyuan Li was the creator of Alluxio, and he returns to the show to discuss his journey taking Alluxio from a research project to a company that has customers including Alibaba, Baidu, Wells Fargo, and Samsung. Alluxio is a...

Redis is an in-memory database that persists to disk. Redis is commonly used as an object cache for web applications. Applications are composed of caches and databases. A cache typically stores the data in memory, and a database typically stores the data on disk. Memory has significantly faster access times, but is more expensive and is volatile, meaning that if the computer that is holding that p...

LinkedIn is a social network with petabytes of data.  In order to store that data, LinkedIn distributes and replicates that data across a large cluster of machines running the Hadoop Distributed File System. In order to run calculations across its large data set, LinkedIn needs to split the computation up using MapReduce-style jobs. LinkedIn has been developing its data infrastructure since the ea...

Cryptocurrencies are a fundamental computer science invention.  Cryptocurrencies crashed in 2018 but the technology remains as promising as ever. Bitcoin is a decentralized currency, and a plausible end state that is implied by Bitcoin’s current trajectory is a permissionless, decentralized financial system. This idea of decentralized finance or “DeFi” begs numerous questions: who will build the c...

Dark Lang is a programming language that is tightly integrated with the cloud.  Dark takes an opinionated approach that most developers are going to want to run their applications in the cloud, and this perspective influences how Dark looks at deployments, IDEs, exception handling, and other aspects of software development. Paul Biggar is the founder of CircleCI, and ran the company for eight year...

Apache Kafka was created at LinkedIn. Kafka was open sourced in 2011, when the company was eight years old. By that time, LinkedIn had developed a social network with millions of users. LinkedIn’s engineering team was building a range of externally facing products and internal tools, and many of these tools required a high-throughput system for publishing data and subscribing to topics. Kafka was...

Modern applications are distributed systems. These applications require an installation mechanism that can run and update the software across multiple nodes.  When a SaaS company starts to work with large enterprise customers, that company needs to figure out a way to deliver their product to the enterprise. This requires the SaaS company to deploy the product to whatever infrastructure the enterp...

Databases go offline. Services fail to scale up. Deployment errors can cause an application backend to get DDoS’d. When an event happens that prevents your company from operating as expected, it is known as an incident. Software teams respond to an incident by issuing a fix. Sometimes that fix returns the software to its ideal state. Other times the software remains in a degraded state, and it tak...