Software Engineering Daily

Google Search allows humans to find and access information across the web. A human enters an unstructured query into the search box, the search engine provides several links as a result, and the human clicks on one of those links. That link brings up a web page, which is a set of unstructured data. Humans can read and understand news articles, videos, and Wikipedia pages. Google Search solves the...

David Cancel has started five companies, most recently Drift. Drift is a conversational marketing and sales platform. David has a depth of engineering skills and a breadth of business experience that make him an amazing source of knowledge. In today’s episode, David discusses topics ranging from the technical details of making a machine learning-driven sales platform to the battle scars from his e...

Engineers who start companies often find themselves building something they have no experience with: a hiring process. Hiring engineers today is not as systematic as building software. We don’t have lots of data that tells us what makes for an effective programming interview question. The smartest tech companies in the world are still making hiring mistakes–often through the “false negative” of re...

I like to write music, and a year ago I started working on an album called “Gig Economy”. The plan for the album was to hire musicians from gig economy platforms like Fiverr and Upwork to perform on songs that I produced. The album is finished and I’m happy with the result, so I’m sharing it on the podcast today as an extra episode released on a Sunday. We’ll be back tomorrow with content about so...

BloxRoute Labs is a company that is developing a blockchain distribution network (BDN). Much like a CDN pushes media files out to the edges of the web to make them faster to access, a BDN pushes out information to miners in the network. Of course, this means that the BDN could potentially be centralized infrastructure. In order to make the BDN effectively decentralized and trustworthy, BloxRoute c...

A smart security camera takes in a high volume of video images and processes those images using a set of machine learning models. Those models can be used to identify interesting snippets of movement throughout the day, and decide which of those snippets to keep. Some of the video snippets might contain movement of birds–but other video snippets might contain footage of intruders. As the video str...

Engineers can build applications faster by using tools that abstract away infrastructure. Major cloud providers offer this tooling in the form of functions-as-a-service, as well as managed services such as Google BigQuery or Azure Container Instances. The term “serverless” refers to these functions-as-a-service and the managed services–because when you use these tools, you are not making calls to...

DevSecOps emphasizes moving security out of a siloed audit process and distributing security practices throughout the software supply chain. In the past, software development usually followed a waterfall development process. Each step in building software was serialized, one after another. First, software was planned. Then it was built. Then it was tested. Finally, the software received a security...

Google Search is a highly interactive JavaScript application. As you enter a query, results are being automatically suggested to you before you even finish typing. When you press enter, some of your search results may be widgets that represent the weather, the price of a stock, a recipe for green bean soup, or a language translation for a phrase. These complex frontend components are loading dynam...

Deepak Singh is the director of compute services at AWS, where he works on cloud products relating to containers, Linux, and High Performance Computing. In today’s show, Deepak describes how the market for containers and serverless has evolved, and how Amazon thinks about product strategy. Back in 2014, Docker containers were becoming a popular way to deploy and manage application infrastructure....

Linux has created much more value for Google than it has for Linus Torvalds. Ruby on Rails has created more value for Airbnb than it has for David Heinemeier Hansson. Successful open source projects create more value than their creators’ capture–and that’s one reason why collections of people on the Internet are often inspired to work together on open source. When an engineer creates an open sourc...

Search is part of almost every application. Users search for movies to watch. Engineers search through terabytes of log messages to find exceptions. Drivers search through maps to find a destination. Search remains an unsolved problem, with lots of room for optimization. Many search applications have been built Elasticsearch, an open source distributed search engine. Elasticsearch is the code that...

Virtual machines are operating system instances that run alongside each other on the same physical host. The virtual machines running on a physical host are managed by a hypervisor running on the physical host. A cluster of two physical servers could have four virtual machines running across those two physical instances. Those four virtual machines can communicate over a virtual switch. A network...

A data warehouse provides fast access to large data sets for analytics, data science, and dashboards. A data warehouse differs from a transactional database, because you often do not need to update specific records. Because of the read-only nature of the access patterns, and the high volumes of data being queried, the design of a data warehouse is very different than a transactional database. With...

The Windows operating system is one of the most widely used pieces of software in history. Windows was started before there was any alternative to a monolithic codebase, because Microsoft was building software before the Internet was widely used by consumers. Networked computers gave rise to web applications, and software engineers began to rethink how to build everything. Software development got...

Google Brain is an engineering team focused on deep learning research and applications. One growing area of interest within Google Brain is that of generative models. A generative model uses neural networks and a large data set to create new data similar to the ones that the network has seen before. One approach to making use of generative models is GANs: generative adversarial networks. GANs can...

Mapillary is a platform for gathering photos taken by smartphones and using that data to build a 3D model of the world. Mapillary’s model of the world includes labeled objects such as traffic signs, trees, humans, and buildings. This 3D model can be explored much like you can explore Google Street view. The data set that underlies Mapillary is crowdsourced from volunteer users who are taking pictu...

When Aran Khanna was a college student, he accepted an internship to work at Facebook. Even before his internship started, he started playing around with Facebook’s APIs and applications. Aran built a Chrome extension called Marauder’s Map, which used Facebook Messenger’s web APIs to track where people lived, what their schedule was, and other highly sensitive information. These were not public fe...

Airbnb began in 2008 as a monolithic Rails application serving the simple purpose of listing homes for rental. Over time, the number of listings increased dramatically, as did the number of people who were renting. With that scale, the Rails app had to be broken into different services, and entire teams were built out to focus on challenges such as pricing, application infrastructure, and search....

We previously released this episode with the wrong audio file and are re-releasing it on a weekend. TIBCO was started in the 90’s with a popular message bus product that was widely used by finance companies, logistics providers, and other systems with high throughput. As TIBCO grew in popularity, the company expanded into other areas through products it developed in-house as well as through acquis...

Log messages are fast, high volume, unstructured data. Logs are often the source of metrics, alerts, and dashboards, so these critical systems are downstream from a log management system. A log management system needs to be highly available, so that a failure in one part of your system will not be correlated with failure of the log management system. Users of a log management system are often buil...

When a database gets large, it can start to perform poorly. This can manifest in slow query speed. You can speed up a query by defining an index, which is a data structure that allows for faster access to the data that is being indexed. As a consequence, whenever you update the database, you will now need to update the index with that new piece of data. The more you index your data, the faster the...

Linkedin is an organization with thousands of employees. An enterprise of that size starts to develop problems with data collaboration. Data collaboration is the process of sharing and analyzing data with multiple users, such as data scientists, business analysts, and engineers. How do data scientists know what questions to ask? How do business analysts know the right way to query a database? How...

Kotlin is a statically typed programming language that started as a JVM language. It gained popularity because it reduces the amount of boilerplate code required for a typical Java project. Many of the early adopters of Kotlin were building Android apps or Java applications, but it has grown to a variety of use cases including at companies like Uber, Pinterest, and Atlassian. Andrey Breslav is the...

Open source software is key to our software infrastructure. Closed source enterprises rely on open source software, but the development processes for closed source and open source software are often different in their approach to continuous integration and delivery. Oren Novotny is a chief architect of DevOps and modern software at BlueMetal Architects where he works with a variety of clients to b...