Software Engineering Daily

Dogecoin was started in 2013 as a joke. Jackson Palmer forked Bitcoin and created his cryptocurrency as a play off the “doge” meme. The currency became popular as a means of reddit users “tipping” each other. If I made a comment on reddit that you liked, you might send me some Dogecoin. This use case allowed people to share the idea of Dogecoin virally, and Dogecoin became valuable, even though th...

There are two factors that limit the rate at which transactions are accepted into the Bitcoin blockchain: block time and block size. Block time defines how often a new block is appended onto the blockchain. Block size defines how many transactions fit into a new block. As of March 2018, the current block time and block size allow for about 7 transactions per second to be accepted into the Bitcoin...

Bitcoin is an immutable, append-only blockchain ledger that reaches consensus through proof-of-work. The contents of the ledger are financial transactions–people sending and receiving Bitcoin currency to each other. Since Bitcoin, there have been other cryptocurrencies that have similar properties–like Ethereum and the IPFS/Filecoin system. Similar to Bitcoin, they use a decentralized, proof of wo...

The easiest way to train a computer to recognize a picture of cat is to show the computer a million labeled images of cats. The easiest way to train a computer to recognize a stop sign is to show the computer a million labeled stop signs. Supervised machine learning systems require labeled data. Today, most of that labeling needs to be done by humans. When a large tech company decides to “build a...

Apache Spark is a system for processing large data sets in parallel. The core abstraction of Spark is the resilient distributed dataset (RDD), a working set of data that sits in memory for fast, iterative processing. Matei Zaharia created Spark with two goals: to provide a composable, high-level set of APIs for performing distributed processing; and to provide a unified engine for running complete...

Steve Herrod led engineering at VMWare as the company scaled from 30 engineers to 3,000 engineers. After 11 years, he left to become a managing director for General Catalyst, a venture capital firm. Since he has both operating experience and a wide view of the technology landscape as an investor, he is well-equipped to discuss a topic that we have been covering on Software Engineering Daily: the i...

On Software Engineering Daily, we have been covering the “serverless” movement in detail. For people who don’t use serverless functions, it seems like a niche. Serverless functions are stateless, auto-scaling, event driven blobs of code. You might say “serverless sounds kind of cool, but why don’t I just use a server? It’s a paradigm I’m used to.” Serverless is exciting not because of what it adds...

Earlier this year we did several shows about Cloud Foundry, followed by several shows about Kubernetes. Both of these projects allow you to build scalable, multi-node applications–but they serve different types of users. Cloud Foundry encompasses a larger scope of the application experience than Kubernetes. Kubernetes is lower level, and is actually being used within newer versions of Cloud Foundr...

Kafka is at the center of modern streaming systems. Kafka serves as a database, a pubsub system, a buffer, and a data recovery tool. It’s an extremely flexible tool, and that flexibility has led to its use as a platform for a wide variety of data intensive applications. Today’s guest is Gwen Shapira, a product manager at Confluent. Confluent is a company that was started by the creators of Apache...

Streaming architecture defines how large volumes of data make their way through an organization. Data is created at a user’s smartphone, or on a sensor inside of a conveyor belt at a factory. That data is sent to a set of backend services that aggregate the data, organizing it and making it available to business analysts, application developers, and machine learning algorithms. The velocity at whi...

When you go to a website where a video is playing, and your video lags, how does the website know that you are having a bad experience? Problems with video are often not complete failures–maybe part of the video loads, and plays just fine, and then the rest of the video is buffering. You have probably experienced sitting in front of a video, waiting for it to load as the loading wheel mysteriously...

Tugdual Grall is an engineer with MapR. In today’s episode, we explore use cases and architectural patterns for streaming analytics. Full disclosure: MapR is a sponsor of Software Engineering Daily. In past shows, we have covered data engineering in detail–we’ve looked at Uber’s streaming architecture, talked to Matei Zaharia about the basics of Apache Spark, and explored the history of Hadoop. To...

Pinterest is a visual feed of ideas, products, clothing, and recipes. Millions of users browse Pinterest to find images and text that are tailored to their interests. Like most companies, Pinterest started with a large monolithic application that served all requests. As Pinterest’s engineering resources expanded, some of the architecture was broken up into microservices and Dockerized, which make...

Over 12 years of engineering, Box has developed a complex architecture of services. Whenever a user uploads a file to Box, that upload might cause 5 or 6 different services to react to the event. Each of these services is managed by a set of servers, and managing all of these different servers is a challenge. Sam Ghods is the cofounder and services architect of Box. In 2014, Sam was surveying the...

When Box started in 2006, the small engineering team had a lot to learn. Box was one of the earliest cloud storage companies, with a product that allowed companies to securely upload files to remote storage. This was two years before Amazon Web Services introduced on-demand infrastructure, so the Box team managed their own servers, which they learned how to do as they went along. In the early days...

Employees often find themselves needing to do work outside of the office. Depending on the sensitivity of your task, accessing internal systems from a remote location may or may not be OK. If you are using a corporate application that shows the menu of your company’s cafe on your smartphone, your workload is less sensitive. If you are accessing the proprietary codebase of your company’s search eng...

Applications need to be ready to scale in response to high-load events. With mobile applications, this can be even more important. People rely on mobile applications such as banking, ride sharing, and GPS. During Black Friday, a popular ecommerce application could be bombarded by user requests–you might not be able to complete a request to buy an item at the Black Friday discount. If you attend th...

Matt Leising is a journalist at Bloomberg who has covered financial markets for 15 years. Today, his reporting has been completely engulfed by cryptocurrencies. There are so many dramatic stories, it’s hard to pick what to focus on. Today, we discuss two topics he has covered recently: Ripple and Tether. Ripple is a company that makes enterprise blockchain solutions for global payments. That sound...

Over the last decade, computation and storage has moved from on-premise hardware into the cloud data center. Instead of having large servers “on premise,” companies started to outsource their server workloads to cloud service providers. At the same time, there has been a proliferation of devices at the “edge.” The most common edge device is your smartphone, but there are many other smart devices t...

How do you build resilient, failure tested systems? Redundancy, backups, and testing are all important. But there is also an increasing trend towards chaos engineering–the technique of inducing controlled failures in order to prove that a system is fault tolerant in the way that you expect. In last week’s episode with Kolton Andrus, we discussed one way to build chaos engineering as a routine part...

The number of ways that applications can fail are numerous. Disks fail all the time. Servers overheat. Network connections get flaky. You assume that you are prepared for such a scenario, because you have replicated your servers. You have the database backed up. Your core application is spread across multiple availability zones. But are you really sure that your system is resilient? The only way t...

On this show, we spend a lot of time talking about CI/CD, data engineering, and microservices. These technologies have only been widely talked about for the last 5-10 years. That means that they are easy to adopt for startups that get founded in the last 5-10 years, but not necessarily for older enterprises. Within a large enterprise, it can be challenging to make significant changes to how techno...

Sue Loh is a software engineer and author of a book with the goal of breaking developer stereotypes. Stereotyping among developers leads to bad outcomes. When incorrect assumptions are made about certain populations, those populations feel marginalized and engineering resources get misallocated. From the perspective of Sue, the main problem is about how children are socialized. Young girls in part...

Functional programming can improve the overall design of an application architecture. Runar Bjarnason has been exploring how writing in a functional style increases modularity and compositionality of software for many years. He is co-author of Functional Programming in Scala, a book that explores the relationship between functional programming and software design. In this interview with guest host...

Training a deep learning model involves operations over tensors. A tensor is a multi-dimensional array of numbers. For several years, GPUs were used for these linear algebra calculations. That’s because graphics chips are built to efficiently process matrix operations. Tensor processing consists of linear algebra operations that are similar in some ways to graphics processing–but not identical. De...