Software Engineering Daily

In a typical data analytics system, there are a variety of technologies interacting. HDFS for storing files, Spark for distributed machine learning, pandas for data analysis in Python–each of these different technologies has a different format for how data is represented. Serialization and deserialization between these different formats causes significant latency across the overall system. Apache...

EconTalk is a weekly economics podcast that has been going for a decade. On EconTalk, Russ Roberts brings on writers, intellectuals, and entrepreneurs for engaging conversations about the world as seen through the lens of economics. Russ Roberts is today’s guest, and it is a treat because I have been listening to EconTalk since 2006 and it was a central point of inspiration for what Software Engin...

On smart thermostats, sensor-driven assembly lines, and electronically monitored farms, the internet of things is producing huge volumes of data. To take advantage of that data, an application needs tools for storing and analyzing that data. Today’s guest is Jean-Cristophe Cimono, the CTO of mnubo, a cloud platform for connected objects. Today we walk through the architecture of mnubo and the use...

Data modeling is the process of creating relationships and rules about objects, so that we can decide how to store them in a database. Data modeling defines how we store and query our database systems. Today’s episode features a discussion of data modeling in Cassandra with Jon Haddad, an evangelist at Datastax. The distributed nature of Cassandra creates some unique rules around how we should mod...

Negotiation is an important skill for software engineers. The salary you negotiate at the beginning of your job could be a difference of tens of thousands of dollars over the course of an engineer’s career, but intimidating recruiters and exploding offers scare many engineers from negotiating at all. Today, Haseeb Qureshi returns to the show to discuss his epic story of salary negotiation. On a pr...

At software conferences, I like to walk around the vendor booths and talk to the representatives from different companies. By talking to the vendors about their marketing pitches, I get an idea of how those companies are positioning themselves for the future, and the complex business landscape of software becomes slightly easier to understand. At recent conferences, many of the big vendors have be...

Lee Atchison spent seven years at Amazon working in retail, software distribution, and Amazon Web Services. He then moved to New Relic, where he has spent four years scaling the company’s internal architecture. From his decade of experience at fast growing web technology companies, Lee has written the book Architecting for Scale, from O’Reilly. As an application scales, it becomes significantly mo...

Scheduling is the method by which work is assigned to resources to complete that work. At the operating system level, this can mean scheduling of threads and processes. At the data center level, this can mean scheduling Hadoop jobs or other workflows that require the orchestration of a network of computers. Adrian Cockcroft worked on scheduling at Sun Microsystems, eBay, and Netflix. In each of th...

Call centers are a vulnerable point of attack for large enterprises. Fraud accounts for more than $20 billion in lost money every year, and a significant portion of that fraud is due to customer service representatives being fraudulent social engineering attacks. Chris Halaschek joins the show today to discuss how Pindrop Security is addressing this attack vector. Every phone call that gets made t...

In 1999, it took $50,000 to buy a server. Once you bought that server, you had to know how to operate and maintain it. Today, cloud service providers have changed how we build software. Servers, load balancers, networking, storage–these hardware concerns have been turned into software. Don Pezet joins the show today to discuss the fundamentals of a cloud service provider. These are the basics that...

At most universities, there is not a course titled “cloud computing”. Most students leave college without an understanding of distributed systems, cloud service providers, and the fundamentals of how a data center works. Kasper Nissen and Martin Jensen are changing that with KubeCloud, a small tangible cloud computing cluster that runs on Raspberry Pis. Kasper and Martin started KubeCloud as a mas...

Twenty years ago, Bill Joy talked about the eight fallacies of distributed computing–these are things such as “the network is reliable”, and “latency is zero”, and “bandwidth is infinite”, and these fallacies are even more relevant today. With the popularity of Docker containers, the networks of distributed systems that we deal with have become even more complex, and with this growing complexity c...

Transferring money from one country to another is expensive, and the banks that facilitate money transfer have tricked us into believing that it should be expensive. On today’s show, Harsh Sinha explains the peer-to-peer system of transferring money with TransferWise, where he works as VP of engineering. Harsh also discusses the larger picture of FinTech companies. The emergence of so many compani...

Apache Cassandra is a distributed database that can handle large amounts of data with no single point of failure. Since 2008, Cassandra has been widely adopted and the software and the community around it have grown steadily. A software developer interacting with Cassandra uses CQL, the Cassandra Query Language. ScyllaDB is another open-source database that has been created to be totally compatibl...

In order to use a remote desktop experience, software engineers have a limited number of options, and most of them are proprietary, like VMWare or Oracle. Remote desktop is a functionality that many engineers use every day, so it is surprising that the open source world has taken awhile to displace the functionality of proprietary software. In 2010, Mike Jumper started working on Guacamole, a way...

18F is an organization that is building the 21st century digital government. In order to build online government services that have the high quality of modern cloud applications, 18F built Cloud.gov, a platform-as-a-service that can be used to stand up web applications for divisions of the government. Aidan Feldman helped build Cloud.gov, and on today’s episode he dissects its architecture. Cloud....

Pieter Hintjens grew up writing software by himself. The act of writing code brought him great pleasure, but the isolated creative process disconnected him from the rest of the world. As his life progressed he became involved in open source communities, and he discovered a passion for human interaction. Open source software succeeds or fails on the strength of the community. One story of success i...

Six years ago, Twitter was experiencing outages due to high traffic. Back in 2010 Twitter was built as a monolithic Ruby on Rails application. Twitter migrated to a microservices architecture to fix these problems. During this migration, the engineers at Twitter learned how to build and scale highly distributed microservice architectures. William Morgan was an engineer at Twitter during that time,...

Mass customization is the process of making customized, personalized products that are accessible to individuals and small businesses. The process involves manufacturing, assembly lines, supply chains, and software at every step along the way. Today’s guests are Jim Sokoloff and Maarten Wensveen, who work on infrastructure and technology at Cimpress, a mass customization platform. Cimpress has t s...

The unit of computation has evolved from on premise servers to virtual machines in the cloud to containers running in those virtual machines. Serverless computation is another stage in the evolution of computational unit management. With a serverless architecture, a function call to the cloud spins up a transient container, calls the function on that container, and then spins down the container. R...

Algorithmia is marketplace for algorithms. A software engineer who writes an algorithm for image processing or spam detection or TF-IDF can turn that algorithm into a RESTful API to be consumed by other developers. Different algorithms can be composed together to build even higher level applications. Diego Oppenheimer is the CEO of Algorithmia, and he joins the show today to explain how Algorithmi...

The Internet of Things is becoming a reality. Factories are being outfitted with sensors, temperature monitors, and other data gathering devices. In agriculture, farms are becoming more efficient thanks to soil monitoring devices and automated pesticide regulation. In our homes, refrigerators, alarm clocks, and mirrors are becoming “smart”. Steve Busby joins the show today to talk about the big pi...

Every software application has secrets. User passwords and database credentials must be managed carefully, because poor access controls can lead to disaster scenarios. Vault is a tool for secret management, developed at Hashicorp, a company that builds software tools for application delivery and infrastructure management. Seth Vargo is a software engineer and open source advocate at Hashicorp, and...

Google’s site reliability engineers are responsible for maintaining the highly available services that power the Google software that we all use on a regular basis. O’Reilly recently published the book “Site Reliability Engineering: How Google Runs Production Systems”, and the book provides a comprehensive window into how the site reliability engineering role works. Todd Underwood is a director of...

Google researcher Jennifer Wang co-wrote a paper called “Gender Differences in Factors Influencing Pursuit of Computer Science and Related Fields”. The paper focuses on a survey of 1700 high school and college students, and takes a statistical approach to understanding why women are not pursuing computer science. In our conversation, Jennifer talks about the two influences that lead to fewer women...