Software Engineering Daily

Kafka has achieved widespread popularity as a popular distributed queue and event streaming platform, with enterprise adoption and a billion dollar company (Confluent) built around it. But could there be value in building a new platform from scratch? Redpanda is a streaming platform built to be compatible with Kafka, that does not require the JVM nor Zookeeper, both of which are dependencies that...

Companies can have a negative impact on the environment by outputting excess carbon. Many companies want to reduce their net carbon impact to zero, which can be done by investing in forests. Pachama is a marketplace for forest investments. Pachama uses satellites, imaging, machine learning, and other techniques to determine how much carbon is being absorbed by different forests. Diego Saez-Gil is...

Rust and Golang are two of the newest lower level languages for doing systems programming. They are often used for applications such as file systems, operating systems, and latency-sensitive applications. How do they compare in terms of safety, speed, and programming ergonomics? Linhai Song is an assistant professor and researcher at Penn State University, and joins the show to talk about his work...

GraphQL has changed the common design patterns for the interface between backend and frontend. This is usually achieved by the presence of a GraphQL server, which interprets and federates a query from the frontend to the backend server infrastructure. Dgraph is a distributed graph database with native GraphQL support. Manish Jain is a founder of Dgraph, and joins the show to talk about its purpose...

Agriculture infrastructure allows plants such as corn, soy, and wheat to move from large scale farms to consumers all around the world. The relevant players in the agricultural infrastructure includes growers, shippers, and planners. These individuals need new technology to interact more efficiently.  Growers need to be able to connect more smoothly with buyers. Farmers need better management of t...

Network discovery allows enterprises to identify what devices are on their network. These devices can include smartphones, servers, desktop computers, and tablets. Being able to index the devices on a network is crucial to figuring out the security profile of that network. HD Moore is a founder of Rumble Networks, a company focused on network discovery and asset inventory. He joins the show to tal...

Large technology companies are a new type of industry. Their power and reach is resistant to a comparison of previous generations, such as big oil. Alex Kantrowitz is a journalist who has covered big technology for much of his career, and he currently runs Big Technology, a newsletter and podcast about the biggest technology companies in the world. He’s also the author of “Always Day One: How The...

Cost management is growing in importance for companies that want to manage their significant cloud bill. Kubernetes plays an increasing role in modern infrastructure, so managing cost of Kubernetes clusters becomes important as well. Kubecost is a company focused on giving visibility into Kubernetes resources and reducing spend. Webb Brown is a founder of Kubecost and joins the show to talk about...

TensorFlow Lite is an open source deep learning framework for on-device inference. TensorFlow Lite was designed to improve the viability of machine learning applications on phones, sensors, and other IoT devices. Pete Warden works on TensorFlow Lite at Google and joins the show to talk about the world of machine learning applications and the necessary frameworks and devices necessary to build them...

Data lakes and data warehouses store high volumes of multidimensional data. Data sources for these pieces of infrastructure can become unreliable for a variety of reasons. When data sources break, it can cause downstream problems. One company working to solve the problem of data reliability is Monte Carlo Data. Barr Moses and Lior Gavish are founders of Monte Carlo and join the show to talk about...

When Tim Wagner worked at Amazon, he invented AWS Lambda. After working on the early serverless infrastructure, he joined Coinbase and worked as VP of Engineering. Since leaving Coinbase, he has started a new company called Vendia. Vendia combines his learnings from the serverless space with the innovations around blockchains to work on the problem of data sharing. Tim and David Wells join the sho...

Serverless has grown in popularity over the last five years, and the space of applications that can be built entirely with serverless has increased dramatically. This is due to two factors: the growing array of serverless tools (such as edge-located key value stores) and the rising number of companies with serverless offerings. One of those companies is Fastly, which originally gained adoption for...

Originally published July 30, 2019 “Internet of Things” is a term used to describe the increasing connectivity and intelligence of physical objects within our lives.  IoT has manifested within enterprises under the term “Industrial IoT,” as wireless connectivity and machine learning have started to improve devices such as centrifuges, conveyor belts, and factory robotics. In the consumer space, Io...

Originally published July 19, 2019 In 2011, Facebook had begun to focus its efforts on mobile development. Mobile phones did not have access to reliable, high bandwidth connections, and the Facebook engineering team needed to find a solution to improve the request latency between mobile clients and the backend Facebook infrastructure. One source of latency was recursive data fetching. If a mobile...

Originally published September 13, 2019 Amazon Web Services first came out in 2006. It took several years before the software industry realized that cloud computing was a transformative piece of technology. Initially, the common perspective around cloud computing was that it was a useful tool for startups, but would not be a smart option for large, established businesses. Cloud computing was not c...

Originally published May 29, 2020 Kubernetes has become a highly usable platform for deploying and managing distributed systems.  The user experience for Kubernetes is great, but is still not as simple as a full-on serverless implementation--at least, that has been a long-held assumption. Why would you manage your own infrastructure, even if it is Kubernetes? Why not use autoscaling Lambda functio...

Originally published April 17, 2019 Drishti is a company focused on improving manufacturing workflows using computer vision. A manufacturing environment consists of assembly lines. A line is composed of sequential stations along that manufacturing line. At each station on the assembly line, a worker performs an operation on the item that is being manufactured. This type of workflow is used for the...

Originally published April 7, 2017 Engineers in Silicon Valley see a world of constant progress. Our work is creative and intellectually challenging. We are building the future and getting compensated quite well for it. But what if we are actually achieving far less than what is possible? What if, after so many years of high margins, gourmet lunch, and self-flattery, we have lowered our standards...

Originally published March 6, 2020 ReactJS developers have lots of options for building their applications, and those options are not easy to work through. State management, concurrency, networking, and testing all have elements of complexity and a wide range of available tools. Take a look at any specific area of JavaScript application development, and you can find highly varied opinions. Kent Do...

Originally published June 21, 2019 Niantic is the company behind Pokemon Go, an augmented reality game where users walk around in the real world and catch Pokemon which appear on their screen. The idea for augmented reality has existed for a long time. But the technology to bring augmented reality to the mass market has appeared only recently. Improved mobile technology makes it possible for a sma...

Originally published July 7, 2017 Airbnb is a company that is driven by design. New user interfaces are dreamed up by designers and implemented for web, iOS, and Android. This implementation process takes a lot of resources, but it used to take even more before the company started using React Native. React Native allows Airbnb to reuse components effectively. React Native works by presenting a con...

Originally published October 18, 2019 Apache Kafka was created at LinkedIn. Kafka was open sourced in 2011, when the company was eight years old. By that time, LinkedIn had developed a social network with millions of users. LinkedIn’s engineering team was building a range of externally facing products and internal tools, and many of these tools required a high-throughput system for publishing data...

Originally published December 9, 2019 Machine learning algorithms have existed for decades. But in the last ten years, several advancements in software and hardware have caused dramatic growth in the viability of applications based on machine learning. Smartphones generate large quantities of data about how humans move through the world. Software-as-a-service companies generate data about how thes...

Originally published September 17, 2019 Ever since Apache Kafka was open sourced from LinkedIn, it has been used to solve a wide variety of problems in distributed systems and data engineering. Kafka is a distributed messaging queue that is used by developers to publish messages and subscribe to topics with a certain message type. Kafka allows information to flow throughout a company such that mul...

Originally published January 25, 2019 When TensorFlow came out of Google, the machine learning community converged around it. TensorFlow is a framework for building machine learning models, but the lifecycle of a machine learning model has a scope that is bigger than just creating a model. Machine learning developers also need to have a testing and deployment process for continuous delivery of mod...