Cloud Engineering Archives - Software Engineering Daily

Stripe processes payments for thousands of businesses. A single payment could involve 10 different networked services. If a payment fails, engineers need to be able to diagnose what happened. The root cause could lie in any of those services. Distributed tracing is used to find the causes of failures and latency within networked services. In The post Stripe Observability Pipeline with Cory Watson...

Monitoring a Kubernetes cluster allows operators to track the resource utilization of the containers within that cluster. In today’s episode, Ilan Rabinovitch joins the show to explore the different options for setting up monitoring, and some common design patterns around Kubernetes logging and metrics gathering. Ilan is the VP of product and community at Datadog. The post Monitoring Kubernetes wi...

Go is a language designed to improve systems programming. Go includes abstractions that simplify aspects of low level engineering that are historically difficult—concurrency, resource allocation, and dependency management. In that light, it makes sense that the Kubernetes container orchestration system was written in Go. Erik St. Martin is a cloud developer advocate at Microsoft, where The post Go...

Tammy Butow has worked at Digital Ocean and Dropbox, where she built out infrastructure and managed engineering teams. At both of these companies, the customer base was at a massive scale. At Dropbox, Tammy worked on the database that holds metadata used by Dropbox users to access their files. To call this metadata system simply The post Database Chaos with Tammy Butow appeared first on Software E...

Software engineers have interacted with operations teams since the software was being written. In the 1990s, most operations teams worked with physical infrastructure. They made sure that servers were provisioned correctly and installed with the proper software. When software engineers shipped bad code that took down a software company, the operations teams had to help The post Site Reliability Ma...

Steve Herrod led engineering at VMWare as the company scaled from 30 engineers to 3,000 engineers. After 11 years, he left to become a managing director for General Catalyst, a venture capital firm. Since he has both operating experience and a wide view of the technology landscape as an investor, he is well-equipped to discuss The post Cloud and Edge with Steve Herrod appeared first on Software En...

On Software Engineering Daily, we have been covering the “serverless” movement in detail. For people who don’t use serverless functions, it seems like a niche. Serverless functions are stateless, auto-scaling, event-driven blobs of code. You might say “serverless sounds kind of cool, but why don’t I just use a server? It’s a paradigm I’m used The post Serverless Systems with Eduardo Laureano appea...

Earlier this year we did several shows about Cloud Foundry, followed by several shows about Kubernetes. Both of these projects allow you to build scalable, multi-node applications–but they serve different types of users. Cloud Foundry encompasses a larger scope of the application experience than Kubernetes. Kubernetes is lower level and is actually being used within The post Cloud Foundry Ov...

Over 12 years of engineering, Box has developed a complex architecture of services. Whenever a user uploads a file to Box, that upload might cause 5 or 6 different services to react to the event. Each of these services is managed by a set of servers, and managing all of these different servers is a The post Box Kubernetes Migration with Sam Ghods appeared first on Software Engineering Daily .

When Box started in 2006, the small engineering team had a lot to learn. Box was one of the earliest cloud storage companies, with a product that allowed companies to securely upload files to remote storage. This was two years before Amazon Web Services introduced on-demand infrastructure, so the Box team managed their own servers, The post Scaling Box with Jeff Quiesser appeared first on Software...

Applications need to be ready to scale in response to high-load events. With mobile applications, this can be even more important. People rely on mobile applications such as banking, ride sharing, and GPS. During Black Friday, a popular ecommerce application could be bombarded by user requests–you might not be able to complete a request to The post Load Testing Mobile Applications with Paulo...

Over the last decade, computation and storage have moved from on-premise hardware into the cloud data center. Instead of having large servers “on-premise,” companies started to outsource their server workloads to cloud service providers. At the same time, there has been a proliferation of devices at the “edge.” The most common edge device is your The post Serverless at the Edge with Kenton Varda a...

How do you build resilient, failure tested systems? Redundancy, backups, and testing are all important. But there is also an increasing trend towards chaos engineering–the technique of inducing controlled failures in order to prove that a system is fault tolerant in the way that you expect. In last week’s episode with Kolton Andrus, we discussed The post Linkedin Resilience with Bhaskaran De...

The number of ways that applications can fail is numerous. Disks fail all the time. Servers overheat. Network connections get flaky. You assume that you are prepared for such a scenario because you have replicated your servers. You have the database backed up. Your core application is spread across multiple availability zones. But are you The post Chaos Engineering with Kolton Andrus appeared firs...

On this show, we spend a lot of time talking about CI/CD, data engineering, and microservices. These technologies have only been widely talked about for the last 5-10 years. That means that they are easy to adopt for startups that get founded in the last 5-10 years, but not necessarily for older enterprises. Within a The post How to Change an Enterprise’s Software and Culture with Zhamak Dehghani...

After two weeks of episodes about Kubernetes, our in-depth coverage of container orchestration is drawing to a close. We have a few more shows on the topic before we move on to cover other aspects of the software. If you have feedback on this thematic format (whether you like it or not), send me an The post Serverless Containers with Sean McKenna appeared first on Software Engineering Daily .

In 2011, platform-as-a-service was in its early days. It was around that time that Gabe Monroy started a container platform called Deis, with the goal of making an open-source platform-as-a-service that anyone could deploy to whatever infrastructure they wanted. Over the last six years, Gabe had a front-row seat to the rise of containers, the The post Container Instances with Gabe Monroy appeared...

Oliver Gould worked at Twitter from 2010 to 2014. Twitter’s popularity was taking off, and the engineering team was learning how to scale the product. During that time, Twitter adopted Apache Mesos and began breaking up its monolithic architecture into different services. As more and more services were deployed, engineers at Twitter decided to standardize The post Service Mesh Design with Oliver G...

Modern applications store most of their data on hosted storage solutions. We use hosted block storage to back databases, hosted object storage for objects such as videos, and hosted file storage for file systems. Using a cloud provider for these storage systems can simplify scalability, durability, and availability–it can be less painful than taking care The post Kubernetes Storage with Bass...

A common problem in a distributed system: how do you take a snapshot of the global state of that system? Snapshot is difficult because you need to tell every node in the system to simultaneously record its state. There are several reasons to take a snapshot. You might want to take a picture of the The post Kubernetes State Management with Niraj Tolia appeared first on Software Engineering Daily .

In the last four years, CoreOS has been at the center of enterprise adoption of containers. During that time, Brian Harrington (or “Redbeard”) has seen a lot of deployments. In this episode, Brian discusses the patterns he has seen among successful Kubernetes deployments–and the pitfalls of the less successful. How should you manage configuration? How The post Kubernetes Operations with Bria...

A backend application can have hundreds of services written in different programming frameworks and languages. Across these different languages, log messages are produced in different formats. Some logging is produced in XML, some are produced in JSON, some is in other formats. These logs need to be unified into a common format and centralized for The post FluentD with Eduardo Silva appeared first...

Kubernetes has become the standard way of deploying new distributed applications. Most new internet businesses started in the foreseeable future will leverage Kubernetes (whether they realize it or not). Many old applications are migrating to Kubernetes too. Before Kubernetes, there was no standardization around a specific distributed systems platform. Just like Linux became the standard The post...

Kubernetes has become the standard system for deploying and managing clusters of containers. But the vision of the project goes beyond managing containers. The long-term goal is to democratize the ability to build distributed systems. Brendan Burns is a co-founder of the Kubernetes project. He recently announced an open-source project called Metaparticle, a standard library The post Kubernetes Vis...

You are requesting a car from a ridesharing service such as Lyft. Your request hits the Lyft servers and begins trying to get you a car. It takes your geolocation, and passes the geolocation to a service that finds cars that are nearby, and puts all those cars into a list. The list of nearby The post High Volume Distributed Tracing with Ben Sigelman appeared first on Software Engineering Daily .