Cloud Engineering Archives - Software Engineering Daily

When you are working on a program, a lot of things are going through your head. In some sense, you become part machine when you are programming. Learnable Programming is a concept that facilitates this, by showing developers what the computer is doing in real time, before compiling. In this episode, Josh Varty, co-founder of The post Developer Tools with Josh Varty appeared first on Software Engin...

Microservices are a widely adopted pattern for breaking an application up into pieces that can be well-understood by the individual teams within the company. Microservices also allow these individual pieces to be scaled independently and updated in isolation. Past Software Engineering Daily episodes have covered the microservice architectures of Twitter, Netflix, Google, Uber, and other The post M...

Slack is a chat client that has reached wide adoption. The rise of Slack has coincided with the rise of chatbots. A chatbot is a simple, conversational interface into a computer program that may have simple functionality, like telling you some simple statistics, or more complex functionality, like helping you manage your continuous integration pipeline. The post Slack Bots with Amir Shevat appeare...

Amazon Web Services changed the economics of building an internet application. Instead of having to invest tens of thousands of dollars up front for hardware, developers can pay for services over time as their application scales. As AWS has grown to be a gigantic platform, the documentation about how to use cloud infrastructure has become The post AWS Open Guide with Joshua Levy appeared first on...

The blueprint for a typical startup involves investing heavily in cloud services–either from Amazon, Google, or Microsoft. The high costs can quickly eat away at all of the money that startup has raised. In today’s episode, Avi Freedman outlines some of the infrastructure mistakes that can set back a company severely–cloud jail, hipster tools, and The post Infrastructure Mistakes with...

Chat bots are your newest co-worker. Slack, HipChat, and other chat clients allow developers and other team members to communicate more dynamically than the limits of email. Companies have started to add bots to their chat rooms. These bots can give you technical information, restart a server, or notify you that a build has finished. The post ChatOps with Jason Hand appeared first on Software Engi...

Kafka is a distributed log for producers and consumers to publish messages to each other. We’ve done many shows about Kafka as a key building block for distributed systems, but we often leave out the discussion of the complexities of setting up Kafka and monitoring it. Kafka deployments can be a complex piece of software The post Managed Kafka with Tom Crayford appeared first on Software Engineeri...

Google Compute Engine is the public cloud built by Google. It provides infrastructure- and platform-as-a-service capabilities that rival Amazon Web Services. Today’s guest Joe Beda was there from the beginning of GCE, and he was also one of the early engineers on the Kubernetes project. Google’s internal systems have made it easy for employees to The post Google Cloudbuilding with Joe Beda appeare...

Cloud computing was something much different in 2011, when Brian Gracely and Aaron Delp started The Cloudcast, a podcast I listen to on a regular basis. The Cloudcast features technical discussions about cloud infrastructure technology, and one of the most recent shows was a monologue by Brian Gracely where he explained his perspective on the The post Docker Cloudcasting with Brian Gracely appeare...

When a user of a social network updates her profile, that profile update needs to propagate to several databases that want to know about such an update–search indexes, user databases, caches, and other services. When Neha Narkhede was at LinkedIn, she helped develop Kafka, which was deployed at LinkedIn to help solve this very problem. The post Kafka Event Sourcing with Neha Narkhede appeare...

The intent of the DevOps movement is to get organizations moving faster and more effectively by breaking down siloes, and improving communication. Gene Kim’s book The Phoenix Project illustrated this by telling the fictional story of a company adopting a DevOps mentality. Although that book was fiction, Gene is an experienced engineer, having worked as The post DevOps Handbook with Gene Kim appear...

  At Netflix, developers write applications with a variety of requirements–from simple requests for a list of movies to more resource-intensive requests like a complex machine learning workflow. Netflix wants developers to be able to request the resources they need from a compute cluster and receive those resources on-demand, without thinking too much about the The post Netflix Scheduli...

Building a monitoring system is a complex distributed systems problem. Events are produced from different points in an application and must be aggregated in order to form metrics. These events are often ingested by a time series database, which forms the backbone of our monitoring system. Theo Schlossnagle is the CEO of Circonus, where he The post Monitoring Architecture with Theo Schlossnagle app...

In order to move software updates from the development team to production, companies do a variety of things. Some teams might email files to each other or use FTP or even floppy disks. Most companies today at least use version control systems like Git together with separate servers for development and production. When code is The post Continuous Delivery with David Rice appeared first on Software...

Kafka Streams is a library for building streaming applications that transform input Kafka topics into output Kafka topics. In a time when there are numerous streaming frameworks already out there, why do we need yet another? To quote today’s guest Jay Kreps “the gap we see Kafka Streams filling is less the analytics-focused domain these The post Kafka Streams with Jay Kreps appeared first on Softw...

Platform as a service can mean different things to different people. The most prominent feature of a PaaS is the ability to abstract away issues that every developer within an organization has to deal with. As an example, developers today don’t need to fear scalability and load balancing issues as much as engineers of the The post Platform as a Service with Sinclair Schuller appeared first on Soft...

Google builds cloud services for developers, such as PubSub, Cloud Storage, BigQuery, and Cloud DataStore. On Software Engineering Daily, we’ve done lots of shows about how these types of services are built. In this episode, we are zooming in on the interaction between the developer using a cloud service and the design and engineering of The post Cloud Clients with Jon Skeet appeared first on Soft...

Batch and stream processing systems have been evolving for the past decade. From MapReduce to Apache Storm to Dataflow, the best practices for large volume data processing have become more sophisticated as the industry and open source communities have iterated on them.   Dataflow and Apache Beam are projects that present a unified batch and The post Cloud Dataflow with Eric Anderson appeared...

Slack is a chat application that is rapidly growing in popularity. The focus of Slack is to create a polished, responsive tool for productivity that cuts down on the emailing, context switching, and useless meetings that take place at a typical enterprise.   Keith Adams, the chief architect at Slack, joins the show to explain The post Slack’s Architecture with Keith Adams appeared first...

Uber has a software architecture with unique requirements. Uber does not have the firehose of user engagement data that Twitter or Facebook has, but each transaction on Uber is both high value and time-sensitive. Users are paying for transportation that they expect to be available and reasonably close by. When Uber’s system is trying to The post Uber’s Ringpop with Jeff Wolski appeared first...

Kubernetes is a cluster management tool open sourced by Google. On Software Engineering Daily, we’ve done numerous shows on how Kubernetes works in theory. Today’s episode is a case study in how to deploy Kubernetes to production at a company with existing infrastructure.   GolfNow is a fifteen year-old application written in C# .NET. It The post Kubernetes Migration with Sheriff Mohamed appe...

In a microservices architecture, a user request will often make its way through several different services before it returns a result to the end user. If a user experiences a failed request, the root cause could be in any of the services along that request path. Even more problematic is the challenge of debugging latency The post Distributed Tracing with Reshmi Krishna appeared first on Software E...

“Serverless” usually refers to an architectural pattern where the server side logic is run in stateless compute containers that are event-triggered and ephemeral. Mike Roberts has written a series of articles about serverless computing, in which he discusses theories and patterns around serverless architecture. In this episode, Mike and I discuss how to reimagine our The post Serverless Architectu...

Unbounded data streams create difficult challenges for our application architectures. The data never stops coming, and we are forced to assume that we will never know if or when we have seen all of our data. Some streaming systems give us the tools to deal partially with unbounded data streams, but we have to complement The post Apache Beam with Frances Perry appeared first on Software Engineering...

Prometheus is a tool for monitoring our distributed applications. It allows us to focus on the services we are deploying rather than the individual machines that make up instances of that service.   The monitoring service itself is a portion of a distributed system that is treated differently than the services we are monitoring. We The post Prometheus Monitoring with Brian Brazil appeared fir...