Cloud Engineering Archives - Software Engineering Daily

Meetup is an online service that allows people to gather into groups and meet in person. Since 2002, the company has been growing and its technology stack has been changing. Today, they are in the process of migrating to the cloud, using both Amazon Web Services and Google Compute Platform. Yvette Pasqua is the CTO The post Meetup Architecture with Yvette Pasqua appeared first on Software Engineer...

When a useful new technology comes out, companies that are in a position to adopt that new technology can gain an edge over competitors. As our industry grows and moves faster, these kinds of changes are coming faster–some recent examples are Docker, ReactJS, and Kubernetes. Evolutionary architecture supports incremental, guided change as a first principle The post Evolutionary Architecture...

Self-contained systems is an architectural approach that separates the functionality of a system into many independent systems. Each self-contained system is an autonomous web application, and is owned by one team. Communication with other self-contained systems or 3rd party systems is asynchronous where possible. As Eberhard Wolff explains in this episode, self-contained systems is not The post S...

Application performance monitoring helps an engineer understand what is going on with an application. An application on a single machine is often monitored by inserting bytecode instructions into the application after it has been interpreted. Distributed cloud applications with functionality broken up across multiple servers often use distributed tracing. Andi Grabner from Dynatrace joins today’s...

Online marketplaces and social networks often have a trust and safety team. The trust and safety team helps protect the platform from scams, fraud, and malicious actors. To detect these bad actors at scale requires building a system that classifies every transaction on the platform as safe or potentially malicious. Since every social platform has The post Antifraud Architecture with Josh Yudaken a...

For many years, software companies have been breaking up their applications into individual services for the purpose of isolation and maintainability. In the early 2000s, we called this pattern “service-oriented architecture”. Today we call it “microservices”. Why did we change that terminology? Did the services get smaller? Not exactly. Jonas Boner suggests that the movement The post Reactive Mic...

Some tasks are simple, but cannot be performed by a computer. Audio transcription, image recognition, survey completion–these are simple procedures that almost any human could execute, but the machine learning models have not gotten consistent enough to do them accurately. Scale is an API for human labor, created by Lucy Guo and Alexandr Wang. Similar The post Scale API with Lucy Guo and Ale...

Caching is a fundamental concept of computer science. When data is accessed frequently, we put that data in a place where it can be accessed more quickly–we put the data in a cache. When data is accessed less often, we leave it in a place where the access time is slow or expensive. Netflix has The post Netflix Caching with Scott Mansfield appeared first on Software Engineering Daily .

When you are working on a program, a lot of things are going through your head. In some sense, you become part machine when you are programming. Learnable Programming is a concept that facilitates this, by showing developers what the computer is doing in real time, before compiling. In this episode, Josh Varty, co-founder of The post Developer Tools with Josh Varty appeared first on Software Engin...

Microservices are a widely adopted pattern for breaking an application up into pieces that can be well-understood by the individual teams within the company. Microservices also allow these individual pieces to be scaled independently and updated in isolation. Past Software Engineering Daily episodes have covered the microservice architectures of Twitter, Netflix, Google, Uber, and other The post M...

Slack is a chat client that has reached wide adoption. The rise of Slack has coincided with the rise of chatbots. A chatbot is a simple, conversational interface into a computer program that may have simple functionality, like telling you some simple statistics, or more complex functionality, like helping you manage your continuous integration pipeline. The post Slack Bots with Amir Shevat appeare...

Amazon Web Services changed the economics of building an internet application. Instead of having to invest tens of thousands of dollars up front for hardware, developers can pay for services over time as their application scales. As AWS has grown to be a gigantic platform, the documentation about how to use cloud infrastructure has become The post AWS Open Guide with Joshua Levy appeared first on...

The blueprint for a typical startup involves investing heavily in cloud services–either from Amazon, Google, or Microsoft. The high costs can quickly eat away at all of the money that startup has raised. In today’s episode, Avi Freedman outlines some of the infrastructure mistakes that can set back a company severely–cloud jail, hipster tools, and The post Infrastructure Mistakes with...

Chat bots are your newest co-worker. Slack, HipChat, and other chat clients allow developers and other team members to communicate more dynamically than the limits of email. Companies have started to add bots to their chat rooms. These bots can give you technical information, restart a server, or notify you that a build has finished. The post ChatOps with Jason Hand appeared first on Software Engi...

Kafka is a distributed log for producers and consumers to publish messages to each other. We’ve done many shows about Kafka as a key building block for distributed systems, but we often leave out the discussion of the complexities of setting up Kafka and monitoring it. Kafka deployments can be a complex piece of software The post Managed Kafka with Tom Crayford appeared first on Software Engineeri...

Google Compute Engine is the public cloud built by Google. It provides infrastructure- and platform-as-a-service capabilities that rival Amazon Web Services. Today’s guest Joe Beda was there from the beginning of GCE, and he was also one of the early engineers on the Kubernetes project. Google’s internal systems have made it easy for employees to The post Google Cloudbuilding with Joe Beda appeare...

Cloud computing was something much different in 2011, when Brian Gracely and Aaron Delp started The Cloudcast, a podcast I listen to on a regular basis. The Cloudcast features technical discussions about cloud infrastructure technology, and one of the most recent shows was a monologue by Brian Gracely where he explained his perspective on the The post Docker Cloudcasting with Brian Gracely appeare...

When a user of a social network updates her profile, that profile update needs to propagate to several databases that want to know about such an update–search indexes, user databases, caches, and other services. When Neha Narkhede was at LinkedIn, she helped develop Kafka, which was deployed at LinkedIn to help solve this very problem. The post Kafka Event Sourcing with Neha Narkhede appeare...

The intent of the DevOps movement is to get organizations moving faster and more effectively by breaking down siloes, and improving communication. Gene Kim’s book The Phoenix Project illustrated this by telling the fictional story of a company adopting a DevOps mentality. Although that book was fiction, Gene is an experienced engineer, having worked as The post DevOps Handbook with Gene Kim appear...

  At Netflix, developers write applications with a variety of requirements–from simple requests for a list of movies to more resource-intensive requests like a complex machine learning workflow. Netflix wants developers to be able to request the resources they need from a compute cluster and receive those resources on-demand, without thinking too much about the The post Netflix Scheduli...

Building a monitoring system is a complex distributed systems problem. Events are produced from different points in an application and must be aggregated in order to form metrics. These events are often ingested by a time series database, which forms the backbone of our monitoring system. Theo Schlossnagle is the CEO of Circonus, where he The post Monitoring Architecture with Theo Schlossnagle app...

In order to move software updates from the development team to production, companies do a variety of things. Some teams might email files to each other or use FTP or even floppy disks. Most companies today at least use version control systems like Git together with separate servers for development and production. When code is The post Continuous Delivery with David Rice appeared first on Software...

Kafka Streams is a library for building streaming applications that transform input Kafka topics into output Kafka topics. In a time when there are numerous streaming frameworks already out there, why do we need yet another? To quote today’s guest Jay Kreps “the gap we see Kafka Streams filling is less the analytics-focused domain these The post Kafka Streams with Jay Kreps appeared first on Softw...

Platform as a service can mean different things to different people. The most prominent feature of a PaaS is the ability to abstract away issues that every developer within an organization has to deal with. As an example, developers today don’t need to fear scalability and load balancing issues as much as engineers of the The post Platform as a Service with Sinclair Schuller appeared first on Soft...

Google builds cloud services for developers, such as PubSub, Cloud Storage, BigQuery, and Cloud DataStore. On Software Engineering Daily, we’ve done lots of shows about how these types of services are built. In this episode, we are zooming in on the interaction between the developer using a cloud service and the design and engineering of The post Cloud Clients with Jon Skeet appeared first on Soft...