DevOps Daily with Fexingo: CI/CD, Kubernetes, and Modern Software Operations

Lucas and Luna dissect a specific CI/CD failure pattern: Docker image layer caching that actually degrades pipeline performance. Using a real-world example of a mid-stage startup whose builds went from 4 minutes to 22 minutes after adopting a naive cache strategy, they explain layer invalidation, cache-miss cascades, and the trade-off between cache reuse and build determinism. Lucas walks through...

In this episode of DevOps Daily, Lucas and Luna dive into a subtle but dangerous Kubernetes configuration trap: pod overhead misassignment. When cluster operators set pod overhead values incorrectly—especially in clusters running multiple sandboxed runtimes like Kata Containers or gVisor—the kubelet can overcommit node resources, leading to unexplained OOM kills and node pressure. Lucas walks thro...

Episode 102 of DevOps Daily dives into a lesser-known risk with Kubernetes Vertical Pod Autoscaler (VPA): its ability to cause cascading instability in production workloads. Lucas and Luna walk through a real-world case where VPA's recommender triggered simultaneous resource resizing across multiple microservices, leading to pod evictions, request timeouts, and a multi-hour incident at a mid-sized...

In this episode of DevOps Daily, Lucas and Luna dig into a subtle but impactful Kubernetes networking issue: how kube-proxy's default iptables mode can cause periodic latency spikes under moderate traffic. They walk through the mechanics of iptables rule evaluation, how conntrack table flushes trigger cascading latency, and why large clusters with frequent Service updates are especially vulnerable...

In this episode of DevOps Daily, Lucas and Luna mark episode 100 by tackling a problem that haunts every multi-team cluster: cost allocation. Lucas explains how Kubernetes's shared infrastructure model makes it nearly impossible to attribute spending accurately to teams, namespaces, or workloads without a purpose-built metrics layer. He breaks down the three common pitfalls — relying on namespace-...

In DevOps Daily Episode 99, Lucas and Luna dive into a dangerous and often overlooked Kubernetes behavior: cascading deletes of namespaces and the resource storms they trigger. Using a real incident at a mid-size SaaS company that accidentally deleted a production namespace, they explain how `kubectl delete namespace` doesn't just remove pods—it tears down every resource in that namespace in paral...

Lucas and Luna dive into a nasty Kubernetes failure pattern that catches teams running GPU workloads: how the Cluster Autoscaler can't handle GPU node pools when pods request partial GPU resources. They break down a real incident at a mid-size AI startup where 12 nodes drained in 8 minutes, trace the root cause to the autoscaler's inability to aggregate fractional GPU requests across nodes, and wa...

Lucas and Luna dive into a surprisingly common Kubernetes failure: MutatingAdmissionWebhooks that silently time out under load, bringing cluster-wide latency to a crawl. They walk through a real incident at a mid-size fintech company where a single webhook — a simple config-injection sidecar — started timing out at around 60 requests per second, causing every API call to pile up. They explain why...

Episode 96 of DevOps Daily with Fexingo dives into a subtle but dangerous Kubernetes failure pattern: taint-based evictions silently blocking node drains. Lucas and Luna break down a real incident where a cluster-wide taint added by an automation tool prevented nodes from draining during a planned maintenance window, causing cascading pod failures and a near-P0. They explain how taint propagation...

In this episode of DevOps Daily with Fexingo, Lucas and Luna dive into a specific gotcha in the Kubernetes PodSecurityPolicy (PSP) migration to Pod Security Admission (PSA). They walk through a real scenario where enabling PSA in audit mode silently suppresses warnings for privileged pods that were previously blocked by PSP. Lucas explains the mechanics: PSP evaluates at admission and blocks; PSA...

Episode 94 of DevOps Daily with Fexingo dives into a subtle but dangerous Kubernetes issue: projected ServiceAccount token expiration causing CronJobs to fail silently. Lucas and Luna walk through a real case where a team's nightly batch processes started failing at 3 AM with no clear error, traced back to tokens expiring mid-job after a cluster upgrade. They explain how bound service account toke...

Episode 93 of DevOps Daily drills into a subtle Kubernetes scheduling failure: pod priority inversion. Lucas and Luna walk through a real-world scenario where a high-priority DaemonSet preempts critical batch jobs, leading to node saturation and cascading evictions. They explain why the default priority class hierarchy is misleading, how the preemption algorithm selects victims, and why monitoring...

Episode 92 of DevOps Daily dives into a dangerous blind spot in Kubernetes cluster operations: the default Cluster Autoscaler's total disregard for spot instance interruption notices. Lucas and Luna break down how AWS, Azure, and GCP issue two-minute warnings before reclaiming spot capacity, yet the Cluster Autoscaler treats those nodes as healthy until they're actually terminated. The result? Pod...

Kubernetes cluster autoscaler is a workhorse for dynamic workloads, but when you're running spot (preemptible) instances, the node autoscaler's default behavior can actually hurt reliability. In this episode, Lucas and Luna break down a real incident from a large ad-tech platform running 2,000+ nodes on AWS EC2 Spot Instances. When spot interruption rates spiked during a regional AZ outage, the cl...

In this episode of DevOps Daily with Fexingo, Lucas and Luna dive into a subtle but dangerous Kubernetes behavior: ConfigMap updates to ingress controllers can trigger memory pressure across worker nodes. They walk through a real incident where a team's single ConfigMap change caused a cascade of OOM kills across 12 nodes in a staging cluster. You'll learn why ingress controllers like NGINX reload...

In this episode of DevOps Daily, Lucas and Luna dive into a hidden performance killer in Kubernetes service meshes: the mutual TLS handshake that happens every time a pod starts up. They explain how Istio's mTLS setup creates a cascade of blocking calls that can delay pod readiness by 10 to 30 seconds in large clusters, using a real-world case from a fintech company running 2,000 microservices. Lu...

Kubernetes 1.24 introduced the ability to expand PersistentVolumeClaims for StatefulSets, but the feature has a dangerous edge case. When a StatefulSet’s storage template is updated, the controller can trigger concurrent resize operations across multiple replicas, overwhelming the underlying storage backend and causing node disk failures. In this episode, Lucas and Luna walk through a real inciden...

Episode 87 of DevOps Daily with Fexingo dives into a subtle Kubernetes scheduling pitfall: topology spread constraints that unintentionally create node hotspots. Lucas and Luna walk through a real scenario where evenly-distributed pod replicas on paper led to imbalanced cluster utilization in practice. They explain how the `maxSkew` parameter interacts with node labels, and why a 1-milli-core reso...

Episode 86 of DevOps Daily with Fexingo dives into a subtle but painful Kubernetes trap: Custom Resource Definition (CRD) versioning. When you update a CRD's schema to add a new field or change a validation rule, existing Custom Resources stored in etcd may not automatically convert to the new version—and controllers that watch the old version can crash or miss updates. We walk through a real inci...

In this episode of DevOps Daily with Fexingo, Lucas and Luna dive into a subtle but costly Kubernetes performance issue: how verbose audit logging can silently degrade etcd performance, leading to increased latency, timeouts, and even cluster instability. They walk through a real-world case from a mid-sized fintech that saw API response times jump from 5ms to over 200ms after enabling audit logs w...

Episode 84 of DevOps Daily with Fexingo. Lucas and Luna investigate a subtle but dangerous bug in Kubernetes volume snapshot workflows. They break down how the CSI snapshot controller, combined with certain storage backends, can lead to silent data corruption when snapshots are taken during active I/O. Lucas walks through the sequence of events: the VolumeSnapshot CRD, the VolumeSnapshotContent ob...

Episode 83 of DevOps Daily with Fexingo dives into a subtle but costly Kubernetes issue: Custom Resource Definitions (CRDs) and the controllers built to manage them can silently leak memory over weeks of operation. Lucas and Luna walk through a real incident at a mid-sized fintech company where in-memory caches of CRD objects grew unbounded, leading to OOM-killed controllers and cascading failures...

In this episode of DevOps Daily, Lucas and Luna dive into a subtle Kubernetes scheduling trap: how PodDisruptionBudgets, meant to ensure high availability during voluntary disruptions, can inadvertently cause rollout stalls in production. Using a real-world example from an e-commerce platform handling Black Friday traffic, they explain the interplay between PDBs, replica counts, and node drains. L...

In Episode 81 of DevOps Daily with Fexingo, Lucas and Luna dive into a specific blind spot in Kubernetes observability: the Node Problem Detector (NPD). They explain how NPD's default configuration relies on static patterns and often misses subtle hardware and kernel-level issues like memory corruption, NIC firmware bugs, and ACPI errors that can silently degrade node health. Lucas walks through a...

In this episode of DevOps Daily with Fexingo, Lucas and Luna dive into a subtle but destructive Kubernetes behavior: how ResourceQuotas can silently evict pods when namespace limits are reached, even when the cluster has ample capacity. They walk through a real incident at a mid-sized e-commerce company where a single namespace's quota misconfiguration caused cascading evictions across 12 microser...