DevOps Daily with Fexingo: CI/CD, Kubernetes, and Modern Software Operations

Lucas and Luna dig into the Kubernetes Vertical Pod Autoscaler's recalculations that often leave memory over-provisioned and CPU under-provisioned. They examine a case study where a production e-commerce cluster saw 22% of VPA-recommended memory requests exceed actual usage by over 40%, while CPU recommendations lagged behind real demand by nearly 30%. The episode explains the recommender's slidin...

Kubernetes CPU Manager static policy seems like a performance win, pinning pods to specific CPU cores. But when you need to drain a node for maintenance, those pinned pods refuse to move—or worse, they crash on restart. In this episode, Lucas and Luna dissect the tension between CPU pinning and node lifecycle. They walk through a real scenario where a 32-core production node stalled a rolling upda...

Lucas and Luna dig into a surprising side effect of Kubernetes NetworkPolicies: latency jitter. When you apply fine-grained network policies, every packet traverses iptables rules that can add unpredictable delay, especially under load. They walk through a real incident at a mid-size fintech where packet drops from conntrack table exhaustion caused 200ms tail-latency spikes. They explain the mecha...

Lucas and Luna dive into a specific production issue that has plagued Kubernetes operators for years: memory leaks in ingress controllers under sustained traffic. They trace the problem to connection tracking and endpoint slice churn, citing a real incident where NGINX Ingress consumed 4GB of RSS in under 48 hours. The episode walks through diagnosis via pprof, root cause in the upstream connectio...

Kubernetes cluster administrators often find that their nodes can't fit as many pods as expected — even when resource requests seem reasonable. In Episode 75 of DevOps Daily, Lucas and Luna dive into the hidden cost of Pod Overhead, a feature introduced in Kubernetes 1.24 to account for sandbox and sidecar resource consumption. They walk through how Pod Overhead is calculated in practice, why it f...

Lucas and Luna dive into a subtle but painful Kubernetes performance issue: the single-threaded scheduler loop. When you have hundreds of nodes and thousands of pods, the scheduler's internal algorithm — which scores nodes sequentially — can bottleneck under load, causing allocation delays of 30 seconds or more. They walk through a real example from a mid-stage startup that saw pod startup times s...

Kubernetes Pod Security Standards (PSS) replaced PodSecurityPolicies in 1.25, but migrating legacy workloads to restricted mode often breaks them silently. In this episode, Lucas and Luna dig into why PSS admission checks fail for statefulsets running on GKE, how the 'privileged' profile leaks capabilities via container runtime defaults, and what the baseline profile actually blocks. They walk thr...

In this episode, Lucas and Luna dive into a subtle but devastating failure mode in Istio-based Kubernetes service meshes: TCP connection timeouts caused by sidecar proxy resource limits. They walk through a real-world incident at a mid-size fintech where a spike in traffic led to Envoy sidecars running out of memory, dropping packets, and triggering cascading timeouts across microservices. They ex...

In this episode of DevOps Daily with Fexingo, Lucas and Luna dive into a common but often overlooked problem in Istio service meshes: sidecar Envoy proxies that cache DNS and configuration data until they exhaust pod memory and trigger OOM kills. They break down why the default 2 GiB memory limit for Envoy in Istio 1.20 is not a safety net but a ticking time bomb, especially when sidecars cache fu...

Episode 70 of DevOps Daily with Fexingo dives into Kubernetes 1.27's in-place pod resize feature and why it's causing unexpected scheduling headaches. Lucas and Luna unpack a real case from a mid-stage fintech where CPU and memory adjustments triggered pod preemption, resource fragmentation, and silent failures in the scheduler's bin-packing logic. They explain the resize policy, what happens when...

Episode 69 of DevOps Daily with Fexingo: Lucas and Luna break down why Kubernetes admission webhooks can silently block deployments and cause production rollbacks. They walk through a real-world case where a mutating webhook injected a sidecar that broke pod startup, and a validating webhook rejected a critical ConfigMap update due to a missing label. You'll learn how MutatingAdmissionWebhook and...

Lucas and Luna dive into a subtle but destructive Kubernetes scheduling failure: priority inversion. When a low-priority critical pod holds a lock that a high-priority pod needs, the scheduler can deadlock the cluster. They break down real scenarios where priority inversion causes mysterious pod evictions, latency spikes, and even node resource starvation. Using concrete examples from production c...

In this episode of DevOps Daily with Fexingo, Lucas and Luna dive into one of Kubernetes' most overlooked failure modes: finalizers that block resource deletion indefinitely. They walk through a real incident at a mid-sized e-commerce company where leftover finalizers on PersistentVolumeClaims caused a storage quota blowout over three months, racking up $12,000 in orphaned cloud disk costs. Lucas...

Kubernetes ServiceAccount tokens are often treated as invisible plumbing, but misconfigured automount and long-lived secrets create real security holes. In this episode, Lucas and Luna walk through a common leak pattern: a CI/CD pipeline that exposes a default token to a compromised sidecar. They explain how bound service account tokens (projected volumes) close the gap, why tools like kube-bench...

Lucas and Luna drill into a specific Kubernetes scaling failure: the 'thundering herd' problem caused by the Horizontal Pod Autoscaler. They walk through a real scenario where a retail app's HPA misconfiguration caused 120 pod replicas to start simultaneously, overwhelming the API server and database connection pool. Lucas explains the mechanism behind HPA's default behavior — how it calculates ta...

Lucas and Luna break down why Kubernetes Pod Security Contexts — the container-level permissions layer — still allow privilege leaks in production. They walk through a real incident at a mid-size fintech where a sidecar container with an innocent-looking securityContext gained host-level access via CAP_SYS_ADMIN and a misconfigured seccomp profile. The episode covers the specific fields that cause...

Kubernetes Pod graceful shutdown is supposed to let applications drain traffic and close connections before being terminated. But in practice, many teams find that clients still see connection errors, timeouts, or incomplete requests right when a Pod shuts down. In this episode, Lucas and Luna dig into the gap between the Kubernetes graceful shutdown spec and real-world behavior. They walk through...

Lucas and Luna dive into a lesser-known Kubernetes storage pitfall: persistent volume claim (PVC) snapshot drift. When teams rely on CSI volume snapshots for backup or cloning, they often assume point-in-time consistency. But in practice, snapshot creation is asynchronous and application-aware snapshots require hooks that most teams skip. The hosts examine a real incident at a mid-sized fintech wh...

Lucas and Luna dive into a persistent blind spot in Kubernetes security: audit logging that sounds comprehensive but routinely misses critical events. They walk through a real incident at a mid-size fintech where a malicious pod went undetected for 11 days because the cluster had default audit-log settings — no metadata level set, no dynamic webhook hooked up, and logs streaming to a stdout sideca...

Episode 60 of DevOps Daily with Fexingo: Lucas and Luna dig into a common but poorly understood Kubernetes failure mode — Pod Disruption Budgets (PDBs) that actually accelerate downtime during spot instance preemptions. Using AWS Spot Node draining as the concrete case (similar dynamics apply on GKE Preemptible and Azure Spot), they trace why PDBs designed to protect availability can paradoxically...

Lucas and Luna dig into a common but overlooked Kubernetes pain point: how ResourceQuotas designed to prevent resource abuse can accidentally deadlock your batch processing pipeline. They walk through a real incident at a mid-size fintech that ran 24,000 short-lived Pods per day, where namespace-level quotas caused cascading failures during a routine data-backfill job. Lucas explains the mechanics...

Lucas and Luna break down how container image layers silently consume node disk space in Kubernetes clusters. Using real numbers from a 500-node production cluster, Lucas explains the hidden cost of unused base images, intermediate layers, and stale tags that build up over time. Luna shares a horror story of a team that had to drain an entire node because garbage collection ran too late. The episo...

Lucas and Luna unpack a specific Kubernetes failure pattern: cluster DNS caching. When one node experiences a DNS resolution error, cached negative responses can propagate across the cluster, causing cascading service discovery failures. They walk through how the NodeLocal DNSCache plugin can actually make this worse under certain conditions, and dig into a real incident at a mid-size fintech wher...

In this episode of DevOps Daily, Lucas and Luna dive into the specific mechanics and benefits of Kubernetes Node Local DNS cache (NodeLocal DNSCache). They explain how it reduces DNS latency by up to 30 percent, prevents conntrack table exhaustion, and alleviates pressure on CoreDNS in large clusters. The discussion covers real-world deployment configurations, including the use of iptables rules t...

Episode 55 of DevOps Daily dives into Kubernetes Quality of Service (QOS) classes — Guaranteed, Burstable, and BestEffort — and how they determine which pods get killed when memory runs out. Lucas explains why a Burstable pod with a 1GB limit but a 500MB request is more likely to be OOM-killed than a BestEffort pod using 100MB, using a real-world case from a fintech startup that lost payment-proce...