The AI Concepts Podcast

This episode closes out Module 6 by tackling the question that has been getting louder since large context windows arrived. If a model can hold hundreds of thousands or even millions of tokens at once, do we still need all the architecture we just spent this module building? We explore why RAG was never just about fitting text into a small prompt, what retrieval is actually doing that a large cont...

This episode addresses the category of questions that vector search fundamentally cannot answer, questions about relationships between things. We explore what a knowledge graph is and why traversing connections between entities requires a completely different data structure than semantic similarity search. We break down Microsoft's GraphRAG approach, how it extracts entities and relationships from...

This episode addresses a retrieval failure that has nothing to do with your index and everything to do with the query itself. We explore the vocabulary gap between how people ask questions and how documents are written, and why even strong embedding models cannot always bridge it. We break down three techniques that fix the query before the search runs: query rewriting to reformulate casual langua...

This episode addresses the fundamental tension between retrieval precision and generation context. We explore why small chunks produce tight embeddings that retrieve well but leave the model without enough surrounding information, and why large chunks give the model context but dilute the embedding and hurt search quality. We break down parent-child indexing as the solution that decouples these tw...

This episode addresses the gap between finding candidate chunks and finding the right ones. We explore the bi-encoder bottleneck, why compressing text into a single vector for comparison loses critical nuance, and how cross-encoders fix this by reading the query and document together in a single forward pass. We introduce ColBERT as a powerful middle ground between speed and accuracy through token...

This episode addresses one of the most common gaps in RAG pipelines, relying solely on semantic search. We explore how dense retrieval works and where it excels, then introduce sparse retrieval with BM25 and why it catches what vector search misses entirely, particularly exact identifiers like part numbers, codes, and proper nouns. We break down how hybrid search combines both approaches using Rec...

This episode is about chunking, the quiet step in a RAG pipeline that decides whether your system retrieves the right answer or a confidently wrong one. It covers why the chunk is the real unit of retrieval, the tradeoff between context and precision, the main strategies teams use to split documents, and why testing your chunks against real questions matters more than picking the perfect size.

This episode is about the step that every RAG system depends on. Before meaning can be stored or retrieved, your raw documents have to become clean text. What goes wrong here breaks the entire pipeline in ways that are surprisingly hard to catch.

This episode is about the infrastructure underneath every RAG system. It covers the purpose-built engine that stores all that meaning and searches millions of vectors in milliseconds, in a way no traditional database can. This is what makes retrieval fast enough to actually work in production.

This episode is about the layer of RAG that makes semantic search possible. It covers how machines turn language into math that clusters similar ideas together, so a question and its answer can find each other even when they share no words in common. Without this, RAG is just keyword search with extra steps.

This episode maps out the full RAG pipeline end to end using one concrete scenario, a defense contractor building an AI assistant for fighter jet maintenance crews. It walks through both phases of the architecture, offline and online, following a real question all the way from a raw document to a grounded answer. It also covers why the architecture is modular and closes with the four failure modes...

This episode kicks off Module 6 with RAG (Retrieval Augmented Generation), the #1 architecture every serious enterprise actually uses. Discover why regular LLMs hallucinate on your private data and high-stakes queries, and how RAG fixes it by forcing the model to retrieve real documents first.  

This episode covers reasoning models, the shift from manually guiding a model's thinking to letting the model reason through complex problems on its own before responding. It explains the concept of test-time compute, why reasoning models take longer but perform dramatically better on hard tasks, and how they change the way you should prompt. It walks through when to reach for a reasoning model ve...

This episode covers structured output, how you get a model to respond in predictable, machine-readable formats like JSON instead of natural language paragraphs. It walks through three approaches, from simply asking in the prompt, to JSON mode, to schema-based constraints, and explains why each level adds more reliability. It uses real-world examples to show how structured output turns AI from a co...

This episode covers system prompts, the invisible instruction layer that shapes every model interaction before the user says a word. It explains the three-role message format, why the model is trained to treat system instructions as higher authority than user messages, and how persona prompting works by shifting which region of the training distribution the model samples from. It walks through the...

This episode covers chain of thought prompting, how asking a model to show its reasoning makes it measurably better at complex tasks, and why that works at a mechanical level. It walks through manual and zero-shot chain of thought, then three advanced extensions: self-consistency, Tree of Thought, and step-back prompting. It closes with when chain of thought actually helps versus when it just adds...

This episode explores in-context learning, the idea that you can dramatically change how a model behaves just by showing it examples inside the prompt, without changing a single weight. It walks through zero-shot, one-shot, and few-shot prompting, when each one tends to work best, and why examples shape not just the answer but also the format, tone, and structure of the response. It also gets into...

This episode explores the hidden layer between your prompt and the model’s response: decoding and sampling. We look at how the model moves from a field of possible next tokens to the one it actually chooses, why the same prompt can produce different outputs, and how that variation is shaped rather than random. We walk through the core strategies you will hear over and over in prompt engineering, f...

In this episode, I explore a question I heard recently that sounds simple, but matters more than it seems.

This episode tackles the lever that turns powerful LLMs into something you can actually run: quantization. We explore what it means to store model weights with fewer bits, why that can cut memory in half at 8-bit and down to roughly a quarter at 4-bit, and the real tradeoff between compression and capability as rounding error accumulates across billions of parameters. We break down why large model...

This episode addresses the real bottleneck after you build an LLM: fitting it into hardware that can actually run it. We explore why GPU memory is the scarce resource, how weights, KV cache, and activations compete for that space, and what that means in practice when prompts get long or concurrency spikes. We compare data center GPUs (high bandwidth HBM) versus local machines like the Mac Studio (...

This episode addresses how Reinforcement Learning from Human Feedback (RLHF) adds the final layer of alignment after supervised fine-tuning, shifting the training signal from “right vs wrong” to “better vs worse.” We explore how preference rankings create a reward signal (reward models plus PPO) and the newer shortcut (DPO) that learns preferences directly, then connect RLHF to safety through the...

This episode addresses how we turn a raw base model into something that behaves like a real assistant using Supervised Fine-Tuning (SFT). We explore instruction and response training data, why SFT makes behaviors consistent beyond prompting, and the practical engineering choices that keep fine-tuning efficient and safe, including low learning rates and LoRA-style adapters. By the end, you will und...

This episode addresses the physical and mathematical limits of a model’s "short-term memory." We explore the context window and the engineering trade-offs required to process long documents. You will learn about the quadratic cost of attention where doubling the input length quadruples the computational work and why this creates a massive bottleneck for long-form reasoning. We also introduce the a...

This episode explores the foundational stage of creating an LLM known as the pre-training phase. We break down the Trillion Token Diet by explaining how models move from random weights to sophisticated world models through the simple objective of next token prediction. You will learn about the Chinchilla Scaling Laws or the mathematical relationship between model size and data volume. We also disc...