AI Post Transformers

This episode explores the 1987 paper on synchronous data flow and how it turns stream-processing programs into analyzable graphs with fixed token production and consumption rates. It explains how those fixed rates let a compiler precompute a repeating execution schedule, prove steady-state consistency through balance equations, and allocate bounded buffers ahead of time instead of relying on expen...

This episode explores a 2017 EPFL paper on turning a soft-input soft-output decoder kernel into a reusable RFNoC FPGA block using Vivado HLS, with a focus on software-defined radio and forward error correction. It explains why SISO processing sits at the core of turbo decoding, walking through the BCJR/MAP decoding logic, trellis-based forward and backward recursions, and the hardware challenges c...

This episode explores how fpgaConvNet turns CNN inference into a Synchronous Dataflow problem so embedded FPGA accelerators can be designed with analyzable schedules, buffers, and resource tradeoffs instead of ad hoc hardware tuning. It explains why CNN deployment on robots, drones, and cars is constrained as much by data movement, latency, and power as by raw arithmetic, and why FPGAs can outperf...

This episode explores how the OOMB training system tries to break the memory bottleneck that makes million-token language model training impractical, focusing on why training long contexts is much harder than simply extending inference-time context windows. It explains the paper’s core ideas in plain language, including chunk-recurrent training that recomputes activations during backpropagation, O...

This episode explores PackKV, a method for shrinking the transformer KV cache during long-context inference by combining low-bit quantization with GPU-friendly repacking and lossy compression. It explains why KV cache growth can dominate memory use in large models, using examples where cache size exceeds model weights, and frames the problem as a systems bottleneck driven more by memory traffic th...

This episode explores a paper arguing that language models can reason more effectively at test time if they are trained to use divide-and-conquer strategies instead of defaulting to a single linear chain of thought. It explains the core distinction between ordinary step-by-step reasoning and structured decomposition into subproblems, then situates that idea alongside prior work such as Tree of Tho...

This episode explores a 2025 survey of reinforcement learning as a statement about how the field now organizes itself, covering value-based, policy-based, model-based, multi-agent, offline, and LLM-related RL. It explains core concepts like Markov decision processes, policies, value functions, delayed credit assignment, and the contrast between direct policy optimization and methods that estimate...

This episode explores whether deep sequence models store knowledge as simple associative lookups or as geometric memories that encode broader relational structure. It discusses a recent paper arguing that, after memorizing graph facts in their weights, sequence models can answer multi-hop path queries as if they were making a much shorter move through embedding space, with learned representations...

This episode explores selective classification in deep neural networks: adding a post-hoc reject option so a trained model can abstain when its confidence falls below a calibrated threshold. It explains the key concepts of coverage, selective risk, and the risk-coverage tradeoff, arguing that a model should be judged not just by how often it is right, but by how often it chooses to answer. The dis...

This episode explores the idea of “self-improving pretraining,” where already post-trained models are used to shape the pretraining of new models rather than waiting to add safety, reasoning, and factuality later. It explains how the approach rewrites training continuations, uses stronger models as judges, and compares original corpus text, teacher-generated suffixes, and learner rollouts to push...

This episode explores how node2vec adapts the word2vec idea to graphs by learning node embeddings from random walks instead of hand-engineered network features. It explains the core technical move in detail: second-order walks controlled by the `p` and `q` parameters, which bias the sampling process toward more local, BFS-like neighborhoods or more exploratory, DFS-like paths. The discussion highl...

This episode explores how DeepWalk helped launch modern graph representation learning by turning random walks over a social network into “sentences” and then applying the Skip-Gram ideas behind word2vec to learn dense node embeddings. It explains why that mattered in 2014: instead of relying on heavy spectral or matrix-factorization methods, DeepWalk offered an online, scalable way to learn reusab...

This episode explores how transformers split prediction between knowledge stored in their weights and information inferred from the current prompt, using the paper’s synthetic “bigram world” to make those mechanisms visible. It explains the distinction between global statistical knowledge and true in-context knowledge, then walks through induction heads as a concrete circuit for recalling earlier...

This episode explores whether learned discrete representations actually improve reinforcement learning, world modeling, and continual adaptation compared with standard continuous latent spaces. It explains how vector-quantized codebook latents, sparse binary-style features, and older ideas like tile coding relate to modern world models, and why the real advantage may come from reduced interference...

This episode explores a 2026 paper on recursive multi-agent systems that asks whether AI collaboration can scale better by replacing text-based agent communication with shared latent-state updates. It explains the difference between agent topology, assigned roles, and communication channels, and argues that natural language may be a costly bottleneck compared with the richer internal representatio...

This episode explores a review of mechanistic interpretability for transformer language models, focusing on how researchers study internal features, circuits, and claims of universality across models. It explains the core toolkit behind the field, including linear probes, hidden-state analysis, intervention methods, vocabulary projection, and sparse autoencoders, while grounding those ideas in tra...

This episode explores DeltaKV, a method for reducing the huge GPU memory burden of KV caches in long-context language model inference without simply discarding old tokens. It contrasts three strategies for handling long contexts: token eviction, dynamic sparse attention, and true compression, arguing that the cache contains structured redundancy that can be exploited rather than treated as disposa...

This episode explores CacheFlow, a systems approach to speeding up long-context LLM serving by restoring transformer KV caches more intelligently. It explains the tradeoffs between recomputing prior attention state, loading it from storage, or combining both, and argues that the real user-facing bottleneck is now time-to-first-token rather than raw generation speed. The discussion focuses on Cache...

This episode explores a speech recognition paper that replaces slow left-to-right transcript generation with a faster draft-and-edit approach, where a speech model produces an initial hypothesis and a bidirectional LLM corrects it in parallel. It explains the tradeoff between CTC-based systems, which are fast but weaker at using linguistic context, and autoregressive decoders, which are more expre...

This episode explores whether large language models can genuinely recognize the limits of their own knowledge or whether they have simply learned to sound uncertain in socially acceptable ways. It examines the paper’s idea of “self-knowledge” through the lens of confidence calibration, including the dangerous case where a model does not know an answer but responds with unwarranted confidence. The...

This episode explores how the 2021 paper “Hopfield Networks Is All You Need” reframes transformer attention as a modern continuous Hopfield network, connecting attention, associative memory, and similarity-based retrieval under one mathematical lens. It explains the core idea of content-addressable memory, contrasts classical binary Hopfield networks with newer differentiable versions, and shows w...

This episode explores the 2014 seq2seq paper by Sutskever and colleagues as a turning point in machine translation, asking what problem it actually solved and why it mattered at the time. It explains how deep LSTM encoder-decoder models reframed translation as end-to-end learning of \(p(\text{output}|\text{input})\), replacing hand-built phrase tables, alignment models, and decoding heuristics wit...

This episode explores the 2014–2015 breakthrough paper that introduced attention to neural machine translation, framing it as a solution to a specific flaw in early encoder-decoder models: forcing an entire source sentence into one fixed-length vector. It explains how pre-attention RNN-based seq2seq systems struggled on long or complex sentences, and how Bahdanau et al.’s “soft alignment” let the...

This episode explores the 2017 ConvS2S paper from Facebook AI Research, which argued that sequence-to-sequence models for machine translation did not need recurrence and could instead use fully convolutional encoder-decoder networks with attention. It explains the seq2seq and neural machine translation setup, clarifies that the paper replaces recurrent computation rather than attention, and breaks...

This episode explores whether large language models can estimate their chances of success before acting, update those estimates during a task, and use them to decide when to abstain from costly work. It explains why that matters for agentic systems in coding and software environments, where overconfidence can lead to wasted effort, unsafe actions, or expensive mistakes, and connects the paper to e...