The AI Concepts Podcast

Shay explains where a transformer actually stores knowledge: not in attention, but in the MLP (feed-forward) layer. The episode frames the transformer block as a two-step loop: attention moves information between tokens, then the MLP transforms each token’s representation independently to inject learned knowledge.

Shay breaks down the encoder vs decoder split in transformers: encoders (BERT) read the full text with bidirectional attention to understand meaning, while decoders (GPT) generate text one token at a time using causal attention. She ties the architecture to training (masked-word prediction vs next-token prediction), explains why decoder-only models dominate today (they can both interpret prompts a...

Shay explains multi-head attention and positional encodings: how transformers run multiple parallel attention 'heads' that specialize, why we concatenate their outputs, and how positional encodings reintroduce word order into parallel processing. The episode uses clear analogies (lawyer, engineer, accountant), highlights GPU efficiency, and previews the next episode on encoder vs decoder architect...

In this episode, Shay walks through the transformer's attention mechanism in plain terms: how token embeddings are projected into queries, keys, and values; how dot products measure similarity; why scaling and softmax produce stable weights; and how weighted sums create context-enriched token vectors. The episode previews multi-head attention (multiple perspectives in parallel) and ends with a sho...

Shay breaks down the 2017 paper "Attention Is All You Need" and introduces the transformer: a non-recurrent architecture that uses self-attention to process entire sequences in parallel. The episode explains positional encoding, how self-attention creates context-aware token representations, the three key advantages over RNNs (parallelization, global receptive field, and precise signal mixing), th...

Shay breaks down why recurrent neural networks (RNNs) struggled with long-range dependencies in language: fixed-size hidden states and the vanishing gradient caused models to forget early context in long texts. He explains how LSTMs added gates (forget, input, output) to manage memory and improve short-term performance but remained serial, creating a training and scaling bottleneck that prevented...

This episode dives into the hidden layer where language stops being words and becomes numbers. We explore what tokens actually are, how tokenization breaks text into meaningful fragments, and why this design choice quietly shapes a model’s strengths, limits, and quirks. Once you understand tokens, you start seeing why language models sometimes feel brilliant and sometimes strangely blind.

This episode explores the hidden engine behind how language models move from knowing to creating. It reveals why generation happens step by step, why speed has hard limits, and why training and usage behave so differently. Once you see this mechanism, the way models write, reason, and sometimes stall will make immediate sense.

This episode is about the hidden space where generative models organize meaning. We move from raw data into a compressed representation that captures concepts rather than pixels or tokens, and we explore how models learn to navigate that space to create realistic outputs. Understanding this idea explains both the power of generative AI and why it sometimes fails in surprising ways.

Welcome to Episode One of The Generative Shift. This episode introduces the core change behind modern AI, the move from discriminative models that draw decision boundaries to generative models that learn the full structure of data. Instead of predicting labels using conditional probability, generative systems model the joint distribution itself, which allows them to create rather than classify. Th...

Hello everyone, and welcome to The Generative AI Series. I’m Shay, and this introductory episode is about why this series exists and who it is for. Generative AI has exploded, but real understanding is still scattered. Between hype, shortcuts, and surface level strategy talk, it is hard to find a clear path from fundamentals to building systems that actually work. This series is for practitioners,...

Welcome to the final episode of our Deep Learning series on the AI Concepts Podcast. In this episode, host Shay takes you on a journey through the world of autoencoders, a foundational AI model. Unlike traditional models that predict or label, autoencoders excel in understanding and reconstructing data by learning to compress information. Discover how this quiet revolution in AI powers features li...

Welcome to the AI Concepts Podcast, where we explore AI, one concept at a time. In this episode, host Shay delves into the transformative world of transformers in AI, focusing on how they have revolutionized language understanding and generation. Discover how transformers enable models like ChatGPT to respond thoughtfully and coherently, transforming inputs into conversational outputs with unprece...

In this episode of the AI Concepts Podcast, host Shay delves into the transformation of deep learning architectures, highlighting the limitations of RNNs, LSTM, and GRU models when handling sequence processing and long-range dependencies. The breakthrough discussed is the attention mechanism, which allows models to dynamically focus on relevant parts of input, improving efficiency and contextual a...

Welcome to another episode of the AI Concepts Podcast, where we simplify complex AI topics into digestible explanations. This episode continues our Deep Learning series, diving into the limitations of Recurrent Neural Networks (RNNs) and introducing their game-changing successors: Long Short-Term Memory (LSTM) networks and Gated Recurrent Units (GRUs). Learn how these architectures revolutionize t...

Welcome to the AI Concepts Podcast! In this episode, we dive into the fascinating world of Recurrent Neural Networks (RNNs) and how they revolutionize the processing of sequential data. Unlike models you've heard about in previous episodes, RNNs provide the capability to remember context over time, making them essential for tasks involving language, music, and time series predictions. Using analog...

Welcome to the AI Concepts Podcast! In this deep dive into Convolutional Neural Networks (CNNs), we unravel their unique ability to process and interpret image data by focusing on local patterns and spatial structures. Understand how CNNs tackle the challenge of vast input sizes and learn to identify features without exhaustive connections, making them ideal for tasks involving images. Explore the...

In this episode of the AI Concepts Podcast, host Shay delves into the complexities of deep learning, focusing on the challenges of training deep neural networks. She explains how issues like internal covariate shift can hinder learning processes, especially as network layers increase. Through the lens of batch normalization, Shea illuminates how this pivotal technique stabilizes learning by normal...

In this enlightening episode of the AI Concepts Podcast, join host Shay as we dive deep into the world of deep learning optimizers. Discover how RMSPROP and ADAM revolutionize the training process by adapting to gradient changes, learn the benefits of learning rate scheduling, and explore the critical role of hyperparameter tuning. But the journey doesn't stop there—find out what makes your AI mod...

Welcome to the AI Concepts Podcast, where host Shay unravels the intricate world of AI through relatable examples and easy-to-understand analogies. In this episode, we continue our dive into deep learning by addressing the challenges and solutions of gradient descent. Learn how traditional gradient descent, which is pivotal in neural network training, sometimes falls short due to its slow speed an...

In this episode of the AI Concepts Podcast, we dive into the fascinating world of gradient descent. Building on the foundation laid in our discussion of backpropagation, we explore how gradient descent serves as a pivotal optimization algorithm in deep learning. Discover how it minimizes loss functions by adjusting model parameters and learn why selecting the right learning rate is crucial. Join u...

Welcome to the latest episode of the AI Concepts Podcast, hosted by Shay, where we continue our exploration of deep learning. In this installment, we delve into the mechanics of backpropagation, the algorithm that empowers neural networks to optimize and learn from their mistakes. We start by revisiting fundamental concepts of neural networks, exploring how data flows forward from input to output....

Welcome to this episode of the AI Concepts Podcast. Join host Shay as we delve into the fundamental architecture behind modern deep learning - the feedforward neural network. In this session, we take a closer look at how data flows through this network, transforming input into output without the need for loops or memory. Learn about the mechanics of feedforward networks, including weights, biases,...

Welcome to this episode of the AI Concepts Podcast's Deep Learning series, where we delve into the fascinating world of neural networks. Neural networks are the backbone of deep learning, modeled loosely after the human brain. This episode explores how these systems, made of artificial neurons, learn to recognize patterns and solve complex problems without explicit programming. We'll break down th...

Welcome to the AI Concepts Podcast, your go-to podcast for exploring AI concepts with clarity and simplicity. Join your host, Shay, as we embark on a series to demystify Deep Learning, the transformative branch of machine learning revolutionizing industries worldwide. Unlike traditional machine learning, discover how Deep Learning systems learn directly from raw data, organizing information throug...