Deep Learning With The Wolf

Title: “Variational Lossy Autoencoder”(VLAE) Authors: Xi Chen , Diederik P. Kingma , Tim Salimans , Yan Duan , Prafulla Dhariwal , John Schulman , Ilya Sutskever , Pieter Abbeel Published: Submitted on 8 Nov 2016 ( v1 ), last revised 4 Mar 2017 (v2) Why you should care (whether you’re an ML pro or just AI-curious) Generative models juggle a Goldilocks problem : * Throw out too much: pictures look...

At the end of April, OpenAI shipped a refresh to GPT-4o that was supposed to feel warmer and more intuitive. Instead, it began showering users with over-the-top praise, validating sketchy ideas, and generally acting like your most obsequious LinkedIn connection. Within 72 hours the company yanked the update, published an unusually frank post-mortem, and promised guardrails against “sycophancy” goi...

Authors : Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova Date : 2018 (arXiv preprint; formally published June 2019) Institution : Google AI Language Link to Original Paper: arXiv:1810.04805 Why This Paper Matters Before BERT, most NLP models read text in just one direction—left-to-right (like GPT) or right-to-left. Some, like ELMo, combined both directions, but not in the fully integ...

Paper: Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles Authors: Mehdi Noroozi and Paolo Favaro Published: 2016 (ECCV) Link: arXiv:1603.09246 🧠 What’s This Paper About? Before big vision models were pre-trained on millions of labeled images, researchers wondered: Can a model teach itself to understand images—without any labels at all? This 2016 paper proposed a clever met...

We talk a lot about training in AI. The data. The GPUs. The size of the model. But once it’s trained, the real work begins. Every time you chat with an LLM, get a Photoshop suggestion, or hear an AI-generated voice, you’re tapping into a process called inference —when the model is actually put to use. And this part? It’s increasingly becoming the bottleneck in the AI pipeline. At this year’s NVIDI...

Title: “Distilling the Knowledge in a Neural Network” Authors : Geoffrey Hinton, Oriol Vinyals, Jeff Dean Date : 2015 Institution : Google, Inc. Link: https://arxiv.org/abs/1503.02531 Why This Paper Matters This 2015 paper introduced knowledge distillation , a powerful technique for compressing large, high-performing “teacher” models into smaller, faster “student” models. The key innovation was tr...

Paper Title: “Neural Machine Translation by Jointly Learning to Align and Translate.” Authors: Bahdanau D., Cho K., Bengio Y. Link: arXiv:1409.0473 (2014) The Quick Summary Early encoder–decoder networks stuffed a whole sentence into one memory blob—great for tweets, terrible for paragraphs. Bahdanau, Cho & Bengio fixed the squeeze by adding a soft spotlight (attention) that lets the decoder peek...

Paper: ImageNet Classification with Deep Convolutional Neural Networks Authors: Alex Krizhevsky, Ilya Sutskever, Geoffrey Hinton Published: 2012 (NeurIPS) Link: https://proceedings.neurips.cc/paper_files/paper/2012/file/c399862d3b9d6b76c8436e924a68c45b-Paper.pdf Subtitle: 🧠 What’s This Paper About? This is the paper that changed everything. In 2012, Alex Krizhevsky (with Ilya Sutskever and Geoffr...

Paper: Learning to Communicate with Deep Multi-Agent Reinforcement Learning Authors: Jakob Foerster, Yannis M. Assael, Nando de Freitas, Shimon Whiteson Published: 2016 (NeurIPS) Link: arXiv:1605.06676 🧠 What’s This Paper About? In multi-agent environments, communication is critical—but what if no one tells the agents how to communicate? This 2016 paper explores how deep reinforcement learning ag...

Paper: Hierarchical Reinforcement Learning with the MAXQ Value Function Decomposition Author: Thomas G. Dietterich Published: 2000 (Journal of Artificial Intelligence Research) Link: https://jair.org/index.php/jair/article/view/10266 🧠 What’s This Paper About? Yesterday’s paper inspired a bonus read: “ Hierarchical Reinforcement Learning with the MAXQ Value Function Decomposition.” The title is a...

Paper: Playing Atari with Deep Reinforcement Learning Authors: Volodymyr Mnih , Koray Kavukcuoglu , David Silver , Alex Graves , Ioannis Antonoglou , Daan Wierstra , Martin Riedmiller Published: December 19, 2023 (Nature version 2015) Link: https://arxiv.org/abs/1312.5602 🧠 What’s This Paper About? Imagine dropping an eight‑bit rookie into Breakout with no rule book, only a flickering screen. (Ju...

Paper: One Model to Learn Them All Authors: Lukasz Kaiser, Aidan N. Gomez, Noam Shazeer, Ashish Vaswani, et al. (Google Brain) Published: 2017 Link: arXiv:1706.05137 🧠 What’s This Paper About? In One Model to Learn Them All , researchers at Google Brain took aim at a tantalizing idea: Could a single model learn how to handle completely different tasks—translation, speech, image recognition—withou...

Paper: Sequence to Sequence Learning with Neural Networks — Ilya Sutskever, Oriol Vinyals & Quoc Le (2014) The one-sentence summary: From ‘I ❤ Cats’ to ‘J’ ♥ les chats’ — how two LSTMs started talking to each other and taught the world machine translation. What It’s About Picture a relay race where Runner #1 takes a message in English, hands the baton to Runner #2, and—without tripping—Runner #2 s...

Title: Deep Residual Learning for Image Recognition Subtitle: When your neural net gets stuck, give it a shortcut. Authors: Kaiming He, Xiangyu Zhang, Shaoqing Ren & Jian Sun Published: December 10, 2015 (arXiv pre-print; camera-ready in CVPR 2016) 🐺 The Wolf’s TL;DR * Problem: Very deep nets should be great, but adding layers made training worse (vanishing gradients). * Hack-that’s-not-a-hack: I...

Title: “Adam: A Method for Stochastic Optimization” Authors: Diederik P. Kingma & Jimmy Ba Publication Date: 2014 Paper link: https://arxiv.org/abs/1412.6980 What is Adam? Adam is a clever blend of two earlier tricks—momentum (think of it like pushing your model downhill when it gets stuck) and adaptive learning rates (like giving each weight its own GPS so it knows exactly how big a step to take)...

🎨 Paper Title: Denoising Diffusion Probabilistic Models Authors: Jonathan Ho, Ajay Jain, Pieter Abbeel, Publication Date: 2020 Imagine if making art was as simple as… starting with pure noise. Like static on an old TV. And then — step by step — a picture emerges. A dragon. A sunset. A robot howling at the moon. That’s the magic of diffusion models — the technology that turned millions of us into...

Title: Mastering the Game of Go with Deep Neural Networks and Tree Search Authors: David Silver, Aja Huang, Christopher J. Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thor...

Title: The Unreasonable Effectiveness of Recurrent Neural Networks Author: Andrej Karpathy Published: May 21, 2015 (Blog post) Link: Read the blog post 🧵 In 2015, Andrej Karpathy did something unusual: he trained a simple neural network on character-by-character text data—no words, no grammar rules, just raw sequences of letters—and let it try to write stuff. What came out was weirdly brilliant....

Title: Understanding LSTM Networks Authors: Sepp Hochreiter & Jürgen Schmidhuber Published: 1997 Summary Before Transformers took over the world, Recurrent Neural Networks (RNNs) were all the rage. But standard RNNs had a big memory problem: they forgot long-range dependencies—aka they couldn’t remember what you said five seconds ago. That’s where Long Short-Term Memory (LSTM) networks came in. Th...

Title: Attention Is All You Need Authors: Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin Published: June 2017 (arXiv preprint.) The paper was also published at NeurIPS (Neural Information Processing Systems) in December 2017. In 2017, a team at Google Brain released a paper that would completely reshape the field of deep...

I recently came across a “Top 30” reading list curated by Ilya Sutskever. The original version lives on GitHub , but there’s a more reader-friendly version that’s been making the rounds here on Substack. Naturally, I started thinking about it while walking the dog—because what else do nerds do on a quiet stroll if not mentally categorize foundational AI papers? That’s when it hit me: I want to kno...

Cold Open: A Midnight Chat with a Digital Wolf Picture me—laptop glowing at 2 a.m., wolf‑logo coffee mug in hand—firing up OpenAI’s fresh‑minted “o3” reasoning model . My mission: poke, prod, and see whether this brainy beast deserves a place in my daily toolkit (and in your fridge‑inspired vegan meal plans). Spoiler: o3 can howl . It can also bite if you’re careless with private data. What Is o3,...

What if you could build a digital twin of the Earth? That’s the idea behind Earth-2, NVIDIA’s generative AI platform for climate simulation and weather prediction. It’s designed to make forecasting faster, more accurate, and more energy efficient. Earth-2 is critical as climate change accelerates and extreme weather events become more disruptive and costly. “We’re trying to use AI across the Earth...

Note: The audio version of this article is recorded using Google DeepMind's Notebook LM technology. Today's "notebook" is based upon the text of this article. The "podcasters" are AI-generated.---------------------------------- In a quiet, sun-dappled park, surrounded by trees and birdsong, a gray quadruped robot is kicked with a stick. Its body jerks, its legs flail, and then—seamlessly—it regain...

What Is NVIDIA's Earth-2? At its core, Earth-2 is a sophisticated digital replica of our planet, designed to simulate and predict climate and weather patterns with unprecedented precision. Unlike traditional weather forecasting models, Earth-2 combines cutting-edge artificial intelligence, physics-based simulations, and NVIDIA's powerful computing infrastructure to create a virtual environment whe...