Pretrained

Pierce, Richard, and Will join for the first in-person interview on Pretrained. For our video episode, check out: https://youtu.be/CInTOIgz-pA They cover: - Will's history growing up in the UK - Getting an MBA and deciding what company to start - Trust building versus activating content - Building a personal brand for engineers and researchers - Entrepreneurship in Europe vs the US - & Much mo...

Kimi's serving architecture, mooncake to offload GPU memory to other chipsets, the ubiquity of vllm, and the growing standard LLM stack

Kimi K2 and Moonshot AI's history, avoiding loss spikes during training, the muon optimizer, and data parallelism

Gemini’s new image model, OpenAI is investing more in protein generation, Cohere’s SOTA generation model, and Anthropic working with DOE on nuclear security Further reading: https://blog.google/products/gemini/updated-image-editing-model/ https://openai.com/index/accelerating-life-sciences-research-with-retro-biosciences/ https://cohere.com/blog/command-a-translate https://red.anthropic.com/2025/n...

Rich and Pierce speculate wildly (well, kind of wildly) on the internal architecture of Genie 3. They go into the history of variational auto encoders and diffusion models, 3D modeling with AI as an alternative to video game designers, a recap of the official Genie 1 paper, and possible applications of world models to the real world. Further reading: https://deepmind.google/discover/blog/genie-3-a...

Richard visits his high school, LLMs in education, is AI a calculator or an oracle, and more

Elevenlabs launches a music generator, Claude Long-Term Memory, Reddit blocks the Internet Archive, NVIDIA’s Massively Multilingual Speech, and Self Questioning Language Models Further reading: https://techcrunch.com/2025/08/05/elevenlabs-launches-an-ai-music-generator-which-it-claims-is-cleared-for-commercial-use/ https://www.theverge.com/news/757538/reddit-internet-archive-wayback-machine-block-...

History of tokenization going back to 70s language modeling, modern tokenization approaches with BPE, and the future of token-free bitestreams

OpenAI releases their open weight GPT model, the release of GPT-5 and our growing attachment to different architectures, obsequiousness of language models, Genie 3 and open world models for reinforcement learning, and Pytorch as a standard in ML research Further reading: https://openai.com/index/introducing-gpt-oss/ https://openai.com/gpt-5/ https://deepmind.google/discover/blog/genie-3-a-new-fron...

Exhausting the data on the Internet by 2026, the value or non-value of transcripts, training on other languages, and subsidizing additional dataset generation

ChatGPT's new study mode for students, Facebook's play to be the first with augmented reality LLMs, Cerebras is trying to compete with Claude Code, and are we really in an "alphago moment" for model architecture discovery? Further reading: https://openai.com/index/chatgpt-study-mode/ https://s21.q4cdn.com/399680738/files/doc_financials/2025/q2/META-Q2-2025-Earnings-Call-Transcript.pdf https://www....

Evaluating models on benchmarks, passing a model vibe check, formal reasoning to synthesize datasets, and what type of datasets researchers prefer

OpenAI introduces their new Agent with full computer use capabilities, Anthropic is now evaluating models with other models, what the data leak at Tea can and can't tell us about vibe coding security, and more reasoning time isn't always better Further reading: https://openai.com/index/introducing-chatgpt-agent/ https://alignment.anthropic.com/2025/automated-auditing/ https://www.nytimes.com/2025/...

Allocating enough time to find your passions, Bell Labs vs Frontier Labs, project based classes, and getting involved in communities

Perplexity releases their AI browser Comet, Grok 4 and xAI's safety alignment strategy, the David vs Goliath of language models, and the Muon optimizer challenging Adam for its title

Anthropic & OpenAI's new approach for announcing models, double vs single blind peer review, and distribution channels for researchers

Inception Lab's new diffusion architecture Mercury, Google's emissions up 51% since 2019, DeepFleet warehouse scheduling robots, and Microsoft's delayed AI chip