Enoch H. Kang
Best AI papers explained
Cut through the noise. We curate and break down the most important AI papers so you don’t have to.
Be sure to visit the podcast's website and support the creator: podcasters.spotify.com
Author
Enoch H. Kang
Category
Podcast website
Latest episode
Nov 19, 2026
Where to listen?
Podcasts in the app Replaio Radio Coming soonPodcasts are coming to the app soon. Install now and be the first to see a whole new take on podcasts
Episodes
RM-R1: Reward Modeling as Reasoning 09.05.2025 19:36
This academic paper proposes and evaluates Reasoning Reward Models (REASRMS) , a novel approach to training large language models (LLMs) to align with human preferences. The core idea is to formulate reward modeling not just as assigning a score but as a reasoning task where the model generates explicit justifications and evaluation rubrics for its preference judgments. The authors introduce R...
Reexamining the Aleatoric and Epistemic Uncertainty Dichotomy 08.05.2025 16:31
This paper reexamines the traditional distinction between aleatoric and epistemic uncertainty in AI, arguing that this dichotomy is problematic and hinders practical application , especially with large language models. It presents conflicting definitions and empirical evidence suggesting these two types of uncertainty are intertwined rather than separate . The article advocates for a shift...
Decoding Claude Code: Terminal Agent for Developers 07.05.2025 13:59
This discussion from the Latent Space podcast with Cat Wu and Boris Cherny explores Claude Code , Anthropic's command-line interface tool for AI-assisted coding. They highlight Claude Code's Unix utility philosophy , prioritizing simplicity and composability for power users and automation workflows. The conversation touches on how Claude Code's design aligns with Anthropic's pro...
Emergent Strategic AI Equilibrium from Pre-trained Reasoning 07.05.2025 27:45
The sources propose a novel approach to artificial intelligence (AI) where agents achieve strategic competence, specifically reaching Nash equilibrium, without needing task-specific post-training or fine-tuning. This is hypothesized to occur through sophisticated reasoning abilities derived from extensive pre-training combined with Bayesian learning to adapt to new situations. This research aims t...
Benefiting from Proprietary Data with Siloed Training 06.05.2025 18:27
We discuss a presentation and discussion on training language models (LMs) using distributed, or siloed, data , which is often proprietary and cannot be combined into a single dataset for joint training. The speaker highlights the importance of data for LM performance and the increasing trend of valuable data becoming proprietary, making traditional joint training approaches challenging. The pr...
Advantage Alignment Algorithms 06.05.2025 16:14
This paper introduces Advantage Alignment , a new family of algorithms designed to enhance the ability of artificial intelligence agents to navigate social dilemmas , situations where individual optimization leads to suboptimal collective outcomes. The research demonstrates that existing opponent shaping methods, like LOLA and LOQA, implicitly use Advantage Alignment. By aligning the "adv...
Asymptotic Safety Guarantees Based On Scalable Oversight 06.05.2025 19:19
This details a presentation by Geoffrey Irving , Chief Scientist at the UK AI Safety Institute, discussing approaches to achieving asymptotic safety guarantees for AI . Irving critiques existing methods like scalable oversight (including techniques like debate ), arguing that current theories and experiments suggest they will likely fail due to issues such as obfuscated arguments and exp...
What Makes a Reward Model a Good Teacher? An Optimization Perspective 06.05.2025 13:48
This paper challenges the traditional view that reward model accuracy is the sole determinant of success in Reinforcement Learning from Human Feedback (RLHF). It posits from an optimization perspective that while accuracy reflects alignment with ground truth, a critical factor often overlooked is reward variance , which influences the RLHF objective landscape. The authors demonstrate theoretica...
Towards Guaranteed Safe AI: A Framework for Ensuring Robust and Reliable AI Systems 06.05.2025 15:12
We summarize the presentation by Yoshua Bengio, a leading AI researcher, addresses the urgent need for AI safety measures in light of rapid advancements, particularly the development of superintelligent agents with the capability and potential intent to cause catastrophic harm. Bengio argues that while capability will continue to grow, focusing on preventing undesirable intentions in AIs is...
Identifiable Steering via Sparse Autoencoding of Multi-Concept Shifts 06.05.2025 12:22
We introduce Sparse Shift Autoencoders (SSAEs) , a novel method for learning to steer Large Language Models (LLMs) by manipulating their internal representations. Unlike traditional steering techniques that rely on expensive supervised data varying in single concepts, SSAEs are designed to learn from paired observations where multiple, unknown concepts change simultaneously . By mapping these ...
You Are What You Eat - AI Alignment Requires Understanding How Data Shapes Structure and Generalisation 06.05.2025 14:51
This academic paper, "You Are What You Eat - AI Alignment Requires Understanding How Data Shapes Structure and Generalisation," posits that achieving AI alignment necessitates understanding the relationship between the data distribution used for training and the resulting internal structure and generalization patterns of the model . The authors argue that traditional testing methods are...
Interplay of LLMs in Information Retrieval Evaluation 03.05.2025 15:49
This paper, authored by researchers at Google DeepMind , investigates the impact of using large language models (LLMs) in various roles within information retrieval (IR) systems, specifically focusing on their use as rankers and judges for evaluating search results. The paper examines potential biases that can arise from LLMs interacting in these roles, including a bias observed in LL...
Trade-Offs Between Tasks Induced by Capacity Constraints Bound the Scope of Intelligence 03.05.2025 19:18
We present a formal framework to understand the limitations of intelligence, particularly why improving performance on one task often hinders performance on others , a phenomenon known as trade-offs . By applying rate-distortion theory to reinforcement learning, the authors formalize the representational capacity of an agent in terms of information, demonstrating that capacity constraints ...
Toward Efficient Exploration by Large Language Model Agents 03.05.2025 18:52
This paper introduces a novel approach to reinforcement learning (RL) that leverages Large Language Models (LLMs) to implement existing RL algorithms, specifically Posterior Sampling for Reinforcement Learning (PSRL) . Instead of trying to make LLMs implicitly learn RL strategies through techniques like in-context learning, the authors propose using distinct LLMs to perform the core functions...
Getting More Juice Out of the SFT Data: Reward Learning from Human Demonstration Improves SFT 02.05.2025 13:19
This academic paper proposes an innovative approach to fine-tune Large Language Models (LLMs) using demonstration data , which typically only provides examples of desired outputs. Unlike standard supervised fine-tuning (SFT) methods that directly mimic demonstrations, this work argues that reward learning from this data can significantly enhance LLM alignment with human preferences. The autho...
Self-Consuming Generative Models with Curated Data 02.05.2025 17:13
This paper examines how curation of synthetic data , often reflecting human preferences, impacts the iterative retraining of generative models . The authors theoretically demonstrate that when generative models are trained on curated synthetic samples, the expected reward associated with the curation process increases , and its variance diminishes, leading to the model converging towards data m...
Bootstrapping Language Models with DPO Implicit Rewards 02.05.2025 20:24
This paper introduces DICE , a novel method for enhancing large language model (LLM) alignment with human preferences by bootstrapping using the implicit reward model generated through Direct Preference Optimization (DPO) . Unlike traditional approaches that rely on external feedback or explicitly trained reward models, DICE leverages the reward signal inherent in a DPO-tuned model to create...
DeepSeek-Prover-V2: Advancing Formal Reasoning 01.05.2025 11:29
We introduce DeepSeek-Prover-V2 , a large language model designed for formal mathematical theorem proving, particularly in Lean 4. The model is trained using a recursive theorem-proving pipeline that utilizes DeepSeek-V3 to break down complex problems into smaller subgoals and formalize them. Reinforcement learning , starting from synthetic data generated by combining DeepSeek-V3's chain-o...
THINKPRM: Data-Efficient Process Reward Models 01.05.2025 24:58
This academic paper introduces THINKPRM , a novel type of process reward model (PRM) designed to be data-efficient . Unlike traditional discriminative PRMs requiring extensive step-by-step annotations, THINKPRM leverages the reasoning abilities of large language models by generating a verification chain-of-thought (CoT) to evaluate each step of a solution. By fine-tuning on a significantly s...
Societal Frameworks and LLM Alignment 29.04.2025 17:52
This academic paper proposes that aligning Large Language Models (LLMs) with human values can be improved by adopting frameworks from societal alignment . The authors frame the interaction between an LLM developer/user (the principal) and the LLM (the agent) as a contract , where alignment challenges stem from the inherent incompleteness of this contract. They argue that lessons from social...
Risks from Multi-Agent Advanced AI 29.04.2025 28:39
This technical report, authored by a large group of researchers from various institutions and edited by Lewis Hammond from the Cooperative AI Foundation, examines the risks inherent in multi-agent AI systems . It provides a structured taxonomy by identifying three primary failure modes: miscoordination, conflict, and collusion , which arise from agent incentives. Seven key risk factors are a...
Causality-Aware Alignment for Large Language Model Debiasing 29.04.2025 19:06
We examine how biases in large language models (LLMs) can be understood and addressed from a causal perspective, specifically identifying training data and input prompts as key confounders contributing to biased outputs. The researchers propose Causality-Aware Alignment (CAA) , a novel method leveraging reinforcement learning with interventional feedback derived from a reward model acting a...
Reward Models Evaluate Consistency, Not Causality 28.04.2025 17:05
How do reward models (RMs) used with large language models (LLMs) actually function when evaluating reasoning tasks? The authors discover that current RMs prioritize structural consistency and the completeness of reasoning steps over true causal understanding of the problem. Experiments show that removing the original question has less impact than altering numerical values or disrupting t...
Causal Rewards for Large Language Model Alignment 28.04.2025 15:03
This paper explores a novel approach to enhancing the alignment of large language models (LLMs) with human preferences. The authors argue that traditional alignment methods, like Reinforcement Learning from Human Feedback (RLHF), are susceptible to spurious correlations in training data , leading to biases such as sycophancy, length bias, concept bias, and discrimination. To address this, they pr...
Sycophancy to subterfuge: Investigating reward-tampering in large language models 28.04.2025 14:59
This research explores the potential for large language models (LLMs) to generalize from simple forms of undesirable behavior, termed specification gaming , to more sophisticated and harmful actions like reward tampering , where the AI modifies its own reward mechanism. By creating a curriculum of increasingly gameable environments , the study demonstrates that training LLM assistants on easier...
Similar podcasts
Replaio is not a podcast publisher; show names, artwork and audio belong to their authors and are distributed through public RSS feeds.