AI Safety - Paper Digest

Dive into the key findings of the first large-scale field study on the adoption, usage intensity, and use cases of general-purpose AI agents, drawing on hundreds of millions of anonymized user interactions with Perplexity’s Comet Assistant. This paper tracks the frontier shift from conversational LLM chatbots to action-oriented AI agents, which are defined as AI assistants capable of autonomously...

This episode of the AI Safety Paper Digest is about the World Economic Forum's new playbook on advancing responsible AI innovation. In cooperation with Accenture, the report provides a practical roadmap for turning responsible AI from an aspiration into a competitive advantage while building public trust. Link to the Report: https://www.weforum.org/publications/advancing-responsible-ai-innovat...

How can we make autonomous driving systems safer through generative AI? In this episode, we explore LD-Scene , a novel framework that combines Large Language Models (LLMs) with Latent Diffusion Models (LDMs) to create controllable, safety-critical driving scenarios. These adversarial scenarios are essential for evaluating and stress-testing autonomous vehicles, yet they’re extremely rare in real-w...

Ever wanted a clear, comprehensive explanation of all the key terms related to Large Language Models (LLMs)? This episode has you covered. In this >1-hour deep-dive, we'll guide you through the essential glossary of LLM-related terms and foundational concepts, perfect for listening while driving, working, or on the go. Whether you're new to LLMs or looking to reinforce your understandin...

In this special christmas episode, we delve into " Best-of-N Jailbreaking ," a powerful new black-box algorithm that demonstrates the vulnerabilities of cutting-edge AI systems. This approach works by sampling numerous augmented prompts - like shuffled or capitalized text - until a harmful response is elicited. Discover how Best-of-N (BoN) Jailbreaking achieves: 89% Attack Success Rates (ASR) on G...

In this episode, we explore the latest advancements in automated red teaming from OpenAI, presented in the paper " Diverse and Effective Red Teaming with Auto-generated Rewards and Multi-step Reinforcement Learning ." Automated red teaming has become essential for discovering rare failures and generating challenging test cases for large language models (LLMs). This paper tackles a core challenge:...

In this episode, we explore the findings from " Insights and Current Gaps in Open-Source LLM Vulnerability Scanners: A Comparative Analysis ." As large language models (LLMs) are integrated into more applications, so do the security risks they pose, including information leaks and jailbreak attacks. This study examines four major open-source vulnerability scanners - Garak, Giskard, PyRIT, and Cybe...

In this episode, we delve into the groundbreaking watermarking technology presented in the paper " Scalable Watermarking for Identifying Large Language Model Outputs ," published in Nature . SynthID-Text, a new watermarking scheme developed for large-scale production systems, preserves text quality while enabling high detection accuracy for synthetic content. We explore how this technology tackles...

In this episode, we dive into PyRIT , the open-source toolkit developed by Microsoft for red teaming and security risk identification in generative AI systems. PyRIT offers a model-agnostic framework that enables red teamers to detect novel risks, harms, and jailbreaks in both single- and multi-modal AI models. We’ll explore how this cutting-edge tool is shaping the future of AI security and its p...

This podcast, " Jailbreaking GPT o1 , " explores how the GPT o1 series, known for its advanced "slow-thinking" abilities, can be manipulated into generating disallowed content like hate speech through a novel attack method, the Single-Turn Crescendo Attack (STCA), which effectively bypasses GPT o1's safety protocols by leveraging the AI's learned language patterns and its step-by-step reasoning pr...

This episode explores the intricate world of red-teaming generative AI models as discussed in the paper " Attack Atlas: A Practitioner's Perspective on Challenges and Pitfalls in Red Teaming GenAI ." We'll dive into the emerging vulnerabilities as LLMs are increasingly integrated into real-world applications and the evolving tactics of adversarial attacks. Our conversation will center around the "...

In this episode, we examine the cutting-edge adversarial strategy presented in " Well, that escalated quickly: The Single-Turn Crescendo Attack (STCA) ." Building on the multi-turn crescendo attack method, STCA escalates context within a single, expertly crafted prompt, effectively breaching the safeguards of large language models (LLMs) like never before. We discuss how this method can bypass mod...

This episode dives into how the Crescendo Multi-Turn Jailbreak Attack leverages seemingly benign prompts to escalate dialogues with large language models (LLMs) such as ChatGPT, Gemini, and Anthropic Chat, ultimately bypassing safety protocols to generate restricted content. The Crescendo attack begins with general questions and subtly manipulates the model’s responses, effectively bypassing tradi...