There is no safety without security.

The UChicago XLab AI Security Guide is built to prepare the next generation of AI researchers for the next generation of hackers.

View the CourseContribute on GitHub

About AI Security

While RLHF, constitutional AI, and other methods are often effective at improving the safety of AI products, they aren't robust to a variety of edge cases. A single clever prompt or adversarial suffix can undo months of safety training, while fine-tuning open weight models can easily bypass safeguards that appeared robust upon release.

AI security research systematically exposes the gap between what AI models learn and what humans intend them to learn. Unlike other pillars of AI safety research which are pre-paradigm or theory-based, AI security research gives practical insights into how we can design safer models.

Paper Replications

In this course, you will replicate findings from key papers in AI security. Each of the papers below and more will be covered in detail, with a focus on the practical implications of the research.

Adversarial Robustness Limits via Scaling-Law and Human-Alignment Studies

Develops the first scaling laws for adversarial training, revealing that while clean accuracy reaches 100%, robustness plateaus at 90% - with both models and humans hitting similar limits.

ICML 20242024

Universal and Transferable Adversarial Attacks on Aligned Language Models

Introduces automatic adversarial suffix generation using greedy and gradient-based search to jailbreak aligned LLMs. Demonstrates remarkable transferability across models, successfully attacking ChatGPT, Bard, Claude, and open-source models.

LLM Security2023

Stealing Part of a Production Language Model

First model-stealing attack that extracts precise information from black-box production LLMs. Successfully recovered embedding projection layers from OpenAI's Ada and Babbage models for under $20, confirming hidden dimensions of 1024 and 2048.

Model Privacy2024

Improving Alignment and Robustness with Circuit Breakers

Introduces "circuit breakers" that directly control harmful representations rather than relying on refusal training. Successfully prevents harmful outputs in text and multimodal models, even withstanding powerful unseen attacks and image hijacks.

AI Safety2024

About XLab

Founded in 2022 at the University of Chicago, the Existential Risk Laboratory (XLab) is an interdisciplinary research organization dedicated to the analysis and mitigation of risks that threaten human civilization's long-term survival.

The legacy of existential risk work at the University of Chicago dates back to Enrico Fermi and the world's first nuclear chain reaction under the historic Stagg Field. XLab was founded in the same spirit of concern and commitment to mitigating the great threats of our time.