State of the Art

Scaling Laws

Bartoldson et al. demonstrated that adversarial robustness follows predictable scaling laws with respect to compute (FLOPs) and data quality. Their work shows that robustness can be significantly improved through optimal allocation of computational resources and high-quality training data.

10^1710^30
0.1 (Perfect)50.0 (Poor)
73.3%
Adversarial Accuracy
10^21
Training FLOPs
1.65
Data FID

Bartoldson et al. Methodology

The authors barrow some methodology from (Wang et al., 2023) to train their models. We highlight two conceptually important details below, but the authors give a detailed account of all their training setttings in the original paper.

  1. PGD-Based Adversarial Training: To generate adversarial images at training time, the authors use 10 steps of PGD with α=2/255\alpha=2/255.
  2. Label Smoothing: Label smoothing (Szegedy et al., 2015) is conceptually similar to defensive distilation which you explored in a previous section. The math is slighly different but if you are interested, we highly reccomend section 7 of the orginal label smoothing paper which presents the math in a clear, concise way.

The approach here is nothing new but we include it for context. The insight that made the authors' approach state of the art was not a clever new algorithm but rahter scaling compute and data in an optimal way.

References

Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2015). Rethinking the Inception Architecture for Computer Vision. https://arxiv.org/abs/1512.00567
Wang, Z., Pang, T., Du, C., Lin, M., Liu, W., & Yan, S. (2023). Better Diffusion Models Further Improve Adversarial Training. https://arxiv.org/abs/2302.04638