The U.S. Commerce Department’s National Institute of Standards and Technology (NIST), which creates and tests technology for the government, businesses, and general public, has reopened a testbed intended to gauge how malicious attacks, especially those that “poison” AI model training data, might impair an AI system’s functionality.
The modular, open source web-based application, named Dioptra (after the traditional astronomy and surveying instrument), was initially made available in 2022 and aims to assist businesses that train artificial intelligence (AI) models as well as the individuals who use them in identifying, evaluating, and monitoring AI hazards. According to NIST, Dioptra can be used as a shared platform for exposing models to simulated threats in a “red-teaming” environment, as well as for benchmarking and research purposes.
One of the objectives of Dioptra, according to a press release from NIST, is to test the impact of adversarial attacks on machine learning models. “The community, including government agencies and small to medium-sized businesses, could conduct evaluations to assess AI developers’ claims about the performance of their systems with the aid of open source software, like generating child, which is available for free download.”
Dioptra made its debut with papers from NIST and the recently established AI Safety Institute at NIST, which outline strategies to reduce some of the risks associated with AI, such as the potential for misuse to produce nonconsensual pornography. It comes after the release of Inspect, a collection of tools from the U.K. AI Safety Institute that is also intended to evaluate model capabilities and overall model safety. Announced at the United Kingdom’s AI Safety Summit in November of last year at Bletchley Park, the United States and the United Kingdom have been working together to develop enhanced AI model testing.
Dioptra is also a result of the AI executive order (EO) signed by President Joe Biden, which requires NIST assistance with AI system testing, among other things. In a parallel vein, the EO sets forth guidelines for AI security and safety, including the need that businesses creating models—like Apple, for example—notify the federal government and disclose the outcomes of all safety testing prior to public release.
As we’ve previously discussed, AI benchmarks are challenging, in part because the most advanced models available today are opaque black boxes, with their infrastructure, training data, and other crucial information kept secret by the firms developing them. According to a report released this month by the nonprofit Ada Lovelace Institute, a research institute based in the United Kingdom that specializes in AI, evaluations by themselves are insufficient to assess an AI model’s safety in real-world scenarios, in part because existing regulations permit AI vendors to pick and choose which evaluations to carry out.
Dioptra is not claimed by NIST to be able to fully de-risk models. However, the organization does suggest that Dioptra can provide insight into the kinds of attacks that could reduce an AI system’s efficiency and measure this influence on performance.
Dioptra’s inability to function natively on models that can be downloaded and used locally, such as Meta’s growing Llama family, is a significant constraint. For the time being, at least, avoid using models that are gated behind an API, like OpenAI’s GPT-4o.