Mistral Introduces An API For Moderating

Mistral, an AI startup, has released a new content moderation API.

According to Mistral, the API—which also drives moderation in the company’s Le Chat chatbot platform—can be customized to meet certain needs and security requirements. It is driven by a refined model (Ministral 8B) that has been trained to categorize text in a variety of languages, including English, French, and German, into one of nine groups: personal identifiable information, sexual, hate and discrimination, violence and threats, dangerous and criminal content, self-harm, health, finances, and the law.

According to Mistral, the moderation API can be used for conversational or raw text.

“We’ve witnessed increasing interest in new AI-based moderation systems in the industry and research community over the last few months, which can help make moderation more scalable and robust across applications,” Mistral stated in a blog post. “By addressing model-generated harms like unqualified advice and PII, our content moderation classifier introduces a practical approach to model safety and leverages the most pertinent policy categories for effective guardrails.”

Theoretically, moderation systems driven by AI are beneficial. However, the same prejudices and technological issues that affect other AI systems can also affect them.

African American Vernacular English (AAVE), the colloquial language spoken by some Black Americans, is disproportionately “toxic,” according to various models designed to identify toxicity. According to studies, popular public opinion and toxicity detection methods frequently highlight social media posts concerning persons with disabilities as more unfavorable or poisonous.

Though it acknowledges that it is still a work in progress, Mistral asserts that their moderation model is extremely precise. Notably, the business did not assess the performance of its API against that of other well-known moderation APIs, such as OpenAI’s moderation API and Jigsaw’s Perspective API.

The business stated that it is “continuing to engage with the research community to contribute safety advancements to the broader field” and that it is “working with our customers to build and share scalable, lightweight, and customizable moderation tooling.”

Today, Mistral also revealed a batch API. The startup claims that by handling high-volume requests asynchronously, it can cut the cost of models served through their API by 25%. For their AI APIs, Anthropic, OpenAI, Google, and others also provide batching choices.