Legal Notice
The contents of this article are for general informational purposes only and do not constitute legal advice. Although we prepare this information with the utmost care, we make no guarantees regarding its accuracy, completeness, or currency. For binding advice on your specific situation, please consult a qualified attorney.
The Key Question: High-Risk or Not?
When introducing an AI system in your business, one question matters more than any other: does the system fall into the “high-risk” category? The answer determines the effort, cost, and legal risk involved. A high-risk system is subject to extremely strict and expensive requirements. A limited-risk system, on the other hand, only has manageable transparency obligations.
Simple chat and voice assistants whose primary purpose is answering customer inquiries generally do not fall under the high-risk areas in Annex III of the EU AI Act. For them, the transparency obligation under Art. 50 is in the foreground. Whether this holds for your use depends on how you use the system.
However, complexity arises when more advanced features come into play. Could sentiment analysis or automatic urgency detection turn your AI assistant into a high-risk system?
When Does a Communication Tool Become a High-Risk System?
The AI Act lists the areas in which AI systems are potentially considered high-risk in its Annex III. For customer service, two points are particularly relevant: systems that decide on access to essential services, or systems that perform profiling of natural persons.
This is where the typical concerns arise:
- Is sentiment analysis (“caller sounds upset”) considered prohibited “emotion recognition” or risky “profiling”?
- If the AI detects an urgency (“urgent matter”), does it then decide on access to an essential service?
These concerns are understandable. Whether they apply depends on your specific use.
The Key: Is the AI Just an Assistant to Humans?
The AI Act provides specific exceptions for when a system is not considered high-risk despite being potentially listed in Annex III. Two of these exceptions are crucial for customer service:
- The AI serves to improve the outcome of a previously completed human activity.
- The AI performs a preparatory task for an assessment that is ultimately carried out by a human.
This is where the key lies. An assistant that only prepares information and presents it to a human for decision-making can fall under these exceptions. Sentiment analysis, for example, then makes no autonomous decision. It performs a preparatory task by providing a human agent with an additional data point (e.g., “caller seems upset”). The agent can use this information to handle the conversation more effectively. The exception does not apply, however, if the system performs profiling of natural persons. You check whether your use falls under an exception.
This principle of “human-in-the-loop augmentation” can play a role in the classification. It does not replace reviewing your specific use. The risk class depends on how you use the system.
Assess Your Own Use
The takeaway for you as a business owner: the risk class depends on how you use the system, and you assess your own use.
An AI assistant like Safina, for example, prepares information and presents it to a human for decision-making. Tell callers at the start of every call that they are speaking with an AI. This is part of the transparency duty under Art. 50 of the AI Act. Safina makes no statement about which risk class your use falls into. Check your use case and get legal advice if in doubt.
