We are releasing COMPL-AI (https://compl-ai.org), the first free and open source EU AI Act Evaluation Framework. COMPL-AI is built by teams at ETH Zurich, INSAIT in Sofia, Bulgaria and LatticeFlow AI. 

The complete release includes:

• The first comprehensive mapping between the EU AI Act regulatory principles and actionable technical requirements. The mapping can also be found here: https://arxiv.org/abs/2410.07959. It is a strong starting point for the recently formed GPAI Code of Practice working groups by the Commission and can serve as a blueprint for other AI regulations and future compliance frameworks. 
• An evaluation framework based on the mapping: COMPL-AI, which can be found here: https://compl-ai.org. The framework is released open-source with all code freely available, and can be extended and built on.
• A comprehensive evaluation of public generative LLMs from OpenAI, Meta, Google, Anthropic, Alibaba using COMPL-AI. The evaluation reveals key gaps in addressing various requirements (e.g., in transparency), while highlighting strong performance in handling harmful content and toxicity.

We would be delighted if the community builds on COMPL-AI, updating the mapping, adding new modalities and new evaluations.