RegTek: Accuracy Evaluation of RAG

The introduction of complex regulations such as the EU's AI Act places new, high demands on precision and efficiency in regulatory processes. At the same time, analyses from bodies including the "Swedish Regulation Council" (Regelrådet) and the OECD show that up to 60% of current Swedish impact assessments do not meet the set requirements, often due to resource constraints and complexity. This deficiency constitutes an obstacle to both responsible innovation and predictable regulatory compliance, especially within fast-moving technology areas like artificial intelligence. The need for more reliable, objective, and time-efficient methods is acute.

Pioneering AI Support for impact assessment

In response to these challenges, the RegTek project addresses the need for modernized regulatory tools. RISE is leading a strategic pilot study to evaluate an advanced language model based on Retrieval-Augmented Generation (RAG), developed by the Swedish National Financial Management Authority (ESV). The RAG technology is particularly promising as it combines the capabilities of large language models with the ability to dynamically retrieve and integrate information from specified, reliable sources. This enables the generation of impact analyses that are both contextually relevant and fact-based, which is crucial for regulatory application.

RAG performance measurement and trustworthyness

RegTek is conducting a pilot study where RISE evaluates a RAG-based language model developed by the Swedish National Financial Management Authority (ESV). The evaluation includes performance measurements and an investigation into how standards and metrological methods can be applied to measure precision and reliability.

Improved compliance, increased trust, and strengthened national AI capability

The RegTek project aims to deliver measurable improvements and strategic advantages:

• Streamlined regulatory processes: Potential for significantly faster and more resource-efficient impact assessments.
• Validated reliability: An objectively evaluated method to measure and ensure the accuracy of AI-generated documentation, strengthening trust among regulators and decision-makers.
• Facilitated AI Act compliance: Improved tools that support organizations' ability to meet the requirements of the EU's AI Act and other complex regulations.
• National competence enhancement: The project strengthens Sweden's collective capability in AI testing, validation, and responsible implementation of AI in the public sector, through the work of RISE and ESV.
• Knowledge dissemination via open source: Developed evaluation methods and key results are planned to be published as open source, promoting transparency and broader benefit.

Through RegTek, RISE contributes to building a foundation for the safe, effective, and reliable use of AI in support of future regulations.