AI-Enabled XR

The goal is to develop a working prototype that integrates these technologies, enhancing language-based interactions in XR. By incorporating the latest advancements in multi-modal LLMs, the project seeks to move beyond basic text-based interactions and explore new, innovative applications in XR environments.

Expected effects and result

We aim to establish a consortium of collaborators through the organisers´ network to develop XR training simulators. By project end, we expect to produce a working prototype of a basic XR training scenario with an LLM-based character for user interaction. Additionally, we will generate a detailed report outlining further development steps, key application areas, challenges of language-specific training (e.g., Swedish), and potential ethical considerations.

Planned approach and implementation

This project emerged from 2023 workshops and inquiries exploring XR and AI, highlighting the need for a dynamic, responsive VR simulator enabled by LLM technology. These simulators could help individuals gain equitable education, overcome anxiety, or develop skills through continuous practice. We will involve representatives from our established EU and US networks, ensuring the expertise needed to develop the project and providing a strong foundation for future cross-country collaborations.

This project is funded by Vinnova for the establishment of collaboration between Swedish and American actors with a focus on Extended reality, XR. This project is the first step in a two-step effort from Vinnova's Future Areas area and part of Vinnova's investment in pioneering technology.