DIDYMOS-XR

The digital transformation and the availability of more diverse and cost-effective methods for 3D capture has led to the creation of digitised representations of parts of our public spaces, machinery and processes, so-called Digital Twins. These Digital Twins are an important basis for building advanced XR applications for cityscapes and the industrial environments. These XR applications need high-fidelity models for accurate localization, which are kept in-sync with the real world. However, currently (i) creating high-fidelity Digital Twins is costly, (ii) integrating data from heterogeneous sensors is still challenging and (iii) the update of Digital Twins requires explicit manual intervention. The vision of DIDYMOS-XR is to enable advanced, more realistic and more dynamic extended reality (XR) applications, powered through artificial intelligence. The project thus focuses on advancing technologies for creating large-scale digital twins, synchronised with the real world. DIDYMOS-XR will research and develop methods for reconstruction and mapping from heterogeneous inputs, including static and mobile sensors, AI-based data fusion, scene understanding and rendering. Digital twin based XR applications also require means of accurately positioning in the environment, aware of the dynamics of the scene, also addressed by the project. The capture of scenes at scale, as well as using cameras and other sensor data for synchronising the digital representation, bears the risk of capturing personal and sensitive data. Hence, the technologies resulting from DIDYMOS-XR must be and will be ethical and privacy-aware by design. In order to demonstrate and validate the enabling technologies researched and developed in DIDYMOS-XR, the project will address use cases of XR applications in two domains that differ in their scale and characteristics. One is urban planning and smart mobility, the other is the collaboration with autonomous mobile robots at an industrial production facility.

Our team was tasked with: 1) NERF-based 3D scene reconstruction (indoors and outdoors), 2) NERF-based localization.

Consortium

JOANNEUM RESEARCH FORSCHUNGSGESELLSCHAFT MBH
American University of Beirut
ETHNIKO KENTRO EREVNAS KAI TECHNOLOGIKIS ANAPTYXIS
FUNDACIO PRIVADA I2CAT, INTERNET I INNOVACIO DIGITAL A CATALUNYA
DIGITALTWIN TECHNOLOGY GMBH
PANEPISTIMIO PATRON
FICOSA ADAS, S.L.
TECHNISCHE UNIVERSITAT BERLIN
HOCHSCHULE HAMM-LIPPSTADT
FIWARE FOUNDATION EV
NEAPOLIS SPAIN
IDEALWORKS GMBH
UNITY TECHNOLOGIES APS