PC5: GENUINE

Overview

By its very nature, a digital twin strongly relies on data visualization and user interactions. Indeed, a DT usually embeds interactive systems that enable users to visualize and interact with both the physical and digital twins and their large amount of associated data.

Those interactive systems can take multiple forms, from fully digital VR immersive systems to hybrid AR systems, but also 2D graphical user interfaces, and systems that imply physical input devices. They often propose collaborative asymmetrical (AV/VR/2D) interactive situations, which can also be a challenge.

Building such interactive systems from scratch or without automation is time-consuming and may prevent the generalization of DT as industrial tools. First, novel approaches are required to build the visual representation of the DT, which can be 3D models (to build, for example, by combining CAD models, 3D scans, pictures, and other media sources) or 2D various kinds of information. Users can then interactively finalize some parts of the DT. Additional visualization tools can also help reduce the bias of automatic reconstruction.

Then, DT come with usages that raise research questions regarding, from an HCI viewpoint, how to help users in visualizing data and interacting with it, especially when we use 3D. The visualization concerns not only the “raw” model of the physical twin but also all the data that can be used to enrich the model (additional information about how to use it). It raises new challenges about how to display large datasets of information and how and when to interact with them.

Finally, all these points require validations, at different scales. To reduce the cost of developing the HCI facets of DT, the project must propose novel techniques that focus on their engineering.

The project focuses on 3D reconstruction, data visualization, interaction engineering, spatio-temporal navigation, and validation with clear objectives: developing semi-automatic construction techniques, creating visualization metaphors, designing interaction approaches, implementing seamless transitions, and establishing usability assessment methods.

Associated Use Cases

Use cases in PC5 span multiple scales:

• Territory Scale: Reconstructing the DT of a city or area and populating it with virtual humans to better understand current limitations and how the DT can enhance it
• Building/Factory Scale: Reconstructing the DT of a reconfigurable assembly line, providing AR augmentation for workers and VR visualization for experts who can help remotely
• Technical Object Scale: Reconstructing the DT of a complex system (car, satellite) to train people for maintenance tasks using VR interactions, then using AR augmentations when physically in front of the physical twin

Investigator & Project Partners

Principal Investigators:

Thierry Duval

Full Professor and Head of Computer Science Department at IMT Atlantique

Thierry Duval is a full Professor and the head of the Computer Science Department at IMT Atlantique in Brest. He is member of the INUIT team at Lab-STICC. He received a PhD degree in Computer Science from Centrale Nantes and the University of Nantes in 1993, and an Habilitation degree from University of Rennes 1 in 2012. His research interests include Virtual, Augmented, eXtended and Mixed Reality, Immersive Analytics, Collaborative Virtual Environments, as well as Software Engineering for interactive systems. He is currently involved in several collaborative projects (5G Metaverse, Reclassif, eNSEMBLE) where he conducts AR/VR/XR/MR/CVE research mainly applied to Industry of the Future where Digital Twins of factories are central elements of the projects. He has published over 100 communications in international conferences and journals.

Arnaud Blouin

Associate Professor at INSA Rennes

Arnaud Blouin is a Senior Lecturer authorized to supervise research at INSA Rennes, France, where he conducts research within the DiverSE research group. He conducts research in the field of software engineering, focusing on human-machine interaction engineering, software maintenance and validation, as well as domain-specific languages and their development environments. He currently co-leads the CNRS GDR-IHM working group on interactive systems engineering.

Participating Partners:

Project Implementation

The project is structured into three complementary work packages, addressing reconstruction and visualization of Digital Twins,their spatial and temporal exploration, and the modeling of their users interactions.