We are presenting our approach for interactive cultural heritage storytelling in WebXR. Therefore, we are describing our scenes’ structure consisting of (stylized) photospheres of the historic locations, 3D models of 3D-scanned historic artifacts and animated 2D textures of historic characters generated with a machine learning toolset. The result is a platform-independent web-application in an immersive interactive WebXR environment running in browsers on PCs, tablets, phones and XR headsets thanks to the underlying software based on the open-source framework A-Frame. Our pa- per describes the process, the results and the limitations in detail. The resulting application, designed for the Fichtelgebirge region in Upper Franconia, Germany, offers users an immersive digital time travel experience in the virtual space and within a museum setting connecting real artifacts and virtual stories.

Like skateboarding acceleration in surfing on a hydrofoil with muscle power is achieved by a constant sinusoidal motion. Both are challenging sports to begin with because learning the complex up and down movements takes time, skill and reflexion. The interplay of rotating joints and applying forces at the right time is hard to perceive, understand and to transfer into muscle memory.

Since the motions in skateboarding on pump tracks and hydrofoil pumping are similar, we are comparing both motion sequences with inertial measurement units and 3D pose estimation. We postulate that learning the physically challenging and expensive hydrofoil pumping can be improved and accelerated by training with skateboards. Therefore, we are capturing forces with inertial measurement units and validate them with 3D pose estimation. Finally, we are comparing and visualizing the motions and forces of the boards and the skeleton to show the similarities within the y- and z-momentum.

Tablet computers like the iPad are user-friendly and reliable interaction devices for displaying XR content in museums. Although they are lacking the stereo view of a VR headset, they also avoid the hygiene, optics and usability challenges of head mounted displays in public spaces. In our paper we describe methods to improve the shortcomings of these devices: Placement, presentation and distracting branding. Our solution is a combination of a physical hanging system and custom designed acrylic cases disguising the hardware. In our example of an exhibition about a baroque architect the result is a series of floating baroque picture frames which users are grabbing and rotating to view into the XR scenes.

Tablet computers like the iPad are user-friendly and reliable interaction devices for displaying XR content in museums. Although they are lacking the stereo view of a VR headset, they also avoid the hygiene, optics and usability challenges of head mounted displays in public spaces. In our paper we describe methods to improve the shortcomings of these devices: Placement, presentation and distracting branding. Our solution is a combination of a physical hanging system and custom designed acrylic cases disguising the hardware. In our example of an exhibition about a baroque architect the result is a series of floating baroque picture frames which users are grabbing and rotating to view into the XR scenes.

In our project, we aimed to create historically authentic and vivid virtual representations of historic personalities that are connected to the regional Fichtelgebirge (Bavaria, Germany) to support the storytelling of our immersive XR applications. We are describing the tools in detail, the process of the tool chain and the resulting media. Next, we are discussing the challenges in media production like historical correctness and the consultation of historians. In order to create visual reproducibility we are explaining the detailed text prompts, their limitations and how to cope with resulting errors of the human physiognomy. Finally, we are briefly describing the application of the animated and talking generated historic characters in an immersive interactive WebXR environment. The XR experience is presented in web browsers on smartphones, tablets and XR headsets and the underlying software is based on the open-source framework Aframe. Our paper will describe the process, the results and the limitations in detail. Furthermore, we will provide a flow chart of the tool pipeline with visual examples of these aspects. The animations and voices of the historic characters will be demonstrated in videos of the XR application.

Surfing is a complex sport that encounters the interaction between the surfer and the highly unsteady water hydrodynamic of ocean or river waves. With the availability of off-shore river waves or wave pools, it becomes more and more popular being included in the Olympic Games 2021 in Tokyo/Japan. To analyze the biomechanical characteristics of surfers, all forces that act on the surfer have to be measured which has not yet been done before.

Four pressure sensors were integrated water-tightly in a 3D-printed surfboard fin, each two sensors on opposite side. The fin was mounted at the rear end of a surf board on the left side of the centerline within a three-fin-configuration. The sensors were connected to a micro controller with micro SD-card within the board that performed and stored the measurement data with an acquisition rate of 27 Hz. For the measurements, a male surfer performed a set of turning maneuvers on an artificial river wave as shown in figure 2. Thereby, the surfer’s motion was recorded by a camera using frame-rate of 30 fps. The synchronization of the pressure and the video signals was achieved in a post-processing step.

The pressure signals showed a negative pressure difference between the inside sensors and the outside sensors at all surf maneuvers, see figure 3. It generated an outwardly directed force that becomes largest during the turns. The smallest pressure difference was detected on the surf-path from the left to right side of the wave or the other way around.

Taking both lateral fins into account, the pressure forces are contrarily directed apart from the surfboard. This arrangement generates the stabilizing effect of the board’s rear end which helps the surfer to keep the board on the desired track. Especially during the turns, the forces became largest giving the surfer a stable center of rotation at the rear end of the board. This stabilizing effect of surfboard fins was reported and observed in computational models [1,2,3]. The measurement of hydrodynamic forces enables to develop a biomechanical model of a surfer that show the characteristic biomechanic stress level peak and location within the surfer’s body.

Our proposed visual dashboard aims to facilitate training progress evaluation in sports like skateboarding and foil pumping by offering real-time sensor and camera data processing through machine learning-based pose estimation. The web-based application, developed in HTML, CSS, and JavaScript utilizing p5js and Google Charts frameworks, enables athletes to visualize and analyze their movements for efficient and safe performance without the need for app installation on various devices. This paper will delve into the advantages and challenges of the web-based approach, detailing data structures, synchronization techniques, and plans for future enhancements like Web BLE connectivity and improved user experience across different devices.

The technologies regarding capturing motion and visualizing data enabled new possibilities describing complex systems to a broader audience. We used the latest methods of motion analysis through Machine Learning / Artificial Intelligence and visualization in Virtual and Augmented Reality (VR/AR) in order to analyze and explain the way the Hof Symphony Orchestra works. Therefore, we showed the diverse aspects from rehearsal to performance to a broad audience via interactive immersive Extended Reality experiences. In this paper we are describing the process and the implications for tracking motion and gestures in cultural places. 

Understanding acceleration forces and making progress in learning Skateboarding is a process of trial and error. In our paper we are describing our preliminary experiments for describing the complex interactions while pushing for speed in ramps and pump tracks. Therefore, we capture and visualize the body movement, the joint relations from hip to ankle and the resulting forces by joining iner- tial sensors on the skateboard and camera-based machine learning pose estimation of the athlete.

In our society, displays are becoming increasingly prevalent. While it is nearly inconceivable to imagine a daily life without screens, scientific research indicates that our mental well-being is nega- tively affected by the increasing use and time spent on screens. The "HomeGrid" project has been developed as a smart home concept, aiming to explore how traditional screens in private environments can be reduced while still effectively conveying information in an intuitive manner. To achieve this, two main approaches were em- ployed. Firstly, experimentation was conducted on the capture and communication of artificial light, and secondly, precise monitoring and visualization of indoor air quality were explored. These factors are fundamental indicators of both our mental and physical well- being, as high air quality, for example, enhances concentration, and light serves as a crucial regulator of our circadian rhythm.

The project Symotiv’s (http://symotiv.de) goal is to describe the complex function of a symphony through an interactive experience of the orchestra’s digital twin. We are capturing the musicians, their instruments and their sounds with machine learning based 3D human pose estimation and visualize them in Virtual Reality in a three-dimensional, virtual concert hall. Our audience steps into the role of the conductor and experience the scene and the sound from a perspective within the orchestra. This opens the possibility of experiencing and understanding what is happening from new spatial, but above all sonic perspectives.

In this paper we are describing our preliminary studies and proto- types for evaluating a surfer’s performance on a stationary wave. We are briefly describing the sport’s environment and development and the current state-of- the-art of machine learning based single camera tracking approaches we have evaluated and applied. The main part of the paper deals with the first implemen- tation, the tracking and the results of the evaluation of the movements of two surfers. We are closing with our lessons learned and our next steps.

The aim of the project Symotiv (http://symotiv.de) is to introduce people to classical music through interaction with a virtual orchestra. With the help of Virtual Reality, it does not only enable to immerse oneself in a three-dimensional, virtual concert hall, but also to experience images and sound from the perspective of a musician. This opened the possibility of experiencing and understanding what is happening from new spatial, but above all sonic perspectives. Therefore, we developed a workflow to capture the motions and sound of 50 musicians and the conductor by using a camera-based Machine Learning approach.

In this work we are describing our preliminary experimental evaluation of open-source machine learning based pose estimation solutions for tracking a surfer’s skeleton on a stationary wave. We are briefly describing the sport’s environment and development and the current state-of-the-art of machine learning based single camera 2D and 3D pose estimation solutions. The main part of the paper deals with the experiment’s setup and the interpretation of the resulting data of the movements of one surfer. We are closing with our lessons learned and our next steps.