OSPRay Studio: Immersive OSPRay

.gitignore

@@ -8,4 +8,6 @@
 !/plugins
 /plugins/*
 !/plugins/example_plugin
+!/plugins/gesture_plugin
+!/plugins/storyboard_plugin
 doc/*.html

README.md

@@ -1,168 +1,185 @@
-# OSPRay Studio
+# Immersive OSPRay Studio
+> This project is part of a larger project called [Immersive OSPray](https://github.com/jungwhonam/ImmersiveOSPRay).
 
-This is release v0.11.1 of Intel® OSPRay Studio. It is released under the
-Apache 2.0 license.
+We extend [OSPRay v2.10.0](https://github.com/ospray/ospray/releases/tag/v2.11.0) to display a single, coherent 3D virtual environment on tiled display walls and use gesture-based interaction techniques to navigate the environment. We provide another mode of running the application with the ability to open multiple windows and coordinate these windows (see [MULTIWINDOWS Mode](#multiwindows-mode)). We provide gestured-based interaction techniques by integrating a separate server application sending tracking user data to a plugin to the rendering application (see [Gesture Plugin](#gesture-plugin)).
 
-Visit [**OSPRay Studio**](http://www.ospray.org/ospray_studio)
-(http://www.ospray.org/ospray_studio) for more information.
-
-See [what's
-new](https://github.com/ospray/ospray_studio/blob/master/CHANGELOG.md)
-in this release.
-
-## Overview
-
-Intel OSPRay Studio is an open source and interactive visualization and
-ray tracing application that leverages [Intel OSPRay](https://www.ospray.org)
-as its core rendering engine. It can be used to load complex scenes requiring
-high fidelity rendering or very large scenes requiring supercomputing resources.
-
-The main control structure is a *scene graph* which allows users to
-create an abstract scene in a *directed acyclical graph* manner. Scenes
-can either be imported or created using scene graph nodes and structure
-support. The scenes can then be rendered either with OSPRay's pathtracer
-or scivis renderer.
-
-More information can be found in the [**high-level feature
-description**](https://github.com/ospray/ospray_studio/blob/master/FEATURES.md).
-
-Building OSPRay Studio
-========================
-
-CMake Superbuild
-----------------
+# Build and Run
+## CMake configuration and build
+```
+git clone https://github.com/jungwhonam/ospray_studio.git
+cd ospray_studio
 
-### Required dependencies for superbuild
+git checkout v0.12.0-alpha.x
 
--   [CMake](https://www.cmake.org) (v3.15+) and any C++14 compiler
+mkdir build
+cd build
+mkdir release
 
-For convenience, OSPRay Studio provides a CMake Superbuild script which will
-pull down its dependencies i.e. GLFW, OSPRay, rkcommon and TBB. It builds OSPRay
-Studio without OpemImageIO and OpenEXR support.  `stb_image` is used for all
-image operations by default instead. 
+cmake -S .. \
+-B release \
+-DCMAKE_BUILD_TYPE=Release \
+-DUSE_PYSG=OFF \
+-DUSE_MPI=ON \
+-DBUILD_PLUGINS=ON \
+-DBUILD_PLUGIN_GESTURE=ON \
+-Dospray_DIR="/Users/jnam/Documents/Test/ospray/build/release/install/ospray/lib/cmake/ospray-2.10.0"
 
-To use the superbuild run with:
+cmake --build release -- -j 5
 
-``` sh
-mkdir build
-cd build
-cmake ..
-cmake --build .
+cmake --install release
 ```
+OSPRay Studio needs to be built with ```-DUSE_MPI=ON```, ```-DBUILD_PLUGINS=ON```, and ```-BUILD_PLUGIN_GESTURE=ON``` in CMake. Also, we need to use [OSPRay we have customized](https://github.com/jungwhonam/ospray/tree/v2.11.0-alpha.x). After building the OSPRay, set ```ospray_DIR``` so CMake can locate OSPRay, e.g., ```/Users/jnam/Documents/GitHub/ospray/build/install/ospray/lib/cmake/ospray-2.10.0```.
 
-For other full set of options, run:
-
-``` sh
-ccmake ..
+## Run the application
+```
+mpirun -n 3 \
+./ospStudio \
+multiwindows \
+--mpi \
+--displayConfig config/display_settings.json \
+--scene multilevel_hierarchy \
+--plugin gesture \
+--plugin:gesture:config config/tracking_settings.json
 ```
 
-or
+```multiwindows```: This option activates our custom mode.
 
-``` sh
-cmake-gui ..
-```
+```--mpi```: This option enables the OSPRay Studio's built-in MPI support, which is a required dependency of our custom mode.
 
-Standard CMake build
---------------------
+````--displayConfig config/display_settings.json````: The JSON configuration file contains information about off-axis projection cameras and windows. Information in the file is used to position and scale windows. See [Display Configuration JSON File](https://github.com/jungwhonam/ConfigurationGenerator#display-configuration-json-file) for details on the JSON file.
 
-For standard cmake process turn off cmake option `OSPRAY_INSTALL` and provide
-following required dependencies with their respective cmake options as will be
-listed in OS-specific building process below. 
+```--scene multilevel_hierarchy```: This option starts the application with the scene opened (optional).
 
-### Required dependencies
+```--plugin gesture```: This option starts the application with the gesture plugin.
 
--   [CMake](https://www.cmake.org) (v3.15+) and any C++14 compiler
--   Intel [OSPRay](https://www.github.com/ospray/ospray) (v2.10.0) and its
-    dependencies - OSPRay Studio builds on top of OSPRay. Instructions on
-    building OSPRay are provided
-    [here](http://www.ospray.org/downloads.html#building-and-finding-ospray).
-    OSPRay and OSPRay Studio have the following common dependencies which Studio
-    can hence leverage from an OSPRay build.
-    -   Intel oneAPI Rendering Toolkit common library
-        [rkcommon](https://www.github.com/ospray/rkcommon) (v1.10.0)
-    -   Intel [Threading Building Blocks](https://www.threadingbuildingblocks.org/)
--   OpenGL and [GLFW](https://www.glfw.org) (v3.3.4) - for the windowing environment
+```--plugin:gesture:config config/tracking_settings.json```: The JSON configuration file contains information about the gesture tracking server and user tracking data. Gesture Plugin uses this file. See [Implementation details](#implementation-details) for details on the JSON file.
 
+> See [example-config](/example-config/) for example JSON files.
 
-### Optional Dependencies
+# MULTIWINDOWS Mode
+OSPRay Studio provides different modes of running the application. We added another mode called ```MULTIWINDOWS```; the mode is similar to the default ```GUI``` mode with these additional features: 1) Position and scale windows based on MPI ranks and 2) Synchronize MPI processes.
 
--   Intel [Open Image Denoise](https://openimagedenoise.github.io) - (v1.4.3 or
-    newer) for denoising frames. To use with OSPRay Studio, OSPRay must be built
-    with `-DBUILD_OIDN=ON` in CMake.
--   [OpenVDB](https://www.openvdb.org/) to support loading VDB formatted volume files.
--   [OpenImageIO](http://openimageio.org/) and [OpenEXR](https://www.openexr.com/)
-    (pre-3.x versions) to support images in a variety of file formats.  Set `OPENIMAGEIO_ROOT`
-    and `OPENEXR_ROOT` to the respective install directories to use these libraries.
-    (tested with OpenImageIO v2.3.16 and OpenEXR v2.5.8)
--   [Python] (3.9.7) (https://python.org) for python bindings
+> See ```app/MultiWindows.cpp```. To implement the mode, we copied and modified ```app/MainWindow.cpp```. 
 
-### Building on Linux and macOS
+### 1. Position and scale windows based on MPI ranks
+This new OSPRay Studio mode takes a command line option, ```--displayConfig```, which points to a JSON configuration file that specifies windows and off-axis cameras. At the start of the application, the JSON file is loaded, and values get stored in a JSON object ```nlohmann::ordered_json configDisplay```. Positioning and scaling GLFW windows are done in a constructor.
 
--   Follow OSPRay's build instructions to install it, which will also
-    fulfill most other required dependencies. Set the following
-    environment variables to easily locate OSPRay and
-    rkcommon during CMake.
+> See [Display Configuration JSON File](https://github.com/jungwhonam/ConfigurationGenerator#display-configuration-json-file) for details on the JSON file.
 
-    
+> See ```void MultiWindows::addToCommandLine(std::shared_ptr<CLI::App> app)``` for implementation.
 
-    ``` bash
-    export ospray_DIR = ${OSPRAY_INSTALL_LOCATION}
-    export rkcommon_DIR = ${RKCOMMON_INSTALL_LOCATION}
-    export TBB_DIR = ${TBB_INSTALL_LOCATION}
-    ```
+### 2. Synchronize MPI processes
+We take additional steps to run these multiple processes in a synchronized fashion. After processing user inputs, the master process updates values in a sharing object. 
+> Currently we only synchronize a camera location and a closing status across processes.
+```
+while (true) {
+
+    ...
+
+    // poll and process events
+    glfwPollEvents();
+    if (sg::sgMpiRank() == 0) {
+        // poll and process events from the server
+        for (auto &p : pluginPanels)
+            p->process("update");
+
+        // update the shared state
+        sharedState.camChanged = true;
+        sharedState.transform = arcballCamera->getTransform();
+        sharedState.quit = glfwWindowShouldClose(glfwWindow) || g_quitNextFrame;
+    }
+    MPI_Barrier(MPI_COMM_WORLD);
+}
+```
 
-    Alternatively, [CMAKE_PREFIX_PATH](https://cmake.org/cmake/help/latest/variable/CMAKE_PREFIX_PATH.html)
-    can be set to find the OSPRay install and other dependencies.
+Then, at the beginning of the next frame, the object is broadcast to other processes, and each process updates its objects and application states based on the shared values.
 
--   Clone OSPRay Studio
+```
+while (true) {
+    MPI_Bcast(&sharedState, sizeof(sharedState), MPI_BYTE, 0, MPI_COMM_WORLD);
+
+    { // process changes in the shared state
+      if (sharedState.quit) {
+        break;
+      }
+
+      if (sharedState.camChanged) {
+        auto camera = frame->child("camera").nodeAs<sg::Camera>();
+        camera->child("transform").setValue(sharedState.transform);
+        camera->child("topLeft").setValue(xfmPoint(sharedState.transform, topLeftLocal));
+        camera->child("botLeft").setValue(xfmPoint(sharedState.transform, botLeftLocal));
+        camera->child("botRight").setValue(xfmPoint(sharedState.transform, botRightLocal));
+
+        sharedState.camChanged = false;
+      }
+    }
+
+    ...
+}
+```
 
-    ``` bash
-    git clone https://github.com/ospray/ospray_studio/
-    ```
+Also, to ensure windows display rendering results simultaneously, processes wait for others to complete the rendering processes before swapping buffers. This is done by calling ```waitOnOSPRayFrame()``` and ```MPI_Barrier(...)``` before ```glfwSwapBuffers(...)```.
 
--   Create build directory and change directory to it (we recommend
-    keeping a separate build directory)
 
-    ``` bash
-    cd ospray_studio
-    mkdir build
-    cd build
-    ```
 
--   Then run the typical CMake routine
+# Gesture Plugin
+<div id="image-table">
+  <table>
+    <td style="padding:4px">
+      <img src="GesturePlugin-UI.png" width="400"/>
+    </td>
+    <td style="padding:4px">
+      <img src="GesturePlugin-System.png" width="680"/>
+    </td>
+  </table>
+</div>
 
-    ``` bash
-    cmake -DCMAKE_CXX_COMPILER=clang++ -DCMAKE_C_COMPILER=clang ... # or use ccmake
-    make -j `nproc` # or cmake --build .
-    ```
+The plugin handles the connection with [Gesture Tracking Server](https://github.com/jungwhonam/GestureTrackingServer), computes gestures from received data, and keeps track of the latest state. When the plugin receives a message from the server, it derives additional information from the body tracking data. The underlying scene is not updated immediately; OSPRay Studio initiates updating the scene from the latest tracking data. When the application is in the phase of processing user inputs, e.g., key-pressed events, it calls a poll event method from the plugin to get the latest tracking result and uses the result to update corresponding 3D objects, e.g.,  changing camera locations. 
 
--   To run OSPRay Studio, make sure `LD_LIBRARY_PATH` (on Linux) or
-    `DYLD_LIBRARY_PATH` (on macOS) contains all dependencies. For
-    example,
+> See codes under ```plugins/gesture_plugin/tracker```.
 
-    ``` bash
-    export LD_LIBRARY_PATH=${OSPRAY_INSTALL}/lib64:...:$LD_LIBRARY_PATH
-    # then run!
-    ./ospStudio
-    ```
+## GUI
+The plugin panel can be opended by clicking ```Plugins/Gesture Panel``` in the menu. As shown in the left figure, the top pane shows information about the server and has a button for starting and closing the connection. ``Configuration`` pane provides options to modify body tracking data received from the server. ``Save`` button saves the current values to a JSON file (the application reads the values at the start). ``Status`` pane shows important updates, e.g., indicating whether a server is connected.
 
-### Building on Windows
+## Implementation details
+<!-- Configuration File -->
+At the start, the information about the socket, e.g., IP address and port number, is read from a JSON configuration file. The file also contains information about how to process the user tracking data.
 
-Use CMake (cmake-gui) to configure and generate a Microsoft Visual
-Studio solution file for OSPRay Studio.
+```
+{
+    "ipAddress": "127.0.0.1",
+    "portNumber": 8888,
+
+    "scaleOffset": [0.001, -0.001, -0.001],
+    "translationOffset": [0.0, -0.1, 1.19],
+    "confidenceLevelThreshold": 1,
+    "leaningAngleThreshold": 1.0,
+    "leaningDirScaleFactor": [1.0, 1.0, 1.0]
+}
+```
+- the first two key/value pairs are information about the gesture tracking server.
+- ```multiplyBy``` is multiplied to position values of joints. This process is needed as Kinect and OSPRay are in two different coordinate systems. 
+- ```positionOffset``` are used to offset the sensor's center. The offset is applied to calibrate the sensor and displays. 
+- ```leaningAngleThreshold``` is a threshhold for activating the flying mode. When a user's body is leaning more than the angle, the flying mode is activated. 
 
--   Specify the source folder and the build directory in CMake
--   Specify `ospray_DIR`, `rkcommon_DIR` CMake
-    variables for the respective install locations
--   Click 'Configure' and select the appropriate generator (we recommend
-    using at least Visual Studio 15 2017)
--   Select x64 as an optional parameter for the generator (32-bit builds
-    are not supported)
--   Click 'Generate' to create `ospray_studio.sln`. Open this in Visual
-    Studio and compile
+<!-- async-sockets -->
+`async-sockets` receives body tracking data from the server. The implementation is based on [async-sockets](https://github.com/eminfedar/async-sockets-cpp) (using the version from the last commit on 2/21/2022).
 
-You can optionally use the CMake command line:
+<!-- TrackingManager -->
+```TrackingManager``` manages a socket connection and keeps track of the latest data from the server. In addition to providing methods for starting and closing the connection, the class keeps track of the latest data received from the server. The data can be accessed by calling ```pollState()```. When the method is called, the object that stores the latest information becomes empty, indicating the data has been used. In the plugin, the method is called in ```process(std::string key)```. 
 
-``` pwsh
-cmake --build . --config Release --target install
 ```
+void PanelGesture::process(std::string key) {
+  if (key == "update") {
+    TrackingState state = trackingManager->pollState();
+    if (state.mode == INTERACTION_FLYING) {
+      context->arcballCamera->move(state.leaningDir);
+    }
+  }
+  else if (key == "start") {
+    trackingManager->start();
+  }
+}
+```
+
+The manager class also figures out current gestures. When a message is received from the server, ```updateState(std::string message)``` is called to process the message. We compute a leaning direction and the current gesture mode in our current implementation.

app/ArcballCamera.cpp

@@ -83,6 +83,11 @@ void ArcballCamera::pan(const vec2f &delta)
   updateCamera();
 }
 
+void ArcballCamera::move(const rkcommon::math::vec3f &dir) {
+  translation = AffineSpace3f::translate(-dir) * translation;
+  updateCamera();
+}
+
 vec3f ArcballCamera::eyePos() const
 {
   return cameraToWorld.p;

app/ArcballCamera.h

@@ -113,6 +113,7 @@ class ArcballCamera
   void zoom(float amount);
   void dolly(float amount);
   void pan(const vec2f &delta);
+  void move(const vec3f &dir);
 
   vec3f eyePos() const;
   vec3f center() const;

app/CMakeLists.txt

@@ -15,6 +15,7 @@ add_executable(ospStudio
   Batch.cpp
   TimeSeriesWindow.cpp
   AnimationManager.cpp
+  MultiWindows.cpp
 
   $<$<BOOL:${USE_BENCHMARK}>:Benchmark.cpp>
 
@@ -80,12 +81,26 @@ install(TARGETS ospStudio
 
 if(OSPRAY_INSTALL)
   get_target_property(OSPRAY_LIBNAME ospray::ospray IMPORTED_LOCATION_RELEASE)
+  if (CMAKE_BUILD_TYPE MATCHES Debug OR CMAKE_BUILD_TYPE MATCHES DEBUG)
+    get_target_property(OSPRAY_LIBNAME ospray::ospray IMPORTED_LOCATION_DEBUG)
+  endif()
 
   string(REGEX MATCH "^.*/" _sharedlib_glob "${OSPRAY_LIBNAME}")
   string(APPEND _sharedlib_glob "*${CMAKE_SHARED_LIBRARY_SUFFIX}*")
   # message(STATUS "_sharedlib_glob: ${_sharedlib_glob}")
   file(GLOB _sharedlibs LIST_DIRECTORIES false "${_sharedlib_glob}")
   # message(STATUS "_sharedlibs: ${_sharedlibs}")
+
+  # get *_debug*.dylib* files from the tbb's lib directory
+  if (CMAKE_BUILD_TYPE MATCHES Debug OR CMAKE_BUILD_TYPE MATCHES DEBUG)
+    get_target_property(TBB_LIBNAME TBB::tbb IMPORTED_LOCATION_DEBUG)
+    string(REGEX MATCH "^.*/" _tbbLib_glob "${TBB_LIBNAME}")
+    string(APPEND _tbbLib_glob "*_debug*${CMAKE_SHARED_LIBRARY_SUFFIX}*")
+    file(GLOB tbb_sharedlibs LIST_DIRECTORIES false "${_tbbLib_glob}")
+    list(APPEND _sharedlibs ${tbb_sharedlibs})
+    unset(_tbbLib_glob)
+    unset(tbb_sharedlibs)
+  endif()
 
   if(WIN32)
     install(FILES