|
| 1 | +--- |
| 2 | +title: A New Approach to Local Multiplayer / Splitscreen Perspective With Raytracing |
| 3 | +date: 2023-12-31 17:30:00 +0100 |
| 4 | +categories: [Programming, Game Development] |
| 5 | +tags: [programming, raytracing, co-op, camera] |
| 6 | +author: petert |
| 7 | +img_path: /assets/2023-12-31-Ray-Coop-Camera |
| 8 | +--- |
| 9 | + |
| 10 | +# Motivation |
| 11 | + |
| 12 | +I'm a huge fan of local multiplayer, and in particular local cooperative multiplayer. |
| 13 | +I'm *also* interested in 3D rendering, and especially raytracing. |
| 14 | + |
| 15 | +This blog post is the result of me **finally** -- barely scraping into 2023 -- writing about an idea I had in 2019, |
| 16 | +and which I implemented as a prototype in 2022. It uses the unique strengths of fully raytraced realtime |
| 17 | +rendering to implement a new kind of screen sharing for local multiplayer. But before that, some history. |
| 18 | + |
| 19 | +> If you're not interested in the history lesson, then you can just skip directly to [the result](#the-result). |
| 20 | +> I swear that the history part is also interesting though! |
| 21 | +{: .prompt-tip } |
| 22 | + |
| 23 | +## Split-Screen and Shared-Screen History |
| 24 | + |
| 25 | +One of the more significant challenges with implementing any kind of local coop is the fact that all players share a single screen. |
| 26 | +Ideally, a game can be designed in such a way to make use of one shared perspective, |
| 27 | +and I generally feel like these games work best for local multiplayer. |
| 28 | +But often, particularly for 3D games, working with a single perspective is not feasible. |
| 29 | + |
| 30 | +{: width="50%" #fancyimg } |
| 31 | +_Traditional Split-Screen Gameplay in Halo:Infinite_ |
| 32 | + |
| 33 | +The traditional solution to this conundrum is splitting the screen into N separate areas for N players, |
| 34 | +which are usually equally sized. Each shows a distinct perspective, and -- in 3D rendering terms -- has its own camera. |
| 35 | +Some games have developed this idea further, either visually or even by integrating it with gameplay. |
| 36 | +In the following I'll go through a few examples of games that I am aware of which did something interesting |
| 37 | +within a single-screen, multi-camera framework. |
| 38 | + |
| 39 | +### Renegade Ops -- Dynamic Splitting |
| 40 | + |
| 41 | +The 2011 co-op twin-stick shooting action game [Renegade Ops](https://store.steampowered.com/app/99300/Renegade_Ops/) |
| 42 | +was the first time I encountered a system which varies the screen layout in multiplayer |
| 43 | +based on the relative position on the players in world space. |
| 44 | + |
| 45 | +{: width="50%" #fancyimg } |
| 46 | +_Renegade Ops' Dynamic Split-Screen Implementation_ |
| 47 | + |
| 48 | +In the game, each player controls a vehicle, and when they are relatively close the screen is shared. |
| 49 | +If they move further away from each other, at some point the game switches over to two separate camera perspectives, |
| 50 | +which are split along a line that responds to the relative positioning. I.e. if the players were to be at the exact same |
| 51 | +postion on the horizontal axis, the line would be perfectly vertical, and if they were on the same vertical position, |
| 52 | +the split line would be perfectly horizontal. Any offset will result in a slanted line, as seen in the screenshot above. |
| 53 | + |
| 54 | +Several more games with a top-down or isometric camera have had this type of dynamic visual style for split-screen since |
| 55 | +Renegade Ops. If you know of anything that does something similar earlier than it, do let us know. |
| 56 | + |
| 57 | +> In internal PH3 proofreading, it turned out that Lego Indiana Jones 2 used dynamic split-screen, and released in 2009. |
| 58 | +> There might well be an even earlier example! |
| 59 | +{: .prompt-info } |
| 60 | + |
| 61 | +### DYO -- Splitscreen as a Gameplay Mechanic |
| 62 | + |
| 63 | +[DYO](https://store.steampowered.com/app/791470/DYO/) is an indie co-op puzzle platformer released in 2018. |
| 64 | +It implemented an idea which, to me, is nothing short of genius: instead of seeing the split-screen as a "necessary evil" |
| 65 | +required to allow two players with individual perspectives on a single screen, it instead re-conceptualized the split as |
| 66 | +a core gameplay mechanic. |
| 67 | + |
| 68 | +{: width="50%" #fancyimg } |
| 69 | +_DYO Uses its Split Screen as a Gameplay Mechanic_ |
| 70 | + |
| 71 | +In the game, players are given control over various aspects related to how and when the screen is split, and can e.g. |
| 72 | +arrange both halves to make a previously impassable obstacle traversable. |
| 73 | + |
| 74 | +> DYO is now [free on Steam](https://store.steampowered.com/app/791470/DYO/), |
| 75 | +> so do check it out if this sounds at all interesting to you! |
| 76 | +{: .prompt-tip } |
| 77 | + |
| 78 | +### Degrees of Separation -- Why Not Both? |
| 79 | + |
| 80 | +2019's [Degrees of Separation](https://store.steampowered.com/app/809880/Degrees_of_Separation/) |
| 81 | +takes both Renegade Ops' dynamic split and DYO's screen-split-as-gameplay and integrates |
| 82 | +them into a brilliant puzzle platformer. |
| 83 | + |
| 84 | +{: width="50%" #fancyimg } |
| 85 | +_Degrees of Separation is Mechanically Interesting, and it's also Beautiful_ |
| 86 | + |
| 87 | +Each of Degrees of Separation's levels introduces a new unique use of dynamic splitscreen as a mechanic, |
| 88 | +and each of those mechanics is in turn explored over many puzzles with quite some depth. |
| 89 | + |
| 90 | +### Ys IX -- Just Don't Split It |
| 91 | + |
| 92 | +Overall, I'm not a huge fan of split screen gameplay, when it is not used in a really clever way as in the games above. |
| 93 | + |
| 94 | +One option which I think is underutilized is simply not splitting the screen, even in fully 3D games. This is how the |
| 95 | +coop modes I implemented for both Ys VIII an Ys IX work. A single 3D camera does its best to keep both players in frame, |
| 96 | +and both players have the shared ability to adjust it. The video below some gameplay using that approach: |
| 97 | + |
| 98 | +<p align="center"><iframe id="fancyimg" width="560" height="315" src="https://www.youtube.com/embed/SNgDIFRG7S8?si=4pviUxcRhcUMSkYr" title="YouTube video player" frameborder="0" allow="accelerometer; encrypted-media; gyroscope; web-share" allowfullscreen></iframe></p> |
| 99 | + |
| 100 | +Of course, this is not without tradeoffs: even with quite a lot of time invested into tweaking the camera, |
| 101 | +it still takes getting used to and good coordination between both players, and there are situations in which it can |
| 102 | +fail to show one or even both actual player characters. However, I know that we and others fully completed these games, |
| 103 | +and the shared camera provides a unique gameplay feeling, so I wish someone else would also give it a try at some point. |
| 104 | + |
| 105 | +> You can read more about local coop in Ys IX [in this Steam post](https://store.steampowered.com/news/app/1351630/view/3100164020394220707) |
| 106 | +> announcing its release. |
| 107 | +{: .prompt-info } |
| 108 | + |
| 109 | +# The Idea |
| 110 | + |
| 111 | +There is one thing that all implementations of split screen gameplay I am aware of at this point have in |
| 112 | +common: regardless of how the screen is divided -- either traditionally in a fixed distribution or |
| 113 | +dynamically -- it is still ultimately just a split into two or more standard camera perspectives, with |
| 114 | +one alloted to each player. |
| 115 | + |
| 116 | +## Technical Background: Rasterization Limitations |
| 117 | + |
| 118 | +This universal truth is not surprising. Realtime rendering in games -- at least until the advent of |
| 119 | +HW-accelerated raytracing -- was based on rasterization, and |
| 120 | +the traditional rasterization pipeline assumes that there is one consistent camera projection across the |
| 121 | +entire screen. In order to accomplish split screen rendering in 3D games, the traditional cross-platform |
| 122 | +approach required basically performing almost the entire work of rendering a frame again, once for each |
| 123 | +perspective. |
| 124 | + |
| 125 | +A technology I'm also quite interested in, but which is unrelated to the topic of this post, actually |
| 126 | +brought some improvements in this respect: **VR**. VR rendering requires two perspectives, one per eye, |
| 127 | +or ideally even more for large-FoV displays. This motivated GPU vendors to provide improved support |
| 128 | +for this use case, though I am not sure if many (or any?) split-screen games actually leverage such |
| 129 | +hardware support. |
| 130 | + |
| 131 | +> NVidia has a thorough and very well-illustrated technical blog post about the VR-related multi-view |
| 132 | +> rendering topic [here](https://developer.nvidia.com/blog/turing-multi-view-rendering-vrworks/). Give |
| 133 | +> it a read if you want to learn more. |
| 134 | +{: .prompt-tip } |
| 135 | + |
| 136 | +However, even with the most advanced hardware support, only up to 4 different views can be rendered at |
| 137 | +once without substantial performance implications -- and even that requires quite a lot of development |
| 138 | +effort. |
| 139 | + |
| 140 | +## The Potential of Raytracing |
| 141 | + |
| 142 | +Raytracing changes these fundamental rules, and essentially eliminates all of these limitations. |
| 143 | +Somewhat simplified, every pixel is generated by shooting an individual ray into the scene and seeing |
| 144 | +what it hits. As such, we can in theory use a different direction and position for every single one of |
| 145 | +these rays. And that's where we can finally get to the idea this is all about: |
| 146 | + |
| 147 | +<p style="font-size: 1.5em; font-weight: bold; font-style:italic"> |
| 148 | +Rather than having entirely distinct and clearly delineated perspectives for each player, dynamically |
| 149 | +adjust the camera(s) at a pixel level. |
| 150 | +</p> |
| 151 | + |
| 152 | +This allows us to not only use an arbitrary distribution of screen space to various players' perspectives, |
| 153 | +but even smoothly interpolate between their perspectives! |
| 154 | + |
| 155 | +> Note that basically no mainstream game which uses raytracing today actually does so for the so-called |
| 156 | +> primary or initial rays (mostly since rasterization is just more effective). It's certainly possible to |
| 157 | +> do so though, as you'll see shorty. |
| 158 | +{: .prompt-info } |
| 159 | + |
| 160 | +# The Result |
| 161 | + |
| 162 | +Without further ado, here's what that could look like: |
| 163 | + |
| 164 | +<p align="center"><iframe id="fancyimg" width="640" height="360" src="https://www.youtube.com/embed/O_LH6aGUzbY?si=dkKKrmzicJlHNOYV" title="YouTube video player" frameborder="0" allow="accelerometer; encrypted-media; gyroscope; web-share" allowfullscreen></iframe></p> |
| 165 | + |
| 166 | +This is a short video showing a basic implementation of this idea. |
| 167 | +As you can see, the scene starts with a shared perspective, and when the two spheres (controlled by the |
| 168 | +two players) move further away from each other, the screen is smoothly split into separate areas for each |
| 169 | +of them. The most interesting part is what happens in the "no-man's land" between, where space appears |
| 170 | +squished as the camera parameters are interpolated. |
| 171 | + |
| 172 | +In case you're wondering about the game: what you're seeing (for the first time) is *Sphere Spectacle*. |
| 173 | +It's something we worked on shortly after the release of the first RTX-series GPUs, so around 2019. The |
| 174 | +idea, at a gameplay level, is to basically be a ball-rolling game. At a technical level, we wanted to |
| 175 | +build something that is **entirely** raytraced, with absolutely no rasterization. We also didn't want to |
| 176 | +use any images/textures at all in the game, so everything is just geometry and shaders. |
| 177 | + |
| 178 | +# Implementation |
| 179 | + |
| 180 | +I won't go into too much detail here, but the most interesting part of the implementation is how to |
| 181 | +actually decide to split the screen. I experimented with a few options here. Mathematically, the problem |
| 182 | +seems similar to trying to draw a Voronoi diagram with the players on-screen positions as the points, |
| 183 | +and then setting some value for the width of the interpolation. I tried making that work, but ended up |
| 184 | +reverting to a more ad-hoc solution. |
| 185 | + |
| 186 | +What you saw in the video is the result of me trying a few different formulas and sticking with something |
| 187 | +that looks vaguely like what I wanted, but there is a wealth of options to explore here. I ultimately |
| 188 | +only spent two working days hacking this demo into Sphere Spectacle -- and that includes implementing |
| 189 | +support for two players! |
| 190 | +As one might expect from that, the code is quite unhinged. The most interesting core aspect is here, |
| 191 | +in this excerpt of the ray generation shader: |
| 192 | + |
| 193 | +{: width="50%" #fancyimg } |
| 194 | + |
| 195 | +I still think it's remarkable that I could get this entire idea working, at least to some extent, in |
| 196 | +just two days of hacking, and it shows how it is fundamentally so much simpler than trying to do |
| 197 | +anything like it in a rasterization pipeline. |
| 198 | + |
| 199 | +# Conclusion |
| 200 | + |
| 201 | +If you've read until this point then thanks for coming with me on this journey through the history |
| 202 | +of local multiplayer camera perspectives and my idea on having it take a new step based on raytracing. |
| 203 | + |
| 204 | +Now we just need someone to build a full game around it! |
0 commit comments