React ui implementation

[the-simian]

Use React for menus and overlays. This is fully aware there are limitations to this sort of UI, such as the inability to pur UI behind the canvas, but its great for menus and things like that. Many game can make ui faster with this approach.

• Semantic ui as the standard library see https://react.semantic-ui.com/introduction

• also the core tool used for create-phsaer-jetpack create-phaser-jetpack #17

• easier google fonts integration

  • we currently have a google fonts loader, right now that takes a simple list of fontnames

• react Vis for visualizations https://uber.github.io/react-vis/documentation/welcome-to-react-vis

  • use skill tree example in create-phaser-app

[the-simian]

Investigate https://github.com/koreezgames/phaser3-ninepatch-plugin when looking into 9-patch support

[aroman]

Love this idea! Some thoughts based on my experience using, then abandoning, and now re-investigating using React for menus/UI in a Phaser game.

So, I tried doing this in a phaser game, but ultimately found it to be more trouble than it was worth, particularly trying to reconcile CSS positioning with Phaser's scaling system (namely, keeping the aspect ratio fixed and scaling uniformly), and keeping Phaser's data (not reactive) in sync with React's preference for reactive data.

That said, I definitely found that laying out UI was a lot easier with CSS/React. Two big reasons I think: 1) flexbox and 2) reactive/data-driven updates. One of the most annoying things in Phaser for me has been sharing data across parts of the application, and balancing caching of data vs redrawing every frame. For example, consider drawing a health bar: in Phaser, I might draw the bar on every frame, but in React, I'd rather just have the health data be a state/prop attribute, so that the bar is only redrawn when the underlying data changes.

Actually, I'm reworking my game's UI right now, and I'm considering switching back to React. I want to do some weird things like having a video/trailer playing on the main menu, and I can't think of a good way to harmonize that with a pure-Phaser UI system (as I ended up adopting).

Do you have thoughts about how to integrate A) scaling and B) data b/w react and phaser?

[the-simian]

@aroman I just wanted to pop in and let you know I saw this. I'm working on the animation state machine at this exact moment. I really appreciate your thoughts, and I think you deserve a lengthier response. I'll respond when I can, yeah i do have some thoughts on those things.

[the-simian]

@aroman ,

Ok I wanted to take some time to reply with my thoughts on this. So firstly, I know that 'the proof is in the pudding' so I am aware that you're going to be skeptical about how well this will work until I actually have a sensible prototype working.

Ok that said there's a few things to keep in mind that I am fully aware about

• clearly you can't truly have an element go 'behind' a phaser element
• if you want to embed something in the html, you need a totally new instance of phaser, and managing state is very complex at this point
• I've seen this work with a game-dev shop in my area; using html+css (they used Vue.js)

CSS positioning with Phaser's scaling system
So lets unpack some of the issues. Firstly for something like a menu, where understanding the position of game elements doesn't matter is clearly easier. You can get the position of the 'game' and scale it with a linear scale to the window. this way any "black bars" stay there, and you use absolute positioning.

So answer to your hard part:
What I know can work is if you read the sprite x and y, and the accoutn for the camera scroll you can fairly easily predict the sprits position in the window if you use a simple linear scale. Let me show you what I mean.

in this game lets say I do this. In the main index.html file I add a little box, that I intend to 'attach' to my player. Maybe this is like.. a dialog box? A weapon select? A healthbar? you get the idea.

You add the box...

    <div id="experimental-popup" style="
        position: absolute; 
        width: 75px; 
        height: 75px; 
        background-color: rgba(0,0,0,0.2);
        border: 1px red solid; 
        color: white;
        display: flex;
        align-content: center;
        justify-content: center;
        align-items: center;
        justify-items: center;
        ">

      XXX

    </div>

Now you can get the element in the scene. I propose a more organized approach, but just to prove the scaling...

let xxx = document.getElementById('experimental-popup');

Great! Got a reference.

Now I need a very simple linear scale function.

function linearScale(domain, range) {
  return function(value) {
    if (domain[0] === domain[1] || range[0] === range[1]) {
      return range[0];
    }
    var ratio = (range[1] - range[0]) / (domain[1] - domain[0]),
      result = range[0] + ratio * (value - domain[0]);
    return result;
  };
}

Ok here's the fancy part.

const scaledX = linearScale([0, HEIGHT], [0, window.innerHeight]);
const scaledY = linearScale([0, WIDTH], [0, window.innerWidth]);

HEIGHT and WIDTH are for my game, defined in the conts.js Of course if I use a scale, you multiply times the scale too. This is a memoized function that translates the gamespace to htmlspace as an absolute position. Now you just read (when you want) off the sprite

    var left = scaledX(this.player.x - this.cameras.main.scrollX);
    var top = scaledY(this.player.y - this.cameras.main.scrollY);

    xxx.style.left = `${left}px`;
    xxx.style.top = `${top}px`;
    xxx.innerHTML = `<span>${top}, ${left}</span>`;

Here is this working in the game:

You can see the top-left point of the box is firmly anchored to the centroid of the sprite (yay)! Now with this we can simply offset, or do whatever we'd do normally if this any other webapp. (like if you had a div following your mouse or popping up on an html element like a tooltip or whatever). Like you could offset this Top and make a health bar, since you also know the height of the sprite!

anyways that's a proof of concept to 'read' off the game and put things in the right spot.

I think generally speaking for things that 'need to know' about the game? you're going to be using position: absolute and reading off the game. For performance its sensible to maybe throttle or do batched updates somehow. I can imagine some sort of system where you 'mark' sprites and make a phaser-html bridge so you can read right off that and everything is prescaled for you. You'd also want to recalculate the x/y if the windows was resized. I think this can be a great way to handle nearly all ui ... health bars, numbers, dialog..whatever.

Here's a quick and dirty 'health bar' experiment

I fully intend to integrate this approach into this boilerplate.

As for the menus, I imagine making an absolutely positioned 'box' sitting right on top of the phaser window that uses this approach to get the bounds, but therein you can use flexbox for its children. SO the top box will be explicitly set width height, top, left andposition absolute, but children can use flexbox or whatever. It should stay on top of the 'screen' part of the phaser game.

I should note there's a performance hit for all of this... the benefit is the nicer ui. I am also curious how this will work in practice

Data synchronization
I think for this, the short answer is that for critical state data for the game must live in an external store an propagate to phaser as well as the html;

Lets say that we had like a color picker that would change the color of something in the game?

The flow would need to resemble something like redux, or some way there is an external store.

So you'd click on your color picker (likely html), and lets say we got a redux-y setup? You click the color picker, and it updates the global state. this causes a recalculation in the ui, as well as on update or something in phaser you'd want to be reading this piece off the store, so it should update both. You would read in the store in the scene, and update the game object on update, just like any other object!

And of course, To go from phaser to the html you can also dispatch, just as you would from the html to the phaser app. The important thing is that 3rd store that feeds both things.

I havent jumped into these things just yet, but I plan to. Heres a few more things:

• I plan to use semantic ui react for a library
• I plan to roll a theme for that that makes sense with a game, so it will depart quite a lot from the base theme
• This isn't quite next on my list, I want to make sure its easy to mange in-game animation and some stuff like that first. Today I rolled out a utility to automatically handle tilemap extrusion. I think right after animation and the player entity, then I do this. So its not quite next, but very very close.

@vantreeseba might have more thoughts on this I think he's also had some success with Integrating HTML with Phaser, but you can see from my demo above its very possible!

[aroman]

Wow, thanks for the very detailed reply, Jesse! Apologies for the long-winded response below...

In general, definitely agree that it is possible to (relatively simply) reconcile the coordinate systems of Phaser and the Window. :-) I appreciate the demo. I'm sure that in practice there are a lot of tricky edge-cases to consider (hi-DPI displays, viewport scaling, etc). I think your prototype could probably be extended to reach reasonable parity with Phaser coordinates, and I can definitely see some wrappers/middleware so that one could say "I want to insert react component at (x,y) phaser-world coordinates." There's some interesting API ergonomics considerations there — would you create a "stub" Phaser game object (a sprite or container or something) that serves as a placeholder for the React component? Such that, as you suggested, you could position the React rectangle using Phaser-native positioning tools (anchoring, etc), the wrapper would actually do the scaling and position it absolutely, and then the API consumer would get some hook for a React component which magically positioned correctly in the react world.

Some specific thoughts/concerns:

1. Lag/position synchronization issues

Ensuring that the "reposition" function (that updates the absolute positioning of the React-in-Phaser components) is called exactly at the right time and frequency may be really tricky and can introduce lag. Is it called by monkeypatching phaser's 60fps (or whatever the actual rate is) update method? Is it called via a separate render loop? In your healthbar demo gif, when the character is falling, you can see some lag where the bar is redrawn. I wonder if this is because of where/when the function that updates the coordinates — if it's not updating in lockstep with phaser's own redraw loop, I'd imagine you'd have issues as we see in the gif. I think there's actually an even bigger concern here, which is that even if you had some code like this:

function update() {
  // Update all phaser object positions
  super.update()
  // Update all react object positions
  phaserReactBridge.components(component => {
    component.style.left = `${left}px`;
    component.style.top= `${top}px`;
  })
}

We don't really have direct control of when the browser recalculates and repaints, right? So even if we had the updates happening at the right time and place, we might still introduce some lag time. Perhaps this is what we're seeing in your gif, I wasn't sure where you were doing the update. Maybe we need requestAnimationFrame() for this?

2. What's the core value?

Is it actually desirable/necessary to use CSS/HTML as the backend for React? Or would it be better to (this is a little crazy, I know) instead of reconciling the coordinate system, create an alternative "backend" for react, replacing react-dom with e.g. react-phaser. This way, we could use nice declarative, reactive, component-oriented UI, but avoid all the potential sources for "sync issues" described above. Ah, but then we'd lose CSS, right? Well, is the real value of react+phaser declarative construction and composition (i.e. HTML), or declarative styling (i.e. CSS)? I don't have strong opinions here, I just wanted to try to unpack what's really the compelling value in trying to use react for UI in Phaser. Maybe what we should really build is some tool to let us style phaser components using CSS?

3. Data synchronization

Not so concerned about this :-) I agree that the approach here would be to use some external data store, probably something like redux. I actually think this is a much simpler problem than the rendering stuff. In fact I think I've seen some gists/blog posts where people actually advocate using redux (without react) for managing data in phaser.

3. Does this belong in create-phaser-app?

To me, a create-foo-app project generator is about being opinionated but also canonical. create-react-app doesn't bundle tachyons, redux, or any other possibly helpful but unnecessary frameworks. It's basically about being quick to get started with, designed for simple/prototype projects, and requiring minimal/zero configuration. If you actually want to configure it, you just eject and you're on your own.

Your bundling of an experimental (even a robust) react+phaser layer in create-phaser-app, imho, pivots it away from the values I described above. And hey, it's your project, you are free to do as you please. But if I were new to Phaser, I would not expect (and possibly be very confused by), a "get up and running quickly" generator that is integrated with a whole other complex framework/system (react, possibly + redux).

4. Summary

Anyway, that's my $0.02. In general, I think the current state of designing UI for Phaser games is crap, and there's clearly a need for something more declarative and easy. I'm not totally convinced that hooking up react-dom and phaser is the solution to that, or that such a system makes sense to bundle in a "get started with phaser easily!" project. But, again, it's your baby, and your call, and I will not be offended if you disagree :-)

5. My interest in this and create-phaser-app in general

FWIW, my personal interest in the project is that I'm trying to put together a similar project as create-phaser-app, except with a hardware focus, actually. Basically I want to create a simple system for helping people get started doing custom input/alt ctrl stuff with Phaser. I recently made a game using that setup, and I think it worked really well architecturally, and am trying to abstract it into a framework for others to use. I'm super behind on documentation, but I describe the architecture in an interview here if you're curious. I'd be super interested in using create-phaser-app as a component of/starting point for my project :)

[the-simian]

I'll answer a few quick points

Here' the answer to points 1, 2 and your second #3:

I think this is worth pursuing; I'll explain why.

I want someone to be able to pull this in and actually be able to build a pretty big chunk of the things they will need for their game. Right now, you can do a lot with HTML 5 games, but a problem is tooling sprawl. One of the things that makes a generator or scaffold good is that it can allow you to quickly get going and encapsulate good practices. Another point of a scaffold or generator is to work fast. I suppose I could stop even now and say 'whelp, webpack works!, all yours!' But IMO there's so many problems! The moment you use a tilemap, you'll see rippling. Ah. you need the tilemap extruder everyone uses. And how should you make a map? Well? Everyone uses Tiled. What if you want to make a mobile build or something? I'm just trying to bring all this together in a place so that its actually useful.

What I am saying is that a create-___-app isn't necessarily as minimal as one would think. Look at create-react-app-again, and look in the packages folder. They have a lot of extra 'niceties' like error overlays; even the build process is quite opinionated. I haunt the issues for that a lot, and they've made a lot of choices, such as not supporting decorators, that definitely funnel the developer into a particular approach to some problems. Its also true as you say they don't include MOBX, Apollo, relay or redux. Some scaffolds do, though. I am using the fact you need to use Tiled here; that's an opinion. Its no different than settling on prettier (instead of standard) as a style. Or not using typescript. Or whatever.

So (in my opinion) should we do html menus and things? I am going to try it. The reason is for, just like you said, making something quickly. Its significantly easier to make things like menus, scoreboards, popups and whatever in html. Its true it will never draw or be as fast as canvas. I think there is a lot of 'mundane' things that can use it. I think if something is a core gameplay element and needs to be 60fps, maybe HTML isn't the right choice. I just picked a healthbar to show you that you can reconcile between phaser and screen space, so something 'sticky' to a sprite. The real value here is that recreating something as basic as a textbox or a table in canvas isn't trivial, its a lot of time. I want to make something that someone at a game jam can pull this down and have the tooling they need, and not need to 'invent' that part of the puzzle. Imagine making a game for a game jam with a textbox to name your player. In HTML its trivial, in canvas, there goes half your jam. Input lag isn't a big deal for that. There's also a lot of generes of games that are text-heavy, or UI heavy, that are kind of not worth using phaser at all for right now. What if I wanted to make a an rpg with lots of dialog? Does the dialog need to be 60 fps? probably not. Having HTML as a first class citizen would be pretty nice though.

Still, I'll try to optimize this as best as I can. I imagine with some backpressure on the data streams and decent abstractions a lot of those problems can be overcome. If they can't be overcome, and its just 'terrible' I'll remove it. I've got to at least try though. I think you mentioned requestAnimationFrame, which I didn't both with for that quick and dirty example, but you're right that is another thing that would help. I believe with more time I could get the lag down a lot, and make an abstraction that is comfortable to work with. Main goal in that prototype was just to broadly address the scaling/ screenspace-to-HTML problem you brought up.

An example to make my point clear, I know the guy that worked on this project (He lives in my city and is a friend): https://github.com/goldfire/CanvasInput . It took an entire weekend for him to make. he said it was surprisingly challenging. For one input, for something html can do out of the box. If you really really need a 60fps input to type in your name for a high score, it will be objectively better to do it in canvas, but from a rapid development perspective - I'd say the dev-speed gains of HTML are self evident. He said the same thing about making a table in canvas. It was a huge time investment. That is the real value. like you said, our options right now are terrible. I think trying to integrate a good, performant UI in HTML might give a lot of benefit right now, rather than trying to reinvent HTML in canvas, which is super hard.

HTML/CSS really is the nicest thing we have, from the dev's perspective imo for building UI. Every other phaser boilerplate (and they do exist) just doesn't even try here, so I'm going to give it a shot. I imagine at the point of scaffolding, I might use something like inquirer.js, and you can opt out of this if you want. Regardless, enough people will get value from it. I feel like need to see if i can make something work.

At the end of the day this project might end up more comprehensive than many other create-___-app's out there, but I think that's the direction this needs to go. If you want something simpler, honestly that already exists - you can go get the es6 boilerplate some guy out there made already. Its a decent tool, and will save a lot of people time. I started on this because I personally needed something more comprehensive for my own projects.

Points 4 & 5:

Wow, that looks incredible! You clearly put a lot of effort into that! Are you using Johnny 5 or Serial.IO ? I'd love to see your framework when you're done.

I can tell you've really been thinking about this in your own projects. I appreciate you taking the time to share your thoughts, and I think you've brought up a ton of great points and some technical considerations to overcome. My hope is I can have something going soon enough you can pull this down in the future and evaluate it.

[GabeAtWork]

I'll add my €0.02 as someone with little gamedev experience. I feel making UI building easier will be a big plus to new devs who already know html/React. That being said, I think clear use cases should be given (like building menus), and limitations of this technique should be explicit. I'm afraid learning devs won't understand easily when to use react for ui, and when to use phaser's tools.

I'll certainly use react for menus if it's available! If clear guidelines and examples are given, I think it's worth it!

[the-simian]

Appreciate the feedback @GabeAtWork