Proposal: Modernizing the Graphics of USDX

[complexlogic]

Due to the recent changes in the theme system, the total amount of theme code has been reduced from 57k lines to just 12k lines. The huge reduction of copy and paste in the themes makes changing them much easier.

I have been working on a local branch which changes the game to render internally at 1920x1080 instead of 800x600, and am adapting the theme files accordingly. You can see me screenshots below (new vs. old). So far I have completed the main menu, option menus, and the navigation bar for all screens.

Eventually, the game will need new high resolution textures to make it look truly good. Hopefully somebody with graphic design skills can help with that. But even just rendering the existing low-res textures at the correct aspect ratio and avoiding the ugly 4:3 to 16:9 stretch makes the game look a lot better.

Adapting the game to render at 1920x1080 not technically difficult work, but it does require some time. It involves converting the elements in the theme file, and then finding the hardcoded elements in the game code and changing those as well.

I am proposing to start a feature branch to change the graphics to render at 1920x1080. If the various contributors to this game are willing to focus efforts on this, I think we can fully convert the game's graphics within the span of a few months. My idea is that we would submit PRs to the feature branch one screen at a time until the game is fully converted to run at 1920x1080.

[barbeque-squared]

In my own local builds I just scale the theme elements (this does cause non-square covers and avatars, I run a custom theme where it just "happens" to look good on widescreen displays) but my version renders at whatever is the display resolution (4k, ultrawide, 800x600, it doesn't care).

It has some shortcomings (resize is completely broken) due to how I implemented/hacked it, but I'll post some screenshots later today because it doesn't look much different from yours I think.

Some other important observations I made thus far (I've been running it like this for probably at least 3 years now):

• 99% of this game is song selection + sing + scores. on all other screens, non-square avatars aren't really a dealbreaker.
• song selection: Chessboard is OP, should be fairly trivial to compute so that things are square, get rid of the zoom animation, all it needs is a frame around the active entry in the coverlist. I don't know why we even still have other modes, but my guess is because the stupid Categories thing is eternally broken on Chessboard and for some reason people really can't live without whatever this does (I've never been able to figure it out).
• sing: is half hardcoded anyway. if we decide to not be a SingStar clone anymore, could still have some theming but also be much nicer.
• score: my custom themes just makes it look like a "table" which wouldn't be very difficult to compute on the fly entirely.

Internally rendering at 1920x1080 would still be an improvement for 95% of today's computers, so I'm not enterily against it, but I also feel it doesn't really address the underlying issue which is that the way theme elements scale is essentially undefined.

[complexlogic]

the way theme elements scale is essentially undefined

The vertices, which are specified on the 800x600 grid, are multiplied with an orthographic projection matrix and then transformed to the viewport of the window. So to get the screen coordinates from the 800x600 vertices:

ScreenX = VertexX * ScreenW / 800
ScreenY = VertexY * ScreenH / 600

If you want to render internally at the actual screen width and height, you could use OpenGL model matrix manipulation functions like glScale* to transform the vertices, specified on a 1920x1080 grid , to the equivalent position in the actual window. To make this worthwhile though, the textures should be resolution-independent, e.g. a generated from a vector image such as SVG.

Anyway, I've uploaded my code below. Could be used as a starting point for converting the graphics to high definition 16:9. If you have other plans for the graphics I understand.

https://github.com/complexlogic/USDX/tree/full_hd

[barbeque-squared]

Attached is parts of the diff that I use in my local builds. It is actually less than half of all my changes, and beware there will be random unrelated things thrown in. It probably won't even git apply or compile straight away, it's just here to serve as what I have been using for the last 3+ years:
native-resolution.patch

You'll want to look especially for occurences of ScreenW, ScreenH, ScreenWidthMultiplier and `ScreenHeightMultiplier, those should give you some idea of what's going on here.

Pro: it runs on any resolution
Con: it doesn't allow window resizes, will still "stretch" things on non-4:3 unless a theme already compensates for it

Another trick I've found is simply not using textures in the first place. The way I run my custom theme, I'd really only need a thing for theme elements to indicate "do not touch my aspect ratio" that then just centers it in whatever the stretched space for it would be, and done. The hardest screen would be the song selection screen, but that's way too complex anyway. But this would require some probably-breaking changes (I've never been able to figure out the end-of-sentence popups, but turns out, if you make the bar simpler and more obvious, you don't even need them anyway) and probably also lose a lot of the "look and feel" of whatever the current theme is trying to clone. But this then also starts hitting things like why the hell we need to individually enable/disable certain theme elements (my money is on party mode).

To illustrate: my sing screen looks like this and by far the lowest-quality things in that screenshot are actually the notes themselves (technically also the golden note sparkles and the lyric indicator thingy, but those move around much more):

Yes, my theme 'corrects' the aspect ratio for the avatars in the theme file, but it's easy to see that it really wouldn't matter if I can just lock the avatar aspect ratios. It really doesn't matter if the avatar moves around a few pixels. This is also why I'm leaning more towards (vastly) simplifying the UI (read: delete anything unneeded) and less towards making the current 4:3-problem a 16:9-problem.

I don't know if glScale can also do these kind of things. The real solution is probably a mix of all these ideas.

[dgruss]

@barbeque-squared i also implemented a scaling mechanism: #1146
The theme simply sets ScreenW and ScreenH to define the aspect ratio. All scaling in game is done in the code then. Tested with 4:3 and 16:9 and 16:10.
Avatars are correctly scaled and placed based on the theme + scaling factor. Elements can specify whether or not to scale with aspect ratio considered or directly.

[dgruss]

@complexlogic i'd be happy to work on this together but would definitely like to have something that does not care whether the screen ratio is 4:3, 16:10, 21:10, or 16:9.

my approach here was that the theme defines the resolution it assumes and for rendering everything is scaled and placed accordingly if the screen ratio does not match. this allows themes for different screen ratios to be designed but still work on other screen ratios reasonably.
as i wrote, for this, every element can/must specify how to scale (proportionally or not). but the results i got are not bad if you look at the screenshots i posted in #1146 with the basic 4:3 theme - you can also try out the artifact in there to look at more screens - it worked surprisingly well

[barbeque-squared]

Finally had a chance to look at both complexlogic's branch as well as dgruss' branch. There's definitely good stuff in both branches, eg complexlogic's font size (I need to try if that fixes the still somewhat blurry font issue on my branch), idk who it was where themes define the resolution they're made for, or dgruss' thing where theme elements can define if/how they get scaled.

All of our branches also have less optimal parts, eg:

• complexlogic's branch doesn't really solve the hardcoding of everything
• dgruss' branch is quite complex because it also does scaling
• my branch can't resize the window at all

But I do wonder, can we split this into multiple separate incremental improvements? For example:

• Render at whatever is the display size, but don't bother preserving the aspect ratio of anything yet (special case: fonts, for those do whatever makes the most sense/least effort). Right now we don't preserve it either, so simply upping the resolution is already an improvement
• (maybe?) allow themes to be defined in a different resolution than 800x600
• better preserve aspect ratios where necessary

It's likely that there will be some more steps, probably refactorings because iirc the area where the notes get ultimately drawn is still hardcoded, for example. On a side note: I don't know the historical significance of the credits screen, but I'll probably take a PR that just removes it and its assets altogether.

[dgruss]

1. Render at whatever is the display size, but don't bother preserving the aspect ratio of anything yet (special case: fonts, for those do whatever makes the most sense/least effort). Right now we don't preserve it either, so simply upping the resolution is already an improvement

That is exactly what i started with. but for this you already need something like the UScale class I introduced, because the coordinates and width/height have to be recomputed now before the rendering instead of by the gl scaling afterwards.

2. (maybe?) allow themes to be defined in a different resolution than 800x600

this one comes for free once you have (1)

3. better preserve aspect ratios where necessary

the amount of code for this is negligible.

if you want i can split the PR into three, covering these three points. not sure we can get the complexity much lower

[barbeque-squared]

this one comes for free once you have (1)

it does not. the note area, with the golden notes etc for example will still get drawn in its 800x600 box, because it's ALL hardcoded

the amount of code for this is negligible.

it works surprisingly well, but I've also already noticed weird things where avatar frames get stretched and now there's empty space inside it. but then the covers have black pillarboxing, which looks really awkward in some places. whereas the current code just stretches everything, so can we please just start by stretching in a better resolution and only that.

It also already has real bugs, for example the lyric engine doesn't understand resizes until you Ctrl-Left the song. I haven't yet checked for other screens, but please stop suggesting that things are "free" or "negligible" because they're not: they're still more than the bare minimum. And anything more than the minimum takes exponentially much time to review/test.

[dgruss]

yeah, it's definitely far from finished.

my point was rather that selecting what behavior you want can be very little code.
the note area still uses a 800x600 coordinate system, same as the themes. it didn't look like they were rendered on that resolution to me. also, just by adding 3 lines of code i changed the scaling of the note area:

doesn't look nicer to me though, i think the note area should be stretched to the window size (but of course rendered on maximum resolution)

[complexlogic]

Would it be unacceptable to break compatibility with existing custom themes? I would really like to completely move away from the 800x600 coordinate system if possible. In my branch, I changed it to 1920x1080. This allows you to design a fully native 16:9 UI, at the expense of breaking compatibility with existing custom themes.

I don't think that implementing scaling hacks on top of the existing 800x600 coordinate system is a good long term solution.

@barbeque-squared's idea to allow the theme to define its coordinate dimensions could work, but it would require exposing the game's many hardcoded graphics attributes in the themes, which would add complexity to the themes.

[dgruss]

the proposal to let the theme define the coordinate system is from my PR ;)

yes, your theme with 1920x1080 would work very well on that branch as no scaling is necessary then :) i think our branches might already be largely compatible.

the idea is quite simple:
you have a new class UScale which computes which scaling factor is needed to get from the coordinate system of the theme to the resolution of the screen.
elements are then either scaled directly, proportionally, or letter/pillar-boxed with the placement using the element center to place it in the right new location.

this works pretty well and gives the most flexibility. of course now we are largely at 16:9 but there might be a future not so far away where things shift to 21:9 and then we would have the same discussion again. i would like to avoid that by simply introducing this scaling method as a basis and then letting the theme decide the coordinate system and the scaling class do the scaling to the actual screen size. and surely, this scaling will have corner cases where it does not look nice, but it will still allow you to use themes without adjustments on different screen ratios than what they were designed for.

there is no need to expose hardcoded attributes, you can look at my PR #1146 - it is not much code and very few changes in the themes to make them scalable.

[complexlogic]

So the scaling is only done if the screen size is different from the theme dimension? That sounds like a good idea.

Also, I totally agree with barbeque's idea to remove the credits scene. It hasn't aged well. Whoever designed it used custom graphics, but the original files aren't available in an editable format as far as I know, so it's not really feasible to extend it. There is a reason why it is basically unchanged since 2008. I would favor replacing the credits scene with a simple pop up with scrolling text, similar to the credits USDB Syncer, US Creator, and US Manager.

The other thing I would recommend reducing is in the song selection screen. UScreenSong.pas is by far the worst of the graphics code, IMO. Is it really necessary to support 7 different user interfaces? It really adds a lot unneeded complexity to the code. Some of user interfaces like "Slide" look really bad. I would propose just keeping Roulette and Chessboard, and removing the rest. That would simplify UScreenSong.pas a lot.

[dgruss]

I only use List... But yeah getting rid of some others might make sense.

[barbeque-squared]

Would it be unacceptable to break compatibility with existing custom themes? I would really like to completely move away from the 800x600 coordinate system if possible. In my branch, I changed it to 1920x1080.

It's not about breaking compatibility. It's about all the currently hardcoded 800x600 stuff and me vetoing changing those numbers for the sake of moving away from 800x600. But fix the underlying problem(s) first, and I'll happily break existing custom themes.

Same with the aspect ratio scaling. I'm not against the idea. But then either make a PR that solves this in the current 800x600 coordinate system. Or first make a PR that makes 800x600 look better (but still stretched), then follow up with another PR that allows for aspect ratio (this second approach seems more sensible to me).

The above paragraphs have a common divider (essentially dgruss' UScale without the theme dimension or aspect ratio scaling). It's a net benefit and also an improvement on its own, and lays some groundwork for the things that each of us is looking for in the long-term. So stop discussing and just make it already! Or wait until I've cherry-picked it, that's also fine. Figuring out these longer-term things is much easier if step 1 isn't part of the discussion anymore.

Agree on replacing the credits screen.

Also agree on removing most of the song selection modes. Keep Roulette, Chessboard and List for now, they're pretty different ways of presentation anyway.