More Shader Support

The XmlShader specification allows for multiple fragment shaders, the output of one being piped into the input of the next. Unfortunately, the existing XML parser  has difficulty with dynamic xml structures (or I had no idea how to read a variable structure). So I wrote a new XML parser. It parses arbitrary XML structures and produces an easy-to-use tree.

Then I altered my existing OpenGL backend modifications to allow multi-pass rendering. This requires the framebuffer object extension, which is widely supported, but it may not be supported. I will have to add some compatibility options later.

In the meantime, almost all of the shaders in the bsnes repository work (https://gitorious.org/bsnes/xml-shaders). Some of them are quite interesting. They scale to arbitrary sizes, although some look better at certain resolutions (scale2x, 5xBr, etc.). Ctrl Alt 9 and 0 switch shaders.

Right now, I am in the middle of nowhere. I'll be back home next week, but until then I will have very limited connectivity. Feel free to send me an email or leave a comment, and I'll read and reply when I get a chance.

Status

I've added some support for multiple shaders using XmlShaderFormat. It looks for all *.shader files and tries to parse and compile them. However there isn't a good way to change shaders. So far, all changes have been in the base OpenGL backend. However, in order to expose the shaders through the graphics mode settings, changes may possibly be made to the SDL backend. Right now, it queries a static class function to figure our the supported graphics modes. However, OpenGL calls to compile the shaders cannot succeed before the OpenGL context is created. So either the class needs to be initialized before the SDL backend queries the graphics modes, or the static class function can guess about what shaders will eventually be compiled (For instance if "CoolFilter.shader" exists in the current directory, but is not a valid XmlShader or it does not contain valid GLSL programs, then it would still be reported as a valid graphics mode.).

My work on scaler plugins has slowed down. I want to get it incorporated into the main project, so the things I need to consider are

• How will the plugins work with existing backends that used the old scalers?
• How will they work with future backends?
• Do more formats need to be supported? Can they be detected at runtime or compile time (Or probably both)?
• How can they be integrated with the build system?
• Options to disable formats (e.g. 32 bit formats which are not used)
• Options to disable plugins. Currently the Edge2x/3x plugin shares the HQ scaler compile option (USE_HQ_SCALERS).

Shaders

I have added some very basic support for shaders into the OpenGL backend. Right now, it looks for a "vertex.glsl" and a "fragment.glsl" to load as vertex and fragment shaders respectively. Eventually I want to have some manifest file that stores the properties of shader programs, but I have not done that yet.

Right now, two uniforms are passed to the shaders. The first is the texture to use called "texture". The second is a vec2 containing the width and height of the texture in pixels called "textureDimensions". This is useful since texture coordinates are in the range [0..1] so it is difficult to tell where one pixel ends and another starts. I have attached a shader program that implements scale2x (Advmame2x) by finding the position within a pixel using these uniforms. It duplicates the functionality of the scaler in the SDL backend, but it looks kinda funky with scale factors != 2. It can probably be modified to use a combination of Advmame3x and Advmame2x depending on subpixel position.

What is great about this is that no development tools are required to change the shader programs. So users can create and load shaders without having to configure build environments and compile ScummVM. Hopefully this can lower the barrier to making some creative works.

Advmame2x shader: https://gist.github.com/3161079
ScummVM branch with enabled shaders: https://github.com/singron/scummvm/tree/opengl

I've Been Away...

But now I'm back. I had limited time last week to do work so I have no progress to show since then. I was working on a spline interpolating filter that could be extended to be a arbitrary size scaler, but it was needlessly complicated for poor results, and I have scrapped it to work on new, more useful things.

Right now I am revising the last API addition I made (comparing the current frame to the previous frame to update only necessary pixels). It forced extra code into the backend, and with so many backends, it would get easier adoption if more of the bookkeeping was migrated to the scaler code. To simplify the addition of new plugins wishing to use this feature, the relevant code has been included in a subclass that new plugins can inherit and get all the bookkeeping for free.

Then I'll be taking a look at the OpenGL backend to implement the scalers as shaders. Hopefully with some improvements, the OpenGL backend can improve performance and quality, giving people a reason to actually use it (it is not even included in many release packages).

Edge2x/3x Finished Up

The performance of the scaler is fine now, even in debug builds without optimization. I have reimplemented the changed pixel detection through a new part of the API. The backend queries the plugin to see if it supports using an old image to compare changed pixels. One problem is that panning the screen causes the whole image to be reupdated and the mouse movements to become choppy. However, it happens rarely enough that it really is not an issue, and in optimized builds it does not matter.

I also templated the function for 32bpp support. This scaler is unique in that it uses the products of interpolation to then compare to other pixels (in other scalers, the products of interpolation are only written to the final image). The existing interpolation functions mangled the alpha channel (in the case of rgba and argb) and padding bits (in rgb888). This caused quirky image defects to happen that were trickier to track down, since the different alpha channels caused the comparisons to be different without changing the color of the pixels.

In the past, I had debugged these problems by simply returning the color red from an interpolation function. Then I would see if the broken pixels turned red. Since this scaler compared the results of the interpolation, whenever I followed a similar technique, the scaler would choose a different path, and the image would change in more chaotic ways (e.g. lines ceasing to anti-alias, black pixels appearing (but not red)). Everything at least appears to be fixed now.

Here are some sample images scaled with the 32bpp scaler.

Edge2x

Edge3x

Clarification on Edge Scaler Optimization

I said some ambiguous statements about what optimizations I disabled in the Edge scaler. Currently, scalers update partial rectangles of the screen based on what pixels have actually changed. However this still causes lots of duplicate pixels to have the same calculations repeated on them. For fast scalers, this is good enough, but the Edge2x/3x scalers needed more speedups.

So the original scaler author included code that took these partial rectangles and tried to reconstruct the source image in the scaler. Then it diffed its own source image with future calls to find out exactly which pixels needed to be updated. However, this involves a lot of guess work about where the rectangles are in the original image. I disabled this particular optimization since the backend can more simply give this information to the scaler through a new part of the API (currently in design). Dirty rectangle updates still work just like they do with the other scalers.

The new part of the API will probably be optionally implemented for backends and scalers. The scalers will request that an old source image be kept by the backend and passed to the scaler so that the scaler can run a diff and update the pixels however it wants. This does not complicate other scalers, does not change backends that would not use the Edge scaler, and provides some needed functionality.

Edge2x/3x scaler

So this week I added the Edge2x/3x scaler. It is the most complex scaler yet, producing the highest quality images and being the most cpu greedy.

It had some pretty ingenious heuristics to detect the entire source image, see exactly what pixels had changed and only update the relevant regions. Of course this was originally written years ago so this kind of hack is not necessary, and I have disabled it. On my laptop, everything seems smooth as long as it is compiled with optimization. Without, it is quite jerky (the compiler must be doing something right). However I am considering adding optional access to the old source image as part of the scaler api so all those hacks would not be necessary.

So here are some comparisons. Click to see larger versions.

Normal2x

HQ2x

PM2x

Edge2x

Notice that the Edge2x scaler manages to anti alias almost every edge whereas the other scalers tend to miss some. In particular, look at the roundness of the coins and the features of Guybrush's body.

EDIT: I have clarified some of the optimizations I disabled in another post.

2xPM scaler

Digitall recently pointed me towards some old patches on the patch tracker for scalers. There I found an old patch for the 2xPM scaler. It is a high quality scaler similar to hq2x. It took a few minutes to copy some files around and to change some variable names, but adding it was painless at first. When I tried to use it, the quality did not look as good as the screenshots I had seen previously. In particular the pattern in the source image:

#|@|#
#|#|@
#|#|#

Would scale the center pixel to:

#|@
@|#

which made no sense and gave previously sharp edges strange dithering effects.

So I compared the patch to the reference implementation it was based on and realized a few mistakes had been made in the copy-paste. The patch was never accepted, so it was probably never checked too thoroughly.

After fixing that, picture quality improved dramatically. 2xPM is arguably better than hq2x. I have included some images that demonstrate the different scalers. The source image is great because it contains a lot of pixel art in different colors in a variety of configurations.

Normal2x


AdvMame2x


2xPM



HQ2x

Images

I thought it was odd that a graphics focused project like mine would not have any images yet.



So among various invisible infrastructure changes I have made, I did add two new modes, Normal4x nearest neighbor scaling, and AdvMame4x scaling. They look nice if you have a high resolution monitor. I plan to add HQ4x too and I will post I similar screenshot. Click the thumbnails for full sized versions.


Normal4x


AdvMame4x

Scaler Progress

The plugin system seems to be working well. It is now integrated with the GUI and configuration file. The naming scheme in the configuration file has changed slightly, but old values will be recognized and replaced. So if you had different settings for every game in your game library, they should all still work.

In addition, I added a 4x nearest neighbor scaler. It works fine except for some minor cursor issues which should be fixed when the cursor api gets changed. This means HQ4x and AdvMame4x will soon be available with very little added code.

Now I am in the process of removing quick hacks I made earlier. Then much of the scalers need to be changed to fully implement more pixel formats.

Scaler Plugin WIP

I have a functioning proof-of-concept implementation for the new scaler plugin system. It has some hacks for now to make it easier to prototype. Also it is missing GUI features and ignores configuration settings.

The benefits are really starting to show. Adding new plugins is really easy and takes no additional configuration from the sdl backend. Also, I have made small changes to the DotMatrix scaler to take advantage of the coordinates passed from the backend. Now it does not change the surrounding pixels when the mouse is moving.

If you want to try it out, my code is on github. "Ctrl alt +" and "ctrl alt -" increase and decrease the size (as usual) but "ctrl alt [" and "ctrl alt ]" cycle through the different scalers.

New Scaler API

This new api  attempts to solve problems with the old scalers that were difficult to fix without a full redesign. I'll give an overview.

void initialize(Graphics::PixelFormat format) 

• This is called before each scaler is used so that it knows what the pixel format is.
• Generating RGBYUV tables and other precomputed data should happen here.

int increaseFactor() 
int decreaseFactor() 
int getFactor()

int setFactor() // For loading from configuration

Common::Array<int> getFactors() // For displaying in GUI

• These give easy ways for setting and manipulating the scale factor.
• These functions always return a valid scale factor so mapping these functions to hotkeys is simple.
• Also this allows different sized versions of the same scaler to be included in the same plugin.

 bool canDrawCursor()

• Some scalers are not suitable for scaling the cursor since they blur the image.
• Since the cursor is not always scaled the same way as the background and can use a 1.5x scaler, the mouse scaling needs extra thought. If you have good ideas for this, leave a message or otherwise contact me.

int extraPixels()

• Since some scalers look at surrounding pixels outside of the area to be scaled, the rects to be scaled need to be expanded, and the surface needs to be extended to prevent memory access errors.

void scale(const uint8 *srcPtr, uint32 srcPitch, uint8 *dstPtr, uint32  dstPitch, int width, int height, int x, int y)

•  This is the scaling function similar to what is currently used.
• The addition of x and y coordinates allows position sensitive scalers like TV and DotMatrix to produce consistant results.

A couple changes could possibly be made.

void transform(Rect *dirty)

• Instead of extraPixels(), this function changes a dirty Rect so that it will behave properly when passed to a scale function.  The backend can change this afterward in case the Rect was expanded off the edge of the surface (or possibly the dimensions of the surface can be given as well).

The same Rect struct could be passed to the scale function as well.

Currently I am working on proof of concept implementations try out some of these ideas and decide what to put in the final version. If you have suggestions please leave comments or message me.

EDIT: Also there is a deinitialize function. It's self explanatory.

Hello ScummVM!

My name is Eric Culp (a.k.a. singron). I'll be working on the scaler plugins project.

I'm very excited to work on a project like this. If you have questions, comments, or witty insults, you can leave them as comments on this blog or catch me on irc. Relevant links to to github will be in the sidebar.