Ending the Year with Dynamic Multi-Pass Shaders in Starling

So, the last day of 2015 and, I believe, just my second post of the year. I think there’s some law which states you have to make at least two blog posts per year to legally say you maintain a blog, so I’m a bit pressed for time.

I would love to say the reason for my absence is all the work I’ve been doing on a secret project which has been taking up all my time – and that would actually be at least partially true. But, let’s face it, I’m a bit of a lazy man, and also have to wonder if anyone, besides myself, that is, actually still reads Flash blogs these days. Hopefully someone.

A Little Bit About Project X

For the past 3 years or so, in my spare time, an hour here, 2 hours there, I’ve been working on a bit of a casual mobile game. While I’m not at the point of releasing or even revealing too much about it, I will say you will eventually get the chance to play a big-nosed bug on a motorcycle (if anyone’s actually able to guess the game title from that little tidbit, I’ll give out a prize. I don’t know what, but something). My New Year’s Resolution is to release in 2016 – which gives me exactly one more year to finally bring this thing to fruition. What is primarily lacking at this point is art (the “Programmer Art” I currently have in place just ain’t gonna cut it), but, unfortunately, I’m not in the position to be able to hire anyone at the moment. And so I send out a simple plea: if anyone reading this has some serious art/UI design skills, a fairly modern Android device, and would just love the opportunity to work on spec or a profit sharing plan, please contact me. I know, it’s a crazy long shot, but, hey, what good is having a blog if you can’t ask the whole world for help?

In any case, while working on this game over the Christmas holiday I came across a series of interesting Starling shader problems I thought I’d share.

Problem I – Animating from a Point

The first thing I needed was a shader that could animate an effect from a given set of x,y coordinates. Imagine, for a moment, a shader that would draw a red circle at a point where an image was clicked then move the circle until it reached the bottom of the image, then remove it. Now, this isn’t at all what I actually needed for my game, but it’s easy to make believe and easier code wise, than what I was actually working on.

Here is a quick example (Click on the image):

 

[kml_flashembed publishmethod=”static” wmode=”direct” fversion=”18.0.0″ movie=”http://blog.onebyonedesign.com/wp-content/uploads/2015/12/demo1.swf” width=”550″ height=”450″ targetclass=”flashmovie”]

[/kml_flashembed]

And this is the FragmentShader implementation that produces the effect:

/** * Copyright (c) 2016 Devon O. Wolfgang * * Permission is hereby granted, free of charge, to any person obtaining a copy * of this software and associated documentation files (the "Software"), to deal * in the Software without restriction, including without limitation the rights * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell * copies of the Software, and to permit persons to whom the Software is * furnished to do so, subject to the following conditions: * * The above copyright notice and this permission notice shall be included in * all copies or substantial portions of the Software. * * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN * THE SOFTWARE. */ package { import flash.display3D.Context3D; import flash.display3D.Context3DProgramType; import flash.display3D.Program3D; import starling.filters.FragmentFilter; import starling.textures.Texture; public class CircleShader extends FragmentFilter { /** AGAL for Circle shader */ private static const CIRCLE_SHADER:String =  // calculate distance (with aspect ratio taken into account) sub ft1.x, v0.x, fc1.x mul ft1.x, ft1.x, ft1.x div ft1.x, ft1.x, fc1.z sub ft1.y, v0.y, fc1.y mul ft1.y, ft1.y, ft1.y mul ft1.y, ft1.y, fc1.z add ft1.x, ft1.x, ft1.y // distance = sqt ft1.x, ft1.x slt ft1.y, ft1.x, fc0.w sge ft1.z, ft1.x, fc0.w mul ft2.xyz, ft0.xyz, ft1.zzz mul ft3.xyz, fc0.xyz, ft1.yyy add ft0.xyz, ft2.xyz, ft3.xyz mov oc, ft0 ]]> /** Shader Program */ private var circleShader:Program3D; /** Circle Radius */ private var radius:Number = .025; /** Speed of Circle */ private var speed:Number = 1.5; /** Width of filtered display object */ private var width:Number; /** Height of filtered display object */ private var height:Number; /** X position of circle */ private var xPos:Number = 0.0; /** Y position of circle */ private var yPos:Number =0.0; /** Shader params ( R, G, B, Radius ) */ private var shaderParams0:Vector. = new [1.0, 0.0, 0.0, 1.0]; /** Shader params ( X, Y, Height/Width Ratio, unused ) */ private var shaderParams1:Vector. = new [0.0, 0.0, 0.0, 1.0]; /** * Create a new Circle Shader * @param width width of filtered display object * @param height height of filtered display object */ public function CircleShader(width:Number, height:Number) { this.width = width; this.height = height; this.yPos = this.height * 2; } /** Dispose */ override public function dispose():void { if (this.circleShader != null) this.circleShader.dispose(); super.dispose(); } /** Add a circle */ public function addCircle(x:Number, y:Number):void { this.xPos = x / this.width * this.width; this.yPos = y / this.height * this.height; } /** Create programs */ override protected function createPrograms():void { this.circleShader = assembleAgal(CIRCLE_SHADER); } /** Activate */ override protected function activate(pass:int, context:Context3D, texture:Texture):void { this.shaderParams0[3] = this.radius; this.shaderParams1[0] = this.xPos / texture.width; this.shaderParams1[1] = this.yPos / texture.height; this.shaderParams1[2] = texture.height / texture.width; context.setProgramConstantsFromVector(Context3DProgramType.FRAGMENT, 0, this.shaderParams0, 1); context.setProgramConstantsFromVector(Context3DProgramType.FRAGMENT, 1, this.shaderParams1, 1); context.setProgram(this.circleShader); this.yPos += this.speed; if (this.yPos >= this.height) { // if circle has travelled the height of the display object, // move it well down and out of sight this.yPos = this.height * 2; } } } }

You’ll notice, though, that if you rapidly click on the image in that demo several times, the effect resets itself so there is only one red circle at any given time. But what if you want more?

Problem II – Animating from Multiple Points

There are actually a few ways to tackle this problem. The laziest and probably worst is to just add more filters. Since Starling 1.6 it has been possible to add multiple filters to a single DisplayObject instance by nesting parents. Which is to say, you can add a filter to a Starling DisplayObject and another to its parent and which will also be applied to the child DisplayObject. But, unless you know from the start how many filters you’ll be applying, this will require some really nasty and hacky parenting and re-parenting of DisplayObjects. Of course each addition will require 2 extra draw calls (one for the parent container and one for the filter added to it) and performance will degrade exponentially.

Another approach is to use multiple passes of the shader in the FragmentFilter implementation. This presents another problem, though. In Starling, every filter pass requires an extra draw call which can have potentially negative impacts on performance. So, then the question is, how to plan the number passes in advance. So, in this red circle case, should the filter perform 5 passes just in case, even though I’ll typically only want 3 red circles? What if I end up wanting 7 red circles? And in the case where there are no circles present at all, do I really want a 5 or 7 pass shader (no, is the answer to that in case you were wondering)?

Turns out one of the beauties of Starling’s FragmentFilter class is the ‘numPasses’ property. As the name implies, this sets the number of passes a given shader will have. The great part though is that this can be set any given time. So in the case of our falling red circles we can start out by applying an identity shader in the first pass. When we click we add a circle, increase the number of passes by one, and in the additional passes we animate those circles. As each circle reaches the bottom of the image, we then lower the number of passes by one and simply stop drawing it. In this way, we can add n number of animating circles (or whatever we need), and performance will only potentially degrade when necessary, but automatically reset itself when our effect is complete.

Here is another example:

[kml_flashembed publishmethod=”static” wmode=”direct” fversion=”18.0.0″ movie=”http://blog.onebyonedesign.com/wp-content/uploads/2015/12/demo2.swf” width=”550″ height=”450″ targetclass=”flashmovie2″]

[/kml_flashembed]

Try rapidly clicking several times again, but keep an eye on the number of draw calls in the performance monitor in the upper left. You’ll see how each red circle adds a new draw call, but automatically removes that call when the circle reaches the bottom. It’s a very handy trick.

And the FragmentFilter implementation for that effect:

/** * Copyright (c) 2016 Devon O. Wolfgang * * Permission is hereby granted, free of charge, to any person obtaining a copy * of this software and associated documentation files (the "Software"), to deal * in the Software without restriction, including without limitation the rights * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell * copies of the Software, and to permit persons to whom the Software is * furnished to do so, subject to the following conditions: * * The above copyright notice and this permission notice shall be included in * all copies or substantial portions of the Software. * * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN * THE SOFTWARE. */ package { import flash.display3D.Context3D; import flash.display3D.Context3DProgramType; import flash.display3D.Program3D; import starling.filters.FragmentFilter; import starling.textures.Texture; public class CircleShader2 extends FragmentFilter { /** AGAL for Circle shader */ private static const CIRCLE_SHADER:String =  // calculate distance (with aspect ratio taken into account) sub ft1.x, v0.x, fc1.x mul ft1.x, ft1.x, ft1.x div ft1.x, ft1.x, fc1.z sub ft1.y, v0.y, fc1.y mul ft1.y, ft1.y, ft1.y mul ft1.y, ft1.y, fc1.z add ft1.x, ft1.x, ft1.y // distance = sqt ft1.x, ft1.x slt ft1.y, ft1.x, fc0.w sge ft1.z, ft1.x, fc0.w mul ft2.xyz, ft0.xyz, ft1.zzz mul ft3.xyz, fc0.xyz, ft1.yyy add ft0.xyz, ft2.xyz, ft3.xyz mov oc, ft0 ]]> /** Circle Shader Program */ private var circleShader:Program3D; /** Default Shader Program */ private var defaultShader:Program3D; /** Circle Radius */ private var radius:Number = .025; /** Circle Speed */ private var speed:Number = 1.5; /** Width of filtered display object */ private var width:Number; /** Height of filtered display object */ private var height:Number; /** Maximum number of circles */ private var maxCircles:int; /** Collection of circles */ private var circles:Array; /** Shader params ( R, G, B, Radius ) */ private var shaderParams0:Vector. = new [1.0, 0.0, 0.0, 1.0]; /** Shader params ( X, Y, Height/Width Ratio, unused ) */ private var shaderParams1:Vector. = new [0.0, 0.0, 0.0, 1.0]; /** * Create a new CircleShader2 * @param width width of filtered display object * @param height height of filtered display object * @param maxCircles max number of circles (extra pass/draw call per circle) */ public function CircleShader2(width:Number, height:Number, maxCircles:int=10) { this.width = width; this.height = height; this.maxCircles = maxCircles; this.circles = []; } /** Dispose */ override public function dispose():void { if (this.circleShader != null) this.circleShader.dispose(); if (this.defaultShader != null) this.defaultShader.dispose(); super.dispose(); } /** Add a circle */ public function addCircle(x:Number, y:Number):void { addCircleInternal(new Circle(x / this.width * this.width, y / this.height * this.height)); } /** Internal add a circle */ protected function addCircleInternal(circle:Circle):void { this.circles.push(circle); // if exceeded maximum number of circles, remove the oldest if (this.circles.length > this.maxCircles) this.circles.shift(); // Adjust the number of passes this.numPasses = this.circles.length + 1; } /** Internal remove a circle */ protected function removeCircleInternal(circle:Circle):void { var idx:int = this.circles.indexOf(circle); if (idx > -1) this.circles.splice(idx, 1); } /** Create programs */ override protected function createPrograms():void { this.defaultShader = assembleAgal(STD_FRAGMENT_SHADER); this.circleShader = assembleAgal(CIRCLE_SHADER); } /** Activate */ override protected function activate(pass:int, context:Context3D, texture:Texture):void { // on first pass, just apply the default shader if (pass==0) { context.setProgram(this.defaultShader); return; } var c = this.circles[pass-1]; this.shaderParams0[3] = this.radius; this.shaderParams1[0] = c.x / texture.width; this.shaderParams1[1] = c.y / texture.height; this.shaderParams1[2] = texture.height / texture.width; c.y += this.speed; context.setProgramConstantsFromVector(Context3DProgramType.FRAGMENT, 0, this.shaderParams0, 1); context.setProgramConstantsFromVector(Context3DProgramType.FRAGMENT, 1, this.shaderParams1, 1); context.setProgram(this.circleShader); // clean up on final pass if (pass==this.circles.length) { for each(var circle:Circle in this.circles) { if (circle.y >= this.height) removeCircleInternal(circle); } // Adjust the number of passes this.numPasses = this.circles.length > 0 ? this.circles.length+1 : 1; } } } } class Circle { public var x:Number; public var y:Number; public function Circle(x:Number, y:Number) { this.x = x; this.y = y; } }

Adjusting the number of passes dynamically as just shown is really the gist of this post, and there’s not much sense reading further, but there is one other problem to consider.

Problem III – Additional Shaders

So we finally have multiple red circles running down our image, but what if wanted our image filtered in some other fashion as well – say, for example, we want red circles running down a sepia toned image? Well, once again we could try applying an additional filter to our DisplayObject’s parent, but once again, that’s the bad programmer’s opt out (lazily, I actually tried this approach in my game for the hell of it and found, when running on device, frame rate was literally cut in half – from 60 to 30 – not the best of all possible options).

Instead of adding additional filters, why not put that default shader that runs in our filter’s first pass to work. Instead of an identity shader, that could really be anything (e.g. a sepia toned shader). Now, you could keep making additional copies of the CircleShader class replacing the default shader with something else, but instead let’s take one last pass at CircleShader.as and make it a bit more extensible:

/** * Copyright (c) 2016 Devon O. Wolfgang * * Permission is hereby granted, free of charge, to any person obtaining a copy * of this software and associated documentation files (the "Software"), to deal * in the Software without restriction, including without limitation the rights * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell * copies of the Software, and to permit persons to whom the Software is * furnished to do so, subject to the following conditions: * * The above copyright notice and this permission notice shall be included in * all copies or substantial portions of the Software. * * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN * THE SOFTWARE. */ package { import flash.display3D.Context3D; import flash.display3D.Context3DProgramType; import flash.display3D.Program3D; import starling.filters.FragmentFilter; import starling.textures.Texture; public class BaseCircleShader extends FragmentFilter { /** AGAL for Circle shader */ private static const CIRCLE_SHADER:String =  // calculate distance (with aspect ratio taken into account) sub ft1.x, v0.x, fc1.x mul ft1.x, ft1.x, ft1.x div ft1.x, ft1.x, fc1.z sub ft1.y, v0.y, fc1.y mul ft1.y, ft1.y, ft1.y mul ft1.y, ft1.y, fc1.z add ft1.x, ft1.x, ft1.y // distance = sqt ft1.x, ft1.x slt ft1.y, ft1.x, fc0.w sge ft1.z, ft1.x, fc0.w mul ft2.xyz, ft0.xyz, ft1.zzz mul ft3.xyz, fc0.xyz, ft1.yyy add ft0.xyz, ft2.xyz, ft3.xyz mov oc, ft0 ]]> /** Default Shader Program */ protected var defaultShader:Program3D; /** Circle Shader Program */ private var circleShader:Program3D; /** Circle Radius */ private var radius:Number = .025; /** Circle Speed */ private var speed:Number = 1.5; /** Width of filtered display object */ private var width:Number; /** Height of filtered display object */ private var height:Number; /** Maximum number of circles */ private var maxCircles:int; /** Collection of circles */ private var circles:Array; /** Shader params ( R, G, B, Radius ) */ private var shaderParams0:Vector. = new [1.0, 0.0, 0.0, 1.0]; /** Shader params ( X, Y, Height/Width Ratio, unused ) */ private var shaderParams1:Vector. = new [0.0, 0.0, 0.0, 1.0]; /** * Create a new CircleShader2 * @param width width of filtered display object * @param height height of filtered display object * @param maxCircles max number of circles (extra pass/draw call per circle) */ public function BaseCircleShader(width:Number, height:Number, maxCircles:int=10) { this.width = width; this.height = height; this.maxCircles = maxCircles; this.circles = []; } /** Dispose */ override public function dispose():void { if (this.circleShader != null) this.circleShader.dispose(); if (this.defaultShader != null) this.defaultShader.dispose(); super.dispose(); } /** Add a circle */ public function addCircle(x:Number, y:Number):void { addCircleInternal(new Circle(x / this.width * this.width, y / this.height * this.height)); } /** Internal add a circle */ protected function addCircleInternal(circle:Circle):void { this.circles.push(circle); // if exceeded maximum number of circles, remove the oldest if (this.circles.length > this.maxCircles) this.circles.shift(); // Adjust the number of passes this.numPasses = this.circles.length + 1; } /** Internal remove a circle */ protected function removeCircleInternal(circle:Circle):void { var idx:int = this.circles.indexOf(circle); if (idx > -1) this.circles.splice(idx, 1); } /** Create programs */ override protected function createPrograms():void { this.circleShader = assembleAgal(CIRCLE_SHADER); createDefaultShader(); } /** Create default shader */ protected function createDefaultShader():void { this.defaultShader = assembleAgal(STD_FRAGMENT_SHADER); } /** Apply default shader parameters */ protected function applyDefaultShaderParams(pass:int, context:Context3D, texture:Texture):void { context.setProgram(this.defaultShader); } /** Activate */ override protected function activate(pass:int, context:Context3D, texture:Texture):void { // on first pass, just apply the default shader if (pass==0) { applyDefaultShaderParams(pass, context, texture); return; } var c = this.circles[pass-1]; this.shaderParams0[3] = this.radius; this.shaderParams1[0] = c.x / texture.width; this.shaderParams1[1] = c.y / texture.height; this.shaderParams1[2] = texture.height / texture.width; c.y += this.speed; context.setProgramConstantsFromVector(Context3DProgramType.FRAGMENT, 0, this.shaderParams0, 1); context.setProgramConstantsFromVector(Context3DProgramType.FRAGMENT, 1, this.shaderParams1, 1); context.setProgram(this.circleShader); // clean up on final pass if (pass==this.circles.length) { for each(var circle:Circle in this.circles) { if (circle.y >= this.height) removeCircleInternal(circle); } // Adjust the number of passes this.numPasses = this.circles.length > 0 ? this.circles.length+1 : 1; } } } } class Circle { public var x:Number; public var y:Number; public function Circle(x:Number, y:Number) { this.x = x; this.y = y; } }

And a quick sepia implementation:

/** * Copyright (c) 2016 Devon O. Wolfgang * * Permission is hereby granted, free of charge, to any person obtaining a copy * of this software and associated documentation files (the "Software"), to deal * in the Software without restriction, including without limitation the rights * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell * copies of the Software, and to permit persons to whom the Software is * furnished to do so, subject to the following conditions: * * The above copyright notice and this permission notice shall be included in * all copies or substantial portions of the Software. * * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN * THE SOFTWARE. */ package { import flash.display3D.Context3D; import flash.display3D.Context3DProgramType; import starling.textures.Texture; public class SepiaCircleShader extends BaseCircleShader { /** AGAL for Default Sepia shader */ private static const SEPIA_SHADER:String =  // sepia dp3 ft1.x, ft0, fc0 dp3 ft1.y, ft0, fc1 dp3 ft1.z, ft0, fc2 mov ft0.xyz, ft1.xyz mov oc, ft0 ]]> /** Sepia parameters */ private var sepia1:Vector. = new [0.393, 0.769, 0.189, 0.000]; private var sepia2:Vector. = new [0.349, 0.686, 0.168, 0.000]; private var sepia3:Vector. = new [0.272, 0.534, 0.131, 0.000]; /** Create a new SepiaCircleShader */ public function SepiaCircleShader(width:Number, height:Number, maxCircles:int=10) { super(width, height, maxCircles); } /** Create default shader */ override protected function createDefaultShader():void { this.defaultShader = assembleAgal(SEPIA_SHADER); } /** Apply default shader parameters */ override protected function applyDefaultShaderParams(pass:int, context:Context3D, texture:Texture):void { context.setProgramConstantsFromVector(Context3DProgramType.FRAGMENT, 0, this.sepia1, 1); context.setProgramConstantsFromVector(Context3DProgramType.FRAGMENT, 1, this.sepia2, 1); context.setProgramConstantsFromVector(Context3DProgramType.FRAGMENT, 2, this.sepia3, 1); context.setProgram(this.defaultShader); } } }

Which will get you this:

[kml_flashembed publishmethod=”static” wmode=”direct” fversion=”18.0.0″ movie=”http://blog.onebyonedesign.com/wp-content/uploads/2015/12/demo3.swf” width=”550″ height=”450″ targetclass=”flashmovie3″]

[/kml_flashembed]

Hope that might help someone out. Have a great New Year, all!

…