Back in September, a Chris posted a comment on this blog asking about a Starling filter that would create a circular mask over an image. Being the lazy sort I am, I never got back to the commenter (Sorry, Chris). Some stuff I was doing at work the other day reminded me of the question, though; and, although it’s a relatively simple thing to do, the task raises two interesting problems I thought deserved a whole post rather than just a quick response.

In plain English to create a circular mask we would want to do something like this: Find the distance of every pixel in an image from the center of our circle. If that distance is greater than the radius of our circle then we should set its alpha to zero. Well, to find distances we can turn to the good ol’ Pythagorean theorem – distance equals the square root of distance x times distance x plus distance y times distance y (I’m still amazed I use information I learned in high school in my day to day life. Who’d a thunk those teachers were on to something?). In a sort of pseudo/AS3 code, our operation may look something like this:

var distancex = pixel.x - center.x; distancex = distancex * distancex; var distancey = pixel.y - center.y; distancey = distancey * distancey; var distance = distancex + distancey; distance = Math.sqrt(distance); if (distance > radius) { pixel.alpha = 0; }

We can start to port that pseudocode over to AGAL like so:

; Assume v0 contains original pixel position ; Assume fc0 contains our circle definition: ; fc0 = [centerX, centerY, radius, 1] sub ft0.x, v0.x, fc0.x mul ft0.x, ft0.x, ft0.x sub ft0.y, v0.y, fc0.y mul ft0.y, ft0.y, ft0.y add ft0.z, ft0.x, ft0.y sqt ft0.z, ft0.z tex ft1, v0, fs0<2d, clamp, linear, mipnone> ; ...

But there lies the rub… We now have our distance value in ft0.z and our texture information (red, green, blue, and alpha) in register ft1.xyzw, but how do we do the if conditional? Ideally, we would like to write something like:

if (ft0.z > fc0.z) mov ft1.w, 0

Unfortunately, AGAL doesn’t provide such explicit branching statements. It does, however, provide four nifty little ‘set’ operations which, with a little ingenuity, can handle such conditionals: SGE (“set if greater than or equal”), SLT (“set if less than”), SEQ (“set if equal”), and SNE (“set if not equal”). These operations will set a register’s component to either 1 or 0 depending on whether or not the conditional passes. Let’s take a look at SLT for a moment. SLT (like the other 3 set operations) takes two arguments. If the first argument is less than the second the result will be 1, otherwise the result will be 0. In Actionscript, the operation would look something like this:

var result:int = arg1 < arg2 ? 1 : 0;

So, how is that helpful? Well, we can use that 1 or 0 in a multiplication operation to get either 0 or the original result. If we go back to our pseudocode and re-write it using a ternary statement similar to the one above, it could look like this:

var distancex = pixel.x - center.x; distancex = distancex * distancex; var distancey = pixel.y - center.y; distancey = distancey * distancey; var distance = distancex + distancey; distance = Math.sqrt(distance); var conditional = (distance < radius) ? 1 : 0; // alpha is now either its original value or 0 if the distance is greater than the circle's radius pixel.alpha = pixel.alpha * conditional;

We know we can do that in AGAL, so now our complete fragment shader can look like this:

sub ft0.x, v0.x, fc0.x mul ft0.x, ft0.x, ft0.x sub ft0.y, v0.y, fc0.y mul ft0.y, ft0.y, ft0.y add ft0.z, ft0.x, ft0.y sqt ft0.z, ft0.z tex ft1, v0, fs0<2d, clamp, linear, mipnone> slt ft0.w, ft0.z, fc0.z mul ft1.w, ft1.w, ft0.w mov oc, ft1

So, the main point here:

If you need to do some basic branching in AGAL, look for a way to use a "Set" operation and multiply the result with another value to get either a 0 or the original value.

If we plunk that AGAL into a Starling filter now, it might look like this:

/** * Copyright (c) 2012 Devon O. Wolfgang * * Permission is hereby granted, free of charge, to any person obtaining a copy * of this software and associated documentation files (the "Software"), to deal * in the Software without restriction, including without limitation the rights * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell * copies of the Software, and to permit persons to whom the Software is * furnished to do so, subject to the following conditions: * * The above copyright notice and this permission notice shall be included in * all copies or substantial portions of the Software. * * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN * THE SOFTWARE. */ package starling.filters { import flash.display3D.Context3D; import flash.display3D.Context3DBlendFactor; import flash.display3D.Context3DProgramType; import flash.display3D.Program3D; import starling.textures.Texture; public class CircleMaskFilter extends FragmentFilter { private var mShaderProgram:Program3D; private var mVars:Vector. = new [1, 1, 1, 1]; private var mCenterX:Number; private var mCenterY:Number; private var mRadius:Number; public function CircleMaskFilter(radius:Number = 100.0, cx:Number = 0.0, cy:Number = 0.0) { mCenterX = cx; mCenterY = cy; mRadius = radius; } public override function dispose():void { if (mShaderProgram) mShaderProgram.dispose(); super.dispose(); } protected override function createPrograms():void { var fragmentProgramCode:String = "sub ft0.x, v0.x, fc0.x \n" + "mul ft0.x, ft0.x, ft0.x \n" + "sub ft0.y, v0.y, fc0.y \n" + "mul ft0.y, ft0.y, ft0.y \n" + "add ft0.z, ft0.x, ft0.y \n" + "sqt ft0.x, ft0.z \n" + "tex ft1, v0, fs0<2d, clamp, linear, mipnone> \n" + "slt ft0.w, ft0.x, fc0.z \n" + "mul ft1.w, ft1.w, ft0.w \n" + "mov oc, ft1" mShaderProgram = assembleAgal(fragmentProgramCode); } protected override function activate(pass:int, context:Context3D, texture:Texture):void { mVars[0] = mCenterX / texture.width; mVars[1] = mCenterY / texture.height; mVars[2] = mRadius / ((texture.width + texture.height) * .5); context.setProgramConstantsFromVector(Context3DProgramType.FRAGMENT, 0, mVars, 1); context.setProgram(mShaderProgram); } public function set centerX(value:Number):void { mCenterX = value; } public function get centerX():Number { return mCenterX; } public function set centerY(value:Number):void { mCenterY = value; } public function get centerY():Number { return mCenterY; } public function set radius(value:Number):void { mRadius = value; } public function get radius():Number { return mRadius; } } }

If you try that filter though, you'll notice something very odd. The circle is there all right, but instead of the outside being transparent, it has kind of a ghosted screen look. An interesting effect, maybe, but not what we wanted. That's because when dealing with shaders in Stage3D, in order to get alpha values, you have to set the blend factors of the Context3D instance. If you check out the Adobe documentation, they even, very helpfully, tell you what blend factors to use. Since, in our case, we want to to use Alpha, back in the activate method of our CircleMaskFilter, just before we set the program of the context, set its blend factors like so:

context.setBlendFactors(Context3DBlendFactor.SOURCE_ALPHA, Context3DBlendFactor.ONE_MINUS_SOURCE_ALPHA);

A little gotcha though - as soon as we're done running our filter, we need to reset the context3D's blend factors back to No Blending. Luckily, the good folks who put together the Starling framework made that simple enough. Just as there is an activate method in the base FragmentFilter, there is also a deactivate method. Simply override that method and reset the blend factors to no Blending:

override protected function deactivate(pass:int, context:Context3D, texture:Texture):void { context.setBlendFactors(Context3DBlendFactor.ONE, Context3DBlendFactor.ZERO); }

Finally, you should get a result like this.

And that's really all there is to it to handle alpha in a Starling filter. Once you get that down, you can do all sorts o' stuff. With a second texture you can easily create complex masks. Or instead of finding the distance between two pixels as we just did, you can calculate the distance between two colors. Why (you ask)? Well, imagine you have a target color and you calculate the distance between it and each pixel in your texture. If the distance is below a give threshold, you set its alpha to 0, and suddenly you have a basic greenscreen application.

So, Chris, I hope that helps out. Better late than never....