MocoCompositor: The Nasty Hack + Videos

So remember a while back I mentioned that I wanted to be able to expose the image plates data structure (ie the image layers + shaders for each layer) used by the MocoCompositor to the main user interface (the web)? Yeah well, turns out it was waaaaaay harder than I’d expected it to be. The exposing of the data structure and UI wasn’t too bad it was the part where I had to rebuild the image plates within the compositor while it was still running (quite multithreaded no less).

Let me explain how the Compositor Configurator (compositor_config.html file) works… this is a simple web page that loads a json file from the server upon onLoad using the /loadjson/ service that the Server already provides (the same one used in saving and loading curve sets). This json file, named “compositor_layers”, contains the current state of the compositor image plates. The compositor_config.html GUI lets the user add a new layer, reorder the layers, change the shader used as well as any parameters necessary for that shader. When the user is satisfied they simply click the “Update Compositor” button and it sends the new image plates as JSON to /publish/compositor_config . Meanwhile the Compositor is sitting there with a subscription to /subscribe/compositor_config (in a thread)… the minute it receives the new plate data structure it saves it to the “compositor_layers” json file and then it “theoretically” rebuilds the image plates within the compositor and starts executing those.

Now I said “theoretically” for a reason… turns out pyglet + shaders + OpenGL + PIL + whatever else really don’t like having things being shuffled from underneath them. I was getting this incredibly annoying Segmentation Fault every time I tried to rebuild the plates after the initial build. I tried everything I could think of including putting a giant COMPOSITOR_LOCK around both the on_draw and rebuild_plates functions hoping that maybe it was just a race condition of the data structures getting used while in an invalid state. No dice, I was still getting SegFaults… there wasn’t going to be a nice clean way for me to rebuild these image plates with all these threads running as well as OpenGL doing shader stuff concurrently.

So I used my nuclear option… suicidal processes… sometimes the only way to cleanly do something is to destroy the process (thereby giving the OS a chance to clear memory, close file handles etc) and start it up again… Now we have the MocoCompositor at start up do nothing but spawn another process of itself (using the subprocess.call function). The child then can create the pyglet window + shader etc… when the child receives the new image plate data structure from the GUI it saves it to the json file and promptly unsubscribes all the subscriptions and exits pyglet as cleanly as it can and exits with an exit value of 1. When the child dies the parent process gets back the exit value (remember he’s blocked on the subprocess.call() call), if the exit value is equal to 1 it knows the child was requesting a restart so he goes ahead and spawns a new child process of MocoCompositor which in turn loads the configuration from the json and bam we’re back in business with the new configurations. If the exit value was anything else the parent assumes a normal exit and doesn’t try to restart the child and just exits himself.

The observant reader would wonder how the MocoCompositor would keep from recursively spawning child processes of itself. We prevent this by passing an additional command line flag (that is picked up in the child’s sys.argv) called ‘–no-subprocess’ in the subprocess.call()… when MocoCompositor sees that flag he assumes he’s the one that should be doing the pyglet stuff rather than spawning… additionally this allows us to run a non-respawning version of MocoCompositor (still will kill itself on a configuration change) by passing the ‘–no-subprocess’ flag to it in the command line.

All this works without the clients that are subscribed to /viewfinder/compositor being any the wiser because of the way the Server implements the Publisher/Subscriber model… it never tells recipients of a disconnection from senders and still keeps live sockets connected. This is what gives me the freedom to kill a publisher/subscriber process at will without hurting others in the network (theoretically).

Our MocoCompositor now has the following shaders:

  • normal – Simply loads an image from a file or an HTTP stream (MJPEG) which becomes the input to the next layer.
  • blackandwhite – Grayscales the previous output image using the luminosity method of grayscaling. http://www.johndcook.com/blog/2009/08/24/algorithms-convert-color-grayscale/
  • greenscreen – Accepts a new file/HTTP stream to use as a foreground plate on top of the previous output layer’s image (background plate). Green pixels in the foreground image are replaced with the pixels from the background image. It also has an option to set a small “feather” value to fine tune the matching for green removal.
  • horizontal_blur, vertical_blur – These are 2 versions of the same blur shader (one blurs horizontally and the other vertically). They accept a mask image where the alpha values in the mask are used to determine the “blurriness” of the background plate. So one can open up photoshop and do a simple gradient on the alpha channel and save it as a PNG and set it as the mask for this filter to make gradient blurs (ie. what’s used in faking tilt-shift shots). These filters also accept a blur_size value (512 seems to be ok, the higher the smoother it seems). I stole the algorithm for these blur shaders from http://www.gamerendering.com/2008/10/11/gaussian-blur-filter-shader/ .

Ok enough talk, let’s see some results… the following are all live MocoCompositor outputs from /viewfinder/compositor that I opened up in a different computer’s browser and screen captured using xvidcap (on Linux)… that also explains the slowish frame-rate.

Here’s a Tilt/Shift of the street outside using a gradient blur mask with the horizontal_blur and a blur_size of 1024.0:

Here’s the same Tilt/Shift scheme with a blur_size of 512.0:

Here’s the cool and addictive greenscreen shader… you’ll notice the background plate is coming from a live Maya virtual camera compliments of MocoMaya (/viewfinder/maya), the foreground plate is the /viewfinder/rig… also a special thanks to the YouMedia group for unknowingly giving us those little oragami thingies… it was a nice square green sheet of paper :).
Note: The reason I keep looking off to the right is because that’s where our big LCD TV is with a full screen Firefox displaying the /viewfinder/compositor and also doing the xvidcap 😛 (terrible weatherman impersonation as well):

Finally we have the blackandwhite shader (for that Film Noir look):

Comments are closed.