03
Apr 12

The new Audi A3 webspecial with webcam gesture control

Audi A3 webspecial

In March 2012 Audi released a new version of its extremely successful Audi A3. The brief for Razorfish was to motivate potential clients to discover the A3s interiour. And to apply simplicity and intuitiveness of the A3s interiour design to the internet. So the concept team came up with the idea to  create a web special that could be fully experienced via gestures.

But how to realize that? Full-scale webcam-based controls for websites have not yet left the realm of experimental microsites. And in contrast to the Kinect platform by Microsoft, Flash does not provide a software framework to recognize gestures via the webcam. Kinect is also equipped with multiple cameras and a depth sensor that precisely captures a persons’ spatial movements.

The aim was to create an experience that is equally interesting when controlled by mouse as well as by the hands throught the webcam. So the gestures would be based on the movement of the mouse cursor. Consequently, we had to transform the hand movements captured by the camera into a hand cursor.

The hand becomes the cursor

Research on the experiments of creative developers on user interaction with a webcam in flash resulted in three approaches that could be suitable for our untertaking.

1st approach: Object Tracking

Object Tracking example The OpenCV (open computer vision) open source framework for C / C++ provides various algorithms for image processing. Some of them concerned with object detection within images have been partly ported to Flash by the Japanese Ohtsuka Masakazu in 2008. Though this technology is extreamly promising, we had soon to realize the code base available in Flash was too limited to support commercial development in time or budget.

2nd approach: Color Tracking

Colour Tracking exampleWith a colour filter it is possible to determine the location of a person’s hands within the image based on his skin colour. We had good results at first. Of course there was the need to filter out the persons head and also to dynamically handle different persons’ skin colours. That could be resolved by leveraging object tracking to determine the face and grab the person’s skin colour from it. But the approach was still instable with backgrounds that resembled the person’s skin colour or changing light conditions.

3rd approach: Motion Tracking

Motion Tracking example

This method compares successive camera images and determines the areas of alteration, which represent potential areas of movement. In the first prototype we implemented the rather simple swipe gesture. While the approach worked fine, was quite stable in different light as well as background conditions and performant, it was yet too imprecise and jumpy to support more complex gestures.

The solution

Advanced Motion TrackingBecause the first approach was not feasable and the second for its own insufficient, the way to go was to tweak the motion detection and maybe combine it with the others. Not an easy task. The enhancements which evolved over time by gaining a deeper understanding may be classified into detection, interpolation and performance optimization. Here are some examples:


Detection

  • Merging the results of serveral motion detections (3 were optimal) added a lot of stability and accuracy. This handles e.g. flicker of a light bulb (not visible to a person but to a camera!) or duplicate camera frames (framerate drops with lower light due to increased exposure time).
  • Two motion detection loops with different settings, the first optimized to handle faster movement and the second optimized to still capture very slow movement.

Interpolaton

  • In what we call triangular interpolation the last three detected coordinates are taken to calulate the triangular balance point as the result point. This smoothes the detected movement a lot and eases undesired jiggle.
  • With a bezier curve interpolation the last 10 result points are taken as control points to calculate a bezier curve. The new result point is set at 70 percent on that curve. While this adds a little delay, it further smoothes the movement.

Performance

  • Because the detection loop runs on every frame, it is crucial that it uses minimal computational resources so it does not impede other parts of the application like playing smooth animations.
  • Furthermore, the emergence of garbage collection which is accompanied by frame drops and a collapse of detection quality should be mininized.
  • The best method to handle both is the consequent deployment of object pools. The reuse of a BitmapData object e.g. is 10x faster than creating a new one!

In the end, the system became so accurate that we renounced the need to combine it with the other approaches (and deal with their disadvantages). The job was done!

The cursor becomes a gesture

This was the comparatively easy part. Simple movement based gestures like swiping are recognized by moving the cursor in a specific direction or over an appointed shape. The hold gesture – the equivalent to the mouse click – is triggered by holding the cursor for a while over a determined shape.

Form based gestures are recognized by comparison with a previously captured form. In case of the rotation gesture this is a circle form. To reduce false detections, we also applied some noise filters. The comparison is based on the 1$ gesture recognizer algorithm, a good AS3 implementation can be found at http://www.betriebsraum.de/.

 

07
Sep 11

TOUCH N CLASH / Awardee of the Adobe Mobile Challenge

At the 6th July Adobe unveiled the European Challenge in order to promote the new mobile development features of the Flash/AIR platform. The mission was to get approval by the Adobe Jury and to publish the application on the Android marketplace, the Apple Store and the BlackBerry AppWorld before the 1st of September.

Yesterday was the decision and luckily TOUCH N CLASH won the Novelty/Innovation Price! Read what the jury said on http://www.adobemobilechallenge.com/winnersbeta/

With TNC we successfully demonstrated that it is possible to develop a crossplatform multiplayer game without any kind of server for the three platforms named iOS, Android and PlayBook.

What is TOUCH N CLASH?

TOUCH N CLASH is a multiplayer game for two up to four players.
The goal is to get the other players out of the game by using a gameball which works like a bomb. The last player in the game wins.
You will need Wi-Fi internet connection to find other players. All players have to be connected to the same Wi-Fi network.

Gameplay

In the game the colored sides of your gamefield represents the other players.
If a gameball appears in your gamefield, you have to pass it to an other player bevore the countdowm time runs out.
To pass a gameball to an other player, touch a gameball and drag it to a colored side of your gamefield.
Sometimes you have the possibility to get an additional gameball.
A transparent gameball will appear in the middle of your gamefield, touch it to activate it.
Keep your gamefield clean of gameballs.
Good luck.

Insights

TNC was developed using Flash Builder 4.5.1. It is using skinned components of the mobile spark skin and the ViewNavigator class for switching between the views.

Communication

For the crossplatform communication we used the Real Time Media Flow Protocol (RTMFP) which was introduced in the Flash Player Version 10. We implemented a local P2P Neighbor lookup to connect devices inside the same WiFi to the overall lobby. All devices together create a P2P mesh which makes it possible to transfer data between ths clients. Each client has the possibility to open a new game the other clients can join. When a client opens a game he switches into a kind of “control-mode” to tell the game players how to behave and what to do.

Gamefield

For the phyisc based game field we used the Box2D engine. We connected the acceleration sensors of the devices to the gravity of the Box2D engine so the players can influence the course of the gameballs by rotating their device.

Performance

Performance was one of our critical points because the game only makes fun when it is running at a good framerate. Choosing the Flex Components as a base for the app was not the best choice at all, but when we ran into the performance issues a switch back to pure AS3 was impossible due to the deadline of the submission.

What we did to ensure a good performing app on all devices:

  • Removing all Spark images where possible and replacing them by Spark BitmapImages
  • Replacing any vector based assets by Bitmaps
  • Optimized the render interval of the Box2D based gamefield
  • Optimized the accuracy (velocity iterations, position iterations) of Box2D to achieve a good ratio between performance and functionality
  • reduced the overall framerate to about 30 fps
  • moving time and CPU consuming processes away from transition processes to ensure that the user has a smooth experience by navigating through the app

We also experimented with CPU vs. GPU to achieve the maximum performance available. The result: we stayed on the CPU. Some devices like the HTC Desire or Desire HD performed better in the GPU mode. On the Apple IPad 1 the game was slower on GPU compared to the CPU. On the iPad 2 we could not really identify a difference.

In general the performance on these devices was awesome:

  • iPad 2
  • Samsung Galaxy S2
  • BlackBerry PlayBook

Problems

Because we do not use any kind of server (not locally and not in the internet) we were totally dependent on the features of the WiFi hotspot. Some WiFis do not have the client to client communication enabled. As other WiFi only multiplayer games TNC will not work in this kind of networks. Because RTMFP uses broadcast technologies to discover the clients this can also be a pitfall.

We are currently investigating some problems in PlayBook only WiFis. Sounds insane but with our current settings the PlayBooks will not be able to connect to another PlayBook until another device type like PC, Mac, Android or iPhone/iPad joins the NetGroup. The other devices will then enable the lookup even between the PlayBooks.

Links

TOUCH N CLASH for iOS

TOUCH N CLASH for Android

TOUCH N CLASH for BlackBerry PlayBook

Winners of the Adobe Mobile Challenge

Screenshots

Credits

Game Director & Game Design & Development
Tobias Richter, Kay Wiegand

Design
Felix Moeckel

 

11
Aug 11

WebGL on heavy rotation

Surprisingly there’s quite a lot of stuff going on related to the Web here at Siggraph. And even more suprising, those are the session that are really fully packed with people! I don’t know, either they’ve been given the smallest rooms or they didn’t expect so many people interested in the latest efforts behind the words Web3D, X3D, WebGL etc.

Full WebGL session

...last row, but at least sitting.

So, after I could not make it to the Web3D BoF on Tuesday (room was bursting!) I made it this morning to the BoF of Khronos Group, one of the key influencers behind OpenGL or WebGL in particular. Almost 15 speakers in 2 hours (!) showed their stuff and here a some of my favourites:

The Jellyfish simulation by Alexander Rodic makes a beautiful, but useless proof of 3D artwork (his own words…). Alexander is no developer at all, but comes from an artists perspective settling from Maya to WebGL directly (without having any experience in any more “basic” GL programming). With that perspective he claims that JavaScript is very fast – well, compared to C++ it still is NOT ;-) check it out, you can play around with it get behind the secrets for yourself.

Just a short note in between made by an employee of nVidia: they are working on WebGL mobile implementation for the Android browser. His non-public demo showed a small 3D aquarium with about 100 texturized fishes running at 6 fps (on a Motorola Xoom device) …ok that’s almost nothing, but interesting though the initiative of nVidia and their support for WebGL.

Myrobotnation.com will finally make my dream come true! Mark Danks of Kodoma Studios presented the robot design studio of their upcoming e-com site where you can design and – yes, they can – 3D print your own robot. Although they would not need too many polygones (the robot must not exceed some limits for the printing precision), the demo was pretty neat. You can even add a lot of textures that would map correctly onto the meshes in realtime, which is straight forward in other technologies but not as easy with the given WebGL standards and JS programming. Anyway, I am looking forward to my own nation ;-)

You should definitely start playing around with PhiloGL from SenchaLabs. It’s a very straight forward framework to start your work on WebGL projects and understand how the parts work together. The site is fully-loaded with examples and the project available on Github. I liked most the realtime color 3D histogram analysis – great work of state-of-the-art web tech!

There’s been another very interesting presentation by Opera Labs. Unfortunately, I couldn’t find anything about it later on the web. Anyway, by trying to develop a modern 3D gaming experience with the abilities of HTML5 they’ve revealed some interesting measurements on performance. For instance, rendering with WebGL comes up to 60 fps while simple 2D drawing on the canvas will result in almost 15 fps (with less quality). Of course, no new fact at all, but it proves how important the knowledge of “real hardcore graphic programming” will be in the future of exceptional experiences in web development.

As all of this stuff is not included on the conference DVD, the guys promised to upload their presentations to the Web GL Blog. Hopefully they’ll hold the promise, so far the site looks a little empty…

11
Aug 11

Catching the light

Accidentally I happen to run into a session called “Show me the pixels” of once again MIT Media Lab’s Camera Culture Group. Amazing stuff. Shooting almost a trillion frames per second they even catch the light (sent out by a laser) and produce somewhat the slowest slow-motion sequence ever recorded. It’s used to gather the outlines of objects that may reside around a corner and in the future it’ll be even more than just the outlined object…

Fortunately they’ve put one of their first presentations online, enjoy!

10
Aug 11

Pixar always makes a nice queue

Have a look at this picture:

Wow, it's Pixar!!

That is the end of a queue and the entrance to a production session…and here:

What's going on here? Pixar?

That is the start of that same queue on the OTHER side of the convention center. Though there were many voices telling “oh the people are queueing for just another pixar-something-session” I made my way into the room (of course in the end while waiting till everybody was inside).

It was worth it. Enrico Casarosa, an italo-american animator and storyteller working at Pixar on movies like “Cars” or “Up”, showcased his animation short “La Luna”. It was previously released on several short film festival, starting with Annecy in June. Enrico held a fantastic presentation. Most likely for an animator he presented a bunch of his slides as small illustrations he made by himself.

Have a look at the trailer:

Not only that it’s a very nice (and sentimental) piece, looking behind the scenes reveals how much work you have to put in those 4 or 5 minute animation shorts. Enrico talked about almost every step – starting with convincing animation titan John Lasseter with some sketches up to storyboarding, animating, lighting, sound production and overall delivery.

As always filmmakers show how important a precise and well-established production cycle is, especially for highly creative and even emotional experiences! It makes perfect sense to…

  • not only think but talk, present and discuss your story, idea, product, project to others right from the start (we tend to often think for ourselves until there is the “right point” to reveal our thoughts)
  • produce sketches, workout prototypes and dummies, even with the risk to FAIL at some point (Enrico had the great idea to take his crew out to a moonshine ride with a small boat to discover the feelings of the fishermen….they learned absolutely nothing, but were freezing to death ;-) )
  • hold your schedule and follow the steps you’ve learned to produce an highly professional outcome (though it takes time and you might think of “oh I did that a thousand times, I would not need that skeleton working scheme stuff…”)

I don’t think the above would prove wrong – in any industry, especially the so-called creative programming.

10
Aug 11

Insights on Google Body Project

You may heard of the Google Body Project, which allows you to browse and search all parts of your precious anatomic details. It features of course the heavy usage of WebGL. So far they’ve reached a peak of about 1.4 M vertices in realtime, within Chrome for sure. I didn’t try it with Firefox but it works very well in the Google universe. Nice fact: the coding team stated they’ve developed everything in their (legendary) 20% research time they’ve been given from their employer.

But the interesting thing is not that it works or that it even exists. The people at Google really tried to work out things with the existing tools that are out there, which is, to be honest, not quite a lot (compared to all the heavy “real” GL stuff around here at Siggraph). Web3D, WebGL et al. are still waiting to break through, but projects like these show some preview of what can be achieved (in fact, if you feel the need to do so without using other tools such as Flash ;-) ) in the first place.

Anyway, I bet some of my colleagues would be very interested about the way they managed to transport all the data via HTTP. They took a nice approach to use UTF-8 encoding for somewhat “compressing” the big bunch of data one needs to display a complex mesh like the human body and respectively all of its parts. I’ll hand over the details soon!

09
Aug 11

First impressions from Siggraph 2011

My very first impression was: huge. That really IS a conference. Almost 10.000 attendees, speakers and student volunteers! And Vancouver perfectly suits to that kind of crowd, especially while being the second most loved city by west coast animation film-makers. All kinds of artists, creatives, nerds, sysadmins (yes, they even have their own BOF sessions here!), managers and production executives seem to inhabit Downtown Vancouver these days. Though they may be very different, they all have at least one thing in common: the “nVidia-Green” badge by the main sponsor ;-)

What a nice location!

What a nice location!

A word on keynote sessions at conferences like Siggraph. They always have the power to either let you fall asleep or bring out the very most enthusiasm about yourself and what your everyday’s work is. Neither of those happened this year, because the speaker – the author Cory Doctorow – spoke about a very important topic in the creative industries: copyright laws.
Though I already knew some of his thoughts (and of course they are not new at all), he managed to let the crowd feel, what they’re problem really is. Companies like Apple (he didn’t want to pinpoint everything on Apple but almost did so anyhow…) follow a DRM policy that in the end he claimed as “you give them your property, they put a lock on it, but do not give you the key”. I think no one who ever tried to listen to a song which was purchased legally on iTunes NOT on iTunes but another distribution system will disagree. The artist however has no chance of giving you the right to do so – only Apple does. Of course not only Apple wants to earn money, but the artist, developer or writer too and they need distribution ways like iTunes.

Inside the conference center

Inside the conference center

I pretty much liked the way he was giving the speech and – he did not talk about “good” and “evil”, he moreover tried to convince the people to think of new models for DRM. In the end he had no solution to the problem at all, but gave some nice thoughts to go out and “ask the right questions” (american way of talking). And this is what keynote sessions are all about, right?

Oh, of course there’s always something bad you recognize right from the start. This time it’s the air conditioning system. It’s way too cold! If I want to warm out, I’d rather go outside – and this is Canada!

08
Aug 11

Siggraph 2011

Started on sunday, Siggraph 2011 is in full swing. Hopefully we’ll get some direct coverage. Meanwhile you can have a look at a great collection of papers and presentations, Ke-Se-Huang maintains a very comprehensive page.

Extract of Rubika ( Le laboratoire – Canal + ) from Ludovic HABAS on Vimeo.

Mr. Choco in Love.

#SIGGRAPH

03
May 11

Realtime ribbon: the new Audi Q3 webspecial

Although it’s kinda hard to tell – all the transitions and stages in our latest webspecial for the Audi Q3 are rendered in realtime 3D (which sounds easier than it was).

In this article we’ll give a few insights in how we built it, using Away3D 3.6.0 in combination with Collada files and textures exported from Cinema4D.

Before the real production started we did several tests on how to build an endless ribbon. The ribbon should display the loading process and had to form the stages for the actual stage content.

The core of the solution is quite easy: A sequencer which connects several imported meshes combined with a texture animation and a camera animation.

The sequencer

Because the shape of a full dynamic ribbon would be hard to handle in realtime we used a set of meshes which were modeled in Cinema4D. The first and the last face of each sequence has to match the specific face of the previous/next sequence.
This was mostly done in the authoring tool (Cinema4D). To achieve a seamless look the corresponding points of the faces are docked in realtime inside the Flash application.
For each stage a special short connector sequence is available, to help overcome the difference in face width between ribbon and stage.
For the travel between the two different routes of the plot (bright and dark side) specialized sequences are used.
All in all we use seven stages, eight connector sequences (the intro stage needs two), four ribbon sequences and one change sequence.

Let the ribbon flow

The animation of the ribbon is based on a simple but brilliant idea from Richard Olsson: a growing bitmap material. Therefore the UV-texture coordinates of the meshes have to be unwrapped in a special way.
The texture parts of the mesh have to be aligned in the order of the faces from top to bottom. To animate the growth of the ribbon you “just” have to unmask the bitmap from top to bottom in a modified Away3D BitmapMaterial.

In this basic variant the relative height of a UV-Segment can be more or less different to the height of a mesh segment. The result is a not linear animated ribbon. Because we had to apply easings to the animations we had to linearize the texture animation.
In the first step we developed a small tool which sorts the mesh faces from top to bottom. This can be done very easily because in this case two neighbor faces will always share two vertices.
The ordered faces are used to calculate a path which lies exactly in the middle of the ribbon (yellow dots and yellow lines). With this data the height of every mesh segment can be determined. When this data is set in relationship to the height of the depending UV-segments a very smooth and nearly linear animation of the texture can be achieved.

Camera path animation

For the camera animation a similar technique is used. To achieve a smooth movement the centers of the mesh segments defines the camera target path. Because the normal path animator of Away3D does not consider the length of each segment we had to write our own path animator class. It is quite simple: due to the knowledge of the total path length and the length of each segment we can do a linear animation along the path.

To achieve a smooth camera animation on the path and on the way to the final camera position on the stage we used two “magnetic” camera targets. The red one follows the generated path on the ribbon. The green one is used for the transition from the ribbon animation to the final stage position.

Each sphere has a force to the real camera target (visualized by a trident in the picture) which is enforced or weaked due to the situation. At the end of the animation all targets are at the exactly same (predefined position) to assure a seamless look in combination with the other 2D elements.

That’s just a small impression into what kind of problems had to be solved for this webspecial. In the end the performance was better than could be expected and the result is a convincing combination of 3D and 2D elements. At least when taking into account that the whole implementation phase lasted less than 6 weeks – including localization and Q&A!

18
Mar 11

When touch just isn’t enough

Always wondered how to control your Parrot AR.Drone in a more exact and stable way? Maybe even over longer distances than the integrated WIFI can provide? Well, look no further – Andreas just set up the Ardudrone project on Googlecode (http://code.google.com/p/ardudrone/).

“This project contains all required binaries and scripts as well as the source code for modifying the Parrot AR.Drone to be controlled by a standard RC Remote Control. The RC receiver is connected directly to the drone and sends the commands directly with the help of an Arduino board. ”