Building a Multitouch GUI

Touchlib is a library for creating multi-touch interaction surfaces. It handles tracking blobs of infrared light, and sends your programs these multi-touch events, such as ‘finger down’, ‘finger moved’, and ‘finger released’.

It includes a configuration app and a few demos to get you started, and will interace with most types of webcams and video capture devices. It currently works only under Windows but efforts are being made to port it to other platforms.

Yes, this is progress!
Yesterday I started to hate the drivers of the Firefly MV, but after a fresh reinstall of Windows XP (because I screwed up a lot of DLL, systemfiles and Device Settings) I was able to make both drivers work.

If you have problems with the Firefly MV driver-installation drop me a line, and I will try to tell you step by step what I did (or better: what you should NOT do)

With functioning DirectShow Functionality the Rest is a peace of cake.
Latency is not perfect right now, but I like the first results:

Next things I will need: a wider lense for the camera, and a short distance projector. Those will be the christmas gifts I will give to myself.

Finally I get my first FTIR-Results, I was starting to doubt this.
Looking back there were some points that where necessary to get the effect in my setup:

- really darken the room, instead of lowering the exposure of the cam.
- having the IR-Leds point into the acrylic in a 45 degree fashion. Not parallel to the glas.

The FireFly MV has a high IR-Sensitivity, and a lot of Parameters are adjustable through Software and SDK.
This post made me aware that it is vitaly necessary to adjust some parameters like “exposure” and “Brightness” or the camera will be in an Auto-Modus that will take control of the parameters and basically do what it thinks is the best. But we dont want that.

With parameter “exposure” Auto-Mode on , you get a Value of 58. The resulting picture is very bright, altough it can see IR, everything else is very bright too. We have no contrast here.

Whitout the auto-mode you can set the parameters freely.
Now I like the results:

I will have to experiment with the best settings, now I just use a Wii-IR-Led-Bar and a 4×7-Led-IR flood to check sensitivity.
Next Week I expect to start experimenting with different high-power IR-Leds to try and finally get the FTIR-Blob-Effect working. I ordered half a dozen different IR-Leds, with different angle of radiation, so this will be fun to learn what works best.

So finally I find a little time and money to continue this task. I will try to make everythin easier and faster this time.:-)

I bought 2 acryl-plates, both are 80cm x 60cm, one with 12 mm the other with 10 mm thickness.

Also 2 adjustable stands, I will have to get my mind together and plan the next steps.

I always feared that the simple red light IR-lamps will create too much heat inside the multitouch setup, which is especially dangerous for components like the beamer and so on.

So I bought some dimmer-units and build my own dimmable IR-lamps. They work good, and the heat can be minimized a lot. So I see many possibilities in lighting out my touchscreen region, so that it will get homogenous IR all around the table.

This is very low cost , and creates very much IR-light (much more then IR-Led based), the only drawback is you have to block the red visible light somehow.

Today I recieved a IR-Camera from point grey research. www.ptgrey.com

It’s the model Firefly MV Black and White, with development-kit and some lenses.
The camera should be perfect. I will show you some tests later on, when I have finished building my own IR-floodlights.

The package contains:

a. IEEE 1394 Card
b. IEEE 1394 Skinny Cable 2m
c. Software CD & Case
d. GPIO Pigtail Assembly for FFMV
e. FFMV Tripod Mounting Bracket with M2 x5 mounting screw
f. Getting started Manual
g. 6mm Microlens
h. BW & Color Microlens Holders

I was looking for a material which is very thick and would not let any light in. So any thin curtain even if it has the darkest black color will not suffice.
Thats why I bought 2 picknick carpet for 10 CHF each (my girlfriend bought an additional third one, that we are really going to use for picknick )

This does do the job.

Next days I will cut a nice hole on one side for the beamer and camera.

From the inside:

Yesterday I recieved specimen of various rear projection and other plexiglas from röhm switzerland. http://www.plexiglas.ch/

I was surprised how little they knew to tell about their rear projection plexiglas sortiment, called PLEXIGLAS XT RP. It seems all very expensive, very special, low on stock, and additionaly some sorts would have to be orderer in other countries.

I was lucky that they had some specimens (the only ones left) to send me.

So i feel lucky in that respect and I want to show you those specimens.

I will hold them in front of the beamer, like this:

First we have here a 7D009 RP. They tell me this stuff actualy exists for decades now, the price per square meter is awesome in the negative way, but also on a basis for negotiation. You might pay between 600 or 1200 CHF for 1 (!) square meter of this material, and you have to buy at least a whole standart size plate which is like 2×1 meters.
Here is how it looks:

You might not get the whole impression through this photo, but let me assure you, it looks like a very clear and bright computer screen (the room appears much darker than it really is,but it has the same light as picture nr. 1, it is the camera that adjusts to the high brightness of the beamer picture this way, sorry about that)
Compared with this the cheap rear projection foils I used so far look and feel like the cheapest crap. What I like about this 7D009 RP is also its thickness of 5 mm and its backside is already antireflex with a nice grey diffusor implemented. It is very robust, and stiff.
Also there is no hotspot of brightness, it is so evenly distributed throughout the whole plate.
Additionaly both front and rear projection is possible with this plate.
Here another picture of 7D009 RP:

The other specimen I got is the 99562 RP (not 99561, I am still waiting for them to send me a 99561).
It will cost around 350 CHF per sqm, which is also the price for the 99561. Sadly it only comes in a standard thickness of 3 mm, so for a big plate this might be just too little for enough stiffness and stability.
You dont want someone to use too much force on this, but still it has a high stiffness and is quite immune to scratching. Also you cannot just order 1 square meter of this model, you need to order a different standard plate of 3×2 m.
Here are 2 pictures:

You can cleary see the difference, a highly visible hot spot region of brightness…
I do not know yet if this concerns only the 99562 RP and NOT the 99561 RP. But apart from that drawback these RP plates are also very bright clear and with much contrast.
Both sides are equally good for rear projection, but front projection looks worse.

Thank you very much to the people of Röhm. They where so nice to also send me thin surface acryl foils (because since you pay sooo much for the RP, you probably dont want to let everybody scratch and touch it with their bare fingers, altough they have a high immunity to scratching). Also they send me their standart sortiment of PLEXIGLAS SATINICE:

Addendum: Let me show how the 2 RP Screens’ transparency is, because of course we want to track fingers and fiducial markers with them:

Original picture:

front and back side of 99562: The transparency is clear and sharp.

front and back side of 7D009: The one Side reflects and is not sharp, the other has a diffusor on it and is sharp.

below the beamer I have enough space now for the camera(s)…

or what about placing a high power IR-Floodlight there or something like that?

I bought some coloured foils, which are usually used for disco and theatrical lights.

If I put red/green and Blue together I get dark black, and the IR is still able to pass through.

It was easy to open the PHILIPS SPC1300NC/00 Webcam and put 3 little pieces of the foil into the case, and it was really blocking all the visible light, but I see now after some testing that the IR-sensitivity of this camera is just too low…

I bought some basic “angles” and wood. So now I was able to attach the mirror to the exakt millimeter, which is very important.

Next days I will visit the wood supplier often and let him cut the planks to the size I want straightaway.

I need to build a basic socket for the projector, so that it keeps a fix distance and position. Also a location for camera(s) and IR-Lamps and -LEDs has to be found and taken into account.

Ok, so I bought a simple Mirror (no surface mirror) worth 50 CHF.
Now I get a rectangular 100 x 60 cm picture on the table, and I can dream on about the project
I believe I could get an even bigger picture size, but then I would also need a bigger mirror. Right now the mirror is completely used, so if I move the projector a little bit the picture is beeing cropped and disappears.

I am glad so far…

But I definitely don’t like the reflection I get underneath the glas table. This will most likely cause reflection of all sorts: beamer- and IR-Rays, and disturb the camera tracking.

Because it is a simple mirror instead a surface mirror, I get “double images”, but right now they do not disturb me so much, also because they have not much contrast…
And: when I am moving and adjusting the mirror position and angle sometimes the double images disappear or are not really visible.

Today I finally recieved a coil of 2 different rear projection materials (Gerriets-OPERA and -REVUE from www.modulor.de).
I am happy with the first result. No beamer-rays pass through the foil, they stay “caught” on the material, which is great. What you can see on the photo/video is the glas table is reflecting the image onto the floor also.
Maybe that is the reason why I will need to put the foil below the glas, thus loosing this nice feeling of touching directly on the “picture” since I am touching the foil itself right now, not the glas, which is approximately 1 cm thick.

As much as I heard OPERA is not very well for tracking objects/fingers ontop of the foil, but I want to test it anyway.

I NEED A NICE BIG MIRROR!

This link here was quite usefull to help me understand what components are necessary to create a DI setup.

1) a Beamer.
I decided to go with this model: PANASONIC PT-AX200E. For 1000 EUR you really get a lot of good value.
I like that it has a high resolution, high contrast and is bright. Also most important for the MultiTouch: you have many possibilities to adjust and shift and tweak the picture.. for example to be able to magnigy the projected picture 2x with the zoom objective…

I got it 2 days ago and I am very happy… this is the perfect home cinema projector.

So right now I am waiting for all the other components to arrive. I have located the beamer on top of a cupboard in my sleeping room and test its basic features. I have it embedded as a second monitor in a extended desktop (win xp) with a laptop providing the signal.

2) acryl or plexiglas or a normal glas…
This is where all the “touching” will take place.
Ok, so i have to find a company that sells me one. Or wait… dont I already have something in my possession that can substitute as a first basic glas plane?
Yes:

I bought this table half a year ago to place it in the dining room. It was cheap, only around 200 CHF. It was 50% off… so I thought… why not , it looks stylish…. (first I wanted to buy a heavy wood table)… Also my girldfriend liked it, and pushed me into buying those red clothed chairs…

I dont yet know if this kind of glas will work. I will find out in the next days though.

3) IR-sensitive Camera (infrared)
Right now I do not have much expertise on what camera to use, also I do not want to make any impulse purchase of rather expensive cameras. So I first try to use what I already have here at home:

I bought a miniature surveillance camera module that is just black and white and analog, and of course IR-sensitive. It was extremely inexpensive (<20 Euro)… Also together with this camera I purchased a video grabber device (USB) so that in future I am able to use analog cameras which have an oldschool cinch- or tv- plug.

But I am also looking forward to test this unusual USB Webcamera:
PHILIPS SPC1300NC/00, unusual, because it has a higher framerate of up to 90 FPS (though I wasnt able to get more than 60 FPS on my computer) and also it has a wider angle than normal cheap webcams.
IR-sensitivity of this webcam is to be tested the next days.

4) IR Lighting
I have 2 different kinds of IR lights I am going to check.
One is a little module wich has 4×7 IR Leds (30 EUR), and can run on 3 - 15 Volt (gets brighter but also warmer with more volts). It has a very narrow angle of emitting. This is a problem… so you will definitely need more then one such module to light up the whole table. (But a Fresnel Lense might help here)

The second are those IR lamps which are used for hatching young chicken. They are cheap, emitting very strong IR (but unfortunately also emitting red light which has to be filtered/blocked with some material as far as I know at this point). Also they seem to get warmer then the IR LEDS… but they do not get hot… so I hope a good Fan and Cooling System will help ommiting too much heat inside the MultiTouch Table.

5) Projection Surface / Diffuser
It has to be a rear-projection Surface. What is important is the right Gain Value (ca. 1.6).
On the back side you project the picture, but additionaly on the front side you want your fingers to reflect the IR light which then passes back through the Surface and goes into the IR-sensitive Camera for the tracking.
I ordered 2 different rather cheap surfaces last week and hope i get them soon to start checking.
I also asked at Gerriets if they can send me a samples folder of their different surfaces, and they send it to me within 2 days. AWESOME!

Gerriets OPTITRANS Sample:

Additional Samples of Gerriets Rear-Projection Surfaces:

Aha, so I learn that there are 2 main ways to create a MTC (Multitouch console):

• FTIR (Frustrated Total Internal Reflection)
• DI (Diffused Illumination)

I have done some reading and a lot of googling and searching, and I believe that I have to start with the DI-Method just because it seems easier to get all the necessary components together.

Because after all, that is what I want as a first step:

to get a basic multitouch working, even if it has crappy resolution, high latency or ugly tracking…

I just want to get a glimpse of a working multitouch table…

All the messy tweaking and polishing has to wait first.

Hello, my name is George Margaris, and I am a fulltime freelancer creating flash applications and games.

So, since the Idea of creating a multitouch GUI spread around the world some time ago (see jeff han and others) I and probably many other people dream about creating such a device themselves.

The thing is that it really seems possible and realistic to try this yourself. This is no rocket science.

Now the main goal of course is to create a multi touch device without having to spend huge amounts of money… so there is this lowcost-aspect that has a high priority.

I hope I can achieve this, and are willing to spend my free time the next months to learn all the necessary things.

So lets get started.