Sunday, August 25, 2013

Oculus, Ar. Drone and Kinect completed


Recently I completed my experiment with the Oculus Rift and a Kinect to drive a Parrot Ar.Drone.
Using Kinect I implemented both voice commands and gestures to drive the drone. With voice commands I issue the takeoff, land, emergency and change camera commands, while with gestures I drive the drone.
The nice thing about the voice recognition of the Kinect is that it works quite well with all the noise made by the drone. As you will see in the video I raised a bit my voice, but I still find it awesome.
The gestures implemented are the following:
  • Both arms forward -> move the drone forward
  • Both hands near the shoulders -> move the drone backward
  • Left arm extended to the left and right arm along the body -> move the drone to the left
  • Left arm extended to the left and right arm forward -> move forward and to the left the drone
  • Left arm extended to the left and right hand near the right shoulder -> move backward and to the left the drone
The right movements are simply the left ones mirrored.
The Oculus, instead, was used to view the live feed of both drone’s cameras (with the stereoscopic filter applied, of course) and the head tracking to move the drone up/down and turn left/right.

Here is a video of it working. In the first part you can see the head movements, while in the second part the gestures.

It was fun to put together all these gadgets and try them out (I mean, having the drone crash everywhere :D)
Here you can find the full source code if you want to try it out.
As a disclaimer, I did this to learn something new and just for fun in my spare time, so the code is nothing to be proud of :D

This is a simple summary, but, if you’re interested, I will make some posts about the code of every part involved (gestures, voice commands, head tracking)

61 comments:

  1. I just took delivery of an AR Drone yesterday to do exactly the same thing. Pity the code is in c#. What did you do for the matched pair of cameras? Are there stereo pairs already available, or did you have to cobble your own together.

    ReplyDelete
    Replies
    1. If you mean the stereoscopic distorsion, I used a simple pixel shader to render the two frames

      Delete
    2. What I meant was that did you put two cameras onboard to get true stereo, or just used the one and split that into two identical images. Sounds like the latter with a little distortion to simulate stereo.

      Delete
    3. Yes, I did not use two cameras, but the one that comes with the parrot and splitted the view via software

      Delete
  2. hello!! where's the file haarcascade_frontalface_default.xml? i got file not found error.

    ReplyDelete
    Replies
    1. Look in the lib forlder. You will find all the dll you must copy in the bin folder including the xml file

      Delete
  3. Very nice project! btw could you make a step by step tutorial how to execute the program and connecting to hardware for dummies? I confuse a lot. Thank a lot.

    ReplyDelete
    Replies
    1. Thanks.
      In the next weeks I wil try to prepare a couple posts on this topic

      Delete
  4. How to connect kinect to ar drone? what kind of tools did you use? warm greetings from Indonesia

    ReplyDelete
    Replies
    1. Hi, If you take a look at the source code you will find everything.
      For the IDE I used Visual Studio

      Delete
    2. you have oculusparrotkinect, oculusparrotkinect.drone, etc. Which program I have to run? I just want to running kinect and ar drone, not with oculus.

      Delete
    3. In the read.me on github there is a short description on how to disable it (drawOculus=false)

      Delete
  5. I couldn't open the project, neither with vs2010 nor with vs2013!
    "incompatible version error"

    ReplyDelete
    Replies
    1. Be sure to have the XNA framework correctly installed. Here you should find what you need https://msxna.codeplex.com/releases Hope it helps

      Delete
  6. Ho do you change the distance between the viewports based on the LensSeparationDistance? this distance is fixed in your code, right?

    ReplyDelete
    Replies
    1. You are right, I get the info from the plugin, nothing more.
      To change the value, look at the documentation in the SDK.
      If I remember correclty, in the samples there is a configuration tool to calibrate these values.

      Delete
  7. Hey !
    I am running your code on Visual Studio 2013, but it does not run. It Says "A project with an Output of Class Library cannot be started directly”. Can you tell me how to fix it please.
    Thanks,

    ReplyDelete
    Replies
    1. You must set the project "OculusParrotKinect" as your statrtup project. Also be sure to have the XNA framework installed

      Delete
    2. Sir, I Already did both of that and still not working.

      Delete
    3. I just tried to compile it with my Visual Studio 2013 professional and everything works fine.
      Can you double check that all the projects in the solution load correctly?
      When you compile what errors do you get?

      Delete
    4. Sir I think I know what is the problem now it add these folder (OculusParrotKinect.Drone, OculusParrotKinect.Kinect & OculusParrotKinect.OculusRift) that is it, and it shows these two file OculusParrotKinect & OculusParrotKinectContent, but beside each one of them it write (incompatible). On Visual Studio 2013. what is the problem ? how can Fix it?
      Thanks in advance,

      Delete
    5. 1. Install DirectX
      2. Install XNA Framework 4.0 Redistribution
      3. Install XNA Game Studio 4.0 Platform Tools
      4. Install XNA Game Studio 4.0 Shared
      5. Install XNA Game Studio 4.0.vsix

      Delete
    6. Already did that Diako, Thanks a lot man. By the way I use windows 8.1 64 bit. I only need the Microsoft Kinect and the Ar Drone 2.0 part.

      Delete
    7. The message"incompatible" for the projects "OculusParrotKinect" and "OculusParrotKinectContent" in your solution, means that the XNA framework is not installed correctly.
      Try to completely remove and reinstall it. Here you will find everything you need: https://msxna.codeplex.com/releases Hope it helps

      Delete
    8. Here are the screenshots of the problem.
      https://plus.google.com/u/0/photos/yourphotos?pid=5999829414248465106&oid=106093317869324934273
      https://plus.google.com/u/0/photos/yourphotos?pid=5999829067313784530&oid=106093317869324934273
      thanks for your helps.

      Delete
    9. Still the same. I think I am going to do it on a different machine and I will let you know. Thanks a lot for your time.

      Delete
    10. I can confirm from your screenshots that the XNA framework is not installed correctly. Sorry I can not be of more help

      Delete
    11. Thanks a lot. I will get back to you once I get my hand one a new machine. Thanks again,

      Delete
    12. Now it runs, but it says Kinect not found. Its plugged in and works on Kinect developer toolkit apps. I also deleted and added back the kinect in the reference manual in the code. I also found in the your codes it's for Microsoft and Xbox kinect. If I am not mistaken. I use Microsoft KInect.

      Delete
    13. The difference in signature should be because I use an sdk from a year ago. I'm in the work of updating it with the kinectv2 preview. I will also try with a recent SDK to see if there are problems.

      Delete
    14. What is your suggestion ? Should I buy Xbox Kinect or wait for you to update it. The problem is I have to make this work in a week. I can not wait more than a week. What you think ?

      Delete
    15. When I developed this I tried with a kinect for windows and a kinect for xbox, and both worked. So I do not know why it isn't working for you. Try to use the same SDK I used at the time (1.7 if I'm not mistaken).
      For you other request, I do not think I will update the project in less than a week. I did this in my spare time and to try something new, so I can not guarantee that I will update anything in a short period of time. Sorry

      Delete
    16. I will try it now and will get back to you for the result. Thanks I know you are just trying to help here. I really appreciate that and I am sorry for being a pain here. lol

      Delete
    17. Still the same. Here is the Screenshot with debug running.
      https://plus.google.com/u/0/106093317869324934273/posts

      Delete
    18. The first warning is related to the version of your platform. Are you using the 32bit version or the 64bit for the kinect sdk? Look in the project properties and correct the target platform. That should solve your problem

      Delete
    19. This is what I mean I fix one and I get another one. I think this screenshot will make it easy for you to see my problem. by the way I fixed that before. I have win 8.1 64, but for the project platform there is only 86bit which is 32bit I guess.
      https://plus.google.com/u/0/106093317869324934273/posts
      Thanks for your helping I mean it Sir, I don't know what I could have done without your help.

      Delete
    20. But now it doesn't go for the black pop up window that shows "Kinect is not found". It stops with these two new warnings.
      https://plus.google.com/u/0/photos/yourphotos?pid=6000231373516550994&oid=106093317869324934273

      Delete
    21. It seems that you still have the dll of version 1.8 and not 1.7. Maybe it reamined from your previous try.
      Do a "clean solution" to be sure all the dlls are wiped out form the bin folder and try to recompile

      Delete
    22. Now it goes back to the black pop up windows still says " Kinect not found". I extracted your code in different place and I redo the whole steps. This time it gives me only this warning. By the way I put all of the properties platform to X86. That's what it should be right ?
      https://plus.google.com/u/0/photos?pid=6000240285721340338&oid=106093317869324934273

      Delete
    23. That should do it. I will try it myself when I go back home this evening

      Delete
    24. Okay Sir, get home safe, lol. I am waiting for your reply. I just want to get it done in a week . On Sunday I will get my Ar drone 2.0 power edition one. by the way. Does your code support Drone 2.0 live video broadcasting because the Ar Drone 2.0 has HD front camera which does not support the Drone 1.0 code which is not HD 720P. I hope I will not get any issue with the Drone 2.0 just like the Kinect.
      thanks

      Delete
    25. Hi, I can confirm that everything works fine. I also used the 1.8 version of the sdk and no problems at all.
      For the video part, I use an Ar.Drone 2.0. For the 720p stream you can't get it via Wifi. You should insert an usb stick in the drone AFAIK

      Delete
    26. Hi, I can confirm that everything works fine. I also used the 1.8 version of the sdk and no problems at all.
      For the video part, I use an Ar.Drone 2.0. For the 720p stream you can't get it via Wifi. You should insert an usb stick in the drone AFAIK

      Delete
    27. I have another pc which NetBook PC with windows 7 starter 32bit. I will try it on that too. I will get back to you soon. Do you have any idea why this is happening with me ? I used the Kinect with other app it works fine. Do you think I should get Xbox Kinect. What is the issue windows or Kinect ?
      Thanks,

      Delete
    28. and I Guess you downloaded the same code you uploaded it right ?

      Delete
    29. I used the code downloaded from github to be sure and I have used a Kinect for windows. So I do not know what is causing all your troubles

      Delete
    30. Okay Sir. let me see it on the other pc. Will get back to you. Thanks a lot for help. I am really sorry for making you to do all of these works.

      Delete
    31. My other pc has a poor graphic card. I couldn't run the pop up window. My computer is fine the code runs fine now without any errors or warnings. The Kinect works fine with the ToolKit samples. When I also run your code and I open SDK 1.8 it says it is connected to OculusParrotKinect and the little red light goes on each time I run the code, but it still says "Kinect not found". I tried everything reinstall, remove and different version. You know I still don't have the drone which I don't think that will make any problem for running the code. I don't get it why its not working there is no any error massages. I worked on it today more than 10 hours.

      Delete
    32. Sorry, but I do not know how to help you more. On my machine everything works as it should. I can not think of other possible problems

      Delete
    33. I know Sir, Thanks I will try it on another computer and will let you know.
      Lots of thanks to you Sir,

      Delete
    34. I just ran it on different computer windows 7 pro 32bit and its just the same. No errors messages only "Kinect is not found" the red light woks once I run it. My Microsoft Kinect is V 1.0 is that the problem ? I am going to install Visual Studio 2013 professional, I am also trying to get Xbox Kinect.

      Delete
    35. And I also don't have the Oculus which I don't think that is the problem. I just have the Kinect + your codes + computer. I don't have the Ar drone yet, I will get it in a couple of days, but I will not have the Oculus.

      Delete
    36. HI, I used the kinect 1.0 too (both windows and xbox). The oculus is not mandatory everything works without it

      Delete
    37. Hi,
      I am doing it from the begging step by step:
      1- I delete the Kinect reference and add it again Kinect.dll v 1.8.
      2- warning is this screenshot. I think we fix that by copying all of the lib files to the all of the folders bin>x86>debug. Am I right here. I don't want to do anything wrong. I want to fall your instructions. I wan to make sure whatever I did till now is exactly like what you did.
      Thanks,

      Delete
    38. The x86 message should be inherent to the oculus project. just change the project to compile to x86 and it should go away. if you have also changed the reference to kinect 1.8 you have done the same thing I have done. So it should work

      Delete
    39. Everything is working fine till it say "Kinect is not found" for unknown reason. It is so sad. it says that maybe because I don't have the Ar drone yet. It search for the drone and gives me back that result. Don't know if you tried that too without drone. I will try some help from some professors, Will let you know about any updates too many thanks for your helps Sir,

      Delete
  8. This comment has been removed by the author.

    ReplyDelete
  9. Thanks again Alessandro for sharing your project. Just want to mention my GazeOculus project here in case somebody is interested to know about the other projects. I used parts of the Alessandro's code and made some changes to emulate a see through head-mounted display with gaze tracking inside the display. It has nothing to do with drone and hand gestures though. Please find more information about my prototype here: http://www.itu.dk/research/eye/?page_id=671

    Also I found another cool and relevant project by Diego Araos: http://bettercoderwannabe.blogspot.dk/2013/08/oculus-ar-drone-and-kinect-completed.html

    ReplyDelete
    Replies
    1. Hey !
      I just ran your code and it works fine. great job, but why the Kinect part is not showing ? did you exclude it. Can you make the Kinect work too Sir please. Look at my comments with Mr Alessandro Colla. He is helping me a lot, but I am still straggling with it and I DO want it to work. I have to finish it in a week. I am not leaving it. By the way Sir I found some codes that you can receive a live video broadcast from the drone camera why don't you plug that in. it will be nice. Can you please help me because I don't know "namedwnam" why it is not working.
      thanks for sharing and great job.

      Delete