Set Up for Kinect-Fanuc Interface This guide walks you through the steps to set up an interface between the

Xbox Kinect camera and a Fanuc Robot arm. The interface allows the Fanuc robotic arm to interact with visual stimulus in its environment through the information obtained from the Kinect camera that is processed in LabVIEW. This setup lets new projects build applications for the Kinect and Fanuc robotic arm without rebuilding the communication between the two. If there are problems encountered during this project set up, send questions to: Andrew Schoer aschoer@wustl.edu Joshua Remba jremba@wustl.edu Ed Richter ed@ese.wustl.edu Preparation For this project setup you will need An engineering department PC o Internet connection o At least 4 GB of RAM o Graphics card that supports DirectX 9 o Windows 7 or newer o Administrator rights on the PC Fanuc Robtic Arm (LRMate 200ic) Xbox Kinect for Windows Visual Studio 2010 LabVIEW 2011 MATLAB 2012

This project is intended to be set up on a Wash U. engineering department machine. The PC and the robot must be on the same network connection for the project to work. Originally, this project was setup on PC that ran Windows 7, a first generation Xbox Kinect, and an LRMate 200ic Fanuc Robot. Newer versions of the hardware would likely work with this setup, but nothing is guaranteed. WARNING: The Fanuc robotic arm is dangerous if used incorrectly. Users of this robot should take the Fanuc Certified Education Robot Training (CERT) course before operating the robot. Everything described in this project setup is specifically for Windows 7. When setting up on another Windows operating system, the steps will be similar, but not exactly as stated

Outline of Steps 1. Download project file and Windows update a. Project .zip file b. Windows 2003 Resource Kit 2. Download Kinect and DirectX Software a. Kinect for Windows SDK v1.7 b. Kinect for Windows Developer Toolkit v1.7 c. Kinect Camera drivers d. DirectX End-User Runtimes e. DirectX Software Development Kit 3. Check PC for these programs, and install if necessary a. Microsoft Visual Studio 2010 i. Microsoft Visual Studio 2010 Service Pack 1 b. MATLAB 2012 c. LabVIEW 2011 (with Vision) 4. Make Registry Changes 5. Add SharedMem to the path 6. Modify DepthWithColor-D3D 7. Change location references in the LabVIEW code Note: If specific instructions for an installation are not given, just follow prompts and press next/accept/finish to get through the installation process.

1. Download project file and Windows Update

a) A .zip file of the project is provided by your project advisor. If you do not already have it, project files are available to download at the report website, <http://eyerobot.weebly.com/download.html>. This .zip file contains all of the necessary files to run the interface code once the setup has been completed, including all of the Kinect, LabVIEW, MATLAB, and SharedMem.DLL code. Be sure to save these files to a location that is easy to find because there are multiple references to the file locations in the code that you will have to edit in step 7. b) Download and install the Windows 2003 Resource Kit which is necessary for the project to function correctly and can be found at < http://www.microsoft.com/en-us/download/details.aspx?id=17657>.

2. Download Kinect and DirectX Software

a) The software download for Kinect for Windows SDK v1.7 can be found at <http://www.microsoft.com/en-us/download/details.aspx?id=36996>. Follow the instructions for installation found at the website which are copied here:

1. Make sure the Kinect sensor is not plugged into any of the USB ports on the computer. 2. If you have the Kinect for Windows v1.0, v1.5, or v1.6 SDK currently installed, close any open samples, the Sample Browser, etc. and skip to step 5. Kinect for Windows v1.7 will upgrade the previous version. 3. Remove any other drivers for the Kinect sensor. 4. If you have Microsoft Server Speech Platform 10.2 installed, uninstall the Microsoft Server Speech Platform Runtime and SDK components including both the x86 and x64 bit versions, plus the Microsoft Server Speech Recognition Language - Kinect Language Pack. 5. Close Visual Studio. You must close Visual Studio before installing the SDK and then restart it after installation to pick up environment variables that the SDK requires. 6. From the download location, double-click on KinectSDK-v1.7-Setup.exe. This single installer works for both 32-bit and 64-bit Windows. 7. Once the SDK has completed installing successfully, ensure the Kinect sensor is plugged into an external power source and then plug the Kinect sensor into the PC's USB port. The drivers will load automatically. b) After the installation of the SDK is complete, you should be prompted to install the Kinect for Windows Developer Toolkit v1.7. If no prompt appears, the download can be found at < http://www.microsoft.com/en-us/download/details.aspx?id=36998> c) Once these installations are complete, you can plug the Kinect Camera into any USB port and the drivers will install automatically. d) DirectX End-User Runtimes can be found at < http://www.microsoft.com/en-us/download/details.aspx?displaylang=en&id=8109> e) DirectX Software Development Kit at < http://www.microsoft.com/en-us/download/details.aspx?displaylang=en&id=6812> After this step, all Kinect applications should be able to run.

3. Check PC for these programs, and install if necessary

a) Microsoft Visual Studio 2010: Some Wash U engineering computers has Microsoft Visual Studio 2010 installed. In this case, skip to the indented text. If you do not have Visual Studio 2010, speak with your project advisor to get it installed on your machine. Wash U already owns this software so it does not need to be purchased again. Note: You may try to run this on a newer version of Visual Studio, but there is no guarantee it will function properly. During the project design, we were unable to edit the programs successfully in Visual Studio 2012. a. Once you have Visual Studio 2010, install the Visual Studio 2010, Service Pack 1 < http://www.microsoft.com/en-us/download/details.aspx?id=23691>

b) LabVIEW 2011: If your machine does not have LabVIEW installed, speak with your project advisor to get a copy put onto your computer. a. The LabVIEW code uses VIs from the LabVIEW Vision extension which can be downloaded from < http://www.ni.com/labview/vision/>. Again, speak with your project advisor if there are any licensing issues with this download. c) MATLAB 2012: If your machine does not have LabVIEW installed, speak with your project advisor to get a copy put onto your computer.

4. Make Registry Changes

Changes to the Windows registry must be made to allow the user to allocate large blocks of memory necessary to the functioning of the interface programs. To make these changes, the user must have administrator rights on the machine. 1. Open the command prompt 2. Type cd \Program Files (x86)\Windows Resource Kits\Tools and press enter 3. Type C:\Program Files (x86)\Windows Resource Kits\Tools>ntrights -u jar4 +r SeLockMemoryPrivilege and press enter a. Replace jar4 with your user name b. Should look like Figure 1 when done 4. Log out and back in to activate your new privileges

Figure 1: Command Prompt SeLockMemoryPrivilege

5. Add SharedMem to the path

The SharedMem.dll must be added to the Windows path so it can be found when the program is called. To do this, open the control panel, click System and Security, then click System. On the left side of the window, click Advanced system settings, marked 1 in Figure 2.

Figure 2: Adding SharedMem to path

In the window that pops up, click the Environment Variables button on the bottom right, marked 2 in Figure 2. In the environment variables window, scroll to Path and double click it in the bottom scroll pane, labeled 3 in Figure 3. Finally, in the Edit System Variables window that pops up, add the SharedMem location to the end of the list shown as 4 in Figure 3. For example, the file location that you add to the end of the Variable value field would look like this: Y:\497-summer2013\KinectFanuc497\SharedMe m\x64\Debug. Be sure to put the location of your SharedMem file, downloaded in Step 1a.

Figure 3: Adding SharedMem to path

6. Modify DepthWithColor-D3D
If you have multiple versions of Microsoft Visual Studio installed, you need to make Visual Studio 2010 the default launch program for the DepthWithColor-D3D.sln solution file. Open Windows Explorer and find the solution file. Highlight it by single clicking. Look at the upper left of the window for the Open drop down menu and click choose default program from this menu, as shown in Figure 4. Selecting Visual Studio 2010 will set this file to always open in Visual Studio 2010 unless otherwise specified.

Figure 4: Setting Visual Studio 2010 as default

Open DepthWithColor-D3D.sln in Visual Studio 2010. Make sure that the platform window at the top says Debug and not Release, and x64 and not Win32 change these settings in their drop down menus if necessary.

Figure 5: Debug and x64

When DepthWithColor-D3D.sln is opened in Visual Studio, there will be a side window called the Solution Explorer that displays all of the different parts to DepthWithColor-D3D. In the solution explorer, right click on the DepthWithColor-D3D header. Click properties in the drop down menu that appears. In the Property Pages window, shown in Figure 6, click Configuration Properties Linker General Additional Library Directories, and add the SharedMem\x64\Debug file to this field. (Example: (Y):\zipfile\SharedMem\x64\Debug)

Figure 6: Adding to Additional Library Directories

Next, in the same Property Pages window, go to Configuration Properties Linker Input Additional Dependencies and add SharedMemDLL.lib to this field, followed by a semicolon.

Figure 7: Adding to Additional Dependencies

In the solution explorer, double click DepthWithColor-D3D.cpp to open it. Edit the second #include statement, on line 8 to be the location of your zip file. (Example: #include "Y:\497-summer2013\SharedMem\SharedMemDLL\SharedMemDLL.h.)

Figure 8: #include Statement

7. Change Location References in the LabVIEW Code

In each of the following files, there is a reference to a file location that needs to be changed. The change is described on the left, and the figure on the right side highlights where in the VI the change needs to occur. After you make the change, you will want to right click on the data structure, select Data Operations and choose Make Default Value. Main When the calibrate case-statement is false, the location of the saved transformation matrix needs to be specified. This is the same file location that the transformation matrix will be saved to which you will change in calibrate.

Figure 9: Main VI

AcquireImage

On the front panel, there is an integrated command line field. This is what is launching the Kinect application when the VI is run. This field needs to be changed to the file location of the .exe file created to run DepthWithColor-D3D in your project setup.

Figure 10: Acquire Image VI

Calibrate In the waitForLearn state, the location that either saves or loads the template needs to be changed to the file location that it will save/load from in your project setup.

Figure 11: Calibrate, waitForLearn

On the right side of the VI, there is a case statement for saving or loading the transformation matrix. This location reference needs to be changed to the location of your transformation matrix. This is the same location mentioned above in changes made to the Main VI.
Figure 12: Calibrate, Transformation Matrix

MATLAB Transform On the block diagram, there is a MATLAB script node. The first line of the script node is a change directory statement. This needs to be changed to the location for your projects MATLAB files. Write cd, and then the file location.

Figure 13: MATLAB Transform

getCoordinates-Multimarker On the left side of the block diagram, at the beginning of the while loop, there is a location reference that leads to the location of the folder that holds the three template images. This file needs to be changed to the file you are using to store the template images.

Figure 14: getCoordinates-Multimarker Templates

Also in getCoordinates-Multimarker, in the findMarker state, there is change directory line in a MATLAB script node that needs to be changed to the project MATLAB folder. This folder is the same as the one changed earlier in the MATLAB Transform VI.

Figure 15: getCoordinates-Multimarker MATLAB

www.eyerobot.weebly.com