EEC-693 793 Applied Computer Vision with Depth Cameras Lecture

EEC-693/793 Applied Computer Vision with Depth Cameras Lecture 17 Wenbing Zhao wenbing@ieee. org

Outline n n Skeleton logging Replaying in Unity with logged skeleton data

Skeleton Joint Data Logging n n You can add logging for skeleton joint data into any of your existing project with skeleton tracking enabled Make sure you add the System. IO into your namespace using System. IO; n We will use Stream. Writer class for logging to a comma separated value file (. csv). Add the following member variable to your Main. Window class: Stream. Writer sw = File. Create. Text("kinect. csv");

Skeleton Joint Data Logging n Add captions for columns. Add the following line in Window. Loaded() sw. Write. Line("Frame. No, timestsamp, MC-X, MC-Y, MC-Z, MC-D, Hip. Center-X, Hip. Center-Y, Hip. Center. Z, Hip. Center-D, Spine-X, Spine-Y, Spine-Z, Spine-D, Shoulder. Center-X, Shoulder. Center-Y, Shoulder. Center. Z, Shoulder. Center-D, Head-X, Head-Y, Head-Z, Head-D, Shoulder. Left-X, Shoulder. Left-Y, Shoulder. Left. Z, Shoulder. Left-D, Elbow. Left-X, Elbow. Left-Y, Elbow. Left-Z, Elbow. Left-D, Wrist. Left-X, Wrist. Left-Y, Wrist. Left. Z, Wrist. Left-D, Hand. Left-X, Hand. Left-Y, Hand. Left-Z, Hand. Left-D, Shoulder. Right-X, Shoulder. Right. Y, Shoulder. Right-Z, Shoulder. Right-D, Elbow. Right-X, Elbow. Right-Y, Elbow. Right-Z, Elbow. Right-D, Wrist. Right. X, Wrist. Right-Y, Wrist. Right-Z, Wrist. Right-D, Hand. Right-X, Hand. Right-Y, Hand. Right-Z, Hand. Right-D, Hip. Left. X, Hip. Left-Y, Hip. Left-Z, Hip. Left-D, Knee. Left-X, Knee. Left-Y, Knee. Left-Z, Knee. Left-D, Ankle. Left-X, Ankle. Left. Y, Ankle. Left-Z, Ankle. Left-D, Foot. Left-X, Foot. Left-Y, Foot. Left-Z, Foot. Left-D, Hip. Right-X, Hip. Right-Y, Hip. Right. Z, Hip. Right-D, Knee. Right-X, Knee. Rigt-Y, Knee. Right-Z, Knee. Right-D, Ankle. Right-X, Ankle. Right-Y, Ankle. Right. Z, Ankle. Right-D, Foot. Right-X, Foot. Right-Y, Foot. Right-Z, Foot. Right-D"); n Log per frame information, including q q q Timestamp Frame number Floor clip plane

Per Frame Information void skeleton. Frame. Ready(object sender, Skeleton. Frame. Ready. Event. Args e) { canvas 1. Children. Clear(); long timestamp = -1; int frameno = -1; float A = 0; float B = 0; float C = 0; float D = 0; using (Skeleton. Frame skeleton. Frame = e. Open. Skeleton. Frame()) { // check for frame drop. if (skeleton. Frame == null) { return; }

Per Frame Info timestamp = skeleton. Frame. Timestamp; frameno = skeleton. Frame. Number; A = skeleton. Frame. Floor. Clip. Plane. Item 1; B = skeleton. Frame. Floor. Clip. Plane. Item 2; C = skeleton. Frame. Floor. Clip. Plane. Item 3; D = skeleton. Frame. Floor. Clip. Plane. Item 4; // copy the frame data in to the collection skeleton. Frame. Copy. Skeleton. Data. To(total. Skeleton); // get the first Tracked skeleton = (from trackskeleton in total. Skeleton where trackskeleton. Tracking. State == Skeleton. Tracking. State. Tracked select trackskeleton). First. Or. Default(); // if the first skeleton returns null if (skeleton == null) return;

Log Per Frame Info if (skeleton != null && this. current. Skeleton. ID != skeleton. Tracking. Id) { this. current. Skeleton. ID = skeleton. Tracking. Id; int total. Tracked. Joints = skeleton. Joints. Where(item => item. Tracking. State == Joint. Tracking. State. Tracked). Count(); string Tracked. Time = Date. Time. Now. To. String("hh: mm: ss"); string status = "Skeleton Id: " + this. current. Skeleton. ID + ", total tracked joints: " + total. Tracked. Joints + ", Track. Time: " + Tracked. Time+"n"; this. text. Block 1. Text += status; } Draw. Skeleton(skeleton); sw. Write(frameno+”, ”); sw. Write(timestamp+", ");

Log Joint Info float xc = skeleton. Position. X; float yc = skeleton. Position. Y; float zc = skeleton. Position. Z; float cmd = A * xc + B * yc + C * zc + D; sw. Write(xc + ", " + yc + ", " + zc + ", " + cmd + ", "); for (int i = 0; i < 20; i++) { float x = skeleton. Joints[(Joint. Type)i]. Position. X; float y = skeleton. Joints[(Joint. Type)i]. Position. Y; float z = skeleton. Joints[(Joint. Type)i]. Position. Z; float dist = A * x + B * y + C * z + D; sw. Write(x + ", " + y + ", " + z + ", " + dist + ", "); } sw. Write. Line(""); }

A Sample Log If you have Excel in your computer, you can double click the csv file to open it

Replay Logged Data in Unity n n n n Create a new Unity project. It is based on the Kinect. Point. Man example as part of the ZDK download Add the floor the same way as before Create an empty game object, name it Kinect. Point. Man Under the Kinect. Point. Man, add 20 3 D spheres, and name them properly using the 20 Kinect joints Create a new C# script, I named it Back. Script. cs Attach the script to the Kinect. Point. Man Connect the 20 spheres in the scene with 20 joints listed in Back. Script. cs Save the scene as Kinect. Player

Replay Logged Data in Unity

Connect the 20 Joints to the Script

Back. Script. cs using Unity. Engine; using System. Collections; using System. Xml; using System. Component. Model; using System. Collections. Generic; using System. Linq; using System. Text; using System. IO; public class Back. Script : Mono. Behaviour { public Game. Object Hip_Center; public Game. Object Spine; public Game. Object Shoulder_Center; public Game. Object Head; public Game. Object Shoulder_Left; public Game. Object Elbow_Left; public Game. Object Wrist_Left; public Game. Object Hand_Left;

Back. Script. cs public Game. Object Shoulder_Right; public Game. Object Elbow_Right; public Game. Object Wrist_Right; public Game. Object Hand_Right; public Game. Object Hip_Left; public Game. Object Knee_Left; public Game. Object Ankle_Left; public Game. Object Foot_Left; public Game. Object Hip_Right; public Game. Object Knee_Right; public Game. Object Ankle_Right; public Game. Object Foot_Right; private Game. Object[] _bones; //internal handle for the bones of the model Game. Object hip 2 spine; Game. Object spine 2 shoulder; Game. Object shoulder 2 head; Game. Object left. Shoulder; Game. Object right. Shoulder; Game. Object left. Upper. Arm;

Back. Script. cs Game. Object left. Arm; Game. Object left. Hand; Game. Object right. Upper. Arm; Game. Object right. Hand; Game. Object left. Hip; Game. Object right. Hip; Game. Object left. Upper. Leg; Game. Object left. Foot; Game. Object right. Upper. Leg; Game. Object right. Foot; public GUIText status. Display; List exitems = new List(); List exfileitems = new List(); List> exjointsitems = new List>(); List exdata = new List();

Back. Script. cs private int guide. Counter = 0; bool skipframe = false; public Vector 3 Scale = new Vector 3(0. 001 f, 0. 001 f); float last. Ts; private Vector 3 root. Position = new Vector 3(); // Use this for initialization void Start () { //store bones in a list for easier access _bones = new Game. Object[] {Hip_Center, Spine, Shoulder_Center, Head, Shoulder_Left, Elbow_Left, Wrist_Left, Hand_Left, Shoulder_Right, Elbow_Right, Wrist_Right, Hand_Right, Hip_Left, Knee_Left, Ankle_Left, Foot_Left, Hip_Right, Knee_Right, Ankle_Right, Foot_Right}; connect. Bones(); } void Update () { if (exdata. Count == 0) return; }

Back. Script. cs void On. GUI() { if (GUI. Button (new Rect (10, 70, 150, 30), "Load Kinect v 1 Data")) { load. Config(true); Debug. Log ("Kinect v 1 data is loaded"); } if (Event. current. Equals(Event. Keyboard. Event("escape"))) { print("Quitting"); Application. Quit(); } if (Event. current. Equals(Event. Keyboard. Event("p"))) { print ("Previous Frame"); if(this. guide. Counter > 2) this. guide. Counter = this. guide. Counter - 2; Display. Next. Frame(); } if (Event. current. Equals(Event. Keyboard. Event("n"))) { print ("Next Frame"); Display. Next. Frame(); } if (Event. current. Equals(Event. Keyboard. Event("s"))) { string filename = Get. Screenshot. Filename(); print("Writing screenshot to " + filename); Application. Capture. Screenshot(filename); } }

Back. Script. cs private void load. Config(bool kinectv 1 data) { Xml. Document xml. Doc = new Xml. Document (); xml. Doc. Load ("test. xml"); Xml. Node. List file. Name = xml. Doc. Get. Elements. By. Tag. Name ("Guide. File"); if (1 != file. Name. Count) { Debug. Log ("expecting exactly one guidefile, got: " + file. Name. Count); return; } Xml. Node. List innerlist = file. Name [0]. Child. Nodes; if (1 != innerlist. Count) { Debug. Log ("guidefile should have only one child, got: " + innerlist. Count); } String fn = innerlist [0]. Inner. Text; float[][] guide. Data = get. Guide. Data (fn); exdata. Add (guide. Data); }

Back. Script. cs private float[][] get. Guide. Data(String filename) { String[] data. Strings = null; if (File. Exists(filename)) { data. Strings = File. Read. All. Lines(filename); Debug. Log("Replay: done loading data"); } else { Debug. Log("file not found: "+filename); return null; } // the first row is the caption, skip it, hence the -1 int size = data. Strings. Length-1; float[][] all. Samples = new float[size][]; 20 joints + misc. Frame Time Jagged Array: all. Samples

Back. Script. cs // the first row is the caption, skip it for (int i = 0; i < all. Samples. Length; i++) { String[] row = data. Strings[i + 1]. Split(', '); int rowlen = 1+20*3; // 1 column for ts, 20 sets of joints (x, y, z) all. Samples[i] = new float[rowlen]; int k = 0; for(int j=0; j= 2 && j <=5) continue; // for each joint, the 4 th column is for vertical distance, skip it too if(j > 6 && ((j-5) % 4)==0) continue; all. Samples[i][k++]=(float)Convert. To. Double(row[j]); if(k == 61) break; } } return all. Samples; }

Back. Script. cs string Get. Screenshot. Filename() { System. IO. Directory. Create. Directory("Screenshots"); int i=1; while (System. IO. File. Exists(System. IO. Path. Combine("Screenshots", "Screenshot" + i + ". png"))) { i++; } return System. IO. Path. Combine("Screenshots", "Screenshot" + i + ". png"); }

Back. Script. cs: Display. Next. Frame() void Display. Next. Frame() { float[][] guide. Data = exdata[0]; float[] frame; if(guide. Counter < guide. Data. Length) { frame = guide. Data[guide. Counter]; if(null == frame) { return; } // first column is frame number; second column is timestamp; first 3 are for rootposition Vector 3 jpt = new Vector 3(); int joint. Count = 20; for(int i=0; i

Back. Script. cs: connect. Bones() void connect. Bones() { this. hip 2 spine = Game. Object. Create. Primitive(Primitive. Type. Cylinder); this. spine 2 shoulder = Game. Object. Create. Primitive(Primitive. Type. Cylinder); shoulder 2 head = Game. Object. Create. Primitive(Primitive. Type. Cylinder); left. Shoulder = Game. Object. Create. Primitive(Primitive. Type. Cylinder); right. Shoulder = Game. Object. Create. Primitive(Primitive. Type. Cylinder); left. Upper. Arm = Game. Object. Create. Primitive(Primitive. Type. Cylinder); left. Hand = Game. Object. Create. Primitive(Primitive. Type. Cylinder); right. Upper. Arm = Game. Object. Create. Primitive(Primitive. Type. Cylinder); right. Hand = Game. Object. Create. Primitive(Primitive. Type. Cylinder); left. Hip = Game. Object. Create. Primitive(Primitive. Type. Cylinder); right. Hip = Game. Object. Create. Primitive(Primitive. Type. Cylinder); left. Upper. Leg = Game. Object. Create. Primitive(Primitive. Type. Cylinder); left. Foot = Game. Object. Create. Primitive(Primitive. Type. Cylinder); right. Upper. Leg = Game. Object. Create. Primitive(Primitive. Type. Cylinder); right. Foot = Game. Object. Create. Primitive(Primitive. Type. Cylinder);

Back. Script. cs: connect. Bones() hip 2 spine. transform. local. Scale = new Vector 3(0. 05 f, 0. 25 f, 0. 05 f); spine 2 shoulder. transform. local. Scale = new Vector 3(0. 05 f, 0. 25 f, 0. 05 f); shoulder 2 head. transform. local. Scale = new Vector 3(0. 05 f, 0. 1 f, 0. 05 f); left. Shoulder. transform. local. Scale = new Vector 3(0. 05 f, 0. 25 f, 0. 05 f); right. Shoulder. transform. local. Scale = new Vector 3(0. 05 f, 0. 25 f, 0. 05 f); left. Upper. Arm. transform. local. Scale = new Vector 3(0. 05 f, 0. 25 f, 0. 05 f); left. Hand. transform. local. Scale = new Vector 3(0. 05 f, 0. 1 f, 0. 05 f); right. Upper. Arm. transform. local. Scale = new Vector 3(0. 05 f, 0. 25 f, 0. 05 f); right. Hand. transform. local. Scale = new Vector 3(0. 05 f, 0. 1 f, 0. 05 f); left. Hip. transform. local. Scale = new Vector 3(0. 05 f, 0. 1 f, 0. 05 f); right. Hip. transform. local. Scale = new Vector 3(0. 05 f, 0. 1 f, 0. 05 f); left. Upper. Leg. transform. local. Scale = new Vector 3(0. 05 f, 0. 25 f, 0. 05 f); left. Foot. transform. local. Scale = new Vector 3(0. 05 f, 0. 15 f, 0. 05 f); right. Upper. Leg. transform. local. Scale = new Vector 3(0. 15 f, 0. 25 f, 0. 15 f); right. Foot. transform. local. Scale = new Vector 3(0. 1 f, 0. 1 f); }

Back. Script. cs void update. Bones() { update. One. Bone(hip 2 spine, Hip_Center, Spine); update. One. Bone(spine 2 shoulder, Spine, Shoulder_Center); update. One. Bone(shoulder 2 head, Shoulder_Center, Head); update. One. Bone(left. Shoulder, Shoulder_Center, Shoulder_Left); update. One. Bone(right. Shoulder, Shoulder_Center, Shoulder_Right); update. One. Bone(left. Upper. Arm, Shoulder_Left, Elbow_Left); update. One. Bone(left. Arm, Elbow_Left, Wrist_Left); update. One. Bone(left. Hand, Wrist_Left, Hand_Left); update. One. Bone(right. Upper. Arm, Shoulder_Right, Elbow_Right); update. One. Bone(right. Arm, Elbow_Right, Wrist_Right); update. One. Bone(right. Hand, Wrist_Right, Hand_Right); update. One. Bone(left. Hip, Hip_Center, Hip_Left); update. One. Bone(right. Hip, Hip_Center, Hip_Right); update. One. Bone(left. Upper. Leg, Hip_Left, Knee_Left); update. One. Bone(left. Leg, Knee_Left, Ankle_Left); update. One. Bone(left. Foot, Ankle_Left, Foot_Left); update. One. Bone(right. Upper. Leg, Hip_Right, Knee_Right); update. One. Bone(right. Leg, Knee_Right, Ankle_Right); update. One. Bone(right. Foot, Ankle_Right, Foot_Right); }

Back. Script. cs void update. One. Bone(Game. Object bone, Game. Object start, Game. Object end) { Vector 3 v 3 End = end. transform. position; Vector 3 v 3 Start = start. transform. position; bone. transform. position = (v 3 End-v 3 Start)/2. 0 f + v 3 Start; bone. transform. rotation = Quaternion. From. To. Rotation(Vector 3. up, v 3 End-v 3 Start); }

To Run The Replay Unity App n n n Make sure in the Unity app folder you have the following two files: test. xml and kinect. csv You need to locate kinect. csv you saved previously for the logging part and copy it here Kinect. csv may have to be cleaned up a little before feeding to the Unity app q n Make sure the last row is complete. Remove the row if incomplete test. xml content: kinect. csv

Challenge Tasks n n n For logging, add a start and stop button for starting and stopping the logging For replaying, also add a start and stop button, and a target-framerate button, for automated replaying (instead of manually display frame by frame). Tweak parameters so that different frame rates can be used: default, 30 frames/second, 20, and 10, etc. For replaying, add display for joint angle for a joint that you choose