The Context Engine A Device to Augment Cognition

The Context Engine A Device to Augment Cognition Perry Mc. Dowell The MOVES Institute 1

Augment Cognition Add to your thinking, hopefully improving it That’s been the whole point of computers We want to make it completely natural and automatic n In some ways, similar to: w An admiral’s aide or efficient secretary w A senior NCO or subject matter expert Want to do this with a wearable computer Move nearer to ubiquitous computing 2

Literary Inspiration for the Military Context Engine “Our suits give us better eyes, better ears… better intelligence… the beauty of it [is] you don’t have to drive it, fly it, conn it, operate it, you just wear it… This leaves you with your whole mind free to handle your weapons and notice what is going on around you… If you load a mud foot down with a lot of gadgets that he has to watch, someone a lot more simply equipped – say with a stone axe – will sneak up and bash his head in while he is trying to read a vernier. ” - Robert A. Heinlein, Starship Troopers 3

Steve Mann’s Six Attributes of Wearable Computing 1. Unmonopolizing of the user's attention 1. 2. 3. 4. It does not cut you off from the outside world. You can attend to other matters while using the apparatus. Assumes computing will be a secondary activity. Provides enhanced sensory capabilities. 2. Unrestrictive to the user 1. You can do other things while using it 4

Steve Mann’s Six Attributes of Wearable Computing(cont) 3. Observable by the user · · It can get your attention continuously if you desire The output medium is constantly perceptible by the wearer. 4. Controllable by the user 5. Attentive to the environment · Environmentally aware, multimodal, multisensory. 6. Communicative to others · Can be used as a communications medium. 5

Steve Mann’s Six Attributes of Wearable Computing(cont) Implied by the above six properties is that it must also be: Constant: always on, running, and ready. Personal: human and computer are inextricably intertwined. n n w w w Prosthetic: you can adapt to it so that it acts as a true extension of mind and body; after time you forget that you are wearing it. Assertive: resists, if you wish, prohibition or requests by others for removal. Private: others can't observe or control it unless you let them. 6

Example of Uses Finding a Restaurant You’re downtown and decide you want to eat out The engine makes a recommendation based upon several factors: ¨ Location ¨ Tonight’s specials ¨ Cost ¨ Friends’ favorites ¨ Favorite foods ¨ Reviews ¨ Favorite restaurants ¨ Current wait 7

Finding a Restaurant Let’s take it one step further: n You can tell it you want to take your wife out for your anniversary - handle it w It knows your favorite romantic restaurants, so it makes a reservation at one w It contacts one of the babysitters you use and arranges services for that night w Orders flowers delivered to your wife’s office w Reminds you to buy a present, or orders one from the web for you w Schedules it with your wife’s contact engine n Tells you that everything is ready to go 8

Examples of Uses Watch Turnover At watch relief, transfer all pertinent data to your relief n List of contacts and their status, current/ pending orders, equipment status, upcoming events, etc. The system can’t forget anything, rush so it can hit the pit, misinterpret what it’s told… The system would also be able to give you advice - “No, you don’t want to turn left…” 9

Example of Uses Crossing the River You are the leader of a platoon of infantry on a covert mission, currently moving to conduct a rendezvous across a river n The bridge you were expecting to use is destroyed What are you going to do now? 10

Crossing the River (cont) Determining your current options, weighing them, and deciding upon a course of action takes time Context engine immediately determines options and the best course of action and makes a recommendation n It can explain its reasoning fairly quickly to you 11

Examples of Uses Summary In the dinner example, it acts mainly as an assistant - it knows your likes and searches data for the best match In the river example, it acts as an expert system, giving you a better solution In the watch example, it combines the two - it stores data for use later, but can have intelligence (either the previous watch officer or from the system) built in 12

Goals Our goal for this project is to take Commercially Off the Shelf (COTS) hardware and software to create a proof of concept for the Context Engine. 13

Difficult Problems Determining the “symbolic content” of the user’s environment Once it is symbolized, interpreting the environment to give information and advice Presenting the information to the user in an unobtrusive and helpful manner 14

Breakdown of the Context Engine’s Processes People Goal 2 Actions Goal 3 Goal 1 Determine Course of Action Outer World Abstraction Goal 4 Visual Display Aural Display Output to User Goal n Location Objects User’s Commands Preferences 15

Initial Process Recognize the World People Actions Outer World Abstraction To course of action Location Objects 16

Symbolic Content Having the computer determine where it is, who/what is present and what is happening n n Computer visualization Wireless recognition w Bluetooth n n Aural clues GPS 17

Current Systems Which Convert Data to Abstraction Face Recognition § Several companies has fairly effective facial recognition systems. w Lau Technology’s daughter company, Viisage, scanned all the faces at the Super Bowl, looking for terrorists § We anticipate getting an SDK from one of these companies to begin our research 18

Current Systems Which Convert Data to Abstraction Wearable belt for outdoor sports § Determines events which might indicate the wearer is incapacitated w High acceleration w Vital Signs § Alerts authorities to potential injury 19

Determining The Course of Action Goal 2 Goal 3 Goal 1 Determine Course of Action From World Recognition Goal 4 To Output to User Goal n User’s Commands 20

Determining the Course of Action Once you know what is in the environment, what does that mean? n Network solution w Take the dynamic extensibility from NPSNET-V w Embedded XML information explains what an object is and how to interact with it n SIMS style smart environment w Objects broadcast what they can do 21

Determining the Course of Action Get input from the user’s body Affective computing takes body measurements to determine your mental/ physical state n The context engine will be able to indicate when medical assistance is required n w Expedite triage also System knows not to deliver bad news when you’re angry/ tired/ morose/ etc. n System can tell you’re beginning to panic and “slaps you in the face” n 22

Affective Computing Gear This affective wearable includes a Pro. Comp sensing system (upper left corner) surrounded by four sensors, clockwise from top: respiration, galvanic skin response (GSR) (center, left), blood volume pressure (BVP), and electromyogram (EMG). This unit attaches to a PC 104 standard based computer (lower right) which receives data from a Twiddler hand-held keyboard (lower left). and displays data with the Private Eye (far right, below respiration sensor. ) (J. Healey) 23

Determining the Course of Action User can input commands which give the computer tasks to determine The user’s commands are evaluated in light of the current goals § Goals are determined either explicitly or implicitly w Explicitly means that the user has previously defined this as a goal w Implicitly means the computer has learned that this goal is important 24

Determining the Course of Action Database sharing n Gives one user the knowledge and insight of another user w Can be as simple as identifying someone w Can be as complex as planning an exercise n Will raise emphasis on training people's machines closer to the level we currently spend on training people w May be possible to train both simultaneously 25

Outputting Information to the User Visual Display From Course of Action Aural Display Output to User Preferences 26

Presenting Information to the User Interface must be: Light n Small n Unobtrusive n Intuitive n Robust n Easy to use n Cheap n Easier said than done 27

Presenting Information to the User “Fitting” the system to specific users n n Everyone wants different information User must find the given information useful è Don’t want the annoying paperclip scenario 28

29

Conclusion Context Engine must be easy and natural to use The problem has three major areas: § Recognizing the world § Determining the course of action § Presenting information to the user 30

NPS/MOVES Work Construct a virtual world and build a more complete and robust engine § Several options as to the input to the world: § Joint SIGINT Avionics Family (JSAF) Data § Plan on using agents: § To determine context from the symbolic representation § To determine recommended course of action 31

Potential Thesis Work You can work in any part of the project discussed in this brief n n n First part will be extremely difficult Second part will be difficult, but you’ll be working on a smaller part of the problem Third part will be very interesting, and is a great opportunity for someone interested in human factors work 32

Questions? Contact Information: Perry Mc. Dowell Lecturer of Computer Science plmcdowe@nps. navy. mil http: //www. movesinstitute. org/~mcdowell 33