Sensory Interfaces Augmented Reality and Edutainment Tools for

Sensory Interfaces, Augmented Reality and Edutainment Tools for Edutainment

Background Wu Pin Fong Virtual Reality Consulting company, Xtensory Inc, California, America Total 25 years industrial experience Done virtual reality a long time – 14 years Emphasis mixed reality support for industry

Virtual Reality Background

Started Worked at Digital Equipment (now part of Hewlett Packard) 1990 Start the virtual reality program at Digital First major US company to support virtual reality

Undocumented VR History n n n Working for a major american computer company Able to work with all the key VR developers VR small community then

Relationships Jaron Lanier, VPL Research Scott Fisher, NASA Dr Tom Furness, HITL Labs, University of Washington Carolina Cruz , CAVE inventor, Iowa State University Rudy Darken, Naval Post Graduate School

Others n William Bricken n n Autodesk HITL Labs Foundation for World. Tool. Kit Device manufacturers Ascension, Polhemius, Intersense, Immersion, etc

Jaron Lanier VPL Research Taught me Virtual Reality n Worked on VR project For SIGGRAPH 1990 n

VPL Research in 1990 n Very exciting n New technology n Mark Bolas (later to start Fake. Space)

Tom Furness Director, HITL Labs n n Came from Aerospace Medical Research Laboratory (USAF), Wright-Patterson AFB, OH. Helped get funding from Digital Equipemt for HITL Labs

Electronic Visualization Lab CAVE n The CAVE is a projection based virtual reality system developed at the Electronic Visualization Lab created by Carolina Cruz-Neira

Naval Post Graduate School n n Rudy Darken, Director MOVES Institute, Monterey Post Graduate School Edutainment game - America's Army

Some Projects NASA American army, navy Wright patterson air force base, OH FAA Bank of America Mayo Clinic Toshiba HITL labs, University of Washington Hasbro Lockheed Schlumberger Monterey naval school

Over. View n n Need make it easy to develop mixed reality games Provide Standard components

Why components n n Many mixed reality games involve people who are not virtual reality experts Most edutainment projects assume n n completely new software Control of actual real data Expert programmers Real world need interface with legacy software

Edutainment games n Out of research into actual industrial use

Today n n n Discuss one standard component device support Example of its use Special device support – Haptics A haptics example Component for building virtual cities for use in mixed reality games

Device Support Component

Xvs. Link Standard device component for device support Paper - A Component-based Sensor Architecture for CAVEs (CAVE Automated Virtual Environments), Proceedings of Immersive Projection Technology Workshop, Iowa State University, 2000.

Industry tested since 1996 n n n Not research Used in actual products Toshiba University of Pennsylvania –virtual human Jack Mayo Clinic

Standard Device Interface Importance of components Example of components – standardised device interfaces

Xvs. Link Purpose n Instrument existing legacy software n Linux, Windows, Sun Solaris, HP, SGI n Easy to use VR devices n Interfaces to non VR devices

Work Closely with Device Manufacturers n So closely actually referenced in their manuals Example Intersense IS 600 manual

Focused - Does not provide n user interaction techniques n scene graph (or transform tree) n graphics techniques

Provides n handle the device control layer n provide communication between hosts n hook to higher levels of an edutainment system

Xvs. Link Functional Diagram Virtual vehicle Device support Session capture & playback Network support Users Developers administrators Legacy software User interaction analysis

One industrial Difference n n Need to monitor user interaction with the game Capture user data for later playback and analysis

Layered Architecture User interaction tools Application Xvs. Link device interface transport Legacy software

Component Device Interface n Provide Easy Access n Common Interface to different devices n n Access to VR peripherals from any application Storage and replay of device data

Xvs. Link n Xvs. Link is a C++ class library with both a C and C++ interface that provides a standard device interface for integrating devices into simulator applications.

Make devices invisible to user n n n Writing a driver for a new device usually a few days Special commands for specific device drivers Windows and linux kernel level drivers

Operation n Open networked, serial, USB, shared memory connections n Invisible to user n Generic devices by functions n connection automatically restored if the device server is shut down and restarted

Design n n generic devices polled or queued local or remote actual or virtual devices identified by logical name

Design n polled or queued- application can specify n polled n n n reads data asynchronously typical example: tracker, valuator queued n n circular typical example: keyboard, button

Local and Remote devices n remote devices treated as local devices n Local client and server n queries for device capabilities

Design n local or remote device n n application unaware of source can use actual or virtual data n Device data from hardware or calculations

Device Mapping n n interfaces specify a set of capabilities device is mapped onto one or more interfaces Client connects to the different interfaces run-time bindings

Design n identified by logical name n Device can map to any name n new names can be created as needed

Extensible n Generic Type devices n Layered devices n Multiple-behavior devices

Design generic devices n n n locator valuator button

Device added dynamically n dynamic device support n examples n n n Intersense tracker Fakespace wand, pinch. Glove Phantom haptic

Support n Linux, windows, SUN, HP, SGI n hooks to other tools, packages n Hooks for legacy systems

Example of Xvs. Link’s Use

Virtual Collaborative Clinic

NASA – Virtual Collaborative Clinic 1998 Dr. Muriel Ross, head of NASA's biometrics research at Ames Out of research into deployment usually no news

Virtual Collaborative clinic

Collaboration Partners

Objectives Medical support in remote locations surgical try-outs in mixed reality after which the most adequate procedure can be stored to be used in the actual operation. project computer images onto the patient in order to guide physicians during surgical interventions.

Special device support – Haptics

Haptic Rendering of Virtual Stimuli

Haptic

Haptic Information n Haptic information is a form of sensory stimulation that is acquired through collisions with objects

Using the Haptic Sensory Channel n n n Force, vibration, texture object shape, compliance, impact, contact, sliding, slipping, torque and kinematic constraint. encoding the feel and movement of real or virtual objects during manipulation or exploration

Haptic Exploratory Procedures

Human Sensing

Force Feedback

Force Feedback

Rutgers

Haptics

Example

Xtensory’s Tactile Controller

Shape Memory Alloy (SMA) a system of long stroke vertical pin-rods, using a coil-form Shape Memory Alloy (SMA) as an actuator. This material can be stretched or deformed from its original shape but would spontaneously return to its original shape when heated.

Tactor Sensor

Tactor Array n n a 16 by 16 array of pinrods high-speed update rates

Tactor Array in Action

Operation n n orchestrating the vertical motion of a dense array of pin-rods. The effect is similar to a children's "Pop up book", where a flat surface transforms into a 3 D form. the driving method is scalable so that tactor arrays of any size can be constructed

A Haptics Example

Call Options Game n n n Done for major american bank to train its traders Conceptual architecture and framework for mixing real and virtual data Added haptically rendering virtual stimuli

The Game - Call Options n n n The options portfolio consist of options with 3 different times to expiration. The gamma value of an option is increasing dramatically just before expiration if the option is about at-the-money. Local peaks or valleys therefore occur for each strike, near to maturity. (Long positions give positive gamma and short positions give negative gamma).

Using Haptics n n Peaks and valleys shown visually rendered haptically Mix real and virtual data

In Operation

Another Haptics Example

Enclosed Hazardous Spaces

Narrow field of View

Constrained Space n n Need sense of touch to move haptic orienting cues play a critical role

Inspection Game n Mix real object – containment vessel n Virtual hazards n Tactile sensing n Training

Virtual City Support Component

Why n n n Many mixed reality games will use actual city buildings, streets, locations Need make it easy to mix real and virtual for actual cities Military – urban operations

Urban Operations n n In the last 10 years, however, the U. S. has committed military forces to farflung urban operations in Panama, Somalia, Liberia, Bosnia and now Kosovo. The Marine Corps alone has been involved in urban operations in 237 of its last 250 deployments.

Urban Operations n Chracterised by extremely short firing ranges imposed by a city's long canyons of high-rise buildings and dense thickets of trackless housing and teeming alleys n Need for local infantry autonomy n No command control capability

Intelligence & Reconnaissance Collection Report Transmission Eyes, Ears Mouth Feet Eyes, Ears Written Optics Horse, Drums, Smoke, Mirrors, etc… Telegraph Aircraft, Camera Image Radio, Computer Motion Imagery, Hyperspectral

Requirements n n Quick and easy way to build virtual cities for training games Acuisition of motion imagery from camera phones, UAVs Build virtual cities from analysis of motion imagery Time Frame - Late 1990’s

Xtensory Contracts n n n Enhanced Multiplatform, Integrated Virtual Urban Warfare Simulation. Navy SBIR N 99 -063. Soldier-Computer Interface for Enhanced Interaction and Communication. Army SBIR A 95035 Immersive Visualization of Complex Situations for Mission Rehearsal. Army SBIR A 95 -089.

Component n n n Motion Imagery analysis (acquired from another company) Polygon processing into low polygon count models, autocad, 3 DS file formats Good way to build virtual cities

Conclusion n Shown how use of component tools help in making industrial strength edutainment games