Systems to Capture Everything Beyond cameras and desktops

Systems to Capture Everything: Beyond cameras and desktops www. My. Life. Bits. com Gordon Bell, Jim Gemmell, Roger Lueder

Outline l My. Life. Bits aka Memex l How has the project evolved? l How do we use My. Life. Bits? l How is it built? l Shape of the database? l CARPE- Continuous archiving and recording of personal experience l What is the vision? l Relevance for devices and software?

I am data

History: Telepresence Tele-presentations Tele-meetings

Ambience and Presence: Being there while being here Dining at home on the “Orient Express”

History: The remote worker rediscovers the PERSONAL computer

Oct 1998 Can we scan your books and put them online? Raj Reddy Sure! Don’t worry about copyright stuff. Microsoft has lots of lawyers

1999 – Scanning starts in earnest “we” start to scan, put content into folders & files

My docs and archive Library/file cab X- Employer Active Employer Library/file cab Employer Self. . Biographical Project Employer Business Invests, family $s, & Legal Library/file cab X-Employer Library/file cab Library/file cab <1980 s Library/file cab Project Personal, including Medical

Now that it’s in Cyberspace How do you remember the 20, 000+ file names? Or in which of 1500 folders they live? What’s about a tool for finding stuff?

Jan 2001 CACM “A Personal Digital Store” l l 16 GB; +2/yr A good place to stop Began search for search engines, especially for email. Jim suggests that we build a system that would be easier to use and have many more capabilities.

2001 Capture goes beyond paper

Gordon, You should be using a database. Jim, I don’t need no stinkin’ database!

Re-discovery of Memex As We May Think, Vannevar Bush, 1945 “A memex is a device in which an individual stores all his books, records, and communications, and which is mechanized so that it may be consulted with exceeding speed and flexibility” l Full-text search, text & audio annotations, and hyperlinks

Even more capture l Telephone calls, more video, all web pages visited, keyboard and mouse usage logging, radio, TV…

2003 - Sense. Cam

Feb 2005 Epiphany! Memex is a database & personal TP system

Demo Clips & Screens

747 Screen…

Vue de jour

Timeline

Pivoting: contact> call> t> web page

GPS Photo location

Reports

The Stew family tree Copyright Mark Stewart, 2004

Vibe report

Quindi Meeting Capture

Sense. Cam

Sense. Cam around Cambridge

My. Life. Bits Software

Everything goes in a database l My. LIfe. Bits need all the features of a database l If we didn’t use one, we’ll eventually create one! Files as blobs; sync with file system for legacy apps We are part of Jim Gray’s Bay Area Research Lab l l (Consistency, Indexing, Pivoting, Queries, Speed/scalability, Backup, Replication) SQL

My. Life. Bits Software Room Capture GPS import & Map display Sense. Cam Import files VIBE logging My. Life. Bits Shell Text annotation tool Screen saver My. Life. Bits store Radio capture & EPG Internet Browser tool Legacy applications database MAPI interface files Pocket. PC transfer tool IM capture Outlook interface TV capture tool Pocket. Radio player Voice annotation tool Telephone capture tool TV EPG download tool Legacy email client

Common ground with Win. FS: Items, Links & Meta-data Photo of Event Caller in Phone Call Annotates

Photo. Finder - Shneiderman and Kang

The Shape & Size of Gordon’s Life. Bits

My. Life. Bits 10/31/2005 242 K items 110 GB by number of Items.

My. Life. Bits 10/31/05 Size (MB) by Type 110 GB 242 K items By Size (GB) Bell Growth: 1 GB/month =1. 1 TB/lifetime

15, 000 photos Year 1997 1999 2001 2002 2003 2005 Mpix. 25 1 2 3 4 5 Manufacturer Ricoh Kodak Canon Sony Panasonic

Monthly & Lifetime Storage Use Item 1 MB Books|reports 5 KB Emails 100 KB Image scans 0. 4 MB Photos 75 KB Web pages|docs 100 MB Music 1 KB/s Listened audio, speech 50 KB Daily photos 2 GB/hr TV Daily number 0. 1 Total* MB|GB Month|Life 3 100 5 10 100 0. 1 40, 000 13 13 100 188 250 1, 000 1, 250 4 200, 000

Observations about use(rs) 1. 2. 3. 4. 5. 6. 7. Cell phone sized device (CPSD) will be the platform! On Applications… think about CPSD as the platform and context l Search is the “killer app” pretty much as Bush described. l Screen savers “memory refreshers” also provide ambience l Where did my day to? Users are unwilling to spend time managing their computers or data. l Meta-data, classification, etc. must be automatic l User-input meta-data e. g. Dublin Core – naïve’ Librarian’s dream. l We have nice scheme for classification using facets. It requires work. Time is the most important meta-data. Photos: place (GPS), subject. Folders are a good and bad idea. l Most users don’t know what they are or how they work l If used, over time, they become useless: too many, miss-file, etc. User should put “every” information fragment into the system. e. g. , to dos, call backs, business cards numbers, attention events. It pays. Same information in multiple places always becomes obsolete.

Capturing Everything: Phone calls in context of cell phone as a platform for communication and capture l Formal Meetings l Rooms l Everything in daily life l Personal health and medical monitoring l Memex for scientists and engineers l

Body. Media Output

Polysomnogram for sleep apnea. Real time health monitoring

Microsoft Research Sens. Cam II Sensors: l VGA camera w/ wide-angle lens l light level in R, G, B and white l ambient temperature l passive infrared for person detection l accelerometers l three, programmable buttons, LEDs, sounder l audio level & audio recording l USB 2 and SD memory. 1 -2 K photos/day l Not GPS

Sense. Cam University Grant Program MSFT supplies money, software, Sense. Cams l l l Memex vision: Notebook for engineers & scientists Medical & health: observations & memory recall, including diet and exercise Education: How do people learn? Help me learn/remember! Tourist e. g. museum experience Plumbing l l Security Filtering many images, voice & location annotation

More real time experience capture Real time medical & health monitoring l MIT. Deb Roy home capture to understan how his children learn l U. of Tokyo. Ubiquitous home l Columbia U. Voice & sound record & profile l MIT. i. Dat. Electronic lab that records everything into your notebook l

Experience Retrieval in a Ubiquitous Home Ubiquitious Home Experience Retrieval in a (chamds, byon, yamasaki, aizawa)@hal. k. u-tokyo. ac. jp

MIT i. DAT Project aka notebook

Samsung challenge l Going beyond plain old photography and videography l l Print, view, and file in scrapbook or shoebox Digitized bits offers worldwide sharing and easy sharing Screensaver is useful, but is it a killer app? The cell phone sized device (CPSD)… one device l l Next generation platform Phones and messaging e. g. sms, mail, web, i. M, blogging Audio, photo, video record and viewing (incl. broadcast) Within 5 years and with supplemental devices, will take on the PC

Capture, storage, retrieval, and display Challenge putting them together l Capture …. l l Storage l l Cell phone sized devices (CPSD). The “killer app”!! Consumer… photo, video, audio… experience Professional Capture Archival Retrieval = f(use). Archive… ambience Display l l l Personal: Cell phone PC Wall

www. My. Life. Bits. com

BONUS SLIDES

Challenges l Data-types l l Going beyond a PC to a distributed environment l l l l Quantity expanding i. e. info explosion New capabilities e. g. real time create new data-types Meta-data to increase value & provide pivots Network environment, including media center Into the cloud. Especially important for social aspects Periphery… smart buildings, objects, Backup, migration, and caching for beyond a Terabyte Expanding network: PC > LANs > web > p 2 p(eer) Schema sharing among disparate systems CARPE (real time data capture) l Rooms, phone calls, Sense. Cam, Health transducers, etc. Security, privacy, forgetfulness, deniability, etc.

More challenges l l l l Dear Appy: Monitoring and automatic migration of files that are unlikely to be understood on future platforms as well as platform migration. Get What I Need: GWIN…Endless, but evolutionary improvements in search: misspellings, stemming synonyms Endless frontier of schema and extensions to them for new applications e. g. making org charts, family relationships. CARPE… a whole new game! Versioning is essential Scaling. . We don’t know what happens at a Terabyte What can, should be, or will be in the cloud? Books… videos Will we be allowed to use such systems? Copyright laws vary: E. g. ripping CDs, copy of anything, photos, conversations

The “dear appy” problem Dear Appy, How committed are you? Please come back to me. Forever yours truly, Lost and forgotten data l Who’s responsible? l. Media or 8 track cassette, 8” floppy l. Evolving platform, file, and database l. Evolving, incompatible standards & formats for legacy data that disregard ancestors l. Evolving and/or disappearing apps

Is Cyberspace a safe store? Don’t your physical records e. g. paper last forever? What about information on your CDs, tapes, hard drives, solid state devices?

Automatic classification problem l l XML on bills and imported content… transactions We need to download classifications rather than build them l l Definitions & synonyms should help find what I want Today it is too expensive to manually classify scanned paper. E. g. “right time” meta-data is critical! We hope “the system” can classify papers and other documents e. g. bills. Ideally, build Dublin Core In 10 years we need all documents to appear electronically & classified with a little help from me