Information management workflow and discovery check -in for

Information management, workflow and discovery /check -in for project definitions Peter Fox Xinformatics – ITEC 6961/CSCI 6960/ERTH-6963 -01 Week 9, April 5, 2011 1

Contents • • Review of assignment 3 presentations Reading (re-organized) Information management Information workflow Information discovery Checking in for project definitions? Next class 2

Discussion – A 3 • What did you learn? • What was easy/ hard? • What did you learn from others? • Will you ever look at an information system the same again? – Sorry? – Happy? 3

Logical Collections • The primary goal of a Management system is to abstract the physical collection into logical collections. The resulting view is a uniform homogeneous collection. – Identifying naming conventions and organization – Aligning cataloguing and naming to facilitate search, access, use – Provision of contextual information 4

Physical Handling • This layer maps between the physical to the logical views. Here you find items like replication, backup, caching, etc. – Where and who does it come from? – How is it transferred into a physical form? – Backup, archiving, and caching… – Formats – More --- naming conventions 5

Interoperability Support • Normally the information does not reside in the same place, or various collections (like catalogues) should be put together in the same logical collection. – Bit/byte and platform/ wire neutral encodings – Programming or application interface access – Structure and vocabulary (metadata) conventions and standards 6

Security • Access authorization and change verification. This is the basis of trusting your information. – What mechanisms exist for securing? – Who performs this task? – Change and versioning (yes, the information may change), who does this, how? – Who has access? – How are access methods controlled, audited? – Who and what – authentication and authorization? – Encryption and integrity 7

Ownership • Define who is responsible for quality and meaning – Rights and policies – definition and enforcement – Limitations on access and use – Requirements for acknowledgement and use – Who and how is quality defined and ensured? – Who may ownership migrate too? – How to address replication? – How to address revised/ derivative products? 8

Metadata • Metadata are data about data. • Metainformation are information about information – How to know what conventions, standards, best practices exist? – How to use them, what tools? – Understanding costs of incomplete and inconsistent metadata – Understanding the line between metadata and when it is blurred – Knowing where and how to manage metadata and where to store it (and where not to) 9

Persistence • Definition of lifetime. Deployment of mechanisms to counteract technology obsolescence. – Where will you put your information so that someone else (e. g. one of your class members) can access it? – What happens after the class, the semester, after you graduate? – What other factors are there to consider? 10

Discovery • Ability to identify useful relations and information inside the collection – If you choose (see ownership and security), how does someone find your information? – How would you provide discovery of collections, versus files, versus ‘bits’? – How to enable the narrowest/ broadest discovery? 11

Dissemination • Mechanism to make aware the interested parties of changes and additions to the collections. – Who should do this? – How and what needs to be put in place? – How to advertise? – How to inform about updates? – How to track use, significance? 12

Summary of Management • • • Creation of logical collections Physical handling Interoperability support Security support Ownership Metadata collection, management and access. Persistence Knowledge and information discovery Dissemination and publication 13

Information Workflow • What it is? • Why you would use it? • Some pointers to workflow systems 14

What is a workflow? • General definition: series of tasks performed to produce a final outcome • Information workflow – “analysis pipeline” – Automate tedious jobs that users traditionally performed by hand for each dataset – Process large volumes of data/ information faster than one could do by hand 15

Background: Business Workflows • Example: planning a trip • Need to perform a series of tasks: book a flight, reserve a hotel room, arrange for a rental car, etc. • Each task may depend on outcome of previous task – Days you reserve the hotel depend on days of the flight – If hotel has shuttle service, may not need to rent a car 16

What about information workflows? • Perform a set of transformations/ operations on a data or information source • Examples – Generating images from raw data – Identifying areas of interest from a large information source – Classifying set of objects – Querying a web service for more information on a set of objects – Many others… 17

More on Workflows • Formal models of the flow of data/ information among processing components • May be simple and linear or more complex • Can process many data/ information types: – Archives – Web pages – Streaming/ real time – Images (e. g. , medical or satellite) – Simulation output – Observational data 18

Challenges • Questions: – What are some challenges for users in implementing workflows? – What are some challenges to executing these workflows? – What are limitations of writing a program? • • • Mastering a programming language Visualizing workflow Sharing/exchanging workflow Formatting issues Locating datasets, services, or functions 19

Workflow Management Systems • Graphical interfaces for developing and executing scientific workflows • Users can create workflows by dragging and dropping • Automates low-level processing tasks • Provides access to repositories, compute resources, workflow libraries 20

Workflow Management Systems 21

Benefits of Workflows • • • Documentation of aspects of analysis Visual communication of analytical steps Ease of testing/debugging Reproducibility Reuse of part or all of workflow in a different project 22

Additional Benefits • Integration of multiple computing environments • Automated access to distributed resources via other architectural components, e. g. web services and Grid technologies • System functionality to assist with integration of heterogeneous components 23

Why not just use a script? • Script does not specify low-level task scheduling and communication • May be platform-dependent • Can’t be easily reused • May not have sufficient documentation to be adapted for another purpose 24

Why is a GUI useful? • • • No need to learn a programming language Visual representation of what workflow does Allows you to monitor workflow execution Enables user interaction Facilitates sharing of workflows 25

Some workflow systems • • Kepler SCIRun Sciflo Triana Taverna Pegasus Some commercial tools: – Windows Workflow Foundation – Mac OS X Automator • http: //www. isi. edu/~gil/AAAI 08 Tutorial. Slides/5 -Survey. pdf • http: //www. isi. edu/~gil/AAAI 08 Tutorial. Slides/ • See reading for this week 26

Recall forms of information • Structured/ un-structured • Presentation and organization • Syntax-semantics-pragmatics • Managed, designed and architected. • Goal of this part of the class is to understand how discovery is enabled or disabled based on these factors 27

Discovery • How does someone find your information? • How would you provide discovery of – collections – files – ‘bits’ • How would you find -> 28

Discovery o Federated Search o Folksonomies (user contributed) o Intelligent Agents o Search Engines o Taxonomies o Find photos of Kim o Boy or girl? 29

Use cases • Find a sound recording of a swallow. • Excuse me? 30

Use cases • Find a sound recording of an African Swallow • Find a sound recording of a bird that sounds like an African Swallow • Media types – how can you discover them? 31

Use cases • Find the movie that Jean Tripplehorn first starred in/ that was her most successful/ was lead actress? • Has anyone gene sequenced a mouse? • Discovery can often involve information integration 32

Three level ‘metadata’ solution for DATA Data Discovery Data Integration Level 1: Level 2: Data Registration at the Discovery Level, e. g. Volcano location and activity Data Registration at the Inventory Level, e. g. list of datasets, times, products Earth Sciences Virtual Database Level 3: Data Registration at the Item Detail Level, e. g. access to individual quantities Ontology based Data Integration Using scientific workflows A Data Warehouse where Schema heterogeneity problem is Solved; schema based integration 33 A. K. Sinha, Virginia Tech, 2006

Three level ‘metadata’ solution? Information Discovery Integration Level 1: Level 2: Level 3: Registration at the Discovery Level, e. g. Find the upper level entry point to a source Registration at the Inventory Level, e. g. list of datasets, using the logical organization Registration at the Item Detail Level, i. e. annotation e. g. tagging Integration using mapping management Catalog/ Index Schema based integration 34 A. K. Sinha, Virginia Tech, 2006

Information discovery • What makes discovery work? – Metadata – Logical organization – Attention to the fact that someone would want to discover it – It turns out that file types are a key enabler or inhibitor to discovery • What does not work? – Result ranking using *any* conventional algorithm 35

Federated search • “is the simultaneous search of multiple online databases or web resources and is an emerging feature of automated, web-based library and information retrieval systems. It is also often referred to as a portal or a federated search engine. ” wikipedia • Libraries have been doing this for a long time (Z 39. 50, ISO 23950) • Key is consistent search metadata fields (keywords) • E. g. Geospatial One Stop http: //www. geodata. gov 36

Search engines (1) • Contains an automated spider or crawler • No theoretical limits in the amount of indexing (limited by hardware) • Support remote indexing • Continual background indexing of content • Custom metatag support (some low-end products do not support this feature) • Support for indexing PDF, . doc, etc (some low-end products do not support this feature) 37 • Supports URL and word exclusions & inclusions

Search engines (2) • • • Server-Side Includes (SSI) supported Search by custom metatags Case sensitive or insensitive searching Simple Customizable search/results pages Boolean Searching capabilities Provide users meta description and page title in search results • Inexpensive – ~$200 (2010) • Easily customizable search/results interface 38

Search engines (3) • Result weighting feature • URL Inclusion list • Require significant memory (RAM) and disk space as the collection grows • Low-end alternatives often do not possess the capabilities to do phrase or natural language searching. 39

Improve www discovery? • Implement metatags on your and your partners web sites • Update content frequently • Register your site with the major search engines (tools exist to aid in this process) • Perform a basic study of where your site results within the major search engine providers • Do not spam the search engine providers • Re-evaluate your web site directory structure to ensure information is appropriately categorized/ described within your URL strings 40

Improve www discovery • Look through your server log files to determine what users are trying to find on your site and/or the path they are using to find information • Perform basic usability testing of your site to determine what users expect and can easily gather from your site. This also may determine why users go to an Internet search engine provider versus accessing your site directly. • Realize that Internet search engines don’t all act the same, index at the same time period, and often value a particular metatag, document date, etc. more than another vendor product. 41

Smart search • Semantically aware search, e. g. http: //noesis. itsc. uah. edu , http: //eie. cos. gmu. edu (Water -> Semantic Search) • Faceted search, e. g. mspace (http: //mspace. fm ), Earth System Grid (http: //esg. prototype. ucar. edu ), exhibit (MIT) 42

NOESIS 43

Faceted search • Semantically aware search, e. g. http: //noesis. itsc. uah. edu • Faceted search, e. g. mspace (http: //mspace. fm ), Earth System Grid (http: //esg. prototype. ucar. edu ) 44

Summary - discovery • Useful to write a few discovery use cases to drive how your design is developed • Evolution of your role in facilitating discovery and what/ how others implement access to your information 45

Reading for this week • Is retrospective 46

Check in for Project Assignment • Analysis of existing information system content and architecture, critique, redesign and prototype redeployment • Or a new use case, development, etc. 47

What is next • No class next week April 12 – GM week • Week 10 – Information integration, life-cycle and visualization 48