Project Proposal on Design of quality assessment tool

Project Proposal on “Design of quality assessment tool for evaluation of quality of e-Content” Standing Committee of National Mission on Education through ICT July 18 -19, 2009 @ New Delhi Centre for Development of Advanced Computing, Hyderabad

Outline • • E-Learning @ CDAC Need for e-Content Quality Assessment Proposal Discussion Centre for Development of Advanced Computing, Hyderabad

Technology based Learning Framework Centre for Development of Advanced Computing, Hyderabad

Multilingual e-Learning Framework (LMS) • e. Sikshak – A Multilingual E-Learning Framework – Component based architecture – Supports Asynchronous & Synchronous communication – Assessment, Course Organizer – Student Personal Space – Short Messaging Service – Indian Language Interface (Hindi, Telugu, Tamil) Centre for Development of Advanced Computing, Hyderabad

Architecture Centre for Development of Advanced Computing, Hyderabad

Two Multilingual Web Architectures Separating presentation from software elements and dynamic pages are served based on language selection content is “dynamically” inserted in page templates Multiple static versions of pages stored in a folder hierarchy by language and navigated by selection mechanism language selection static web page is selected and displayed content is stored in databases by language / culture Centre for Development of Advanced Computing, Hyderabad

Indian Language Interface q Internalization approach q Easy to add new languages (Local, International) q Unicode based Support Processing Code User Choice UI Code Resource Bundles: Different Languages Interface Centre for Development of Advanced Computing, Hyderabad

e. Sikshak Hindi Interface Centre for Development of Advanced Computing, Hyderabad

e-Sikshak in English Centre for Development of Advanced Computing, Hyderabad

Uploading assignment Centre for Development of Advanced Computing, Hyderabad

Whiteboard in English Centre for Development of Advanced Computing, Hyderabad

Discussion forums in Hindi Centre for Development of Advanced Computing, Hyderabad

Online Courses Certificate Course on Intellectual Property Rights, Indian Law Institute, New Delhi Course on Sustainable Agriculture, India Development Gateway Course on Right To Information Act 2005 , India Development Gateway Centre for Development of Advanced Computing, Hyderabad

Internet Course Delivery Framework for Cyber Security Resources Content Repository Internet Power. Point HTML Flash Video Other content Content Interface ·Unit o. Module §Lesson Video Lectures Delivery Learner Centre for Development of Advanced Computing, Hyderabad

Video Lectures Centre for Development of Advanced Computing, Hyderabad

LTSA Framework’s System components Learner Entity Multimedia Delivery Behavior Interaction Context Loc ato r Learning Preferences Learning Locator Content Catalog Info Learning Resources Query Coach A s sse sm Evaluation ent (history) Performance/ Preferences (new) Performance (current) Centre for Development of Advanced Computing, Hyderabad Learner Records

Server and Grid Side U Environment-Client Side Database Centre Storage Grid Learning History DB Learner’s Profile DB Content DB sensors Support Rules DB Service Agents Profiler Agent Presentation Agent Sensor Network / Common Store Learning Objects Learning Tasks Filtering Criteria Filtering Application Desktop Environment Rendering Criteria Mobile Environment Learning Exposition Context Aware Tool Kit INTERNET LMS / Virtual Learning Environment camera Microphone LAN Listener Agent Content Mgmt Agent Clients PDA Laptop Learning Communications Administrative Functions Rendering Application Desktop PC Smart Phones 17

Virtual Class Room Experimentation • Access grid experimentation with – DV Camera for High quality video transmission – Microphone with Amplifier – Setting up of business meeting with Access grid between CDAC Hyderabad, Pune & Bangalore – Study on Access grid integration with LMS Centre for Development of Advanced Computing, Hyderabad

Pedagogy & Instructional Design Content Development, Delivery and Tools e-Learning and Its Societal Impact Learning Grids & New Technologies Case Studies Quality Assurance, Standards and Open Source Centre for Development of Advanced Computing, Hyderabad

Centre for Development of Advanced Computing, Hyderabad

Quality Analytics Framework (QAF) • Derive quality metrics that can be used for quantifying the various quality parameters of a e-learning tool and the content. • Develop a prototype tool that can be used by end users/developers to measure effectiveness of the tool and compare different e-Learning tools. Centre for Development of Advanced Computing, Hyderabad

Quality Indicators • • • Scalability Accessibility Usability Content Reusability & Portability Security Centre for Development of Advanced Computing, Hyderabad

QAF Conceptual Diagram Centre for Development of Advanced Computing, Hyderabad

Service Oriented Architecture Centre for Development of Advanced Computing, Hyderabad

Workflow Diagram Centre for Development of Advanced Computing, Hyderabad

Evaluation Constraints – Scalability evaluation should be carried out in isolated manner. – If evaluation of scalability of any LMS is being carried out on a particular node then no other quality metric evaluation should be taken up on the same node. – If evaluation of quality metric other than Scalability is being carried out on a particular node then scalability evaluation cannot be taken up on the same node. Centre for Development of Advanced Computing, Hyderabad

Evaluation Criteria • Technical evaluation of standards compliance • Semi automated evaluation to gauge user satisfaction level using questionnaires • Weight-age based grading methodology Total quality of an e-Learning environment, in terms of the weight-ages assigned to the indicators and quality index evaluated obtained from evaluators can be defined as n n Q = (∑ pi. Wi ) / Σ Wi i=1 n = Number of Quality Indicators Q = Total Quality Index pi = Performance Index of a particular quality indicator Wi = Weightage of a particular quality indicator Centre for Development of Advanced Computing, Hyderabad

Sample Analytical Report of LMS Usability Centre for Development of Advanced Computing, Hyderabad

Evaluation Reports Centre for Development of Advanced Computing, Hyderabad

E-Learning @ CDAC • Learning Management System – e-Sikshak (A Multilingual e-Learning Framework) • SCORM based Cyber Security Course content development • Quality Analytics Framework for E-Learning Tools & Content • Expertise on development of multimedia based interactive solutions • Development of Ubiquitous Learning Environment called USikshak • National & International Seminars on e-Learning and e-Learning Technologies since 2001 Centre for Development of Advanced Computing, Hyderabad

Our Proposal • Development of a quality assessment manual for e-Content through identification of appropriate standards or guidelines or best practices related to quality of e-Content. • Development of web enabled automation tool for evaluation of quality of the e-Content based on quality assessment manual for e-Content. Centre for Development of Advanced Computing, Hyderabad

Steps involved in e. Learning content development Text Production Media Production LCMS/ LMS Visual Design Learning Process Consulting Instructional Design Integration Quality Assurance Hosting Centre for Development of Advanced Computing, Hyderabad

Roles involved in e. Learning content development… Stages Roles User Requirement Analysis Instructional Designer (ID), PM Design ID, Visual Designer (VD), Developer (Dev), PM Prototype ID, VD, Instruction Writer, Graphics Developer, Dev, PM Development/Production Instruction Writer, Graphics Developer, Dev, Editors Testing Deployment Delivery Instruction Writer, Graphics Developer, Dev, QA Developer Project Manager Centre for Development of Advanced Computing, Hyderabad

Content Design - Guidelines • • Goal oriented Modularize Consistency in Look and Navigation Interactivity Support for team based activity Illustrate and don’t decorate (Graphics) Consider accessibility, Multicultural Centre for Development of Advanced Computing, Hyderabad

Content Design - Guidelines • Quiz for recall and assessment for evaluating the learning effectiveness • Focus on value added content (hyperlink) • Template based • Pedagogically Sound Instruction Strategy • Each learning chunk should be 3 -5 screens max. If larger, provide internal links within the page. Centre for Development of Advanced Computing, Hyderabad

Effectiveness of online courses • Course Types – Self contained courses – Interactive courses – Hybrid courses • Learning Styles – – active/reflective sensing/intuitive visual/verbal sequential/global • Teaching Styles – – – Expert Facilitator Delegator Formal Authority Personal Model Centre for Development of Advanced Computing, Hyderabad

Quality Content Principles • Principles of the quality content – – – Accessibility Interactivity Understandability Adaptability Reach-ability Centre for Development of Advanced Computing, Hyderabad

Content Related Standards • Sharable Content Object Reference Model • Question and Test Interoperability Standard • Web Content Accessibility Guidelines (Level A, AAA) • Instructional Design Standards by Michigan Virtual University • ASTD’s E-Learning Courseware Certification Standards Centre for Development of Advanced Computing, Hyderabad

Best Practices Around the World • Quality On the Line: Benchmarks for Success in Internet-Based Distance Education (IHEP, Washington DC) – 24 Benchmarks (Institutional Support, Course Development, Teaching/Learning, Course Structure etc) • Standards for Quality Online Courses by Michigan Virtual University – Technology, Usability, Accessibility & Instructional Design • Western Cooperative for Educational Telecommunications – Curriculum and Instruction, Institutional Context & Commitment, Evaluation and Assessment • e-Learning Courseware Certification for National Science and Technology Program for e-Learning – Content, Navigation, Instructional Design and Instructional Media • Instructional Design Guides & Forums by Penn State Learning Design Community Hub – Web Design, Learning Objectives, Learning Styles & Teaching Styles • Research based Web Design & Usability Guidelines by U. S. Department of Health and Human Services Centre for Development of Advanced Computing, Hyderabad

Proposed Quality Characteristics • Understandability (Instructional Design) • Content Design Principles (Color, Font Size etc) • Accessibility • Evaluation Criterion Centre for Development of Advanced Computing, Hyderabad

Approach & Methodology • Identification of sub characteristics for proposed parameters related to e-Content quality evaluation. • Applying suitable evaluation criterion to evaluate quality of the content. • Development of software package incorporating evaluation criterion. • Organizing seminars/meetings of experts comprising content designers, developers and users to aggregate opinion on content quality aspects. This would be useful in defining guidelines/best practices for evaluating content. Centre for Development of Advanced Computing, Hyderabad

Instructional Design • Learning Objectives: – Learning Objectives and goals are clearly stated and – presentation and organization of content, as well as related activities, supports the learning objectives and goals. • Interactivity: – Learners control their own pace and are informed about their progress – Learners are presented with relevant problems to solve; exemplary solutions are included • Cognition/conceptual changes – Learners will be able to demonstrate or apply the concepts – Mechanisms are provided so learners can monitor their own understanding • Content – Scope of the content is appropriate for the intended learning objectives – Structure of the knowledge to be learned is clearly conveyed • Multimedia Use – Multiple representations are used to help learners construct inter-related knowledge – Media elements are of high visual and aural quality Centre for Development of Advanced Computing, Hyderabad

Content Design Principles • Which image format is suitable for which type of illustrations based on images ? – Use the GIF format simple illustrations with large areas of solid color. – Use the JPEG format for photographs, watercolor images, and pencil or charcoal drawings • Which media format for audio & video should used taking its support into account in most browsers ? • Which font style and text size is suitable for writing text in electronic media ? Centre for Development of Advanced Computing, Hyderabad

Accessibility • Every non text element should be accompanied with equivalent text description. • Organize documents so they may be read by assistive technologies properly. • Check the quality of the image used to convey any message. • Control of audio/video play is available. • Check whenever visual presentation changes appropriate text based captions/audio is also updated. • Clearly identify changes in the natural language of a document’s text and any text equivalents. And many more … Centre for Development of Advanced Computing, Hyderabad

Related Standards/Specification Development Orgs: Education & Learning Technology IEEE, ANSI, ISO: Accredited Standards LTSC (P 1484) Specs submitted by Consortia/Fora Liaison Education & Learning Technology Orgs: AICC, ADL, IMS, ARIADNE Centre for Development of Advanced Computing, Hyderabad

Architectural View Centre for Development of Advanced Computing, Hyderabad

Training/Workshops/Expert Meetings • Meetings/Workshops/Seminars with experts – Four such meetings will be organized per year – Instructional Design Courses Centre for Development of Advanced Computing, Hyderabad

Clarity on Purpose Feature DIT Project MHRD Proposal Remarks Accessibility WCAG version 1. 0 √ WCAG version 2. 0 √ To evaluate content accessibility guidelines on other formats of the content also such as PPT, Flash, Video & Audio etc. Understandability Learning Objectives Outcomes Comprehension √ Instructional Design related aspects based on learning pedagogical models Content Design Principles √ Source: Penn State Learning Design Hub Community Evaluation Criterion √ Apply suitable statistical technique to grade content according to different Levels. Centre for Development of Advanced Computing, Hyderabad

Deliverables • 1 st Year: 1. Literature survey on quality characteristics of e-Content and identification of experts from academic, corporate and R&D institutes for deliberations on content quality criteria. 2. Requirements gathering – Identification of quality metrics for proposed characteristics and relevant best practices/guidelines/standards 3. Define scope of quality characteristics of e-Content & preparation of Software Requirement Specification for the tool. 4. Design of questionnaire for evaluation of quality of the e-Content 5. Prototype of e-Content Evaluation Automation Software 6. Conduction of meeting on content quality with experts • 2 nd Year: – Software Design – Development of the software – Conduction of meeting on content quality with experts Centre for Development of Advanced Computing, Hyderabad

Contd… • 3 rd Year: – Development of the software – Testing – Conduction of meeting on content quality with experts – Evaluation of the online course content using the developed software Centre for Development of Advanced Computing, Hyderabad

Proposed Plan of Activity 1. 2. 3. 4. 5. 6. 7. 8. 9. Literature survey on quality characteristics of e-Content and identification of experts from academic, corporate and R&D institutes for deliberations on content quality criteria. (3 rd Month) Requirements gathering – Identification of quality metrics for proposed characteristics and relevant best practices/guidelines/standards(4 th Month) Define scope of quality characteristics of e-Content & preparation of Software Requirement Specification for the tool. (6 th Month) Design of questionnaire for evaluation of quality of the e-Content (8 th Month) Prototype of e-Content Evaluation Automation Software(12 th Month) Design of e-Content Evaluation Automation Software (14 th Month) Implementation of quality assessment tool. (24 th Month) Testing of quality assessment tool. (26 th Month) Evaluation of the software with online course content (36 th Month) Centre for Development of Advanced Computing, Hyderabad

Total Budget Outlay S. No. Head 1 st Year 2 nd Year 3 rd Year Total 1 Capital Equipment 10. 00 5. 00 20. 00 2 Consumable items/Components 5. 00 15. 00 3 Staff Salaries 32. 40 97. 20 4 Travel 3. 00 9. 00 5 Training 10. 00 30. 00 6 Contingencies 10. 00 30. 00 Total 70. 40 65. 40 201. 2 CDAC’s Contribution : 21. 6 Lakhs MHRD Fund : 179. 6 Lakhs Centre for Development of Advanced Computing, Hyderabad

Thank You Centre for Development of Advanced Computing, Hyderabad