A REVIEW OF RESEARCH METHODS FOR ASSESSING CONTENT

A REVIEW OF RESEARCH METHODS FOR ASSESSING CONTENT OF COMPUTERMEDIATED DISCUSSION FORUMS Reporter：Chia-Yen Feng Date： 2007/12/08 1 Marra, Rose. (2006). A Review of Research Methods for Assessing Content of Computer. Mediated Discussion Forums. Journal of Interactive Learning _Research, 17(3), 243 -268.

INTRODUCTION (1/6) Instructors and students rely on these asynchronous forums to engage one another in ways that replace face-to-face communication. The goal of such interactions is to promote critical thinking, meaningful problem solving, and knowledge construction (Kirschner, Strijbos, Kreijns, & Beers, 2004). In order to assess any meaning that results from these discussions, it is necessary to perform some kind of semantic analysis of them. This article provides an overview of current methods of quantitative and qualitative research paradigms for analyzing the content of asynchronous computermediated discussion forums. 2

INTRODUCTION (2/6) In discussing online learning Interactivity as the most striking characteristic of CMC and the factor with the greatest potential to impact learning. Web-based learning courses must rely on online discussion forums to create these interactions (Similarly, Garrison, Anderson, and Archer , 2001). In CMC, participants who were trying to increase communication efficiency and decrease typing requirements also decreased the use of unneeded elaborative statements and repetitions(Condon and Cech , 1996). 3

INTRODUCTION (3/6) CMC has the ability to promote knowledge construction and meaningful learning. Asynchronous CMC may promote knowledge construction because online environments emphasize student-centered interaction through collaboration with other students. It is through these interactions that learners construct their knowledge and can learn more effectively (Romiszowski & Mason, 2004). 4

INTRODUCTION (5/6) The quality of online postings by focusing on four dimensions – social (e. g. , "I'm feeling great today“) interactive (statements that refer to other postings) metacognitive (statements about reasoning) Cognitive (five types of reasoning skills) elementary clarification in-depth clarification Inference Judgment strategies (Henri's work , 1992) 5

INTRODUCTION (6/6) Henri's model Defines not only the types of skills and interactions demonstrated in online postings, but also attempts to qualitatively define the nature and content of online interactions that evidence cognitive development and meaningful learning. Henri's model has provided researchers with a structure for many ensuing qualitative analysis methods. 6

CMC DISCUSSION RESEARCH METHODS (1/6) Quantitative - Descriptive Statistics “computer-generated statistics about logons, messages sent and read, levels of participation and number and length of entries” (Mason , 1992). Many course management software provide access to quantitative data easily and allow instructors to track student logons and their usage of course materials. Scaffolded discussion forums accompanied with more sophisticated tools for analyzing the quantitative and qualitative nature of those contributions. 7

CMC DISCUSSION RESEARCH METHODS(2/6) Mason (1992) lists the number of messages sent and read, levels of participation, and the number and length of entries for each discussion forum. number of postings per student for overall forum; number of times each posting is read; total depth per thread; postings per student listed by depth of postings the number of postings at depth level 1 - n where n is the number of the most embedded posting(ex. Figure 1). The metrics are directed towards threaded discussion forums - that is an online dialog or conversation that takes the form of a series of hierarchically linked messages. 8

9

CMC DISCUSSION RESEARCH METHODS (3/6) Qualitative Content Analysis two CMC discussion forum analysis protocols Interaction Analysis Model (IAM) (Gunawardena, Lowe, & Anderson, 1997). Critical thinking model(Newman, Webb and Cochrane's , 1996; Newman, Webb, Johnson, & Cochrane, 1997). Interaction Analysis Model (IAM) social presence, teacher presence, and cognitive presence （Garrison et al. , 2000). attempt to further understand describe the processes of negotiating meaning and knowledge coconstruction in a collaborative online discussion environment (Gunawardena et al. , 1997). 10

CMC DISCUSSION RESEARCH METHODS (4/6) Phase 1, 2 provide further detail on points that have already been made in prior postings Phase 3 negotiation of meaning or knowledge co-construction, where students build on prior postings in order to construct the new idea 11

CMC DISCUSSION RESEARCH METHODS (5/6) Critical thinking model (CT) 1) identification, 2) definition, 3) exploration, 4) evaluate 12

CMC DISCUSSION RESEARCH METHODS (6/6) “Critical thinking ratio" (CT) model The Newman et al. (1996) based on the frequencies of plus (+) or minus (-) codes for each letter category. CT = (x+ - x-) / (x+ + x-) User-Classified Forum Content - Constrained CMC Forums designed to encourage certain types of thinking by the participants. The most commonly used constraint system supports argumentation, however, a limitless combination of rhetorical structures are possible. The forum participants used message types that corresponded to a formal argument in order to develop argumentation skills that were hypothesized to enable solving ill-structured problems (Oh, 2004). 13

14

CMC DISCUSSION RESEARCH METHODS (7/) constrained discussion Message types included hypothesizing cause ("What is the cause? “) solution generation ("What should I do about it? “) research evidence ("Research shows. . . " or, "A scholar says. . . “) experiential evidence ("My experience is. . . " or, "I believe. . . “) elaboration ("Your explanation is unclear; can you be more specific? “) rebuttal ("I don't agree because. . . ") 15

CMC DISCUSSION RESEARCH METHODS (8/) Discussion Analysis Tool (DAT)( Jeong, 2003) help researchers examine and measure student interactions and sequences of interactions that may lead to critical thinking outcomes in threaded discussions the probabilities generated by DAT can be converted into graphical depictions that provide what may be a useful visual representation of student interactions. 16

17

METHODS (1/4) Research Questions and Results 18

METHODS (2/4) Reliability an indication of consistency of measures across time. To ensure reliability, researchers should re-check calculations or when using automated tools, check for the accuracy of the input data. Validity The validity of these metrics is less well defined. Validity refers to whether a measure, in fact, measures what it purports to. These metrics are not open to much interpretation and as long as researchers do not attempt to attribute more meaning to them than they have, then the metrics are valid 19 (face value , the total number postings).

METHODS (3/4) IAM CT The IAM's validity hinges on whether the results produced when the protocol is reliably applied are in fact an indication of the degree to which knowledge construction occurred in the forum. Although the CT ratios are easy to calculate, one must question how to meaningfully interpret them. Constrained forums The reliability and validity issues associated with the user categorized constrained forums center around the fact that the users themselves do the coding or labeling of the messages. Reliability is largely a question of consistency of results. 20

METHODS (4/4) Constrained forums The validity of the user categorizations in constrained discussion forums depends on how accurately users label their messages. Another validity issue for these constrained forums concerns the content validity or meaning of the suite of message types. DAT For DAT because it is algorithmic, the reliability can be assumed to be nearly perfect - assuming that the underlying programming is sound and that the same data is provided as input for the tool. The validity of DAT is directly tied to the validity of the coding scheme that provides the data on which DAT operates. 21

CONCLUSION (1/2) Each method is appropriate for addressing different types of research questions. Further, each method offers its own unique strengths and weaknesses. Sequential analysis methods(DAT) builds on content analysis methods and provides quantitative data on discussion interaction sequences. the software program DAT are completely reliable and the method has the potential to answer more complex questions regarding discussion boards, the method implicitly inherits all the reliability and validity issues of the content analysis protocol that it uses to do the initial coding. 22

CONCLUSION (2/2) Researchers need to choose an analysis methods based upon what questions they wish to answer ability to carry out the chosen methodology in the fashion intended (e. g. , does my research team have the qualitative methods expertise to conduct a content analysis? ). Although none of the methods are perfect in terms of their applicability to all situations or in terms of their validity and reliability, all the methods reviewed - when applied appropriately can provide researchers with effective tools for describing online discussions. 23