Intelli Metric Assessment Tools and Online Essay Grading

Intelli. Metric® Assessment Tools and Online Essay Grading The Gold Standard of Essay Scoring

What is Intelli. Metric®? Intelli. Metric® is an intelligent automated essay scoring system that: üProvides immediate holistic and analytic feedback for extended or short answers üMay be used as an instructional tool or for high stakes assessment üIs based on more than a decade of research and development with continual innovations and advancements üWas first commercially used in 1998 for use in college course placement assessments

Who’s Using Intelli. Metric®? • Association of American Medical Colleges (AAMC) for the writing section of the Medical College Admission Test (MCAT). • Graduate Management Admission Council (Analytical Writing Assessment (AWA) portion of the GMAT). • The College Board (ACCUPLACER® Write. Placer and SAT Prep). • ACT, Inc. (COMPASS e-Write™, COMPASS ESL, ACT Prep, Work. Keys ®). • Medical College Admission Test (MCAT) • State Departments of Education and School Districts throughout the country.

How does Intelli. Metric® work? ü Creates a unique scoring solution (model) for each prompt üLearns the characteristics of the score scale through exposure to examples of previously-scored responses ü Uses Artificial Intelligence to learn how to score essays ü Trained with a set of known papers (previously scored by experts) ü Modeled after traditional expert scoring process

How is Intelli. Metric® trained? Prompt specific training mirroring the human scorer calibration process: üTraining set of approximately 300 representative papers previously scored by expert scorers üMust have definition at tails in order to avoid truncating the scale üMust be consistently scored- garbage in, garbage out üRoughly normal distributions provide strongest models üValidation set kept blind

Continued……How is Intelli. Metric® trained? üIntelli. Metric® infers rubric and learns the characteristics of the score scale from training set Intelli. Metric® is inductive rather than rule based üModel is created and used to score validation set üPerformance data is reviewed (comparing human score and IM score for validation set) Agreement Rates Pearson correlation (measure of linear relationship)

How is an Intelli. Metric® model created?

What does the data say about the quality of the scoring? Hundreds of studies provide support that Intelli. Metric®: üAgrees with expert scoring more often than two skilled human raters will agree with each other. üAccurately scores open-ended responses across a variety of grade levels, subject areas and contexts üIM is perfectly internally consistent. It will consistently score the same essay the same way every time. This is not true of human scorers. üShows stable results across samples.

What makes Intelli. Metric® unique? ü Uses over 400 semantic, syntactic, and discourse-level features üContent and Structure ü Neurosynthetic approach ü Learning engine ü Systemic ü Inductive ü Uses Multiple “Judges” üLinear Regression, Bayesian Analysis, Latent Semantic Analysis, Proprietary Algorithms ÖOther automated essay scoring systems use a more narrow approach.

Demo Let’s take a look at how Intelli. Metric® works.