Скачать презентацию Computer assisted assessment of essays z Advantages y Скачать презентацию Computer assisted assessment of essays z Advantages y

44bc213c14beaf50fc33011a66d5cf2f.ppt

  • Количество слайдов: 17

Computer assisted assessment of essays z Advantages y Reduces costs of assessment x Less Computer assisted assessment of essays z Advantages y Reduces costs of assessment x Less staff is needed for assessment tasks y Increases objectivity x More than one assessor can be used without doubling the costs x Automated marking is not prone to human error y Instant feedback x Helps students z As accurate as human graders y Measured by correlation between grades given by humans and system z Training material y y Basis of scores given by computer Human graded essays Training is done separately for each assignment Usually 100 to 300 essays are needed z Surface features, structure, content

Computer assisted assessment of essays z Surface Features y y y Total number of Computer assisted assessment of essays z Surface Features y y y Total number of words per essay Number of commas Average length of words Number of paragraphs The earliest systems where based solely on surface features z Rhetorical Structure y Identifying the arguments presented in essay y Measuring coherence z Content y Relevance to the assignment y Use of words

Analysis of Essay Content z Information retrieval methods y Vector Space Model y Latent Analysis of Essay Content z Information retrieval methods y Vector Space Model y Latent Semantic Analysis y Naive-Bayes text categorization z Ways to improve efficiency y Stemming, term weighting, use of stop-word list y Stemming x Reduces the amount of index words x Reducing different word forms to common roots x Finding words that are morphological variants of the same word stem • apply -> applying, applies, applied

Analysis of Essay Content y Term weighting x Raw word frequencies are transformed so Analysis of Essay Content y Term weighting x Raw word frequencies are transformed so that they tell more about the words’ importance in the context x Amplifies the influence of words, which occur often in a document, but relative rarely in the whole collection of documents x Information retrieval effectiveness can be improved significantly x Term-frequency – inverse document frequency (Tf. Idf), Entropy Local term weight Global term weight (entropy) y Stop-word list x Removing the most common words • For example prepositions, conjunctions, nouns and articles (a, an, the, and , or. . . ) x Common words have no additional meaning to the content of the text x Saves processing time and working memory

Comparison of Essay evaluation systems z Assessment systems y y Project Essay Grade (PEG) Comparison of Essay evaluation systems z Assessment systems y y Project Essay Grade (PEG) Text Categorization Technique (TCT) Latent Semantic Analysis (LSA) Electronic Essay Rater (E-Rater) y Content refers to what the essay says and style refers to the way it is said y System can simulate the score without great concern about the way it was produced (grading simulation) or measure the intrinsic variables of the essay (master analysis)

Project Essay Grade (PEG) z One of the earliest implementations of automated essay grading Project Essay Grade (PEG) z One of the earliest implementations of automated essay grading y Development began in 1960’s z Primarily relies on surface features and no natural language processing is used y Average word length y Number of commas y Standard deviation of word length z Regression model based on training material y Scoring by using regression equation

Text Categorization Technique (TCT) z Measures both content and style z Uses a combination Text Categorization Technique (TCT) z Measures both content and style z Uses a combination of key words and text complexity features z Naive-Bayes categorization y Assesment of content y Analysis of the occurrence of certain key words in the documents y Probabilities estimating the likelihood that essay belong to a specified grade category z Text Complexity Features y Assesment of style y Surface features x Number of words x Average length of words

E-Rater z A hybrid approach of combining linguistic features with other document structure features E-Rater z A hybrid approach of combining linguistic features with other document structure features z Syntax, discourse structure and content y Syntactic features x Measures the syntactic variety x Ratios of different clause types x Use of modal verbs y Discourse structure x Measures how well writer has been able to organize the ideas x Identifies the arguments in the essay by searching “cue” words or terms that signal where an argument begins and how it is been developed y Content x Analyzes how relevant the essay is to the topic by considering the use of words x Vector Space Model

Latent Semantic Analysis (LSA) aka Latent Semantic Indexing (LSI) z Several Applications y Information Latent Semantic Analysis (LSA) aka Latent Semantic Indexing (LSI) z Several Applications y Information Retrieval y Information Filtering y Essay Assessment z Issues in Information Retrieval y Synonyms are separate words that have the same meaning. They tend to reduce recall. x For example: Football, soccer y Polysemy refers to words that have multiple meanings. This problem tends to reduce precision. x For example: "foot" as the lower part of the leg or as the bottom of a page or as a specific metrical measure y Both issues point to a more general problem x There is a disconnect between topics and keywords z LSA attempts to discover information about the meaning behind words z LSA is proposed as an automated solution to the problems of synonymy and polysemy

Latent Semantic Analysis (LSA) z Documents are presented as a matrix in which each Latent Semantic Analysis (LSA) z Documents are presented as a matrix in which each row stands for a unique word and each column stands for a text passage (word-bydocument matrix) z Truncated singular value decomposition is used to model latent semantic structure z Resulting semantic space is used for retrieval z Can retrieve documents that share no words with query. z Singular Value Decomposition y Reduces the dimensionality of word-by-document matrix y Using a reduced dimension new relationships between words and contexts are induced when reconstructing a close approximation to the original matrix y These new relationships are made manifest, whereas prior to the SVD, they were hidden or latent y Reduces irrelevant data and “noise”

Latent Semantic Analysis (LSA) z Word-by-document matrix Latent Semantic Analysis (LSA) z Word-by-document matrix

Latent Semantic Analysis (LSA) z Singular value decomposition Latent Semantic Analysis (LSA) z Singular value decomposition

Latent Semantic Analysis (LSA) z Two dimensional reconstruction of word-bydocument matrix Latent Semantic Analysis (LSA) z Two dimensional reconstruction of word-bydocument matrix

Latent Semantic Analysis (LSA) Latent Semantic Analysis (LSA)

Latent Semantic Analysis (LSA) z Semantic space is constructed from the training material z Latent Semantic Analysis (LSA) z Semantic space is constructed from the training material z To grade an essay, a matrix for the essay document is built z Document vector of essay is compared to the semantic space Word-by-document matrix z Grade is determined by averaging the grades with the most similar essays

Latent Semantic Analysis (LSA) z Document comparison y Euclidean distance y Dot product y Latent Semantic Analysis (LSA) z Document comparison y Euclidean distance y Dot product y Cosine measure z Cosine between document vectors x. Dot product of vector divided by their lengths A B

Latent Semantic Analysis (LSA) z Pros y Doesn’t just match on terms, tries to Latent Semantic Analysis (LSA) z Pros y Doesn’t just match on terms, tries to match on concepts z Cons y Computationally expensive, its not cheap to compute singular values y Choice of dimensionality is somewhat arbitrary, done by experimentation z Precision comparison of LSA and Vector Space Model at 10 recall levels