Automated Question Answering Motivation support for students

CHALLENGES PROCESSING MESSAGES

Poor message style Incorrect POS tagging due to spelling errors when/WRB I/PRP tried/VBD to/TO generate/VB the/DT sample/NN , /, it/PRP said/VBD the/DT data/NNS is/VBZ available/JJ. /.

XML within messages Detected as single sentence

Eclipse console listing within message Line breaks not recognised as end of sentence

Open-source NLP problems • Sentence detection failures: – Bad style (capitalisation, punctuation) – Ellipsis (i tried. . . it failed. . . error message. . . ) – XML, BPEL segments concatenated to single sentence • Tokenisation failures: – Multiple punctuation ? ? ? , !!! (student emphasis) – Abbreviations (im, cant, doesnt, etc. ) • POS errors – Spelling, grammar

Purpose built tools • Tokeniser – Re-coded for typical forum content/style • Multiple punctuation • Abbreviations • Common contractions • Sentence detector – New detector based on token sequences • Pre-filter messages – Remove XML, console listing, error messages

Message pre-filters • Short-forms – i’m, i m – can’t, can t • • • i am can not Line numbers Repeated punctuation (!!!, ? ? ? , . . . ) Smilies Salutations (Hi all, Hiya, etc. ) Names, signature, course codes

Filtered message Raw message containing Eclipse console listing Filtered message ready to process

PRELIMINARY RESULTS: QUESTION CLASSIFICATION

Message-set properties • • • Number of messages: 1051 (100%) Number of questions(M): 777 (73. 9%)(100%) Number of questions(A): 756 (97. 3%) False Positives (A not M): 58 (7. 4%) False Negatives (M not A): 79 (10. 2%) Approx 90% success rate M = manually annotated question, A = automatically annotated question

Message-set properties – cont. • Average # pattern matches: • Min # pattern matches: • Max # pattern matches: 2. 7606 1 12 • Average # of lines (ASCII linefeed) • Min # Lines in a message • Max # Lines in a message 7. 9 1 68 • Average # of sentences • Min # Sentences in a message • Max # Sentences in a message 5. 0 1 89 • Messages containing XML • Messages containing BPEL 17 37

Distribution of pattern match count 350 Number of messages 300 295 240 250 200 174 150 95 100 42 50 32 9 7 7 8 1 2 2 2 9 10 11 12 0 0 1 2 3 4 5 6 Number of pattern matches

Challenges: false positives

Challenges: false negatives

Challenges: detecting the question

Number of messages Messages matching question pattern Pattern IDs Pattern ID

Common question patterns (10) • any – (advice|clarification|clue|comment| – further thought|guidance| – help|hint|idea|opinion| – pointer|reason|suggestion|taker)(s)? • . * • appreciated|welcomed 216 matches Terms added over time to improve detection of questions

Sample question match (10)

Common question patterns (50) • get|getting|gives|got|receive • . * • error(s)? 102 matches

Sample question match (50)

Number of messages Discrimination vs Classification Pattern ID Low discrimination >>> Increases successful classification at the risk of false-positives High discrimination >>> Reduces successful classification and risk of false-positives

Does process transfer? • Tested against TT 380 forums 04 J – 07 J – Preliminary results look promising – Need to manually tag >4000 messages – Review message pre-filters • Need access to Humanities course material

PRELIMINARY RESULTS: QUESTION TOPIC IDENTIFICATION

Basic method • Identify named entities – NEs are block-specific – Majority of questions linked to assignments • Parse sentence for dependencies – Nouns (that are NEs) – Verbs

Named entities: inconsistent usage Message body Message subject Error handling Exception handling

Deep parsing: dependencies advmod(delete-5, How-1) aux(delete-5, can-2) nsubj(delete-5, I-3) advmod(delete-5, properly-4) dobj(delete-5, PLTs-6) conj_and(PLTs-6, PLs-8) conj_and(PLTs-6, roles-10) det(project-13, the-12) prep_from(delete-5, project-13) prep_in(delete-5, order-15) aux(have-17, to-16) xcomp(delete-5, have-17) det(sheet-20, a-18) amod(sheet-20, clean-19) dobj(have-17, sheet-20) advmod(have-17, again-21) How can I properly delete PLTs and PLs and roles from the project in order to have a clean sheet again.

Sentences per message 200 Number of messages 180 160 140 120 100 80 60 40 20 0 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 20 21 22 23 24 25 30 31 38 48 51 89 Sentence counts under-estimated due to spelling /grammar errors. Of the 120 single-sentence questions >80% are multiple sentences.

Guess the topic Excuse me for directing this question at you, but when I try to contact my tutor through my homepage i still go to the details for John Stephenson but I am sure that he is ill at the moment. My question refers to the entities described in ECA part 2 page 2, it states that the term identifier must be unique within the UK business domain. I thought Buyers ID and Sellers ID could be their email address, however, I am stuck on the Order ID which might refer to a depatch note as I do not know what standard these identifiers have to conform to in UK business. I would appreciate being directed as to where I can find this information.

Current status • Unable to establish question topic for the 95% of detected questions • Current NLP techniques (anaphora and co-reference resolution) for multi-sentence questions not well established.

Pattern matching in console listing

Practical work: exact patterns • Process|Assign|Invoke|Scope|Reply • . * • Completed with fault: • invalid. Variables|uninitialized. Variable|join. Failure Provide direct link to FAQ or teaching materials

Future work • Further work on sentence detection – Everything else depends on this • Create patterns to identify content – “how do i (. *)” – “are you now saying (. *)” – “(. *) word count” • Establish relationships between initial message and replies • Build tool to process Eclipse console listings – Could address 5% of all ECA related questions