ITU-T Workshop on Telecommunications relay services for persons

ITU-T Workshop on “Telecommunications relay services for persons with disabilities ” (Geneva, 25 November 2011) The NER model to assess accuracy in respeaking Pablo Romero-Fresco (Roehampton University, CAIAC research centre) Juan Martínez (Respeaking consultant) Geneva, 25 November 2011

Accuracy in Respeaking Quality in respeaking Delay Accuracy

Accuracy in Respeaking 97 -98% accuracy

Basic requirements for a model 1) Functional and easy to apply 2) Include the basic principles of WER calculations in SR 3) Different programmes, different editing 4) Possibility of edited and yet accurate respeaking 5) Compare subtitles with original spoken text 6) Include other relevant info (delay, position, speed) 7) Provide both percentage and food for thought in training

Traditional WER methods US National Institute of Standards and Technology Accuracy Rate N - Errors ------------ × 100 = % N But. . . Well, you know, you have to try You have to try to put out a good and put out a good performance, performance. It’s a stepping stone. I mean, yeah, it’s kind of a stepping stone, isn’t it, really?

Traditional WER methods US National Institute of Standards and Technology Accuracy Rate N - Errors ------------ × 100 = 16% N But. . . Well, you know, you have to try You have to try to put out a good and put out a good performance, performance. It’s a stepping stone. I mean, yeah, it’s kind of a stepping stone, isn’t it, really?

N–E–R Accuracy ------------× 100 = % N Correct editions: Serious errors: Assessment: Spain = SDH guidelines Different European countries UN Accessibility Focus Group

NER Model Accuracy 205 – 3 – 2 ------------ × 100 = 98. 6% 205

NER Model Accuracy 226 – 13 – 1 ------------ × 100 = 93. 8% 226 Assessment: poor editing (not quantity, but quality)

NER Model Accuracy 257 – 13 ------------ × 100 = 94. 3% 257 Assessment: poor recognition (including serious mistakes)

WGBH: “There is a wide range of error types in real time captioning and they are not all equal in their impact to caption viewers”. “Treating all errors the same does not provide a true picture of caption accuracy”.

Types of errors (feedback from DTV 4 ALL project) 1) “There are errors, yes, but you can easily figure out what the correct form was meant to be. Now I’m bilingual –I can speak English and teletext” 2) “Live subtitles? - Sound like gobbledygook to me” 3) “As far as I’m concerned they are not errors, but lies”

Types of errors (feedback from DTV 4 ALL project) 1) Minor edition or recognition errors (0. 25) 2) Normal edition or recognition errors (0. 5) 3) Serious errors (1)

Minor Errors What a great goal by a Ryan Giggs! Simon brown has been appointed new chairman of Rolls Royce. For people are still missing following Sunday’s tornado.

Standard Errors He’s a buy you a bull asset. Is it really attend Tatian?

Serious errors Public funding for universities has been cut by 15% this year. He never talks dirty.

Serious errors Public funding for universities has been cut by 15% this year. He never talks dirty. He never talks to Rudy.

NER MODEL N–E–R Accuracy ------------× 100 = % N Correct editions: Comments: Target = 98%

NER Orange with apples

Graciñas Pablo Romero-Fresco (p. romero-fresco@roehampton. ac. uk)

ITU-T Workshop on “Telecommunications relay services for persons with disabilities ” (Geneva, 25 November 2011) The NER model to assess accuracy in respeaking Pablo Romero-Fresco (Roehampton University, CAIAC research centre) (p. romero-fresco@roehampton. ac. uk) Juan Martínez (Respeaking consultant) Geneva, 25 November 2011