CS 626 -460 Language Technology for the Web Natural

CS 626 -460: Language Technology for the Web/Natural Language Processing Pushpak Bhattacharyya CSE Dept. , IIT Bombay Topic: Hindi Wordnet, Formalization etc.

Lexical Matrix

Creation of Synsets Three principles: • Minimality • Coverage • Replacability

Synsets {house} is ambiguous. {house, home} has the sense of a social unit living together; Is this the minimal unit? {family, house , home} will make the unit completely unambiguous. For coverage: {family, household, house, home} ordered according to frequency. Replacability of the most frequent words is a requirement.

Synset creation From first principles – Pick all the senses from good standard dictionaries. – Obtain synonyms for each sense. – Needs hard and long hours of work.

Synset creation (continued) From the wordnet of another language in the same family – Pick the synset and obtain the sense from the gloss. – Get the words of the target language. – Often same words can be used- especially for t%sama words. – Translation, Insertion and deletion. Hindi Synset: Anau. Bava. I jaanakar ma. Mjaa hu. Aa (experienced person) Marathi Synset: Anau. Bava. I t& jaa. Nata &ata

Gloss and Example Crucially needed for concept explication, wordnet building using another wordnet and wordnet linking. {earthquake, temblor, seism} -- (shaking and vibration at the surface of the earth resulting from underground movement along a fault plane of from volcanic activity) Hindi Synset: {Ba. Uk. Mp, Ba. Ucaala, Ba. UDaola}; p. Rqva. Iko p. RYz. Baagaka ihlanaa; gaujaratmao. M hue Ba. Uk. Mpmao. M Anaok laaoga maaro gayao. (shaking of the surface of earth; many were killed in the earthquake in Gujarat) Marathi Synset: {Ba. Uk. Mp, Qar. Na. Ik. Mp}; p. Rqva. Icaa p. RYz. Baaga hala. Nyaaca. I ik/yaa; gaujaraqama. Qyao Jaalaolyaa Ba. Uk. Mpat Anaok laaok maarlao gaolao.

Word. Net Sub-Graph Hyponymy Dwelling, abode Hypernymy Meronymy kitchen Hyponymy bckyard veranda M e r o n y m y bedroom house, home Gloss A place that serves as the living quarters of one or mor efamilies Hyponymy study guestroom hermitage cottage

Ontology • Needed for word sense disambiguation. • Makes explicit the semantic relations. • Tries to link correctly the exact place of a • particular sense in the structure of a language. Conceptual categories of nouns, verbs, adjectives and adverbs are placed in a directed acyclic graph structure.

Wordnet defines an ontology earthquake, temblor, seism -- => geological phenomenon - => natural phenomenon - => phenomenon -Property inheritance possible. Important for sense disambiguation Ontology is shallow for non-noun POS.

Hindi Wordnet

A small part of Hindi Wordnet

उपय गकरत ट पपण 1. गजब। त र फ करन क ल ए शबद नह । भ रत य मन ष कहत ह क ज उपय ग ह वह सदर ह। मझ जस भ ष व द क ल ए अतयत उपय ग । आभ र। आच रय क मत 2. बहत महतवपरण क अदभत -ड ॰ 3. परस द क रय और अनपमय ह य क रय कह ए ह। जगद श वय म http: //deshduniya. blogspot. com/2006/07/blog-

ह नद शबदक श सकत -सथल • http: //www. cfilt. iitb. ac. in/wordnet/ webhwn/hindi_version. html

Synset Entry Interface

Marathi WN from Hindi WN

Word. Net Building Approaches • Building Word. Net from scratch • Time consuming and needs extensive manual efforts • Alternative approach can be building the Word. Net by using other Word. Net as base • Marathi Word. Net is being built by using same approach from Hindi Word. Net

Building MWN from HWN • Consider a Synset from HWN corresponding to some concept. For e. g. Synset in hindi for the concept of “tree” is: {ped, vriksh, paadap, drum, taru, vitap, ruuksh, ruukh, adhrip, taruvar} • Construct Marathi Synset with the same id representing same concept as follows: {jhaad, vriksh, taruvar, drum, taru, paadap}

Building MWN from HWN(cont. ) • If some sense is only in Hindi, but not in Marathi, corresponding Synset in Marathi can’t be created. For e. g. {daadaa, baabaa, aajaa, daddaa, pitaamaha, prapitaa} • If some sense is only in Marathi but not in Hindi, then create synset with different id. For e. g. {powaadaa} – Song praising the bravery of Maratha Warriors

Building MWN from HWN(cont. ) • When sense is present in both, then semantic relations in HWN are borrowed directly in MWN. • Lexical relationships are added manually. • If the sense is only present in Marathi, then all the relationships are to be established manually.

Challenges • The quality and effectiveness of WN depends largely on how the base WN is. • Some words are polysemous in one language, but not in other. • Same word can have drastically different meaning in two languages. • Words which have subtly different meaning in two languages can be misunderstood to have same meaning.

Hindi WSD

Approach to WSD …. Hindi Document Context Bag Intersection Similarity Hindi Wordnet Semantic Bag

The WSD Algorithm Parameters • Wordnet Relations: synonymy, hypernymy, hyponymy, meronymy relations, their Glosses and Example sentences for semantic Bag. • Word Context Size: Current , previous and following sentences in which word forms for context Bag.

The WSD Algorithm…. • Let ‘w’ be the word whose • • disambiguation is to be done. Construct the context Bag. Construct the semantic Bag. Using the ‘Intersection Similarity’, find the Overlap. Output the sense ‘s’ as the most probable sense which has the maximum Overlap.

Evaluation • Presently, the system disambiguates nouns only. • The test corpora has been taken from CIIL, Mysore. • The system has been tested on corpus from 8 domains and each corpus containing around 2000 words on an average.

Result

Discussion • Agriculture has given maximum correct • result while children literature has given minimum correct result. 25 % of the words are found relevant though they don’t match exactly the sense.