Design of an Electronic Sanskrit Reader SALA XXI

Design of an Electronic Sanskrit Reader SALA XXI, Konstanz, October 2001 Gérard Huet INRIA SALA October 2001

History • 1994 - personal lexicon in Te. X • 1996 - available on Internet • 1998 - 10000 entries - invariants design • 1999 - reverse engineering • 2000 - Hypertext version on the Web, sandhi processor, grammatical engine • 2001 - Segmenter, tagger SALA October 2001

SALA October 2001

SALA October 2001

SALA October 2001

Processing chains Internet Web site (html) Data base source Abstract structure Index a . . . h CGI bin (Ocaml) devnag tex dvi ps pdf SALA October 2001

Close-up view of functionalities CGI Dico. ps Index Engine Dico DB Grind Entries Flexed Dico. pdf Web site (html) Index a Grammar Engine . . . h Flex SALA October 2001

Tools used Printable document Hypertext document • Knuth Te. X, Metafont, La. Te. X 2 e • W 3 C HTTP, HTML, CSS • Velthuis devnag font & ligature comp • Unicode UTF-8 • Adobe Postscript, Pdf, Acrobat • Chris Fynn Indic Times Font Processing & Search • INRIA Objective Caml SALA October 2001

Each entry is a (typed) tree Type entry = [ | Entry Crossref ] . . syntax usage opt cogs N. B. Syntax is really morphology, usage is part of speech roles plus meanings SALA October 2001

Grammatical information type gender = [ Mas | Neu | Fem | Any ]; type number = [ Singular | Dual | Plural ]; type case = [ Nom | Acc | Ins | Dat | Abl | Gen | Loc | Voc ]; SALA October 2001

The verb system type voice = [ Active | Reflexive ] and mode = [ Indicative | Imperative | Causative | Intensive | Desiderative ] and tense = [ Present of mode | Perfect | Imperfect | Aorist | Future ] and nominal = [ Pp | Ppr of voice | Ppft | Ger | Infi | Peri ] and verbal = [ Conjug of (tense * voice) | Passive | Absolutive | Conditional | Precative | Optative of voice | Nominal of nominal | Derived of (verbal * verbal) ]; SALA October 2001

Governance templates (Grammatical valence) word{ga. n}. . . sem{imputer qqc. à qqn. } ca{chandayati} … sem{gratifier qqn. de } word{niyuj}. . . sem{confier qqc. à qqn. } root{krii}. . . sem{acheter (qqc. à qqn. )} Other specific notations for synonyms, antonyms, cross-references. SALA October 2001

Key points • Each entry is a structured piece of data on which one may compute • Consistency and completeness checks : • every reference is well defined once, there is no dangling reference • etymological origins, when known, are systematically listed • lexicographic ordering at every level is mechanically enforced • • • Specialised views are easily extracted Search engines are easily programmable Maintenance and transfer to new technologies is ensured Independence from input format, diacritics conventions, etc. The technology is scalable to much bigger corpus SALA October 2001

Generic reuse of the technology The structure of the dictionary makes separate as much as possible 3 layers : • sanskrit • french • generic dictionary structure Thus the french meanings, at the leaves, could be replaced by e. g. english definitions or glosses. SALA October 2001

Morphological analysis, sandhi • • Sanskrit is pronounced as written … and thus is written as pronounced Phonetic alliteration is rendered by morphology junction (sandhi) The sentence is formed of words joined by external sandhi Compound words are also formed by external sandhi Whereas flexion, prefixing and suffixing use internal sandhi External sandhi is local, internal sandhi is less Sandhi analysis is non-deterministic and sometimes involves sem SALA October 2001

Grammatical engine • In sanskrit, declension is determined by stem and gender • Sanskrit is very regular, since the classical language was frozen by Pânini (4 th century BC) who invented context-free notation • But it spans about 35 centuries, and thus there are many exceptions • Substantive (adjectives, pronouns, numerals) declension may be arranged in 84 tables of 24 endings (3 numbers * 8 cases) • Then internal sandhi is applied to a stem and an ending • Two applications : • online declension of words given with gender (cgi-bin) • offline computation of flexed forms (2000 pages of double-column fineprint) SALA October 2001

Interactions lexicon-grammar • The index engine, when given a string which is not a stem defined in one of the entries of the lexicon, attempts to find it within the flexed forms persistent database, and if found there will propose the corresponding lexicon entry or entries • From within the lexicon, the grammatical engine may be called online as a cgi which lists the declensions of a given stem. It is directly accessible from the gender declarations, because of an important scoping invariant: • every substantive stem is within the scope of one or more genders • every gender declaration is within the scope of a unique substantive stem SALA October 2001

Inverting external sandhi • External sandhi rules are of a finite-state nature • The flexed forms lexicon index may be seen as the graph of a deterministic finite automaton recognizing its words • This tree may be uniformly decorated by relevant sandhi rules seen as non-deterministic choice points • This structure may be evaluated as a finite-state transducer graph segmenting an input text as words joined by sandhi SALA October 2001

Examples of segmentation Chunk: o. mnama. h"sivaaya may be segmented as: • [ om with sandhi m|n ->. mn ] • [ namas with sandhi s|"s ->. h"s ] • [ "sivaaya with no sandhi ] Chunk: kusuma. mgopiibhya. hk. r. s. nodadati may be segmented as: • [ kusumam with sandhi m|g ->. mg] • [ gopiibhyas with sandhi s|k ->. hk] • [ k. r. s. nas with sandhi as|d -> od] • [ dadati with no sandhi] SALA October 2001

From segments to tagged lemmas Chunk: kusuma. mgopiibhya. hk. r. s. nodadati may be lemmatized with tags as: • [ kusumam < acc. sg. n. of kusuma | nom. sg. n. of kusuma | voc. sg. n. of kusuma > with sandhi m|g ->. mg ] • [ gopiibhyas < abl. pl. f. of gopa | dat. pl. f. of gopa > with sandhi s|k ->. hk ] • [ k. r. s. nas < nom. sg. m. of k. r. s. na > with sandhi as|d -> od ] • [ dadati <…> with no sandhi] SALA October 2001

Future work • • • Verb conjugation tables preparation - full flexed forms database Fixing sandhi analysis for bahuvrihi compounds Choice of taggings from concord and valency constraints Semantic guidance from ontology classification and we shall then able to semi-automatically index corpuses towards • • computer-aided concordance of corpus computer-aided preparation of critical editions statistical analysis of corpus (co-occurrence, style, etc) computer-aided accretion of lexicon fully indexed citations extraction of corpus-specific lexicons diachnony control of lexical information SALA October 2001