Shahid Mushtaq Bhat
Rupesh Rai
Linguistic Data Consortium for
Indian Languages
Central Institute of Indian
Languages
Abstract:
Morphological
analyzer is a fundamental tool in Natural Language Processing (NLP) that
generates the morphological analyses of a given word-form. It can be used in
enhancing the accuracy of POS-Tagging, Chunking, Syntactic Parsing, Word Sense
Disambiguation (WSD), Information Retrieval (IR) & Machine Translation (MT)
Systems. This paper describes an ongoing effort to develop Nepali morphological
analyzer, using an open source platform-Apertium (LT-Toolbox). Since, it is the
initial stage of this project; we have confined our work to inflectional
morphology. So far, we have covered all the possible categories, as per LDC-IL1
POS tag-set of Nepali. Currently, the coverage of Nepali Morph-Analyzer is
20,000 words, classified into 219 paradigms.
Keywords: Morphological
analyzer, Word and paradigm model, Apertium, LT-Tool Box, Paradigm,
Concatenative Morphology, Machine Translation, Devnagri, Transliteration.
1. Introduction
Automatic
morphological analysis is the basic level of analysis in NLP. The primary
requirement in the development of an automatic morphological analyzer is the
choice of an appropriate theoretical model and the associated tool that can
handle all morphological operations of a language. Owing to the limitations of
Item and Arrangement (IA) & Item and Process (IP) models, it was decided
that Word & Paradigm-WP (Hockett, 1954) is one such model that can capture
all the morphological information of all Indian Languages including Nepali.
Since, Apertium is an open-source tool that uses finite-state transducers &
operates on the basis of Word and Paradigm (WP) model; we use it for developing
Nepali-Morphological Analyzer.
Given the fact that
most of natural languages construct words by concatenating morphemes together
in strict orders, concatenative morphotactics is highly productive in Nepali
and other Indian Languages, particularly, in agglutinative languages like
Manipuri, Tamil, Kannada, etc. However, there are also languages like Arabic to
which we can’t extend the principle of concatenation as their main
morphological operation is infixation. It constitutes
1 Linguistic data consortium for Indian languages is an 11th
plan project of Govt. of India set in Central Institute of Indian Languages
(CIIL), Mysore, on the lines of LDC at University of Pennsylvania, USA. Its
primary goal is to create annotated quality data of 22 Scheduled Languages of
Indian Union.
what is called
Non-Concatenative morphology also known as Templatic or Root & Pattern
morphology (MacCarthy, 1981). Beyond these concatenative &
Non-concatenative operations, some Indian Languages like Urdu, Kashmiri, etc
show interplay of both types of morphological operations.
But, Nepali morphology
is essentially concatenative in nature. Our experience with in handling
different types of morphological patterns confirms our assumption that Apertium
is adequate to capture all morphological operations of Nepali. This paper
mainly focuses on Nepali inflectional morphology and its implementation in
Apertium.
2. Motivation
Since, the primary
requirement in the development of morphological analyzer is the choice of an
appropriate theoretical model and the associated tool/algorithm that can handle
all morphological operations of a language. Our choice of the same was
motivated partly by economy of procedures and partly by the reasons given
below:
i. Theoretical
Model
Instead of
positing combination of morphemes into word-forms or generation of such
word-forms from the stem by the application of rules, WP (word-based) approach
proposes generalizations about the relationships that hold between the
word-forms of an inflectional paradigm.
Both IA
(morpheme-based-approach) and IP (lexeme-based-approach) fail miserably in
explaining the irregular morphology. For example: They assume there are two
different morphemes (or two different rules) involved in the formation of 3rd
person singular verbal form and plural nominal form (-s) in English but the
distinction between them turns out to be artificial. WP model treats such forms
as whole words that are related to each other by some analogical rules. Words
are categorized on the basis of pattern they fit into.
ii. Tool or
Algorithm
Since, using an
open source tool is a better choice and an economic way (both in terms of
procedures and finances) of developing computational resources for resource
poor languages, decision to choose Apertium is a credible one. Further,
Apertium is designed to implement WP (by means of Finite State Transducers)
which motivates us even more to use it.
3. Nepali in
the Context of Indian Languages
Nepali is National
language of Nepal and also one of the 22 schedule languages of the Indian
Union. This makes it very significant Indo-Aryan language like Urdu. It is
mainly spoken in Nepal but also in its neighboring countries like India, Bhutan
& Myanmar, by approximately 45 million speakers (Hardie et al.,
2011)2. In India it is spoken in Darjeeling district of West Bengal, some parts
of Jalpaiguri District, Sikkim, Assam, Manipur, Mizoram, Nagaland and Bhaksu.
Dehradun of
Himachal Pradesh. It serves as the lingua franca for Nepali speech community
present in this part of South Asia which is highly multi-lingual in nature. It
is written in Devnagrai script like many Indian Languages (Hindi, Marathi,
Gujarati and Dogri).
Like other Indian
languages, Nepali is a resource poor language. However, some computational
resources have been developed for Nepali under Project- NeLRaLEC (Nepali
Language Resources & Localization for Education and Communication) also
known as the Bhasha Sanchar Project3 (Yadava et al, 2008). This project
created POS annotated corpus resources for Nepali & proved instrumental in
NLP awareness in Nepal but more thorough research in Nepali Language Processing
was started in the Grammar Checker Development under PAN Localization Project4.
This led to the
study of the structure of Nepali grammar and language as well as the components
required for development of Grammar Checker. Hence, the work on Morph-Analyzer
and Stemmer started in the same project (Bal Krishna Bal & Prajol Shrestha,
2004-2007). However, in India such work started more than a decade back but for
some languages only but need was felt to promote the technological development
in all Indian languages. So, a serious effort for developing computational
resources for all the 22 scheduled languages under a single umbrella-project
started with the inception of Linguistic Data Consortium for Indian Languages
(LDC-IL), set in CIIL. Therefore, the work on Nepali resource creation &
language processing was also undertaken.
Like other Indian
languages, Nepali is morphologically rich with highly productive inflectional
and derivational morphology. Nepali nouns forms show agreement variants for
number, gender and case. They also show diminutive variant forms. Marked
adjectives show similar agreement changes for number, gender and case. Nepali
verbs inflect to show agreement for number, gender, and case. In addition to
these features, Nepali also has different inflections for infinitive, past,
non-past, habitual and imperative forms. All these forms for a regular verb are
duplicated for transitive and causative forms, thus producing a huge number of
inflected variations.
4. Nepali
Morphology
Nepali morphology is
essentially concatenative or linear in nature with abundance of affixes
attached to the roots. It is clearly visible in the feature strings. Dionysius
Thrax’s Techne (C.100 B.C) – a grammatical sketch of Greek – is not only a role
model for contemporary POS descriptions in European languages; it has also
become a role model for POS descriptions in Indian languages. It includes a
scheme of 8 POS-categories (noun, verb, pronoun, preposition, adverb,
conjunction, particle, and article) but the present scheme for capturing
inflectional morphology of Nepali includes 9 POS categories & 29 subcategories.
2 Himalayan Linguistics,
Vol. 10(1). © Himalayan Linguistics 2011 ISSN 1544-7502
3 http://www.bhashasanchar.org
4 http://www.PANl10n.net
i. Noun (N)
A noun is a word
that refers to people, animals, objects, substances, ideas, concepts and
feelings etc. In Nepali, a noun is usually inflected for gender, number, and
case. It is of four types: Common Noun (NC), Proper Noun (NP), Verbal Noun
(NV), Spatio-temporal Noun (NST).
CATEGORY: N
TYPE: NC, NP, NV,
NST
ATTRIBUTES: Number
(0/sg/pl)5
Gender (0/mas/fem)
Case (0/dir/obl)
Case marker
Particle
Quantitative
6FEATURE STRING:
ghara (घर) \
[N.sg.mas.dir.0.0.0]
ii. Verb (V)
A verb refers to
events, actions and states. In Nepali, verb is inflected for tense, aspect,
mood, and honorificity. It also shows agreement for gender, number and person.
It is of two types; Main verb (VM) & Auxiliary verb (VA).
CATEGORY: V
TYPE: VM, VA
ATTRIBUTES: VM=
Person, Number, Gender, Tense, Aspect, Mood
Finite, Voice,
Negation, Particle, Honorific
VA= Person,
Number, Gender, Tense, Aspect, Mood, Finiteness, Negation, Particle
FEATURE STRING:
jA(जा)\[VM.2.sg.0.npst.0.imp.fn.act.0.0.0],
cha(छ)\[VA.3.sg.mas.npst.0.imp.fn.0.0.0]
iii. Pronoun
(P)
Pronoun is a word
which inflects & functions like a noun and substitutes a noun or a noun
phrase. It is of four five types; Pronominal (PPR), Reflexive Pronoun (PRF),
Relative Pronoun (PRL), Reciprocal Pronoun (PRC), Wh-Pronoun (PWH).
CATEGORY: P
TYPE: PPR, PRF,
PRL, PRC, PWH
ATTRIBUTES: PPR=
Person, Number, Gender, Case, Case marker,
Particle,
Definiteness, Dimension, Quantitative
PRF= Number,
Gender, Case, Case marker, Particle,
5 Every attribute has a set of
values e.g. [Number has (0.sg.pl), Gender has (0.mas.fem), Tense has
(0.pst.npst), etc]
6
Feature strings are actually morphological features [e.g. (N.sg.mas.dir.0.0.0)]
associated with a given word-form [e.g. ghara (घर)]
without any reference to the actual division of morphemes within the word-form
which is not necessary for the current work.
Quantitative
PRL= Number,
Gender, Case, Case marker,
Definiteness,
Particle, Quantitative, Dimension
PRC= Number,
Gender, Case, Case marker, Particle
PWH=
Number, Gender, Case, Case marker, Particle
FEATURE STRING:
ma(म)\
[PPR.1.sg.0.dir.0.0.0.y.prx.0],
AphnO
(own)\[PRF.sg.mas.obl.gen.0.0],
jaskO(जसको)\[PRL.sg.mas.obl.gen.0.0.0.0],
EkaApasa(एकआपस)\[PRC.0.0.dir.0.0],
kO(को)\[PWH.0.0.dir.0.0]
iv. Nominal
Modifier (J)
Nominal modifier
modifies nominals (i.e. nouns as well as pronouns).
a. Adjective
(JJ)
An adjective is a
word that modifies nouns. Morphologically, adjectives inflect for gender,
number, and case. Adjectives have comparative and superlative forms.
b.
Quantifier (JQ)
A quantifier
refers to the quantity of the noun described. They also show Gender, number,
and case inflections like Adjectives. A quantifier can be numeral (cardinal,
ordinal) or non-numeral.
CATEGORY: J
TYPE: JJ, JQ, JINT
ATTRIBUTES:
JJ=Number, Gender, Case, Particle, Degree,
JQ = Gender, Case,
Numeral, Particle
FEATURE STRING:
rAmrO (राम्रो)\ [J.
sg.mas.obl.0.pos]
EuTA(एउटा)\[JQ.0.pl.crd.0]
c.
Demonstrative (D)
Demonstrative is
like pronoun but has a deictic function, i.e. pointing out. So, it will always
be followed by a noun, a pronoun or an adjective.
CATEGORY: D
TYPE: DAB, DRL,
DWH.
ATTRIBUTES:
Person, Number, Gender, Case, Case marker, Particle,
Definiteness,
Dimension
d.
Quantitative
DRL= Number,
Gender, Case, Case marker, Particle, Quantitative
DWH= Gender,
Number, Case, Case marker, Particle, Definiteness, Quantitative
FEATURE STRING:
tyO(त्यो)\[DAB.3.sg.mas.dir.0.0.y.dist.0]
jasalE(जसले)\[DRL.0.0.obl.erg.0.0]
kasakO(कसको)\[DWH.sg.mas.obl.gen.0.0.dist]
e. Adverb
(A)
An
adverb is a word that modifies or qualifies the verb. Nepali has two kinds of
adverbs, basic and derived. Some basic adverbs are phutta
(फुत्त),
besari (बेसरी), chiTO (छिटो),
etc. Derived adverbs are from pronouns, and adjectives.
CATEGORY: A
TYPE: AMN
ATTRIBUTES: 000
FEATURE
STRING: pratidina (daily)\[A.000]
f.
Postposition (PP)
Postpositions
denote a relationship between two entities. It is placed after the noun or
pronoun. Functionally, a postposition and a case marker are same but when it is
attached to the noun or pronoun it is called case marker, when it is separate
it is called a postposition. In Nepali, with pronouns it is written together
whereas with nouns it is written separately.
CATEGORY: PP
TYPE: - 0
ATTRIBUTES:
Number, Gender, Case marker, Particle
FEATURE STRING:
kO(को)
[PP.sg.mas.gen.0]
g.
Participles (L)
Participle is a
non-finite verb form that has some of the characteristics and functions of both
verbs and adjectives.
CATEGORY: L
TYPE: LPRV, LPRF,
LIPFV
ATTRIBUTES:
LPRV=Negation, Particle
LPRF=Number,
Gender, Negation, Particle
FEATURE STRING:
khAEra (खाएर)\[PFV.0.0]
khAEkO (खाएको)\[PF.sg.mas.0.0]
h. Particle
(C)
CATEGORY:C
TYPE: CSIM, CLA,
CINT, etc.
ATTRIBUTES:
CSIM=Number, Gender, Case
CLA= Number,
Gender, Case
FEATURE STRING:
jastO(जस्तो)\[CSIM.sg.mas.dir]
vaTA(वटा)\[CLA.sg.0.0],
E(ए)\[CINT.000]
5. Apertium or
LT-Toolbox
Apertium
an open-source platform (both LINUX & WINDOWS compatible) for creating
rule-based machine translation (MT) system (Armentano Oller et al.,
2006; Corbi Bellot et al, 2005). It was developed through a number of
projects like Open-Source Machine Translation for the Languages of Spain and
EurOpen Trad: Open-Source Advanced Machine Translation for the European
Integration of the Languages of Spain by the Transducers Research Group. It was
initially designed for closely-related Romance languages pairs (such as
Spanish–Portuguese, Czech–Slovak, Swedish–Danish etc), but has also been
adapted to work better for less related languages like the Indian Languages. It
consists of pipeline of lexical processing modules: deformatter,
morphological-analyzer, categorical disambiguater, structural and lexical
transfer module, morphological-generator, post-generator and reformatter. It is
now well established that morphological-analyzer cum generator module can be
utilized to develop morphological analyzers and generators for all Indian
languages. The only effort that one has to be made in this direction is to
identify paradigms and add language specific data in the XML schema of the
tool.
6.
Transliteration Scheme TOOL
Since the tool,
morphological analyzer-cum-generator, supports only roman characters properly,
Indic or Persio-Arabic words need to be transliterated first before one starts
the actual work on paradigm building. For this purpose, a well-defined
transliteration module is required. There are many open source romanizing tools
available based on different transliteration schemes. One such tool has been
developed by LDC-IL for romanizing all Indian Scripts. Such tools vis-à-vis
schemes are very essential to benefit from the computational tools developed
across the world which mostly support roman character.
In
the present work, we have used the LDC-IL transliteration tool which is
predominantly based on different mapping schemes but script grammar rules have
been also incorporated to set constraints on the mapping process for better
results. The transliteration system has made the present work very easy. The
Roman-Devanagari transliteration scheme for Nepali is as under:
[अ-a,
आ-A, इ-I,
ई-I, उ-u,
ऊ-U, ऋ-x,
ॠ-X, ए-E,
ऐ-ai, ओ-O,
औ-au, क-ka,
ख-kha, ग-ga,
घ-gha, ङ-ng’a,
च-ca, छ-cha,
ज-ja, झ-jha,
ञ- nj’a, ट-Ta,
ठ-Tha, ड-Da,
ढ-Dha, ण-Na,
त-ta, थ-tha,
द-da, ध-dha,
न-na, प-pa,
फ-pha, ब-ba,
भ-bha, म-ma,
य-ya, र-ra,
ल-la, व-va,
श-sha, ष-Sa
स-sa, ह-ha,
ड़-D’a,
ढ़-D’ha,
क्ष-kSa,
त्र-tra,
ज्ञ-
jnj’a]
So,
a well-defined transliteration scheme as given above & the associated
transliteration tool as shown in Figure 1 is prerequisite for computing
morphology of Nepali, using LT-Toolbox.
Figure
1. Transliteration Tool interface
7. Computing
Nepali Morphology
The process of
computing morphology of Nepali, simply, involves the manipulation of strings or
words forms (letter transducers), with or without much consideration to real
morphemic division. For illustration, we will show the implementation of
inflectional morphology which can be divided into the following steps:
Step 1.
Defining Elements
All the characters and
terms <sdefs> (Characters, Categories, Subcategories, Attributes and
Attribute values) that are used in Transliteration scheme and Morphological
analysis are defined in the XML code as shown below for Noun:
<?xml version=“1.0”?
<dictionary>
<alphabet>abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ-’</alphabet>
<sdefs>
<sdef n=“cat:N” c=“Noun”/>
<sdef n=“subcat:NC” c=“Common noun”/>
<sdef n=“subcat:NP” c=“Proper noun”/>
<sdef n=“subcat:NV” c=“Verbal noun”/>
<sdef n=“subcat:NST” c=“Spatio-Temporal noun”/>
<sdef n=“gender:m” c=“masculine”/>
<sdef n=“gender:f” c=“feminine”/>
<sdef n=“gender:0” c=“feminine”/>
<sdef n=“number:sg” c=“singular”/>
<sdef n=“number:pl” c=“plural”/>
<sdef
n=“case:d” c=“direct”/>
<sdef
n=“case:o” c=“oblique”/>
</sdefs>
Step 2.
Developing Paradigms
All
the possible paradigms (<pardefs>) of a language are first defined which
includes splitting of a Romanized word-form (string of characters) into the
left-unchangeable string which is considered as computational root (not
necessarily a linguistic root) and the right- variable string which is
considered as computational inflection (not necessarily a morpheme). Both are
separated by a splitter (slash). By choosing a word-form (frequent one) for
defining a paradigm, we are actually naming/tagging that very paradigm by the
name of the word-form chosen (e.g. kETO). The variable part of string on the
right side of the splitter gets substituted while generating the inflectional
paradigm (all possible word-forms) for the same word as shown below:
Noun Paradigms
<pardefs>
<pardef n=“kET/O_Nm”>
<e><p><l>O</l><r>O<s
n=“cat:N”/><s n=“subcat:NC”/><s n=“Gender:m”/><s
n=“Number:sg”/><s n=“Case:dir”/><s n=“Case_marker1:0”/><s
n=“Case_marker2:0”/><s n=“Case_marker3:0”/><s
n=“Particle1:0”/><s n=“Particle2:0”/><s n=“Particle3:0”/><s
n=“Quantitative:0”/><s n=“Distributive:0”/></r></p></e>
</pardef>
<pardef n=“AmA/_Nf”>
<e><p><l></l><r><s
n=“cat:N”/><s n=“subcat:NC”/><s n=“Gender:f”/><s n=“Number:sg”/><s
n=“Case:dir”/><s n=“Case_marker1:0”/><s
n=“Case_marker2:0”/><s n=“Case_marker3:0”/><s
n=“Particle1:0”/><s n=“Particle2:0”/><s n=“Particle3:0”/><s
n=“Quantitative:0”/><s n=“Distributive:0”/></r></p></e>
</pardef>
Pronoun Paradigms
<pardef n=“ma/_P”>
<e><p><l></l><r><s
n=“cat:P”/><s n=“subcat:PPR”/><s n=“Gender:0”/><s
n=“Number:0”/><s n=“Case:dir”/><s n=“Case_marker1:0”/><s
n=“Case_marker2:0”/><s n=“Case_marker3:0”/><s n=“Particle1:0”/><s
n=“Particle2:0”/><s n=“Particle3:0”/><s n=“Quantitative:0”/><s
n=“Distributive:0”/><s n=“Person:1”/><s
n=“Definiteness:y”/><s n=“Dimension:0”/></r></p></e>
</pardef>
<pardef n=“tam’/_P”>
<e><p><l></l><r><s
n=“cat:P”/><s n=“subcat:PPR”/><s n=“Gender:0”/><s
n=“Number:sg”/><s n=“Case:dir”/><s n=“Case_marker1:0”/><s
n=“Case_marker2:0”/><s n=“Case_marker3:0”/><s
n=“Particle1:0”/><s n=“Particle2:0”/><s n=“Particle3:0”/><s
n=“Quantitative:0”/><s n=“Distributive:0”/><s
n=“Person:2”/><s n=“Definiteness:y”/><s
n=“Dimension:0”/></r></p></e>
</pardef>
Nominal
Modifier Paradigms
<pardef n=“catura/_J”>
<e><p><l></l><r><s
n=“cat:J”/><s n=“subcat:JJ”/><s n=“Gender:0”/><s
n=“Number:0”/><s n=“Case:0”/><s n=“Particle1:0”/><s
n=“Distributive:0”/><s n=“Degree:0”/></r></p></e>
</pardef>
<pardef n=“budhdU/_J”>
<e><p><l></l><r><s
n=“cat:J”/><s n=“subcat:JJ”/><s n=“Gender:0”/><s
n=“Number:0”/><s n=“Case:0”/><s n=“Particle1:0”/><s
n=“Distributive:0”/><s n=“Degree:0”/></r></p></e>
</pardef>
Verb Paradigms
<pardef n=“bas/_V”>
<e><p><l></l><r><s
n=“cat:V”/><s n=“subcat:VM”/><s n=“Gender:0”/><s
n=“Number:sg”/><s n=“Particle1:0”/><s n=“Person:2”/><s
n=“Negation:0”/><s n=“Tense:npst”/><s n=“Aspect:0”/><s
n=“Mood:imp”/><s n=“Voice:act”/><s n=“Finiteness:fin”/></r></p></e>
</pardef>
<pardef n=“rU/_V”>
<e><p><l></l><r><s
n=“cat:V”/><s n=“subcat:VM”/><s n=“Gender:0”/><s
n=“Number:sg”/><s n=“Particle1:0”/><s n=“Person:2”/><s
n=“Negation:0”/><s n=“Tense:0”/><s n=“Aspect:0”/><s
n=“Mood:imp”/><s n=“Voice:act”/><s
n=“Finiteness:fin”/></r></p></e>
</pardef>
Adverb
Paradigms
<pardef n=“acAnaka/_A”>
<e><p><l></l><r><s
n=“cat:A”/><s n=“subcat:AMN”/><s
n=“Particle1:0”/></r></p></e>
</pardef>
<pardef n=“acAklI/_A”>
<e><p><l></l><r><s
n=“cat:A”/><s n=“subcat:AMN”/><s
n=“Particle1:0”/></r></p></e>
</pardef>
Participle
Paradigms
<pardef n=“khAikana/_L”>
<e><p><l></l><r><s
n=“cat:L”/><s n=“subcat:LPRF”/><s n=“Particle1:0”/><s
n=“Particle2:0”/><s n=“Particle3:0”/><s
n=“Negation:0”/></r></p></e>
</pardef>
<pardef n=“khAEra/_L”>
<e><p><l></l><r><s
n=“cat:L”/><s n=“subcat:LPRV”/><s n=“Particle1:0”/><s
n=“Particle2:0”/><s n=“Particle3:0”/><s
n=“Negation:0”/></r></p></e>
</pardef>
Postposition
Paradigms
<pardef n=“lAgi/_PP”>
<e><p><l></l><r><s
n=“cat:PP”/><s n=“subcat:PP”/><s n=“Gender:0”/><s
n=“Number:0”/><s n=“Case_marker1:bnf”/><s
n=“Case_marker2:0”/><s n=“Case_marker3:0”/><s
n=“Particle1:0”/><s n=“Particle2:0”/><s n=“Particle3:0”/><s
n=“Distributive:0”/></r></p></e>
</pardef>
<pardef n=“sAtha/_PP”>
<e><p><l></l><r><s
n=“cat:PP”/><s n=“subcat:PP”/><s n=“Gender:0”/><s
n=“Number:0”/><s n=“Case_marker1:0”/><s
n=“Case_marker2:0”/><s n=“Case_marker3:0”/><s n=“Particle1:0”/><s
n=“Particle2:0”/><s n=“Particle3:0”/><s
n=“Distributive:0”/></r></p></e>
</pardef>
Particle Paradigms
<pardef n=“ani/_C”>
<e><p><l></l><r><s
n=“cat:C”/><s n=“subcat:CCD”/></r></p></e>
</pardef>
<pardef n=“EvaM/_C”>
<e><p><l></l><r><s
n=“cat:C”/><s n=“subcat:CCD”/></r></p></e>
</pardef>
Step 3.
Dictionary Building
Lemmatization
is done manually and computationally feasible lemmas are included in the
dictionary along with the paradigm name, they follow (already defined). Here,
compromising of linguistic feasibility can’t be ruled out. Sufficient number of
dictionary entries is to be made in order to assure wider coverage of the
morphological analyzer/generator. Computationally feasible lemmas with their
respective paradigm tag are show below in the XML encoded dictionary.
<section
id=“main” type=“standard”>
<elm=“kET”><i>KET</i><par
n=“kET/O_Nm”/></e>
<elm=“acAn”><i>acAn</i><par
n=“kET/O_Nm”/></e>
<elm=“acAnak”><i>acAnak</i><par
n=“acAnaka/_A”/></e>
<e
lm=“garn”><i>garn</i><par n=“garna/_V”/></e>
<e
lm=“aDn”><i>aDn</i><par n=“garna/_V”/></e>
<e
lm=“ghar”><i>ghar</i><par n=“ghara/_Nm”/></e>
<e
lm=“AkAr”><i>AkAr</i><par n=“ghara/_Nm”/></e>
<elm=“akAs”><i>akAs</i><par
n=“ghara/_Nm”/></e></section>
</dictionary>
Step 4.
Compiling & Processing
After
defining each paradigm and finishing the associated dictionary building, we can
run the system to check if the given word form is properly analyzed (i-e
assigned the analysis of the proper paradigm). For running purpose, we first
compile dictionary into a sort of compact & efficient representation by a
default command called lt-comp.
Syntax:
lt-comp lr
dictionary.dix outputFile.bin ←
Then, we process
the compiled data for getting the analysis of the word form, using another
default command called lt-proc.
Syntax:
lt-proc
–c outputFile.bin ←
9. Interface
Since, the entire process of resource building (i.e. paradigm defining
& dictionary building) is manual and labor intensive, involving direct
manipulation of the XML code called Meta-Data. There are much chances of
mishandling & thus, even a minute alteration in the XML schema renders the
whole system non-functional. Moreover, the inputting is very prone to
repetitions & redundancies. One can’t remember the entire list of words
that has been entered in the lexicon. So, to avoid such complexities, a more
interactive interface has been developed to facilitate our work and reduce the
time taken for defining a single paradigm. Besides, preventing the
mishandling of
Meta-data and repetition of lemma entries, the interface additionally reduced
the burden of manual lemmatization as it lemmatizes automatically.
As
shown in Figure 2, the window of the interface has the two parts; the upper
paradigm defining part and the lower lexicon building part. We can now; either
input the word directly into the text box or load it from the word list and get
its lemma easily, just by assigning an appropriate paradigm from the drop down
list.
Figure 2. Apertium interface
10. Quantum and
Coverage
Currently, the quantum
of our work is very less, only 20,000 Ws. Since, the time consuming analysis
part of the work is almost over. It is expected to show speedy growth in near
future as our work will be mostly confined to dictionary building. So far, we
have covered 9 categories and 29 Sub-categories in our analysis. The number of
paradigms and lemma per category is shown in Table 1.
Table 1. Details of
Nepali Morph-Analyzer
|
Categories
|
No. of Sub-Categories
|
No. of Paradigms
|
No. of Lemma (Ws)
|
|
Noun
|
04
|
045
|
-
|
|
Pronoun
|
05
|
016
|
-
|
|
Demonstrative
|
03
|
003
|
-
|
|
Nom-Modifier
|
03
|
031
|
-
|
|
Verb
|
02
|
023
|
-
|
|
Adverb
|
01
|
016
|
-
|
|
Postposition
|
01
|
008
|
-
|
|
Participles
|
01
|
006
|
-
|
|
Particle
|
09
|
071
|
-
|
|
Total
|
29
|
219
|
20,000
|
11. Conclusion
The above discussion
reveals that developing a Finite-state morphological-analyzer for Nepali using
LT-Toolbox is an efficient way given the elegance of the combinatory effect of
word and paradigm model and the finite state transducers on speedy processing.
Further, using an open source tool in creation of computational resource for
resource poor Indian languages is need of hour, as it involves low cost and
less time. Finally, for developing morphological analyzer with this approach
one doesn’t require any programming skill but a little working knowledge of
computers and morphology. So, every one irrespective of having programming
background and strong linguistic skills can develop language specific resources
and can contribute in the technological development of Indian language. In the
current work we have finished the development of inflectional morphological
analyzer for Nepali. Almost all inflectional paradigms have been captured and
increasing the size of dictionary will continue until extracting and using
unique words from 1 million text corpus of Nepali is over. Further, once
inflectional morphology is done with sufficient coverage, derivational
morphology will be taken care of in future to make a robust morph analyzer of
Nepali.
References
Adhikari, H. R.
(1993). Samsamayik Nepali vyaakaran. Kathmandu: Kunjal Prakashan.
Bharti, A.,
Chantanya, V. & Sanghal, R. (1995). Natural language processing: A
Paninian perspective. New Delhi: Prentice Hall.
Baerman, M.,
Brown, D., & Corbett, G. G. (2005). The syntax-morphology interface: A
study of Syncretism. Cambridge University Press.
Boye, G. (1999).
Nepali verb morphophonology. In P. Yogendra, P. Yavada & W. Warren (eds.). Topics
in Nepalese linguistics (pp. 118-169). Kathmandu: Royal Nepal Academy.
Forcada, M. L., Bonev,
B. I., Rojas, S. O., Ortiz, J. A. P., Sanchez, G. R., Martinez, F. S.,
Armentano-Oller, C., Montava, M. A., & Tyers, F. (2010). Documentation of
the open-source shallow-transfer machine translation platform Apertium.
Retrieved on December 3, 2012 from http://xixona.dlsi.ua.es/~fran/apertium2-documentation.pdf
Hussain, S.
(2004). Finite-state Morphological analyzer for Urdu. Unpublished MS
thesis, Center for Research in Urdu Language Processing, National University of
Computer and Emerging Sciences, Pakistan.
Jurafsky, D. &
Martin, J. (2005). Speech and language processing: An introduction to
natural language processing computational linguistics and speech recognition.
Boulder: University of Colorado Boulder.
LDC-IL
Transliteration Chart For Indian Languages, Retrieved on December 3, 2012 from
http://www.ldcil.org/download/Transliterationstandards.pdf
Stump, G. T.
(2001). Inflectional morphology. A theory of paradigm structure.
Cambridge: Cambridge University Press.
Uma Maheshwar Rao
G. & Prameshwari, K. (2010). On the description of morphological data for
morphological analyzers and generators: A case study of Telgu Tamil &
Kannada. Knowledge Sharing Events. LDC-IL, CIIL, Mysore.
Vaidhya, Ashwini
and Dipti Mishra.2009.Using Paradigms for Certain Morphological phenomenon in
Marathi, 7th International Conference on NLP (ICON-2009. New Delhi: Macmilan.
Zwicky, A.M. (1985).
How to describe inflection. Linguistic Society 11, 372-386.
