[This article and all those to come are all extractions from my university BA research paper. Changes, updates and improvements are to be made when necessary. All rights reserved for the owner.][Part 8] Note: Compared to the original paper, various parts are ignored for multiple reasons, most of which is due to major updates have taken place and the issues found with the older methods. (For example, the interface was made with Python Tkinter GUI framework. And How the results were preprocessed, proceeded and presented. Bugs fixed and imporvements in processing speed) We shall, however, discuss the updates in the upcoming articles. 1. Evaluation The purpose of this study was to answer a couple of questions and to test them on thereal world. First, we asked if we could construct a model that could stem (a) given input(s)string(s) into its (or their) basic form(s) or root(s). Secondly, we argued that the absence oforthographic standards for writing Moroccan Arabic poses challenges and complications onthe text processing level of language. Both the queries were met and validated. We created amodel that focuses most on inflectional morphology. It also supports simple derivativemorphology, but since our goal was to preserve the meaning of the input string for uses suchas that of a simple word translation, we only stem the surface level affixes of the derivativemorphemes, thus preserving the meaning. E.g. "‫"فالمدرسة‬ meaning "at (the) school" isstemmed to "‫"مدرسة‬,"school" rather than to the basic derivative root "‫"درس‬, meaning "lesson".As we have illustrated on table 3.3 on the previous section, we validate our secondquestion, that is the absence of orthographic standards for writing in MA does actually raisecomplications. Still, we have tried to show that some of these complications tend to be ofsimilar nature, thus maintainable and predictable, such as whether to place an "‫"ا‬ or not, whenand where. Some approaches have been developed to go around this issue, but there are ofcourse more to cover as this research's claim is that modeling such phenomenon as stemmingin MA is possible. Of course our stemmer would not always succeed in correctly stemming agiven string, for instance given the two verb strings "‫"مشات‬ , and "‫"كتبات‬, given the numbers 1and 2 respectively: Original input:‫مشات‬Transliterated Input:m$AtFound stem:‫مش‬Parsed results:[perfect_3rd_sng_personal_fem_prn-At-She + VERB_past_stem-m$]Translation:Not found--------------------------------------------------Original input:‫كتبات‬Transliterated Input:ktbAtFound stem:‫كتب‬Parsed results:[perfect_3rd_sng_personal_fem_prn-At-She + VERB_past_stem-ktb]Translation:('‫'كتب‬, 'write') Both the two verbs are: Conjugated in the past tense, 3rd person singular, and feminine.Verb 1 has a length of four characters, verb 2 has a length of five characters. The affix ("","At") is recognized as the 3rd person singular feminine pronoun, "she" in English. It isstemmed and stems "‫"مش‬ and "‫"كتب‬ are returned. The problem is that verb 1 has the sameending as that of the beginning of the attached suffix "‫"ات‬, thus when we stem it, we get thestem or root "‫"مش‬, which would be the correct stem if the "A" of the suffix has not beentrimmed with it. Still, again, we could go around and create some rules that check if suchsituations exist in a given string, then if affirmative, we either strip the suffix "At" and thenadd an "A" to the stem. This "A" would be replaced with a "Y" on the next processing level,thus we get the stem "m$Y", detransliterated as "‫"مشى‬, which is, by means of this research,the correct form. Or when we meet such criteria, we could trim only the "t" from the suffix"At" but then save the suffix as "At". This might look a little fuzzy, but when written in code,it makes more sense. This problem has already been taken care of at a later phase of thisresearch. When you process the word "‫"مشات‬ you would get the valid correct form "‫"مشى‬.The point from the above instance is that we could have all sorts of issues and exceptions thatmay occur, but throughout research, as long as these exceptions raise to the surface to beknown, a solution is always there. Regarding the part of translation (word look-up), it was included as stated before to show how veryhelpful stemming could be when integrated with other tools and/ or programs such as that oftranslation itself. For a native speaker of MA, most if not all of the possible, feasible andmeaningful combinations of a given stem and affixes could easily be understood, but imaginea foreigner who has just started learning the Moroccan Arabic, this could be rather painful andfuzzy for him or her. He or she may know that the MA word "‫"مشى‬ means "went" and is averb, but what if he or she was given the word "‫"ممشيناش‬ or "‫?"مامشيناش‬ The latter two wordsare both the same one with the only difference in the "‫"ا‬ after the negation prefix "‫"م‬ ,meaning "we have not gone". Our stemmer would stem both and return the stem "‫"مشي‬,making it easy to recognize. To conclude this evaluation, we must state that we could develop our stemmingalgorithm, including the patterns and rules defined. We could add other rules and/or patterns,or we could eliminate existing ones or make them better. Another thought that came along theway with this research is that we could also modify and reverse our algorithm to do somethingyet useful. We could use it to, instead of stemming affixes off of the root or stem word,generate inflectional words with only the stem given. For example given the input string"‫"كتب‬, we could generator inflected words such as: ‫كتبنا‬ ,‫كتبوا‬ ,‫كتبت‬ , ‫كتبتي‬ , etc. 2. Limitations Although this research has nearly met what it promised. Still, many limitations werepresent. For example: the shortage of time, resources and expertise. Great amounts of timehave been invested on both the literature part and the programming part of this research.Writing the algorithm and code was not the most time consuming part, but it was that ofdebugging and developing the code to suit its purpose as best as it could. In addition, mappingsuch a phenomenon requires deep understanding of the morphology of Moroccan Arabic,which is not very well documented on the works known to this research at the time of writing.This takes us to the next point which concerns the very small number of participants whoengaged in the questionnaire, exactly eight participants. This is because at the time when thequestionnaire was set up, the students were preparing for their exams and the classes werealmost over. 3. Delimitations This research does not claim to be a well defined guide on how to build or develop suchtools as stemmers, though it specified all the necessary steps of doing so. This work tries tooutline some of the issues found throughout the journey of programming and developing the modeldiscussed in it. This is rather a humble attempt. The stemming algorithm with the patterns and rules have been altered everyonce in a while, for a reason or another. We do not claim that the stemmer at current state andversion could stem any given string. It fails in doing so sometimes, that is how we haveovercame those failings all the way long. At the beginning of this research, the stemmer couldnot stem simple strings such as "‫"كتبت‬ meaning "I wrote", but it could stem other strings suchas "‫"مكتبش‬ meaning "He did not write". Although the latter one may look a little complex thanthe former one, it is made only of the stem "‫"كتب‬ and the negation circumfix ‫ش‬###‫م‬.Through development, our stemmer could successfully stem complex strings such" ‫نكتبواش‬ ‫مغا‬ ‫و‬ " meaning "and I will not write it" and "‫"بدارنا‬ meaning "in/at our house". The twostrings are stemmed correctly, but wrong information has been assigned to the function andform of some of their affixes. Here is why: The verb "‫"كتبوا‬ meaning "they wrote" and theform of the same verb "‫"كتبنا‬ meaning "we wrote" has different endings "‫"وا‬ and "‫"نا‬. Thisallows our stemmer to recognize the two endings and assign them values:[perfect_3rd_plr_personal_prn-wA-They + VERB_past_stem-ktb] : ‫كتبوا‬[perfect_1st_plr_personal_prn-nA-We + VERB_past_stem-ktb] : ‫كتبنا‬Until now, the stem is correct and valid and the parsed results are also correct for boththe strings. The two strings are verbs in the past. Our earlier input is negation in the future.Let us explain briefly what happens in the background: The string "‫"ومغانكتبواش‬. ‫‬‫مغانكتبواش --&gt; ومغانكتبواش‬: Conjunction "‫"و‬ removed.‫غانكتبوا --&gt; ‫مغانكتبواش‬‬: Negation circumfix "‫ش‬###‫"م‬ removed.‫كتبوا --&gt; ‫غانكتبوا‬‬: The future action prefix "‫"غان‬ removed.‫كتب --&gt; ‫كتبوا‬‬: The "‫"وا‬ here is assumed to be belonging to that of the 3rd person plural personalpronoun "‫"وا‬ meanig "they". Because the stemmer only recognizes patterns and rules, itcannot tell that the given string "‫"كتبوا‬ and the reduced stem "‫"كتبوا‬ are different since they arethe same in orthography. In fact, it is hard even for a human to decide which is which. Thestem "‫"كتبوا‬ could mean: <ol><li>They wrote</li><li>Write! (order for plural)</li><li>He wrote it</li><li>Write it! (order)</li></ol> 	All kind of issues are present and it takes time and precise guidelines and rules untilmost, if not all, these issues and exceptions are tackled. We could speak of two issues in termsof delimitations, over-stemming and under-stemming. Over stemming is when a given stringis overly stemmed until it loses its core root or meaning. e.g. the string "‫"لعبنا‬ is overlystemmed to "‫"عب‬ instead of "‫"لعب‬. This is not otherwise true when we negate the same string:"‫"ملعبناش‬ meaning "We have not played" were it is stemmed to the correct root "‫"لعب‬. This isbecause the stemmer falsely identified the "‫"ل‬ in "‫"لعبنا‬ as the definite article "‫"ل‬ meaning"the". This has not occur with the negation form of the same string, this is because the stepwhere the stemmer searches for determiners comes before the one where it searches for thenegation circumfix. And since the determiner "‫"ل‬ cannot be an infix, the stemmer justassumes there is none, and indeed, none there is in this string. Under-stemming is the oppositeto over-stemming. It is when a given string is not fully stemmed to the basic unit or root.To conclude, the more testing is done, the more delimitations found and fixed. Thisresearch, bearing in mind the amount of time that available, could never cover all theirregularities or label the delimitations.

Darija NLP: Text Processing And Morphological Stemming - Discussion

Majdi Mahfoud