[This article and all those to come are all extractions from my university BA research paper. Changes, updates and improvements are to be made when necessary. All rights reserved for the owner.]

[Part 6]

Note: Compared to the original paper, various parts are ignored for multiple reasons, most of which is due to major updates have taken place and the issues found with the older methods. (For example, the interface was made with Python Tkinter GUI framework. And How the results were preprocessed, proceeded and presented. Bugs fixed and imporvements in processing speed) We shall, however, discuss the updates in the upcoming articles.

1. Tokenization

Before we speak of tokenization, we shall mention a preprocess that comes way before

it, normalization. Normalization in the context of this research refers simply the omission of

Arabic punctuation. We normalize an input so we could avoid many problems that might rise

during processing such as unexpected behaviors that would surely interfere and affect the

processing as well as the results. We could now talk about tokenization.

Tokenization is the process by which we split or tokenize an array of data on specific

criteria. As both the algorithm and its flowchart show on the previous chapter, no processing

is done before this simple, yet very crucial step. We could speak of, for example, an input x

which could be one single string, and we could also speak of an input with multiple strings.

Theoretically, we could have an input with an infinite number of strings or tokens, but, surely,

we are limited by the technology and resources employed in this research. To compare

between tokenization in English and in Moroccan Arabic, here is some instances:

1. English: "And they will not write it"

tokenized as: ["And", "they" , "will", "not", "write", "it"]

2. Moroccan Arabic: "‫"ومغيكتبواهاش‬

tokenized as: ["ومغيكتبواهاش"]‬

The two instances are equivalents in meaning, the first one is well indented and the

boundaries between each lexical item is clear and thus is treated as a list of tokens, whilst in

the second one, no boundaries or indentation is present, and thus is treated as one single string

to be processed. This is due to process by which words are constructed in MA, both

inflectionally and derivationally. An example of a possible input with more than one string

regarding MA would be as the following: "ومغيكتبواهاش اليوم"

("ومغيكتبواهاش اليوم"‬, "wmgyktbwAhA$ Alywm", "And they will not write it today")

tokenized as: ["اليوم", "ومغيكتبواهاش"]

Here we have two strings and each is processed individually. After a successful tokenization,

we may want to proceed to the next step, stemming.

2. Stemming

Stemming is the process by which we stem or remove affixes off a given word or string.

The whole purpose of stemming is to try to find the basic stem of a string. When we find the

stem of a string we could do more processing such as POS-tagging and word translation. So

how does our stemming algorithm really work? The best approach to this question is

illustrated by the following example:

We pass our previously tokenized list of strings to our stemmer and hope it finds the right

patterns and rules: ["اليوم", "ومغيكتبواهاش"]

When we hit the process button, we get the following:

Original input:

‫ومغيكتبواهاش‬

Transliterated Input:

wmgyktbwAhA$

Found stem:

‫كتب‬

Parsed results:

[conjuction-W-And + Negation_prefix-m-Not + Negation_suffix-$-Not +

future_action_prefix-gy-Will + object_suffix-hA + pronoun_suffix-wA +

VERB_past_stem-ktb]

Translation:

('‫'كتب‬, 'write')

--------------------------------------------------

Original input:

‫اليوم‬

Transliterated Input:

Alywm

Found stem:

‫يوم‬

Parsed results:

[DET-Al-The + NOUN-ywm]

Translation:

('‫'يوم‬, 'today')

--------------------------------------------------

3.1 Output of our strings

We notice that the stemmer successfully identified all the affixes, stripped them off, and

then returned the basic words of both the strings: "‫"كتب‬ and "‫"يوم‬ meaning "write" and "today",

respectively. We could also notice that the output provided us with some meaningful data.

Such as the parsed results part, where the stemmer give us information on the affixes found,

their function, form, or either, and sometimes the equivalent of the stem in the English

language. The parsed results may seem a little fuzzy, so here is some basic notation:

conjuction-W-And + each color starting from left to right given the numbers 1, 2, 3, 4

respectively. Number one, as just stated above, describes the function, form, or either of the

affix; number 2 is the found stem, of course being it transliterated with the BuckWalter

scheme, "W" in this example for instance when mapped back to the Arabic character

equivalent is "‫;"و‬ number 3 is the English equivalent of the affix found, you may have noticed

that it is not always present as sometimes ambiguity rises to which affix is or functions as

what; lastly, number 4, the plus sign basically denotes that there is yet another affix following

the current one. In the returned data, we also notice that there is a "Translation" label. After a

successful stemming, the found stem is looked up against a small database of basic stems and

words, if any match found, it returns it. Regarding the translation part, we must also mention

that it is also done when no stem is found. The reasons explained next.

3. Translation

We include the translation (word look-up) part for two main reasons:

Our input string could be the basic stem itself, e.g. "‫"كتب‬. Our stemmer would not recognize the given string for the simple raison that it looks for words with any possible affixes, the stem word "‫"كتب‬ has none but it is one, so when no stems are found, the program assumes that the string is either the basic form of the word, or an error occurred during stemming and thus failed to stem, or the user provided an invalid input. In either cases, the program searches the database for any results so it can provide us with any useful data instead of printing "Not found".
We employed translation to illustrate how very useful stemming could be. Again, to show what this means, lets observer the following chart.

Chart 3.2. Inflectional words stemmed to their basic stem

All the instances included in the above chart, and other possible ones, are stemmed into

four basic stems: "‫"كتب‬, "‫"مشى‬, "‫"دار‬, and "‫"سوق‬. You might notice in Sample 2, some

inflected instances have a "‫"ي‬ or "‫"ا‬ as part of their root as in: "‫"مشيت‬ and "‫"مشات‬, but the end

result is always "‫"مشى‬. This is because we cannot have different basic stems or roots for the

same word, so instead of getting the following:

مشيت --> ‫مشي‬

‫مشات --> مشا‬‬

We employ a little function that replaces both the "‫"ي‬ and "‫"ا‬ with "‫"ى‬ when special

criteria and patterns are satisfied. This technique falls rather under what is called lemmatizing,

which makes sure that the produced stem is always an actual word. Lemmatizing usually

uses wordlists (database of valid words of a language) for such purposes. We, however, rely

on some predefined criteria for this task. Some examples our lemmatizer could handle, in

addition to the one shown above, as follows:

Without the lemmatizing technique:

مدرستنا‬ --> ‫مدرست

‫مخبزتنا‬ --> ‫مخبزت‬

With the lemmatizing technique:

‫مدرستنا‬ --> ‫مدرسة‬

‫مخبزتنا‬ --> ‫مخبزة

If we looked up our database before the stems of both the above words were lemmatized, no

results would be found though the database contains the valid forms of both the words. This is

basically because, for example, " ‫مدرست‬ " is not " ‫"مدرسة‬. Speaking of dictionary database, we

must mention that the database used in this work is of a very small size and number of words.

No more than 100 queries. Of course this is only to illustrate how word translation could be

made easy when employing techniques such as stemming. The database could be easily

extended however.

In chart 3.2. some instance are aligned together, e.g. " ‫غيكتبوا‬ ,‫غيكتبو‬ " and " ‫غيمشوا‬ ,‫غيمشو‬ "

in sample 1 & 2, and "‫لدار‬ ,‫"الدار‬ and " ‫لسوق‬ ,‫السوق‬ " in samples 3 & 4. These are aligned

because they have the same meaning. We do not mean they are synonyms, but they are the

same word except for sample 1 & 2 one of each has a final "‫"ا‬ and the other, for each, does

not. In the samples 3 & 4, a similar situation in noted, except this time it is at the initiate

position, "‫"ال‬ and "‫"ل‬ here both mean the definite article "‫"ال‬. Then why specify both if they

are the same thing? Good question. This is because different users tend to write the same

word differently sometimes. Some other times the same user could write the same word

differently at different places and or times. This is not to blame the user, of course, but it is

due to the absence of orthographic standards for written in Moroccan Arabic. This proves to

be challenging and messy when trying to map such phenomenon as that of which patterns

occur which positions.

Darija NLP: Text Processing And Morphological Stemming - Results