[This article and all those to come (Under this specific category) are all extractions from my university BA research paper. Changes, updates and improvements are to be made when necessary. Parts I think won't contribute much or are becoming obsolete shall also be ignored. New suggestions shall be presented eventually as an update. All rights reserved for the owner.]

[Part 4]

Note: Compared to the original paper, various parts are ignored for multiple reasons, most of which is due to major updates have taken place and the issues found with the older methods. (For example, the interface was made with Python Tkinter GUI framework. And How the results were preprocessed, proceeded and presented. Bugs fixed and imporvements in processing speed) We shall, however, discuss the updates in the upcoming articles.

4. English NLP Vs. Arabic and Moroccan Arabic NLP

Natural Language Processing presents some profound challenge:

It is hard from the standpoint of the child, who must spend many years

acquiring a language, it is hard for the adult language learner, it is hard for the scientist

who attempts to model the relevant phenomena, and it is hard for the engineer who

attempts to build systems that deal with natural language input or output. These tasks

are so hard that Turing could rightly make fluent conversation in natural language the

centerpiece of his test for intelligence. (Mathematical Linguistics. 2010: 248)

4.1. English language processing

Unlike Arabic and its verities, natural language processing of English is nothing new, it

begun teens of years ago. Fleeing with his family from the Nazi Regime towards America,

Joseph Weizenbaum became a professor of Computer Science (CS) at MIT in 1964. It was in

that same year and place that Weizenbaum created something extraordinary, ELIZA. "ELIZA

is a program designed to establish natural language conversation with a patient by emulating a

Rogerian therapist" (www.informationweek.com)

Although Weizenbaum is considered one of the fathers of modern artificial intelligence, he

also became one of its leading critics. He described his whys in his influential book

"Computer Power and Human Reason", 1976. (Joseph Weiznbaum. 1976). Although ELIZA

responded in a human-like way sometimes, it had no real NLP behind it, just patterns and

conditionals, but it has given the rise for real NLP to come later.

The reason we chose to talk of English NLP is to compare the difficulties regarding

levels of language of both English and Moroccan Arabic. English language processing

regarding encoding proves no difficulties, all the twenty six letters are part of the ASCII

encoding, a code for information exchange between computers, it uses seven binary digits to

represent each character and is used in microcomputers (Wordweb dictionary). Being this

code implemented in most microcomputers, English remains safe concerning encoding. In

addition, unlike Arabic script, English have no diacritics regarding the manner of writing, and

is written left-to-right, which is the default of most of systems.

Myriad numbers of books regarding English NLP has been written. Various methods

has implemented to tackle different problems, from the hardcoded rule-based approaches to

the very core of Deep Learning (DL) using Artificial Neural Networks (ANN). Having said

that, let us look at what Arabic and Moroccan Arabic NLP is like.

4.2. Arabic and Moroccan Arabic NLP

The Arabic language is both challenging and interesting, it is interesting because of its

history (Kees 1997). It is also challenging because of its complex linguistic structure (Attia,

2008).

Almost all of the research and tools that have been developed and deployed to handle

written texts regarding Arabic are written in and for MSA. Since there are major significant

variations in grammar, syntax and expressions from one variety to another and even between

the dialects themselves. The results of any attempts to apply rules and tools written for MSA

to variants such as Egyptian or Levantine are far from accurate. Moreover, the resources for

developing NLP applications for dialects and variants are very rare, mainly grammars and

dictionaries (Farghaly and Shaalan. 2009)

The Arabic language demonstrates some serious challenges regarding the creation of

NLP applications that process either text, speech, or both. One of the challenges that raise

when trying to process the Arabic language and its variants is the Arabic script itself. Arabic

is a phonetic language, written from right to left, and since it does not have dedicated letters

to represent the vowels, it uses diacritics. But in the contemporary writings of Arabic,

diacritics have been disappearing . (Farghaly and Shaalan. 2009)

Regarding the phonological aspect of the standard Arabic, it has 28 consonants, 3 short

vowels, 3 long vowels, and 2 diphthongs (Diab and Habash. 2014). To make things far more

complicated, Arabic letters have different shapes depending on their position in the word,

mainly, the initial, the median, and the final position. (Farghaly and Shaalan. 2009)

An example of the different shapes of the same letter, for instance let's take the letter ( "‫"ه‬,

"h") which is the standalone form of the letter. Here is the initial, median and final form of the

same letter:

Initial: "ههه"
Median: "ههه"
Final: "ههه"

If NLP in general is so hard and languages such as Arabic and its variants are even more

harder, why do we need to process languages after all, let us say Arabic?

Arabic Natural Language Processing (ANLP) applications developed in the Arab world have

some specific objectives and are approached using different methods including rule-based and

machine-learning approaches. Here are some goals of ANLP. We mention two objectives of

ANLP applications Farghaly and Shaalan stated:

Transfer of knowledge to the Arab World: Most scientific research that is done is published in the English language, this research is not accessible to Arabs with little to no knowledge or competence in English
Modernize Arabic linguistics: Creating NLP applications requires some precise grammars unlike the traditional grammar used in ANLP applications.

There are of course many reasons why we would want to have a working NLP system

for Arabic, Moroccan Arabic and/or any language in the world: It makes life much easier.

Now, let us get back to coding and let us modify and use our script 1.1 so we could find

the Arabic definite article "‫"ال‬. We also create a file with some Moroccan Arabic text:

العلم هو واحد الحاجة اللي زوينة بزاف

A transliteration of the above MA text using the Buckwalter Scheme (We shall adopt it

throughout the whole research):

Transliteration : "AlElm hw wAHd AlHajp zwynp bzAf"

Translation:" Science is something beautiful"

This is our script 1.2:

# -*- coding: utf-8 -*-
text = open("my_text2.txt", "r")
for line in text:
    for word in line.split():
        if word.startswith("‫)"ال‬:
            print("Found a match:", word)

Script 1.1 Arabic determiner "‫"ال‬ finder.

When we run the above script, we expect everything to run just fine and get all the words

which start with the determiner "‫"ال‬, but what we get is the following:

Fig. 1.2. The output of our script 1.1

The above screenshot is the output of our little script, it displays an error regarding encoding,

what we need to do is to first encode our input data which in this case is our Arabic text.

We wouldn't have this error if our input is written using the English alphabet, but we are

dealing with Arabic and its variants, therefore we need to handle that somehow. The

following few lines of code prints the Arabic text without specifying an encoding. Here the

interpreter uses the default universal encoding which is ASCII and which supports only a

small range of characters (128) of which Arabic characters are not part of. Afterwards, we

print the Arabic text encoded with UTF-8.

Here is the code: (s = our Arabic text)

s = " ‫بزاف‬ ‫زوينة‬ ‫الحاجة‬ ‫واحد‬ ‫هو‬ ‫العلم‬ "
print("--> Moroccan Arabic text:\n ", s)
encoded = s.encode('utf-8')
print("--> Moroccan Arabic text after encoding with UTF-8:\n ", encoded)
decoded = encoded.decode('utf-8')
print("--> Moroccan Arabic text decoded with UTF-8:\n ", decoded)

Script 1.2. Encoding the Arabic Alphabet with UTF-8

And here is the output:

Fig. 1.3. The output of script 1.2

The above image demonstrates how the Arabic input is treated by the prompt command with

the default encoding (ASCII), and ,afterwards, encoded with a proper encoding (UTF-8*).

This is fine for machines, still hard to be read by humans.

We could keep demonstrating all kind of challenges and/errors that would be triggered once

our scripts start including some non-ASCII characters such as those of Arabic.

4.3. Moroccan Arabic is hard to process

To be honest, it's not almost ,genuinely, all about the Moroccan Arabic, it's more about

the Arabic itself, mostly the Arabic orthography, the complex morphology and syntax of both

the Arabic language and the Berber language which make up most of MA.

As we shall focus more on the morphology of MA, let us see some samples:

‫ممشيتوش mɑmʃItuʃ

*not have gone-I you not

Translation: "You have not gone" (plural)

This one word include many morphological rules that construct such tokens, or words: The

negation circumfix "‫"م‬ and "‫"ش‬ (El Haloui. 2014), the plural subject for the 2nd person "‫"و‬,

and, of course, the stem "‫"مشيت‬, with the root word "‫"مشى‬ (gone, went = the 3rd person

singular masculine).

Now if we observe the English equivalent, it is rather a complete sentence indented with

a space between each word, or lexical item. We refer to a word or set of words as a token or

set of tokens. In English we could easily tokenize (split into chunks of tokens) tokens based

only on the white space indentation. In contrast, we cannot achieve the same results when it

comes to languages such as Arabic and its variants since one single word could be made of

multiple affixes and/or clitics that could inflect tense, number, gender, case and others (Aqel,

Alwadei, Dahab. 2015). Therefore, we already know traditional tokenization is not enough,

hence we shall come up with another technique to solve the problem.

Now, we know we can write a lexicon dictionary with entries such the negation item

"not", or the personal pronoun "you" or the past participle of verb "go", "gone". We could

loop through each word and look up its meaning in our dictionary, but our dictionary should

be of a limited size, of a logical, practical size. We also know that creativity is one of natural

languages' properties, thus we could have an infinite number of sentences, so we know we

cannot construct a dictionary of sentences. This is to some degree similar to our case. We

cannot have a dictionary with every instance of the same stem or lemma, at least not one of

word inflections. When you look up a dictionary with a word such as "friends", most digital

dictionaries would return a valid definition of the word "friend", but how? Does this mean the

developers had to include the same definition for the word "friend" and "friends"? Sure not.

Many dictionaries, apart from smart backend-engineering techniques designing the database,

use stemmers* or lemmatizers* to reduce word forms to a common stem or lemma. Imagine

we wanted to create a simple dictionary of Moroccan Arabic lexis, we already know we

cannot have all the different forms of each word, at least not the inflectional forms.

For example our word ( " ‫ش‬ ‫تو‬ ‫كتب‬ ‫م‬ " , "you haven't written [plural]"):

Inflectional forms examples:

...‫مكتبناش‬ ,‫مكتبتيش‬ ‫مكتباتش‬,‫مكتبوش‬

...‫كتبنا‬ ,‫كتبتي‬ ,‫كتبات‬ ,‫كتبوا‬

Derivational forms examples:

... (library) ‫مكتبة‬ , (office) ‫مكتب‬ , (book) ‫كتاب‬

We know we could have an input where it is a defined noun, for example: "‫"فالمكتبة‬.

Even if the word "‫"مكتبة‬ (= library) exists in our database and we tried to check " ‫مكتبة‬ ‫ال‬ ‫ف‬ " (=in

the library) against the database, though one word, it would not be found for the simple raison

that " ‫مكتبة‬ ‫ال‬ ‫ف‬ " is not as " ‫مكتبة‬ ‫ال‬ " and is not as "‫"مكتبة‬ although it is written as one word, and we

already know we cannot have every instance of every word, thus stemming comes in handy.

We could use some programming tricks where we check if our word is a subset of an existing

word in the database, for example if our word was "‫"كتب‬ and we applied this technique, it

would match any occurrence of the word given in our database, including, for instance:

" ‫كتب‬ ‫م‬ ", ‫ة‬ ‫كتب‬ ‫م‬ , and "‫"كتب‬ itself. This is both unpractical and useless. Therefore stemming proves

to be useful.

Stemming is the process by which we remove affixes if there are any. To do stemming is

to have knowledge of word roots, the different affixes and the patterns a language uses to

form both inflected and/or derived words from the same root(s), thus to have knowledge of

the morphological level of the language under investigation.

5. Morphology

"The study of internal structure of words, and of the rules by which words are formed is

morphology" (Fromkin, Rodman, Hyams. 2014). The basic concept of morphology is a

morpheme which is the smallest expressive unit of a language (Aqel, Alwadei, Dahab. 2015).

For instance the MA word (fdArnA, "‫"فدارنا‬, "in/at our house" ) is made up of three

morphemes or affixes: (f, "‫"ف‬, "in/at"), (dAr, "‫"دار‬, "house") and (nA, "‫"نا‬, "ours").

Arabic and Moroccan Arabic are agglutinative languages.

Unlike English and most languages, Arabic has a complex word structure. An

agglutinative language constructs complex words that often contain affixes and clitics

representing various parts of speech. for example, a verb may embed within itself its

subject and object as well as other clitics signifying tense, gender, person, number and

voice.

(Farghaly, Shaalan. 2009)

An agglutinative is defined in the Concise English Dictionary simply as: " forming derivative

or compound words by putting together constituents each of which expresses a single definite

meaning"

Moha Ennaji et al (2004) state that MA is not always consonantal, nor is it always

triconsonantal as there exists biconsonantal roots also. This will prove to be rather another

complication regarding the building of a system that could look for affixes based on their

length. And Just as we demonstrated earlier, Moha Ennaji also exemplified that affixation in

MA plays a crucial role in many derivational and inflectional processes:

MA has a typically Semitic morphology in which morphological

categories are represented not only by prefixes and suffixes, but also by non-

concatenative structures built on non-concatenative stems, each of which is being

composed of a radical made up of non-syllabics (consonants and semivowels), vowel

melody and prosody (i.e. suprasegmental features).

(Moha Ennaji, et al, 2004)

Now that we have grasped what NLP is and how we could use a computer hand in hand

with a programming and the knowledge of a language or a level of a language such as the

morphology of Moroccan, we surly must put that knowledge into some practical use, but we

definitely need a methodology.

6. Algorithm flowchart [removed]

7. Data collection [removed]

Darija NLP: Text Processing And Morphological Stemming - Literature Review - Continue