Hello there and welcome back to this new post on how to train a chatter-bot with your Messenger chat. Of course, we will be using Python as usual. I am using Python version 3.8.1. To get started, please make sure you first download your Messenger Messages log from your Facebook account.

Go to Settings --> Your Facebook Information --> Download Your Information.

You will see a list of all the fields you could select to download. They are selected by default, so just go and deselect them all, and only select the Messages box.

Now in the Request Copy section, choose a format of JSON and a quality of medium or low since we won't be using the media included such as images. The time by which your downloadable copy will be ready is proportional to how much data is has, but it won't take long.

When your copy is ready and downloaded, you should see a zip folder. Extract it, you will one folder named messages. You'll get the following folders inside it:

Go and delete all the folders except for inbox. It's the one that has all your chats with every user. Now, inside the inbox folder, you'll see as many folders as many users you have interacted with * 2. The * 2 because each user is given two main folders, one containing the JSON chat files, and the other containing the media such photos, videos, etc. As I mentioned before, we will avoid the media ones. If you open any user folder, you should see something like:

Where each JSON file contains your chats. How many files is also proportional to how much you have exchanged. Now let us write some code that we will only give the source directory for the chat, inbox. Then this code shall look in every folder and grab only the JSON files in a reverse order for each user. Why reverse? As in the image the JSON file message_1.json is not the start of the chat, it's message_8.json. message_1.json is the latest messages. You could open up a file and check that up. If you do so, you'll also notice that line 1 is not the start of the conversation, but the last lines. For example message_8.json has 10000 lines, your first messages are in the lines 9000 or and the latest ones are at the beginning of the file. We understand now that we not only need to reversed the order of the files, but also of the content of the files since we don't wish for our data to be in backward. E.G :

A: I'm fine, thanks

B: How are you?

Before going to extract the data, we shall first install our ChatBot. In your terminal:

pip install chatterbot

After the installation is complete, let us begin by importing our ChatBot

from chatterbot import ChatBot
chatbot = ChatBot("LangandCodeBot")

We have now imported our ChatBot and created an instance we named LangandCodeBot. To train our ChatBot, we have a few trainers:

chatterbot.trainers.ListTrainer
chatterbot.trainers.ChatterBotCorpusTrainer
chatterbot.trainers.UbuntuCorpusTrainer

We will be using the ListTrainer to train our ChatBot, which expected a list of data. We need to render our data as such.

Don't freak out, but here is the complete code at first glance, from extracting data to feeding it to the Trainer:

import json
import os
from chatterbot import ChatBot
from chatterbot.trainers import ListTrainer


basedir = os.path.abspath(os.path.dirname(__file__))


class LangandCodeBot(object):

    def __init__(self, bot_name, basedir=basedir):
        self.slash = '/'
        self.basedir = basedir
        self.path_to_messenger = basedir+self.slash+'inbox'
        self.sub_directories = [ f.name for f in os.scandir(self.path_to_messenger) if f.is_dir() ]

        self.bot = ChatBot(bot_name,
            logic_adapters=[
                'chatterbot.logic.MathematicalEvaluation',
                {'import_path': 'chatterbot.logic.BestMatch',
                'default_response': 'I am sorry, but I do not understand.',
                'maximum_similarity_threshold': 0.90},
                'chatterbot.logic.BestMatch'
            ],
            preprocessors=[
                'chatterbot.preprocessors.clean_whitespace',
                'chatterbot.preprocessors.convert_to_ascii'
                ]
            )

        self.list_trainer = ListTrainer(self.bot)


    def get_messenger_files(self, directory):
        return sorted([f.name for f in os.scandir(
            self.path_to_messenger+self.slash+directory) if f.is_file() ])


    def load_json(self, path):
        with open(path, 'r') as f:
            return json.load(f)


    def parse_json(self, file_obj):
        previous_participant = None
        previous_message = None
        chat = []
        for m in reversed(file_obj['messages']):
            if 'content' in m:
                if previous_participant == m['sender_name'] and previous_message != None:
                    previous_message = previous_message +". "+ m['content']
                else:
                    if previous_message != None:
                        chat.append(previous_message)
                    previous_participant = m['sender_name']
                    previous_message = m['content']
            pass
        return chat


    def train_messenger_corpus(self):
        for d in self.sub_directories:
            print("Loading directory", d)
            for f in reversed(self.get_messenger_files(d)):
                print("Processing file", f)
                chat = self.parse_json(
                    self.load_json(self.path_to_messenger+self.slash+d+self.slash+f
                    )
                )
                print("Finished processing file", f)
                trainer = self.list_trainer
                print("Training with dataset {} ".format(f))
                trainer.train(chat)
                print("Trained with dataset {} ".format(f))


    def console_run_bot(self):

        print('Type something to begin...')

        # The following loop will execute each time the user enters input
        while True:
            try:
                user_input = input()

                bot_response = self.bot.get_response(user_input)

                print(bot_response)

            # Press ctrl-c or ctrl-d on the keyboard to exit
            except (KeyboardInterrupt, EOFError, SystemExit):
                break


bot = LangandCodeBot("Adam")
bot.train_messenger_corpus()
bot.console_run_bot()

Take a look at the code and see what it does. Let's start with the first things first, we load the essential modules, we then define a basedir of our script as our inbox directory is at the same level as our script, so we avoid hard-coding the path. We then create a class named LangandCodeBot to host all of our code. It is not essential, but I like keep things organized. This class takes in one argument, the ChatBot name, we named it here Adam. In our __init__ method, we define a few elements such as the full path to the messages, the sub-directories, and the bot itself. To learn more about the Chatterbot itself, please refer to their documentation the get_messenger_files gets all the JSON files of a given directory, we use the built-in method sorted to sort them as they can be loaded unordered. We use the method load_json to...yes load JSON, of course! Then we define two more boys, parse_json and train_messenger_corpus. The first one takes in a file object that has been loaded using the load_json method. Our parse_json takes care of a little more things for us. To illustrate them we must recall that our Trainer expects a list of elements, where the second element is the reply to the first one. But we all have some bad habit such as:

Adam: Hey

Adam: How are you doing today?

Adam: Still up for that trip?

Lana: Oh hey!

Lana: Yeah, of course.

I bet you can already see what's wrong with this: We get a list of concurrent messages but the same user. To fix that, I added a little validation where I check if the previous message is sent by the same sender of the current one, if so I concatenate them together, if else, go as usual. This is not the best method, but it is rather better than the mess in Adam and Lana's chat. So we would end up we something like this:

Adam: Hey. How are you doing today?. Still up for that trip?

Lana: Oh hey!. Yeah, of course.

A little better? If you come up with a better solution, please let me know so.

Now to our main method: train_messenger_corpus which makes use of all the aforementioned methods and attributes. with a use It loops through the list of sub-directories, and for each directory it grabs all its files in a reversed order, and then it parses each file and returns one list of all the chat logs of a user, then this chat is feed to our ListTrainer and the process continues until we have looped through every file and directory. This is all done by running:

bot.train_messenger_corpus() # to train
bot.console_run_bot() # to start the bot when it's trained.

When you run the code above, you should see something like:

[nltk_data] Downloading package averaged_perceptron_tagger to
[nltk_data]     /home/fsociety/nltk_data...
[nltk_data]   Package averaged_perceptron_tagger is already up-to-
[nltk_data]       date!
[nltk_data] Downloading package stopwords to
[nltk_data]     /home/fsociety/nltk_data...
[nltk_data]   Package stopwords is already up-to-date!
Loading directory xxxxxxx_2TWpSnbSRg
Loading directory xxxxxxxb_y0vqn2uxeg
Processing file message_1.json
Finished processing file message_1.json
Training with dataset message_1.json
List Trainer: [####################] 100%
Trained with dataset message_1.json
Loading directory eyuk_w9ycg
Processing file message_2.json
Finished processing file message_2.json
Training with dataset message_2.json
List Trainer: [####################] 100%
Trained with dataset message_2.json
Processing file message_1.json
Finished processing file message_1.json
Training with dataset message_1.json
List Trainer: [####################] 100%
Trained with dataset message_1.json
Loading directory xxxxxxxouni_mwdekbsyew
Processing file message_1.json
Finished processing file message_1.json
Training with dataset message_1.json
List Trainer: [####################] 100%
Trained with dataset message_1.json
Loading directory gxxxxxxxiamine_rtlnra3gpa
Processing file message_1.json
Finished processing file message_1.json
Training with dataset message_1.json
Trained with dataset message_1.json
Loading directory facebookuser_wnmxhddijg
Processing file message_1.json
Finished processing file message_1.json
Training with dataset message_1.json
Trained with dataset message_1.json
Loading directory Tomxxxxxxx_GmXfVCCQUA
Loading directory tomflowers_o0rpi6w2ha
Processing file message_1.json
Finished processing file message_1.json
Training with dataset message_1.json
Trained with dataset message_1.json
Loading directory error_9na1aao0ig
Processing file message_1.json
Finished processing file message_1.json
Training with dataset message_1.json
List Trainer: [####################] 100%
Trained with dataset message_1.json
Loading directory xxxxxha_-56typ4kqq
Processing file message_1.json
Finished processing file message_1.json
Training with dataset message_1.json
List Trainer: [####################] 100%
Trained with dataset message_1.json
Type something to begin...

To keep using the ChatBot from the script, just comment out the train method so you don't re-train your model each time your run it.

#bot.train_messenger_corpus() # to train
bot.console_run_bot() # to start the bot when it's trained.

Here is the final result:

I hope this has been useful to you, If so, please share and recommend. The source code in this tutorial can be found in the GitHub repository . Keep this code with you, try to improve it if possible. In the coming tutorials, we will be building on the same class LangandCodeBot and we will add more data sources and use other trainers. When we have trained our model with different sources of data, I will a tutorial on how to deploy in into production using Flask as the back-end and of course good ol' JavaScript at the front-end.

Thank you

Train Chatterbot With Your Messenger Chat

Majdi Mahfoud