Add R-to-Python syscall workflow for sentiment analysis training and prediction in a Kaiaulu Notebook

[geraldmjhuff]

This proposal outlines a workflow where users never leave the Kaiaulu R Notebook. Instead, a standalone Python script (for both training and prediction) is exposed to Kaiaulu via tools.yml and invoked from R using system calls.

This shifts the entire analysis workflow (data download, parsing, training, prediction, and results) into a single Kaiaulu Notebook.

Left side: Kaiaulu

R Scripts (R/sentiment.R)
pysenti_train_model(pysenti_path,reply_dt,model_save_path, model) Calls a Python script (train_or_predict.py) that passes in a data table to train a sentiment model.
pysenti_predict(pysenti_path, reply_dt, model_save_path, model): Calls the same Python script to that passes in a data table to predict sentiment on new data.
get_pysenti_path("pysenti", "../conf/tools.yml"): Fetches the path to the Python script based on a configuration (tools.yml).

Configuration (tools.yml)
Defines the path to the Python script for training/prediction:

pysenti: ~/path/to/train_or_predict.py

Vignette (vignettes/sentiment_analysis.Rmd)
Download data
Parse
Train or predict using pysenti_train_model() or pysenti_predict()
Load table with results

Right side: pysenti

Script (exec/train_or_predict.py)
Handles train and predict commands.
Receives the model path and the parsed data table from R, then executes the corresponding Python functions in:

API functions (api/model.py)
train_model(This function already exists):
train_model(parsed_dt, model_saved_path, model_select=0): Takes in a data table (w/ columns "Text" and "Polarity" where Polarity is assumed to have already correct values to train the models) Trains a model and saves it to a path "model_saved_path" and returns that path.

We will add this function:
predict_sentiment(), this is similar to the existing test_model(). test_model() predicts sentiment, compares it to the flat true labels, then combines both labels into a data table and returns it. Our new function predict_sentiment() only predicts sentiment, then returns a data table with updated sentiment values.
predict_sentiment(parsed_dt, model_saved_path, model_select=0): Takes in a data table (w/ columns "Text" and "Polarity") then applies predicted sentiment values and returns that table.

[geraldmjhuff]

@carlosparadis Please let me know if this architecture needs revisioning at any point and I will reflect those revisions on both the diagram and the text that describes the diagram in the top comment.

[carlosparadis]

Left side

Let's presume I will change the repo name after this project is over to pysenti and name the API in Kaiaulu accordingly since script is too generic and will cause issues on the long run with other tools. It is OK that this is still in R/sentiment.R in so in the future we can still add sentiment related code here. As such:

train_model(script_path, parsed_dt, model_saved_path, model_select=0):

rename to

• pysenti_train_model(pysenti_path,reply_dt,model_save_path)

I am not sure what model_select =0 means. Can you clarify? If this means the model selection, then please use the model name and expect a vector that checks for the value typed exists (e.g. bert, etc). Do not assume defaults. Note I removed the "d" from saved since this function is saving.

predict_model(script_path, reply_dt, model_saved_path, model_select=0):

likewise:

• pysenti_predict

Note I removed model from the name since the model is predicting data, not itself. Here I left past sentence since it presumes a model exists somewhere.

• Lastly get_pysenti_path.

It is OK the functions to predict and train within them rename and drop columns not used on the Notebook side (originally I said to pass that to the Python notebook, but in arch 3 it is best it is located in these R functions).

Right side

Parameter is OK to be parsed_dt since it was modified within the R function.

Project Config

• I believe we need to add the save_path in the config rather than the notebook. Can you propose a sentiment section to the Notebook? Have a look how others are created. Let's specify there the model path and the prediction path.

Tools.yml

• Call the attribute pysenti instead of sentiment_analysis.

Diagram

• The order of the second set of arrows is incorrect. The syscall goes first again for predict, and then it returns the prediction.

Thanks!

[geraldmjhuff]

@carlosparadis
Regarding the model_select=0, it is used to loop through running all models on a data set.

Snippet of the notebook Train.ipynb:

# Train each model and save the trained model files
for i, model_name in enumerate(MODEL_NAMES):
    model_save_path = f"{root}/{model_name}_model"
    print(f"Training {model_name} model...")
    train_model(train_df_final, model_save_path, model_select=i)

This is how its used in train_model() in api/model.py:

MODEL_NAMES = ['bert', 'xlnet', 'roberta', 'albert']

MODELS = [(BertForSequenceClassification,BertTokenizer,'bert-base-cased'),
          (XLNetForSequenceClassification, XLNetTokenizer,'xlnet-base-cased'),
          (RobertaForSequenceClassification, RobertaTokenizer,'roberta-base'),
          (AlbertForSequenceClassification, AlbertTokenizer,'albert-base-v1')
        ]


def train_model(train_df, model_save_path, model_select=0):

    seed_torch(42)

    cur_model = MODELS[model_select]
    m_name = MODEL_NAMES[model_select]

We see that the function itself only goes through one model at a time. The parameter "model_select=0" exists because it serves as the iteration, which is why in the for loop above, model_select=i to loop through all the models. Assuming we are taking out the "model_select=0" parameter, and we are shown that all models are supposed to be ran on a data set, should I have the functionality of creating a for loop and running all models inside train_model() opposed to the current "for loop outside the function to run all models" circumstance?

[carlosparadis]

Oh.. I didn't realize we have multiple underneath. That's..not good. It made sense for test_model since you need to predict multiple to compare and evaluate, but predict should not fire all 4.

So what you want is to refactor this into train_model_bert, train_model_xlnet, etc. And the model_select=0 goes away altogether.

Your Python script in turn needs an additional flag for the predict option, which will be bert, xlnet, roberta, or albert. It needs a switch case based on that.

Then your R function for predict requires a new parameter which is also bert, xlnet, roberta, or albert. Have the R function check which.

And that's it. The prediction column would be 4 columns if you fired all 4, you just want one. Be careful however that the test_model on Python side remains there and has the predict 4 as it is used to since someone using the Python side will want to evaluate them before choosing the best to do predictions.

Out of curiosity I will also note: IF we wanted to keep all 4, the proper way to do this would take a list parameter that specifies the name of the models. The loop happens within. The function parameter is the user interface, it should hide the implementation details and make it easy for the user. Having the use guess the index of which model is which, or map from the docs instead of just typing the name is unnecessarily confusing and prone to error!

[geraldmjhuff]

@carlosparadis
I've created the appropriate train functions (Not committed yet):

train_model_bert(train_df, model_save_path)
train_model_xlnet(train_df, model_save_path)
train_model_roberta(train_df, model_save_path)
train_model_albert(train_df, model_save_path)

and the prediction functions (Not committed yet):

predict_bert(test_df, model_save_path)
predict_xlnet(test_df, model_save_path)
predict_roberta(test_df, model_save_path)
predict_albert(test_df, model_save_path)

On a side note, considering that the original train_model() (and the 4 newly added test_model_*) takes in a data table that already has correct "Polarity" values in order to train the models, how would this be implemented in Kaiaulu? Namely, if we are calling
pysenti_train_model(pysenti_path,reply_dt,model_save_path, model)
It is assumed the reply_dt would already have correct "Polarity" values for it to work. train_model() as of now relied on a .csv that has already correct "Polarity" values.

Perhaps it would not make sense to send in a reply_dt when calling pysenti_train_model(pysenti_path,reply_dt,model_save_path, model) because (once we implement a "Polarity" column to the reply_dt) it would contain all zero's, therefore the model can't train on it. How should this be approached? Should we hardcode .csv's used for training in the python repo, then drop reply_dt so its now pysenti_train_model(pysenti_path, model_save_path, model) , then when syscalling train_or_predict.py, train_or_predict.py calls train_model_*(train_dt, model_save_path) where train_dt argument is basically the .csv that the original train_model() uses?

[carlosparadis]

I'm confused. I thought we were creating predict_bert, predict_xlnet, etc? Where does test comes from?

In fact, for future proofing I recommend calling them predict_sentiment_bert, etc.

[carlosparadis]

I am also confused about your confusion (!).

Training a model requires a person to manually label it, there is no way around it. What you should do in your Notebook is use fread() on the labeled data .csv that is on Drive. I.e. save to your computer (don't add to the config), fread it to the notebook and explain in the training section the first step is to manually label sentiment comments so a model can be trained.

Then you can further say you can use that data to train any of the 4 models using the 4 train functions in Kaiaulu. You do not pass a reply_dt to the train functions, but rather the ground truth, which should have a text and sentiment label.

The next section proceeds to explain now you have a trained model, you can predict sentiments of any new data in Kaiaulu. You can then call the predict functions to get labels. You should show tables along the way to build intuition for the user.

And yes, it is OK that you train in one dataset and predict in another (not ideal, but it is what we have).

So what happens with test? Let me put this in a separate comment so you don't get confused with what you should do, with what ideally we would like to do but can't. This comment is about what you should do.

If you fulfill the requirements of this comment, you will have in essence created a Notebook that allows users to download new data, parse it into a table, format in your notebook, and call the Python script to predict sentiment to it.

Your notebook will also allow people to train models to be used for prediction, using existing labeled data, which is also great, because there is more labeled data out there rather than just this .csv. So we could easily swap the files, clean up column names, and voila we have new predictions using the same models and other datasets.

[carlosparadis]

So what would be the ideal test_model () we would make?

For that we would want a notebook that downloads data, parses it, and then subsequently it is explained how it was manually labeled. That is, we would have the pipeline to the table that was manually annotated. There is no way to reconstruct this for the current .csv we have, since the Notebook for that was lost to time.

For evaluation, what I recommend is that you find a library online or do by hand and just do it on the R Notebook, don't add to the Kaiaulu API, because it is the standard way to evaluate any model. Add to the notebook a section that says "Model Evaluation". You want to take that ground truth labeled .csv, and then split it into three ways: Train, Dev, and Test. Give 60% to train, 15% for dev, and 15% for test. Be careful you split this chronologically: Otherwise you re using the future to train the past (!).

Then the train dataset goes in your train function. You can then compare the values you get from the predict table versus the ground truth table. Look up a function for creating a "confusion matrix". This helps understand false positives, false negatives etc.

Here is where you would alter the model parameters based on the performance. You can leave this bank for now.

Then supposedly you changed the parameters of each model to your hearts content and now the performance is good. Then and only then you use the final trained model to compare against the test set. The results of the test set is what a user would report in a paper.

As you can see, the test_model part can be entirely done in R. You are just comparing predict versus manually labeled. It is beyond the scope of Kaiaulu to go too fancy on the metrics.

Hope this makes more sense.

Edit: You want to compare train dataset prediction vs train ground truth ; train dataset prediction vs dev ; and only after all is done and ready to report train dataset prediction vs test set.

This is more ML 101 than Kaiaulu helping in data pipeline, so I think it is ok we just explain that in the notebook. The more important thing here is that we offer the capability to download, parse, and predict sentiment. Having the confusion matrix and the train/dev/test would be helpful for guidance for other users, but not required.

[geraldmjhuff]

@carlosparadis
So to clarify the beginning flow; we download data, we parse into a reply table, apply filters, so now its a filtered reply table. Now when i call train_model_(), we actually use the already existing .csv (w/ ground truths) in the google drive to train it? then once we call predict_model_() after, this is where pass in the filtered reply table?

Or is it instead after we parse and filter into a filter reply table, I manually put correct values myself in the "Polarity" column so that this filter reply table is passed to both train and predict functions?

I have not included in this comment but also aware a dataset has to be split 3 different ways, my question is moreover simply does reply_dt (w/ manually added ground truth lables) go in both train and predict functions, or does the already existing .csv (w/ ground truths) go in the train, and then reply_dt in the predict.

[carlosparadis]

The first section of the Notebook, let's call it "Preparing Dataset" for now does this:

So to clarify the beginning flow; we download data, we parse into a reply table, apply filters, so now its a filtered reply table.

The second section is "Training Models".

Here you will prepare code that allows you to do this:

Or is it instead after we parse and filter into a filter reply table, I manually put correct values myself in the "Polarity

So to say, you will explain the table created in the prior section needs now to be manually annotated for the sentiment scores OR that the user can use labeled data already published (I will get you some references, or maybe even some tables to add here, please remind me next meeting if I don't by then). At this point, you can explain that you are using one curated dataset and cite Haotian's thesis.

In short: Either the reader comes up with their table, or the reader uses one already published! You here won't be manually labeling anything :)

Now when i call train_model_(), we actually use the already existing .csv (w/ ground truths) in the google drive to train it?

The above is correct, but please make sure your narrative doesn't make it sounds the reader has to use this table. Make it clear it can be anything so long they respect the table format we want. At the scope of the notebook, I recommend you abide by the reply_dt format. So, maybe rename the text | polarity column to match in name.

Prediction Section

I have not included in this comment but also aware a dataset has to be split 3 different ways, my question is moreover simply does reply_dt (w/ manually added ground truth lables) go in both train and predict functions, or does the already existing .csv (w/ ground truths) go in the train, and then reply_dt in the predict.

You do not have a reply_dt with manually added ground truth labels. So you have two options that remain (you forgot one):

does the already existing .csv (w/ ground truths) go in the train, and then reply_dt in the predict.

Alternatively, you can have the existing labeled dataset go on both train and predict (but you need to do the 3 way split before then).

[geraldmjhuff]

@carlosparadis
I have created 3 get functions in Kaiaulu in R/config.R :

get_pysenti_path <- function(config_file) {

  pysenti_path <- config_file[["pysenti"]]

  if (is.null(pysenti_path)) {
    warning("Attribute does not exist.")
  }

  return(pysenti_path)
}

get_sentiment_model_folder_path <- function(config_file) {

  model_folder_path <- config_file[["tool"]][["sentiment"]][["model"]]

  if (is.null(model_folder_path)) {
    warning("Attribute does not exist.")
  }

  return(model_folder_path)
}

get_sentiment_prediction_folder_path <- function(config_file) {

  prediction_folder_path <- config_file[["tool"]][["sentiment"]][["prediction"]]

  if (is.null(prediction_folder_path)) {
    warning("Attribute does not exist.")
  }

  return(prediction_folder_path)
}

get_pysenti_path() is used to get the path to exec/train_or_predict.py in the pysenti key in tools.yml (path can be subjected to change) This is assuming we are in the path kaiaulu\vignettes when calling get_pysenti_path() within the notebook, and kaiaulu and sentiment_classifier are within the same folder:
pysenti: ..\..\sentiment_classifier\exec\train_or_predict.py

Given this comment:

I believe we need to add the save_path in the config rather than the notebook. Can you propose a sentiment section to the Notebook? Have a look how others are created. Let's specify there the model path and the prediction path.

I've created get_sentiment_model_folder_path() where the notebook workflow use of it is:

conf <- parse_config("kaiaulu.yml")
model_path <- get_sentiment_model_folder_path(conf)
pysenti_train_model(pysenti_path, reply_dt, model_path, model)
pysenti_predict(pysenti_path, reply_dt, model_path, model)

I've also been told to create a prediction path, so I've created get_sentiment_prediction_folder_path() but unsure of its use case as of yet. I'm assuming this path stores the predicted table that pysenti_predict() returns. So would the notebook workflow be that after pysenti_predict() returns a table, that table is stored in the path that get_sentiment_prediction_folder_path() uses, and after subsequent use of calling the predict function, more tables are added to the folder?

get_sentiment_model_folder_path() and get_sentiment_prediction_folder_path() in kaiaulu.yml:

tool:
  sentiment:
    model: ../../kaiaulu/sentiment/sentiment_classifier/model/
    prediction: ../../kaiaulu/sentiment/sentiment_classifier/prediction/

These folder paths follow the structure of already existing folder paths such as:

issue_tracker:
  jira:
    project_key_1:
      # Obtained from the project's JIRA URL
      domain: https://sailuh.atlassian.net
      project_key: SAILUH
      # Download using `download_jira_data.Rmd`
      issues: ../../rawdata/kaiaulu/jira/sailuh/issues/
      issue_comments: ../../rawdata/kaiaulu/jira/sailuh/issue_comments/
  github:
    project_key_1:
      # Obtained from the project's GitHub URL
      owner: sailuh
      repo: kaiaulu
      # Download using `download_github_comments.Rmd`
      issue_or_pr_comment: ../../rawdata/kaiaulu/github/sailuh_kaiaulu/issue_or_pr_comment/
      issue: ../../rawdata/kaiaulu/github/sailuh_kaiaulu/issue/
      issue_search: ../../rawdata/kaiaulu/github/sailuh_kaiaulu/issue_search/
      issue_event: ../../rawdata/kaiaulu/github/sailuh_kaiaulu/issue_event/
      pull_request: ../../rawdata/kaiaulu/github/sailuh_kaiaulu/pull_request/
      commit: ../../rawdata/kaiaulu/github/sailuh_kaiaulu/commit/

[geraldmjhuff]

@carlosparadis

Now when i call train_model_(), we actually use the already existing .csv (w/ ground truths) in the google drive to train it?

The above is correct, but please make sure your narrative doesn't make it sounds the reader has to use this table. Make it clear it can be anything so long they respect the table format we want. At the scope of the notebook, I recommend you abide by the reply_dt format. So, maybe rename the text | polarity column to match in name.

If we want the train and predict functions to expect reply_body | Polarity, I would have to change the column in the existing .csv called in train function Text to reply_body. We then check to see if Polarity exists, if it doesn't, stop function, and let user know to add manual ground truths. (train() will always work with the .csv. Works with the data table the user created, as long it has reply_body and Polarity)

Similarly, predict will also check to see if reply_body and Polarity exists, if Polarity doesn't exist, its added with values of arbitrary numbers (since it does not matter what values are in Polarity initially) (predict() will always work with both the .csv or the data table the user created as long as it has reply_body. Polarity not necessary since this function will add it if it doesn't exist)

How does this implementation sound?

[carlosparadis]

In hindsight: Have the table expect text (lower case) and polarity contained in the table. However, throughout the notebook you should reference it as a reply data.table. Right before you call the function, you can rename in the notebook. This should keep it future proof..

[carlosparadis]

@geraldmjhuff

I see the last comment was my own here, so let me know if you have questions.

Also, since this is the last full week, what I would like you to show me Thursday is a final version of everything on a Notebook that runs on the PR by Friday (I need the weekend to review).

This should be a PR ready to merge as much as possible, and a Notebook I can download and run (provided I have the necessary files). You also need to narrate the notebook with english as a user would read it, etc.