You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.md
+19-7Lines changed: 19 additions & 7 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -98,18 +98,22 @@ As for the test samples, they can be found in `data/testing/test.csv`
98
98
99
99
```bash
100
100
python3 training/variable_predictor.py \
101
-
--training_csv_file data/training/train.csv
101
+
--training_csv_file data/training/train.csv \
102
+
--device <DEVICE>
102
103
```
104
+
If your machine has an NVIDIA GPU, set the `device` argument to `cuda`. If not specified, the default device will be `cpu`.
103
105
104
106
Where `training_csv_file` is the CSV file containg the training samples that was generated in Part 1.2. Alternatively, you could use the file that we have provided in the same section. The model's checkpoints will be saved in `training/var_runs`.
105
107
106
108
## Part 3: Evaluation
107
109
1. Model evaluation is done through the `evaluation/model_eval.py`,
108
110
```bash
109
111
python3 evaluation/model_eval.py \
110
-
--testing_csv_file data/testing/test.csv
112
+
--testing_csv_file data/testing/test.csv \
111
113
--checkpoint training/var_runs/<CHECKPOINT-NAME>
114
+
--device <DEVICE>
112
115
```
116
+
Make sure to set `device` to `cuda` to run inference/evaluation on the GPU.
113
117
114
118
We also provide the model checkpoint hosted on [Huggingface](https://huggingface.co/spaces/scam2024/ReIdentify/tree/main?clone=true). To clone the repository, make sure to have `git lfs` installed on your machine. You can do so by following the official [documentation](https://git-lfs.com/).
115
119
Once `git lfs` is properly installed, executing these commands,
The checkpoint name is `model_26_2` and the evaluation can be executed as follow,
123
127
```bash
124
128
python3 evaluation/model_eval.py \
125
-
--testing_csv_file data/testing/test.csv
129
+
--testing_csv_file data/testing/test.csv \
126
130
--checkpoint ReIdentify/model_26_2
127
131
```
128
132
129
-
2. GPT4 evaluation: In the paper, we have run experiments to compare our model's performance in generating identifier names compared to GPT4 on the same test set mentioned in previous sections.
130
-
Specifically, we used OpenAI's API service, hence, you will need an [OpenAI API key](https://platform.openai.com/docs/api-reference/authentication).
133
+
2. GPT4 and Gemini Pro evaluation: In the paper, we have run experiments to compare our model's performance in generating identifier names compared to larger models, namely GPT4 and Gemini Pro, on the same test set mentioned in previous sections.
134
+
Specifically, we used OpenAI's API service, hence, you will need an [OpenAI API key](https://platform.openai.com/docs/api-reference/authentication). Similarly, a Gemini API key is also required. It can be obtained through the following [link](https://ai.google.dev/gemini-api/docs/api-key).
0 commit comments