Pre-processing steps

[aravindcheruvu]

Hi,

Thank you for open-sourcing this work. I have a few questions:

1. What are the pre-processing steps that are applied before releasing the datasets?
2. Do we need to apply all the pre-processing scripts mentioned in the dataset?
3. I still see profane words in the dataset, does that mean we have to apply the profane word filtering?

Thanks.

[ehsk]

Apologies for the late reply. We don't actively monitor this repo.

The pre-processing scripts can be found here. The datasets that we released were already pre-processed. We already filtered profane words based on this file. If you found problems that our list does not cover, then yes, you need to apply it on the data.