New task. Textual Anomaly Detector.

[kennynakamura]

Would a textual anomaly detector be interesting? Find phrases outside the context of a conversation that could represent secret codes.

https://github.com/kennynakamura/Detector-de-Anomalias

[lfcnassif]

For sure. What is the license of the suggested library? Did you evaluate the number of false positives and negatives of the proposed approach on some dataset?

[lfcnassif]

I think I understood the general approach. Is there some kind of sentence boundary detection or text is always broken by lines? And could a second ranked line in one block have a better score than the top ranked line of another block? Maybe allowing the user to set the minimal ranking score could improve a bit?

Anyway, things can always be enhanced later and I think a first version could be very helpful.

[lfcnassif]

PS: Thanks for clarifying the license :)

[kennynakamura]

Most of the code is data cleanning. I did it in a rustic way for not knowing so well regex, but the main process is to separate the sentences and remove punctuation. That's because long sentences don't work well. Separating them is better.

Example:
Original:
Na primeira semana, meus primos foram lá em casa, mas como estava chovendo, nós não pudemos sair para brincar!
Modified:
na primeira semana
meus primos foram lá em casa
mas como estava chovendo
nós não pudemos sair para brincar

There are also processes to filter only the texts of whatsapp messages. In this case the messages have many Portuguese errors, abbreviations, laughter, etc. But I had tests that within the 5% was the message with code that I added to the conversation.

Example
Original:
10/12/20 19:42 - Kenny: Opa kkk, de Sábado pode ser?
Modified:
Opa
de sábado pode ser

Yes, it's possible. Good idea!! I hadn't thought that. I will test this, but I already have a good feeling.

In code there is also parallel processing by CPU.
With parallel processing by CPU I got reasonable time process, but I don't know if with real cases this process time is acceptable.
BERT model = 1000 lines in less than 20 minutes
Spacy model = 1000 lines in less than 5 minutes
16gb RAM, i7, 4 cores in parallel
I don't know if a GPU is necessary, as these are texts, but I also have no experience in GPU inference

And it also has a loading screen in a other thread (Carregando ... )

If you think that any of these processes is not necessary or may make it difficult to function in IPED, I can change the code.
I am studying the code of FaceDtection to understand how to apply the python code in the program in java too, if there is something to change, please let me know.

[kennynakamura]

PS: Each block is separate after data cleannig
Example:
Original Text = 870 lines
After data cleannig (separate sentences, etc) = 1000 lines
so the whole process

[lfcnassif]

You can also take a look at Yahoo NSFW based nudity detection: https://github.com/sepinf-inc/IPED/blob/master/iped-app/resources/config/conf/scripts/NSFWNudityDetectTask.py

It does not open external processes (simpler) but does process items in batches (more complex).

[kennynakamura]

I did some tests with the possibility of the user choosing the minimal ranking score, but the range is minimal to choose without many lines being selected. I did the tests with whatsapp group messages from university students, things tend to be a bit "chaotic" too, many messages are really strange. In email with a formal language the results were neither better nor worse.

I was looking at the python scripts, there are "FaceRecognitionFaceProcess.py" and "FaceRecognitionFaceTask.py", the script "Process", I understood. But o "Task" I didn't understand it very well, my knowledge in Python + Java is almost none.
To make a pull request, you need both?

The spacy model can be accessed offline, but needs to be downloaded previously, or it would be better to leave access to it online?

I am studying about the batches!

[lfcnassif]

Sorry about late reply, lots of things happening behind the scenes...

Just a ...Task.py script is needed. Take a look at the basic example, it explains some things about the basic Task structure and how to enable it: https://github.com/sepinf-inc/IPED/blob/master/iped-app/resources/config/conf/scripts/PythonScriptTask.py

Offline processing is better for sure.

I would not recommend trying to do batch processing of items for now, things can become complicated and buggy. I just pointed the NSFWNudityDetectTask.py because it is a single script, I should have pointed the basic PythonScriptTask.py before.

[kennynakamura]

No problem, i thank you for your attention. Good luck out there!

I'm reading "PythonScriptTask.py", realized that it is in Object-oriented programming wich I not much familiar. I realized that it's about the iped software part, right?
In "FaceRecognitionFaceTask.py" and inside off Class is a "def process(self, item):" which is common in "PythonScriptTask.py" too, and the python script process is called over there. But i'm lost on how to write all the code. Can you help me with that?
Sorry for not knowing a lot about how to adapt the code.

If I learn well how to do all the parts, I can create other features that use the same spacy lib. I was thinking about a threat detector in conversations.

[lfcnassif]

The main method is def process(...) as you noticed, in that method you can access the decoded text of files calling item.getParsedTextCache() like shown at the end of the basic example, so you can use your code to process item text and store the results like shown in the last line of the example.

Early versions of python tasks were not object oriented, but that was changed to easy the implementation of thread safe code, your code will be called by several threads at the same time by iped engine.

[lfcnassif]

You should write all your code in one task script file. FaceRecognitonTask was broken in two scripts to be executed in external processes because of an infamous global lock bottleneck of CPython, see the details on #361. Some smarter libs take care of releasing and acquiring the GIL internally when crossing python<->C boundaries, like tensorflow, so external processes would not be needed, ideally they shouldn't...

[kennynakamura]

https://github.com/kennynakamura/Detector-de-Anomalias/blob/main/AnomalyDetectorTask.py
https://github.com/kennynakamura/Detector-de-Anomalias/blob/main/AnomalyDetectorConfig.txt
https://github.com/kennynakamura/Detector-de-Anomalias/blob/main/EmailTeste_Resultado_An%C3%A1lise.txt

Hello!
Just an update to the code i'm writing.
In the def process(..) I put almost all the code, in def init(..) i put the model loading and parameters (filtro) from Config file.
The model was download from here https://spacy.io/models/pt#pt_core_news_lg
When testing in IPED, putting it in the scripts folder and activating in XML , it works, but I'm in doubt where to save the analysis made.
I commented on the code, but if something is unclear, please ask me!

[lfcnassif]

Or another approach is running other algorithm before to tag suspicious emails, then just those would be submitted to this algorithm...

[kennynakamura]

I saw that there is a function of clustering files (Powerful file grouping (clustering) based on ANY metadata), if the email cluster with the least amount is analyzed? but it would probably contain a lot of emails yet, just something that I tried to think.

[lfcnassif]

It is a bit overstated. That phrase refers exactly to Metadata filter panel located at the bottom-left of the GUI, it groups results based on metadata and allow filtering by them.

[kennynakamura]

I wait for the #24 or, if you think my algorithm is fine, i can make a pull-request?

[lfcnassif]

I think this will need to wait #24 because IPED main purpose is to process large amounts of files and, as I understood, currently this algorithm assumes the texts always have anomalies, which may be true just for a few cases. If someone else has a better idea...