Add rrcf outlier

[Tatsuya-hasegawa]

Hello Ian,

I implemented Robust Random Cut Forest class and the outlier function.
RRCF is more useful when we detect anomaly of time series data as you know.

imported Robust Random Cut Forest python library
MIT License Copyright (c) 2018 kLabUM
https://klabum.github.io/rrcf/

Best regards,
Tatsuya

[Tatsuya-hasegawa]

The PoC notebooks are located in https://github.com/Tatsuya-hasegawa/MSTICPy_utils/tree/main/analysis_rrcf_outliers

RRCF can also use "from msticpy.analysis.outliers import plot_outlier_results" function, however that of RRCF is very slow.
On the other hand, I think RRCF anomaly detection accuracy for time series data is worth.

For instance, this is one of DNS traffic anomaly case.

Isolation Forest result is like below.

Robust Random Cut Forest is smart like below.

Since scikit-learn does not support Robust Random Cut Forest, I implemented RRCF in msticpy in Python instead of Cpython natively.
Although it is slower, it is certainly a better anomaly detection algorithm for time series event data.

[ianhelle]

I'll have to trust your expertise on how this works and is implemented (which I do).
You might want to include one of the notebooks and add it to the notebooks folder.

I have a couple of lightweight comments - more about format than functionality.

[Tatsuya-hasegawa]

Hi , Thanks for your many advices.
I think I was able to modify all the points from you except for including one of the notebooks and add it to the notebooks folder.
I will do it on another PR next time when I have get time.
Best regards,

[ianhelle]

Hi Tatsuya,
There are a bunch of typing warnings from mypy for outliers. In many cases it maybe as simple as declaring params as int | float.
Use:

from __future__ import annotations

at the top of the imports to ensure that Py3.8 supports this syntax.

Also - not a hard requirement - if you have any documentation to add about this (now impressive) module, it would def make it more visible to others. Also if you want to publish an article somewhere (we have a msticpy blog that hasn't been used for ages), we can include that in the release and linkedin/X posts.
Or just include the notebooks in the PR.
Great stuff, thx for the submission.

[Tatsuya-hasegawa]

Hi Ian,

Thank you for the advices.
I fixed the typing warning and my local check was fine.
FYI, I accidentally did a force push, but I changed it back, pulled, and then pushed again.

(base) hacket@hackeTlab msticpy % git commit -m "fixed CI/CD errors"  
check yaml...........................................(no files to check)Skipped
check json...........................................(no files to check)Skipped
trim trailing whitespace.................................................Passed
black....................................................................Passed
pylint...................................................................Passed
flake8...................................................................Passed
isort (python)...........................................................Passed
pydocstyle...............................................................Passed
ruff.....................................................................Passed
check_reqs_all...........................................................Passed
[add_rrcf_outlier c5037219] fixed CI/CD errors
 1 file changed, 19 insertions(+), 16 deletions(-)

On the other hand, I'm not sure for the compatibility to Python3.8...
After some research with AI, I found that the "type OR operator (short for Union)" like "int | str" that msticpy is trying to migrate is a syntax introduced in Python 3.10 and later, and it doesn't seem to be supported in 3.8 even if you use from future import annotations.
So, please run CI/CD workflow again to check it.

Also, about notebook documentation and artcle, I'm preparing some notebook files both the current IsolationForest outlier and this RRCF outliers. I'm examining the results of differences for the same datasets.

For time series data, Isolation Forest is known to be effective at detecting simple spikes, while RRCF is effective at detecting trend changes and correlations in multidimensional features. For non-time series data, Isolation Forest is also found to be faster and more accurate.

[ianhelle]

On the other hand, I'm not sure for the compatibility to Python3.8... After some research with AI, I found that the "type OR operator (short for Union)" like "int | str" that msticpy is trying to migrate is a syntax introduced in Python 3.10 and later, and it doesn't seem to be supported in 3.8 even if you use from future import annotations. So, please run CI/CD workflow again to check it.

This should work fine - we're using future annotations in multiple places. Am building now.

[Tatsuya-hasegawa]

OMG, I'm sorry.
6 typing errors are still existing. I'll fix it day after tomorrow with also adding jupyter notebook documents.

In addition, I'll add the joblib package to pip requirement as same as rrcf package.
Missing packages:
joblib:analysis.outliers.py

[ianhelle]

No worries, take your time. :-)

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[Tatsuya-hasegawa]

Hi Ian,

I have finished my tasks with adding some jupyter notebooks!

Best regards,

(base) hacket@hackeTlab msticpy % git commit -m "fixed the rest typing errors and add jupyter notebooks for comparing the outlier algorisms"
check yaml...........................................(no files to check)Skipped
check json...............................................................Passed
trim trailing whitespace.................................................Passed
black....................................................................Passed
pylint...................................................................Passed
flake8...................................................................Passed
isort (python)...........................................................Passed
pydocstyle...............................................................Passed
ruff.....................................................................Passed
check_reqs_all...........................................................Passed
[add_rrcf_outlier 2d66adab] fixed the rest typing errors and add jupyter notebooks for comparing the outlier algorisms
 8 files changed, 5673 insertions(+), 6 deletions(-)
 create mode 100644 docs/notebooks/Outliers-IsolationForest.ipynb
 create mode 100644 docs/notebooks/Outliers-IsolationForest_timeseries.ipynb
 create mode 100644 docs/notebooks/Outliers-RobustRandomCutForest.ipynb
 create mode 100644 docs/notebooks/Outliers-RobustRandomCutForest_timeseries.ipynb

[Tatsuya-hasegawa]

Weird...
What's that? Many errors are probably out of scope.

[ianhelle]

Yeah - looks like something changed with maybe a mypy update and possible a respx update.
I will checkout your PR and fix these