Binary Classifier of Spurious Closure of DHSV

[ricardoevvargas]

Would you like to share in this repository a new approach for this specific problem?

We encourage you to submit a pull request with a file named problems\01_binary_classifier_of_spurious_closure_of_dhsv\[your_name_here].ipynb that you developed to solve this problem. If we like your approach, your file could be listed in this repository as a 3W toolkit's example of use.

Thanks for the contribution!

[lucasalvesmartins36]

Hi, Ricardo. How do I get the labels of the test data (X_test)? The function extract_test_samples doesn't return a variable y_test? Thank you in advance.

[ricardoevvargas]

Hi, @lucasalvesmartins36. No, this method does not return y_test. The basic idea of ​​the 3W toolkit is for you to estimate the y_test without knowing the real y_test. See that there is a method to calculate metrics from the estimated y_test. Since the data and code are open, it is possible to "rack" the real y_test, but this is not a good practice. We do not intend to expose the real y_test, but I would like to understand your needs. Maybe we can evolve the 3W toolkit with new methods. What do you need the real y_test for? Can you detail your needs/proposal?

[lucasalvesmartins36]

My intention is to evaluate the accuracy of my model. If I am not wrong, I should calculate y_pred using my model and so I use the method fold.calculate_partial_metrics(y_pred, event_labels) to see the accuracy of this model, right? Thank you! It seems obvious but I wanted to be sure.

[ricardoevvargas]

Yes, i believe you get the idea. To make it even clearer, in each fold:

1 - Use 3W toolkit methods to extract samples (tô get X_train, y_train, and X_test);
2 - Estimate y_test with your approach, that must use only X_train, y_train, and X_test. You should be creative and make/test hypotheses;
3 - Use fold.calculate_partial_metrics to calculate the default metrics;
4 - Use folds.get_metrics to get global metrics (considering all folds).

We hope to tweak/evolve this and other methods collaboratively. You can propose changes, for example.

By the way, you and everyone else should feel free to ask anything about the 3W. This space is just for discussion, which is a genuine form of collaboration. Thanks!!!

[leopinheiro2]

Hi @ricardoevvargas. One thing I didn't understand well. If I want to implement, for instance, a SEVERE_SLUGGING experiment based on code above I have an error saying that the file folds_clf_03.csv does not exist. I can see that for DHSV there is a file folds_clf_01.csv into dataset/folds and this csv file is read. How would I create the fold files for other experiments? How was that one created for DHSV?

[ricardoevvargas]

Hi, @leopinheiro2. Thank you for your questions, which allow us to provide additional and more targeted clarification.

As described in README.md, specific problems will be incorporated into this project gradually. At this point, with the 3W toolkit current version (1.0) we can work on binary classifier of Spurious Closure of DHSV.

This explains why folds_clf_02.csv exists and folds_clf_03.csv does not. To experiment with other types of algorithms with the 3W dataset (binary classifiers, multiclass classifiers, clustering, one class classification, etc.), there are basically two ways:

1 - You can directly access the data in the 3W dataset and then develop whatever functions you need (or use whatever is ready in other tools). In this case, as you will not be using features/definitions from the 3W toolkit, it will be quite complicated to compare results obtained by you and others. This is because many different methodologies can be used.

2 - You can develop features/definitions needed for a kind of experimentation in a standardized way and then propose a merge in the 3W toolkit. In this way, it will be quite simple to compare results obtained by you and by other people who reuse this development (with specific and fixed methodology). We followed this path in the case of the binary classifier of Spurious Closure of DHSV. The folds_clf_02.csv file was built with random choices, but fixed for results to be reproducible. We chose to work with 5 folds (0 to 4). The fold -1 is additional for when you want to work with simulated data. The is_ova = False when the instance is type 2 or True otherwise. You and others can propose other ways, as long as the results are reproducible and the limitations of the 3W dataset are taken into account.

We strongly recommend you and others to follow 2nd option for us to develop the 3W project collaboratively.