- Domain : Cyber-Security
- Type of problem: Multi-class Classification
- Objective: Predict the probability of a data-point belonging to each of the nine classes of malware.
- Performance Metric: Reduce the Multi-class log loss.
- Solution:
- For every malware we have two files: .asm file and .bytes file
- Total dataset consists of 200 GB of data, one of the largest publicly available datasets.
- Extracted features from files (Unigrams, Bigrams, Opcode, Pixel Intensity, Size). Used batch processing for text extraction and preprocessing to avoid memory overshoot.
- Performed Hypothesis testing method (Chi-Squared) for feature selection to reduce dimensionality.
- Designed and Calibrated (wherever needed) KNN, Logistic Regression, Random Forest and XGBoost on different subsets of features.
- Achieved a Multi-Class log loss of 0.018.
- Utilized Google Cloud Platform (GCP) Compute Engine for memory intensive tasks and Colab for GPU requisite tasks.