New wandb fixes

.env.example

@@ -0,0 +1,3 @@
+WANDB_API_KEY = 
+WANDB_BASE_URL="https://api.wandb.ai"
+WANDB_SILENT="true"

.gitignore

@@ -3,6 +3,9 @@ __pycache__/
 *.py[cod]
 *$py.class
 
+.DS_Store
+*.onnx
+
 # C extensions
 *.so
 
@@ -60,7 +63,7 @@ cover/
 local_settings.py
 db.sqlite3
 db.sqlite3-journal
-
+*.db
 # Flask stuff:
 instance/
 .webassets-cache
@@ -163,3 +166,14 @@ testing/
 
 # Editors
 .vscode/settings.json
+datasets
+data
+wandb
+ecosystem.config.js
+
+
+keys
+
+
+local_datasets/
+*.csv

.pylintrc

@@ -0,0 +1,14 @@
+[MESSAGES CONTROL]
+disable=W1203,  # logging-fstring-interpolation
+        C0111,  # missing-docstring
+        C0103,  # invalid-name
+        C0114   # missing-module-docstring
+
+[FORMAT]
+max-line-length=120
+
+[BASIC]
+good-names=i,j,k,ex,Run,_,id,ge
+
+[MASTER]
+ignore=migrations

CHANGELOG.md

@@ -0,0 +1,27 @@
+# Changelog
+
+## 0.6.2
+
+- Establish version_key to match weights_version subnet hyperparameter
+
+## 0.6.1
+
+- TLDR for validators
+- Safe-Scan banner for validator script
+
+## 0.6
+
+- added blacklisting of miners
+- various logic fixes
+- protection from model copying
+- chain synchtonization fixes
+- fixes for miner CLI
+- stability improvements
+
+## 0.5.1
+
+- Various fixes for miner script
+
+## 0.5
+
+- Initial release of the subnet code

DOCS/COMPETITIONS.md

@@ -0,0 +1,68 @@
+# Safe Scan: Machine Learning Competitions for Cancer Detection
+
+Welcome to **Safe Scan**, a platform dedicated to organizing machine learning competitions focused on cancer detection. Our goal is to foster innovation in developing accurate and efficient models for cancer detection using machine learning. Here, you will find all the details needed to participate, submit your models, and understand the evaluation process.
+
+## Table of Contents
+
+1. [Overview](#overview)
+2. [Competition Trigger and Data Batch Handling](#competition-trigger-and-data-batch-handling)
+3. [Model Submission Requirements](#model-submission-requirements)
+4. [Evaluation and Scoring](#evaluation-and-scoring)
+5. [Configuration and Development](#configuration-and-development)
+6. [Command-Line Interface (CLI) Tools](#command-line-interface-cli-tools)
+7. [Communication Channels](#communication-channels)
+8. [Contribute](#contribute)
+
+## Overview
+
+Safe Scan organizes dynamic competitions focused on cancer detection using machine learning. These competitions provide participants with the opportunity to develop and test their models in a responsive and adaptive environment driven by real-world data.
+
+## Competition Trigger and Data Batch Handling
+
+- **Competition Initiation**: Competitions are triggered by data batch insertions from external medical institutions, creating a steady stream of new, non-public data for testing purposes.
+- **Data Handling Process**: Medical institutions upload each new batch of data to a central reference repository on Hugging Face, along with a reference entry for the new data batch file.
+- **Automatic Detection and Competition Start**: Validators monitor this centralized repository for new data batch entries. When new data is detected, validators initiate a competition by downloading and processing the data batch.
+
+## Model Submission Requirements
+
+- **Model Submission**: Participants, referred to as miners, must submit their trained models at the end of each competition.
+- **Format**: All models must be in ONNX format. This ensures uniform testing and allows for broad deployment options, including on mobile and web platforms.
+- **Training Code**: Each submission should include the code used for training the model to ensure transparency and reproducibility.
+- **Upload Process**: Models are uploaded to Hugging Face at the end of each test. Miners then submit the Hugging Face repository link on the blockchain for evaluation by validators.
+- **Timing Constraint**: Only models submitted at least 30 minutes before the competition start time are eligible for evaluation. This requirement ensures that models have not been retrained with the new data batch, maintaining fairness and integrity across the competition.
+
+## Evaluation and Scoring
+
+- **Independent Evaluation**: Each validator independently evaluates the submitted models according to predefined criteria.
+- **Scoring Mechanism**: Detailed scoring mechanisms are outlined in the [DOCS](/DOCS/competitions) directory. Validators run scheduled competitions and assess the models based on these criteria.
+- **Winning Criteria**: The best-performing model, according to the evaluation metrics, is declared the winner of the competition.
+- **Rewards**: The winner receives the full emission for that competition, divided by the number of competitions held.
+- **Rewards Time Decay**: If a miner stays in the top position for more than 30 days, their rewards start to decrease gradually. Every 7 days after the initial 30 days, their share of the rewards decreases by 10%. This reduction continues until their share reaches a minimum of 10% of the original reward.
+
+## Command-Line Interface (CLI) Tools
+
+- **Local Testing**: Miners are provided with an easy-to-use command-line interface (CLI) for local testing of their models. This tool helps streamline the process of testing models, uploading to Hugging Face, and submitting to the competition.
+- **Automated Data Retrieval**: Code for automating the retrieval of training data for each competition is available to integrate with the model training process. The script is defined in [scripts/get_dataset.py](/scripts/get_dataset.py).
+
+## Communication Channels
+
+Stay connected and up-to-date with the latest news, discussions, and support:
+
+- **Discord**: Join our [Safe Scan Discord channel](https://discord.gg/rbBu7WuZ) and the Bittensor Discord in the #safescan channel for real-time updates and community interaction.
+- **Dashboard**: Access the competition dashboard on [Hugging Face](https://huggingface.co/spaces/safescanai/dashboard).
+- **Blog**: Visit our [blog](https://safe-scan.ai/news/) for news and updates.
+- **Twitter/X**: Follow us on [Twitter/X](https://x.com/SAFESCAN_AI) for announcements and highlights.
+- **Email**: Contact us directly at [[email protected]](mailto:[email protected]) for any inquiries or support.
+
+## Development
+
+- **Software Lifecycle**: The project follows a structured software lifecycle, including Git flow and integration testing. These practices ensure robust development and encourage community contributions.
+
+
+## Contribute
+
+We welcome contributions to this project! Whether you're interested in improving our codebase, adding new features, or enhancing documentation, your involvement is valued. To contribute:
+
+- Follow our software lifecycle and Git flow processes.
+- Ensure all code changes pass integration testing.
+- Contact us on our [Safe Scan Discord channel](https://discord.gg/rbBu7WuZ) for more details on how to get started.

DOCS/COMPETITIONS.md.old

@@ -0,0 +1,75 @@
+
+
+
+
+
+# Safe Scan: Machine Learning Competitions for Cancer Detection
+
+Welcome to **Safe Scan**, a platform dedicated to organizing machine learning competitions focused on cancer detection. Our goal is to foster innovation in developing accurate and efficient models for cancer detection using machine learning. Here, you can find all the details needed to participate, submit your models, and understand the evaluation process.
+
+## Table of Contents
+
+1. [Overview](#overview)
+2. [Competition Schedule](#competition-schedule)
+3. [Dataset and Model Submission](#dataset-and-model-submission)
+4. [Evaluation and Scoring](#evaluation-and-scoring)
+5. [Configuration and Development](#configuration-and-development)
+6. [Command-Line Interface (CLI) Tools](#command-line-interface-cli-tools)
+7. [Communication Channels](#communication-channels)
+8. [Contribute](#contribute)
+
+## Overview
+
+Safe Scan organizes continuous competitions focused on cancer detection using machine learning. These competitions aim to advance the field by providing participants with the opportunity to develop and test their models in a structured environment.
+
+## Competition Schedule
+
+- **Frequency**: Competitions are held multiple times a day, at specific hours, continuously. This allows participants to join at different times that suit them best.
+- **Timed Events**: Each competition starts with a dataset release 5 minutes before testing, providing a short window for participants to prepare.
+- **Testing and Evaluation**: Models are evaluated immediately after each test, ensuring a quick feedback loop for participants.
+
+## Dataset and Model Submission
+
+- **Dataset Release**: A new dataset is provided for each competition, which is released exactly 5 minutes before testing begins. This dataset is used for training the models.
+- **Model Submission**: Participants, referred to as "miners," are required to submit their trained models at the end of each competition.
+  - **Format**: All models must be in ONNX format. This ensures uniform testing and allows for broad deployment options, including on mobile and web platforms.
+  - **Training Code**: Each submission should include the code used for training the model to ensure transparency and reproducibility.
+  - **Upload Process**: Models are uploaded to Hugging Face at the end of each test. Miners then submit the Hugging Face repository link on the blockchain for evaluation by validators.
+
+## Evaluation and Scoring
+
+- **Independent Evaluation**: Each validator independently evaluates the submitted models according to predefined criteria.
+- **Scoring Mechanism**: Detailed scoring mechanisms are outlined in the [DOCS](/DOCS/competitions) directory. Validators run scheduled competitions and assess the models based on these criteria.
+- **Winning Criteria**: The best-performing model, according to the evaluation metrics, is declared the winner of the competition.
+- **Rewards**: The winner receives the full emission for that competition, divided by the number of competitions held.
+- **Rewards time decay**: If a miner stays at the top position for more than 30 days, their rewards start to decrease gradually. Every 7 days after the initial 30 days, their share of the rewards decreases by 10%. This reduction continues until their share reaches a minimum of 10% of the original reward.
+
+## Configuration and Development
+
+- **Competition Configuration**: Each competition is configured through a `competition_config.json` file. This file defines all parameters and rules for the competition and is used by both miners and validators.
+- **Tracking Changes**: Changes to the competition configuration are tracked via a GitHub issue tracker, ensuring transparency and allowing for community input.
+- **Software Lifecycle**: The project follows a structured software lifecycle, including Git flow and integration testing. This ensures robust development practices and encourages community contributions.
+
+## Command-Line Interface (CLI) Tools
+
+- **Local Testing**: Miners are provided with an easy-to-use command-line interface (CLI) for local testing of their models. This tool helps streamline the process of testing models, uploading to Hugging Face, and submitting to the competition.
+- **Automated Data Retrieval**: Code for automating the retrieval of training data for each competition is available to integrate with the model training process. The script is defined in [scripts/get_dataset.py](/scripts/get_dataset.py).
+
+## Communication Channels
+
+Stay connected and up-to-date with the latest news, discussions, and support:
+
+- **Discord**: Join our [Safe Scan Discord channel](https://discord.gg/rbBu7WuZ) and the Bittensor Discord in the #safescan channel for real-time updates and community interaction.
+- **Dashboard**: Access the competition dashboard on [Hugging Face](https://huggingface.co/spaces/safescanai/dashboard).
+- **Blog**: Visit our [blog](https://safe-scan.ai/news/) for news and updates.
+- **Twitter/X**: Follow us on [Twitter/X](https://x.com/SAFESCAN_AI) for announcements and highlights.
+- **Email**: Contact us directly at [[email protected]](mailto:[email protected]) for any inquiries or support.
+
+## Contribute
+
+We welcome contributions to this project! Whether you're interested in improving our codebase, adding new features, or enhancing documentation, your involvement is valued. To contribute:
+
+- Follow our software lifecycle and Git flow processes.
+- Ensure all code changes pass integration testing.
+- Contact us on our [Safe Scan Discord channel](https://discord.gg/rbBu7WuZ) for more details on how to get started.
+

DOCS/competitions/1-MELANOMA-V3.md.old

@@ -0,0 +1,70 @@
+# Description of Melanoma Competition
+
+## Overview
+This competition invites participants to develop a machine learning model that **aids in detecting the possibility of melanoma**. The goal is to create a model that can identify patterns in data that are associated with an increased likelihood of melanoma in visual recognition.
+
+### Objective
+The primary objective is to develop a model that can analyze photos taken by users of their skin lesions or areas of concern. 
+The model should **assist users** by providing a risk assessment or likelihood score that helps them decide if they should seek further medical advice.
+As a result, best model will be released in Skin Scan mobile app to run locally on the phone, and a website that will host it, free for anyone to use. 
+
+## Evaluation Criteria
+Models will be evaluated based on described **performance metrics** of the model.
+The evaluation will be calculaded on following metrics with described weights.
+
+### Performance Metrics
+
+ The models will be assessed on the following metrics with the corresponding weights:
+
+| **Metric**  | **Description**                                       | **Weight** |
+|-------------|-------------------------------------------------------|------------|
+| **F-beta**  | Prioritizes recall, with a high beta to emphasize it. $\beta = 2$ | 0.60       |
+| **Accuracy**| Measures the overall correctness of predictions.      | 0.30       |
+| **AUC**     | Evaluates the model's ability to distinguish classes. | 0.10       |
+
+### Mathematical Formulas
+
+1. **F-beta Score $F\_\beta\$**
+
+
+   $$F_\beta = \left(1 + \beta^2\right) \cdot \frac{\text{Precision} \cdot \text{Recall}}{\left(\beta^2 \cdot \text{Precision}\right) + \text{Recall}}$$
+
+
+   Where:
+   - **$\beta$** is the weight of recall in the combined score
+   - in our case $\beta = 2$ for higher recall importance
+
+2. **Accuracy**
+
+   $$\text{Accuracy} = \frac{\text{True Positives} + \text{True Negatives}}{\text{Total Number of Samples}}$$
+
+3. **Area Under the Curve (AUC)**
+
+   AUC is the area under the Receiver Operating Characteristic (ROC) curve. It is calculated using the trapezoidal rule:
+
+   $$\text{AUC} = \int_0^1 \text{TPR} \, d(\text{FPR})$$
+
+   Where:
+   - **TPR** = True Positive Rate
+   - **FPR** = False Positive Rate
+
+
+## Model Inputs and Outputs
+
+### Inputs
+- **Input Format**: Multiple images in JPEG or PNG format.
+- **Input Features**: During preprocessing, images are resized to 224x224 pixels. Images are converted to numpy arrays with a datatype of `np.float32`, normalized to the range [0, 1].
+
+### Outputs
+- **Output Format**: A numerical value between 0 and 1, represented as a `float`. This value indicates the likelihood or risk score of the area of concern warranting further investigation.
+
+### Submission Requirements
+- **Model Submission**: Models must be submitted in ONNX format. They should be capable of handling dynamic batch sizes and accept inputs with the shape `(batch , 3 , 224 , 224)`, where `batch` represents the batch dimension. This ensures that the model can process a variable number of images in a single batch.
+
+
+## Rules and Guidelines
+
+- **Timeline**:
+- Competitions are triggered dynamically by new data batch uploads from external medical institutions, with no predefined schedule. Competitions may occur at any time based on the timing of new data insertions.
+- Each time a new data batch is detected in the central reference repository on Hugging Face, a new competition is initiated immediately by validators.
+- Results of competition will be available on the dashboard

DOCS/competitions/1-MELANOMA.md

@@ -0,0 +1,70 @@
+# Description of Melanoma Competition
+
+## Overview
+This competition invites participants to develop a machine learning model that **aids in detecting the possibility of melanoma**. The goal is to create a model that can identify patterns in data that are associated with an increased likelihood of melanoma in visual recognition.
+
+### Objective
+The primary objective is to develop a model that can analyze photos taken by users of their skin lesions or areas of concern. 
+The model should **assist users** by providing a risk assessment or likelihood score that helps them decide if they should seek further medical advice.
+As a result, best model will be released in Skin Scan mobile app to run locally on the phone, and a website that will host it, free for anyone to use. 
+
+## Evaluation Criteria
+Models will be evaluated based on described **performance metrics** of the model.
+The evaluation will be calculaded on following metrics with described weights.
+
+### Performance Metrics
+
+ The models will be assessed on the following metrics with the corresponding weights:
+
+| **Metric**  | **Description**                                       | **Weight** |
+|-------------|-------------------------------------------------------|------------|
+| **F-beta**  | Prioritizes recall, with a high beta to emphasize it. $\beta = 2$ | 0.60       |
+| **Accuracy**| Measures the overall correctness of predictions.      | 0.30       |
+| **AUC**     | Evaluates the model's ability to distinguish classes. | 0.10       |
+
+### Mathematical Formulas
+
+1. **F-beta Score $F\_\beta\$**
+
+
+   $$F_\beta = \left(1 + \beta^2\right) \cdot \frac{\text{Precision} \cdot \text{Recall}}{\left(\beta^2 \cdot \text{Precision}\right) + \text{Recall}}$$
+
+
+   Where:
+   - **$\beta$** is the weight of recall in the combined score
+   - in our case $\beta = 2$ for higher recall importance
+
+2. **Accuracy**
+
+   $$\text{Accuracy} = \frac{\text{True Positives} + \text{True Negatives}}{\text{Total Number of Samples}}$$
+
+3. **Area Under the Curve (AUC)**
+
+   AUC is the area under the Receiver Operating Characteristic (ROC) curve. It is calculated using the trapezoidal rule:
+
+   $$\text{AUC} = \int_0^1 \text{TPR} \, d(\text{FPR})$$
+
+   Where:
+   - **TPR** = True Positive Rate
+   - **FPR** = False Positive Rate
+
+
+## Model Inputs and Outputs
+
+### Inputs
+- **Input Format**: Multiple images in JPEG or PNG format.
+- **Input Features**: During preprocessing, images are resized to 224x224 pixels. Images are converted to numpy arrays with a datatype of `np.float32`, normalized to the range [0, 1].
+
+### Outputs
+- **Output Format**: A numerical value between 0 and 1, represented as a `float`. This value indicates the likelihood or risk score of the area of concern warranting further investigation.
+
+### Submission Requirements
+- **Model Submission**: Models must be submitted in ONNX format. They should be capable of handling dynamic batch sizes and accept inputs with the shape `(batch , 3 , 224 , 224)`, where `batch` represents the batch dimension. This ensures that the model can process a variable number of images in a single batch.
+
+
+## Rules and Guidelines
+
+- **Timeline**:
+- Competitions are triggered dynamically by new data batch uploads from external medical institutions, with no predefined schedule. Competitions may occur at any time based on the timing of new data insertions.
+- Each time a new data batch is detected in the central reference repository on Hugging Face, a new competition is initiated immediately by validators.
+- Results of competition will be available on the dashboard