Update doc

Author: lizekai-richard

README.md

@@ -152,17 +152,17 @@ python setup.py install
 ```
 ### Quickstart
 
-Below is a step-by-step guide on how to use our `ddranking`. This demo is based on LRS on soft labels (source code can be found in `demo_soft.py`). You can find LRS on hard labels in `demo_hard.py` and ARS in `demo_aug.py`.
+Below is a step-by-step guide on how to use our `ddranking`. This demo is based on LRS on soft labels (source code can be found in `demo_lrs_soft.py`). You can find LRS on hard labels in `demo_lrs_hard.py` and ARS in `demo_aug.py`.
 DD-Ranking supports multi-GPU Distributed evaluation. You can simply use `torchrun` to launch the evaluation.
 
 **Step1**: Intialize a soft-label metric evaluator object. Config files are recommended for users to specify hyper-parameters. Sample config files are provided [here](https://github.com/NUS-HPC-AI-Lab/DD-Ranking/tree/main/configs).
 
 ```python
-from ddranking.metrics import SoftLabelEvaluator
+from ddranking.metrics import LabelRobustScoreSoft
 from ddranking.config import Config
 
-config = Config.from_file("./configs/Demo_Soft_Label.yaml")
-soft_label_metric_calc = SoftLabelEvaluator(config)
+config = Config.from_file("./configs/Demo_LRS_Soft_Label.yaml")
+lrs_soft_metric = LabelRobustScoreSoft(config)
 ```
 
 <details>
@@ -194,7 +194,7 @@ random_data_path = "./random_data"          # Specify your random data path
 save_path = f"./results/{dataset}/{model_name}/IPC{ipc}/dm_hard_scores.csv"
 
 """ We only list arguments that usually need specifying"""
-soft_label_metric_calc = SoftLabelEvaluator(
+lrs_soft_metric = LabelRobustScoreSoft(
     dataset=dataset,
     real_data_path=real_data_dir, 
     ipc=ipc,
@@ -233,9 +233,9 @@ syn_lr = torch.load('/your/path/to/syn/lr.pt')
 **Step 3:** Compute the metric.
 
 ```python
-metric = soft_label_metric_calc.compute_metrics(image_tensor=syn_images, soft_labels=soft_labels, syn_lr=syn_lr)
+lrs_soft_metric.compute_metrics(image_tensor=syn_images, soft_labels=soft_labels, syn_lr=syn_lr)
 # alternatively, you can specify the image folder path to compute the metric
-soft_label_metric_calc.compute_metrics(image_path='./your/path/to/syn/images', soft_labels=soft_labels, syn_lr=syn_lr)
+lrs_soft_metric.compute_metrics(image_path='./your/path/to/syn/images', soft_labels=soft_labels, syn_lr=syn_lr)
 ```
 
 The following results will be printed and saved to `save_path`:

book/config/overview.html

@@ -155,7 +155,8 @@ <h1 class="menu-title">DD-Ranking API Documentation</h1>
                 <div id="content" class="content">
                     <main>
                         <h1 id="config"><a class="header" href="#config">Config</a></h1>
-<p>To ease the usage of DD-Ranking, we allow users to specify the parameters of the evaluator in a config file. The config file is a YAML file that contains the parameters of the evaluator. We illustrate the config file with the following example.</p>
+<p>To ease the usage of DD-Ranking, we allow users to specify the parameters of the evaluator in a config file. The config file is a YAML file that contains the parameters of the evaluator. We illustrate the config file with the following examples.</p>
+<h2 id="lrs"><a class="header" href="#lrs">LRS</a></h2>
 <pre><code class="language-yaml">dataset: CIFAR100                 # dataset name
 real_data_path: ./dataset/        # path to the real dataset
 ipc: 10                           # image per class
@@ -166,12 +167,14 @@ <h1 id="config"><a class="header" href="#config">Config</a></h1>
 tea_use_torchvision: true         # whether to use torchvision to load teacher model
 
 teacher_dir: ./teacher_models     # path to the pretrained teacher model
+teacher_model_names: [ResNet-18-BN]     # the list of teacher models being used for evaluation
 
 data_aug_func: mixup              # data augmentation function
 aug_params:
     lambda: 0.8                   # data augmentation parameter; please follow this format for other parameters
 
 use_zca: false                    # whether to use ZCA whitening
+use_aug_for_hard: false             # whether to use data augmentation for hard label evaluation
 
 custom_train_trans:               # custom torchvision-based transformations to process training data; please follow this format for your own transformations
   - name: RandomCrop
@@ -189,28 +192,38 @@ <h1 id="config"><a class="header" href="#config">Config</a></h1>
 
 custom_val_trans: null              # custom torchvision-based transformations to process validation data; please follow the format above for your own transformations
 
-use_aug_for_hard: false             # whether to use data augmentation for hard label evaluation
-
 soft_label_mode: M                  # soft label mode
 soft_label_criterion: kl            # soft label criterion
-temperature: 30.0                   # temperature for soft label
+loss_fn_kwargs:
+    temperature: 30.0               # temperature for soft label
+    scale_loss: false               # whether to scale the loss
+
 optimizer: adamw                    # optimizer
 lr_scheduler: cosine                # learning rate scheduler
 weight_decay: 0.01                  # weight decay
+momentum: 0.9                       # momentum
 num_eval: 5                         # number of evaluations
+eval_full_data: false               # whether to compute the test accuracy on the full dataset
 num_epochs: 400                     # number of training epochs
-default_lr: 0.001                   # default learning rate
 num_workers: 4                      # number of workers
 device: cuda                        # device
+dist: true                          # whether to use distributed training
 syn_batch_size: 256                 # batch size for synthetic data
 real_batch_size: 256                # batch size for real data
 save_path: ./results.csv            # path to save the results
+
+random_data_format: tensor          # format of the random data, tensor or image
+random_data_path: ./random_data     # path to the save the random data
+
 </code></pre>
 <p>To use config file, you can follow the example below.</p>
-<pre><code class="language-python">from dd_ranking.metrics import SoftLabelEvaluator
+<pre><code class="language-python">from dd_ranking.metrics import LabelRobustScoreSoft
 
 config = Config(config_path='./config.yaml')
-evaluator = SoftLabelEvaluator(config)
+evaluator = LabelRobustScoreSoft(config)
+</code></pre>
+<h2 id="ars"><a class="header" href="#ars">ARS</a></h2>
+<pre><code class="language-yaml">
 </code></pre>
 
                     </main>

book/datasets/overview.html

@@ -166,6 +166,7 @@ <h3 id="parameters"><a class="header" href="#parameters">Parameters</a></h3>
 <li><strong>data_path</strong>(<span style="color:#FF6B00;">str</span>): Path to the dataset.</li>
 <li><strong>im_size</strong>(<span style="color:#FF6B00;">tuple</span>): Image size.</li>
 <li><strong>use_zca</strong>(<span style="color:#FF6B00;">bool</span>): Whether to use ZCA whitening. When set to True, the dataset will <strong>not be</strong> normalized using the mean and standard deviation of the training set.</li>
+<li><strong>custom_train_trans</strong>(<span style="color:#FF6B00;">Optional[Callable]</span>): Custom transformation on the training set.</li>
 <li><strong>custom_val_trans</strong>(<span style="color:#FF6B00;">Optional[Callable]</span>): Custom transformation on the validation set.</li>
 <li><strong>device</strong>(<span style="color:#FF6B00;">str</span>): Device for performing ZCA whitening.</li>
 </ul>
@@ -198,6 +199,15 @@ <h3 id="parameters"><a class="header" href="#parameters">Parameters</a></h3>
 <li><strong>std</strong>: <code>[0.229, 0.224, 0.225]</code></li>
 </ul>
 </li>
+<li><strong>ImageNet1K</strong>
+<ul>
+<li><strong>channels</strong>: <code>3</code></li>
+<li><strong>im_size</strong>: <code>(224, 224)</code></li>
+<li><strong>num_classes</strong>: <code>1000</code></li>
+<li><strong>mean</strong>: <code>[0.485, 0.456, 0.406]</code></li>
+<li><strong>std</strong>: <code>[0.229, 0.224, 0.225]</code></li>
+</ul>
+</li>
 </ul>
 
                     </main>

book/getting-started/quick-start.html

@@ -155,24 +155,26 @@ <h1 class="menu-title">DD-Ranking API Documentation</h1>
                 <div id="content" class="content">
                     <main>
                         <h2 id="quick-start"><a class="header" href="#quick-start">Quick Start</a></h2>
-<p>Below is a step-by-step guide on how to use our <code>dd_ranking</code>. This demo is based on soft labels (source code can be found in <code>demo_soft.py</code>). You can find hard label demo in <code>demo_hard.py</code>.</p>
+<p>Below is a step-by-step guide on how to use our <code>dd_ranking</code>. This demo is for label-robust score (LRS) on soft labels (source code can be found in <code>demo_lrs_soft.py</code>). You can find the demo for LRS on hard label demo in <code>demo_lrs_hard.py</code> and the demo for augmentation-robust score (ARS) in <code>demo_ars.py</code>.
+DD-Ranking supports multi-GPU Distributed evaluation. You can simply use <code>torchrun</code> to launch the evaluation.</p>
 <p><strong>Step1</strong>: Intialize a soft-label metric evaluator object. Config files are recommended for users to specify hyper-parameters. Sample config files are provided <a href="https://github.com/NUS-HPC-AI-Lab/DD-Ranking/tree/main/configs">here</a>.</p>
-<pre><code class="language-python">from ddranking.metrics import SoftLabelEvaluator
+<pre><code class="language-python">from ddranking.metrics import LabelRobustScoreSoft
 from ddranking.config import Config
 
-&gt;&gt;&gt; config = Config.from_file("./configs/Demo_Soft_Label.yaml")
-&gt;&gt;&gt; soft_label_metric_calc = SoftLabelEvaluator(config)
+&gt;&gt;&gt; config = Config.from_file("./configs/Demo_LRS_Soft_Label.yaml")
+&gt;&gt;&gt; lrs_soft_metric = LabelRobustScoreSoft(config)
 </code></pre>
 <details>
 <summary>You can also pass keyword arguments.</summary>
 <pre><code class="language-python">device = "cuda"
 method_name = "DATM"                    # Specify your method name
 ipc = 10                                # Specify your IPC
-dataset = "CIFAR10"                     # Specify your dataset name
-syn_data_dir = "./data/CIFAR10/IPC10/"  # Specify your synthetic data path
+dataset = "CIFAR100"                     # Specify your dataset name
+syn_data_dir = "./data/CIFAR100/IPC10/"  # Specify your synthetic data path
 real_data_dir = "./datasets"            # Specify your dataset path
 model_name = "ConvNet-3"                # Specify your model name
 teacher_dir = "./teacher_models"		# Specify your path to teacher model chcekpoints
+teacher_model_names = ["ConvNet-3"]      # Specify your teacher model names
 im_size = (32, 32)                      # Specify your image size
 dsa_params = {                          # Specify your data augmentation parameters
     "prob_flip": 0.5,
@@ -184,23 +186,31 @@ <h2 id="quick-start"><a class="header" href="#quick-start">Quick Start</a></h2>
     "ratio_crop_pad": 0.125,
     "ratio_cutout": 0.5
 }
-save_path = f"./results/{dataset}/{model_name}/IPC{ipc}/datm_ranking_scores.csv"
+random_data_format = "tensor"              # Specify your random data format (tensor or image)
+random_data_path = "./random_data"          # Specify your random data path
+save_path = f"./results/{dataset}/{model_name}/IPC{ipc}/dm_hard_scores.csv"
 
 """ We only list arguments that usually need specifying"""
-soft_label_metric_calc = SoftLabelEvaluator(
+lrs_soft_metric = LabelRobustScoreSoft(
     dataset=dataset,
     real_data_path=real_data_dir, 
     ipc=ipc,
     model_name=model_name,
     soft_label_criterion='sce',  # Use Soft Cross Entropy Loss
     soft_label_mode='S',         # Use one-to-one image to soft label mapping
+    loss_fn_kwargs={'temperature': 1.0, 'scale_loss': False},
     data_aug_func='dsa',         # Use DSA data augmentation
     aug_params=dsa_params,       # Specify dsa parameters
     im_size=im_size,
+    random_data_format=random_data_format,
+    random_data_path=random_data_path,
     stu_use_torchvision=False,
     tea_use_torchvision=False,
-    teacher_dir='./teacher_models',
+    teacher_dir=teacher_dir,
+    teacher_model_names=teacher_model_names,
+    num_eval=5,
     device=device,
+    dist=True,
     save_path=save_path
 )
 </code></pre>
@@ -213,16 +223,18 @@ <h2 id="quick-start"><a class="header" href="#quick-start">Quick Start</a></h2>
 &gt;&gt;&gt; syn_lr = torch.load('/your/path/to/syn/lr.pt')
 </code></pre>
 <p><strong>Step 3:</strong> Compute the metric.</p>
-<pre><code class="language-python">&gt;&gt;&gt; metric = soft_label_metric_calc.compute_metrics(image_tensor=syn_images, soft_labels=soft_labels, syn_lr=syn_lr)
+<pre><code class="language-python">&gt;&gt;&gt; lrs_soft_metric.compute_metrics(image_tensor=syn_images, soft_labels=soft_labels, syn_lr=syn_lr)
 # alternatively, you can specify the image folder path to compute the metric
-&gt;&gt;&gt; metric = soft_label_metric_calc.compute_metrics(image_path='./your/path/to/syn/images', soft_labels=soft_labels, syn_lr=syn_lr)
+&gt;&gt;&gt; lrs_soft_metric.compute_metrics(image_path='./your/path/to/syn/images', soft_labels=soft_labels, syn_lr=syn_lr)
 </code></pre>
-<p>The following results will be returned to you:</p>
+<p>The following results will be printed and saved to <code>save_path</code>:</p>
 <ul>
-<li><code>hard_label_recovery mean</code>: The mean of hard label recovery scores.</li>
-<li><code>hard_label_recovery std</code>: The standard deviation of hard label recovery scores.</li>
-<li><code>improvement_over_random mean</code>: The mean of improvement over random scores.</li>
-<li><code>improvement_over_random std</code>: The standard deviation of improvement over random scores.</li>
+<li><code>HLR mean</code>: The mean of hard label recovery over <code>num_eval</code> runs.</li>
+<li><code>HLR std</code>: The standard deviation of hard label recovery over <code>num_eval</code> runs.</li>
+<li><code>IOR mean</code>: The mean of improvement over random over <code>num_eval</code> runs.</li>
+<li><code>IOR std</code>: The standard deviation of improvement over random over <code>num_eval</code> runs.</li>
+<li><code>LRS mean</code>: The mean of Label-Robust Score over <code>num_eval</code> runs.</li>
+<li><code>LRS std</code>: The standard deviation of Label-Robust Score over <code>num_eval</code> runs.</li>
 </ul>
 <!-- - `dd_ranking_score mean`: The mean of dd ranking scores.
 - `dd_ranking_score std`: The standard deviation of dd ranking scores. -->