You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.md
+32-33Lines changed: 32 additions & 33 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -77,11 +77,11 @@ Revisit the original goal of dataset distillation:
77
77
>
78
78
79
79
The evaluation method for DD-Ranking is grounded in the essence of dataset distillation, aiming to better reflect the informativeness of the synthesized data by assessing the following two aspects:
80
-
1. The degree to which the original dataset is recovered under hard labels (hard label recovery): $\text{HLR}=\text{Acc.}{\text{full-hard}}-\text{Acc.}{\text{syn-hard}}$.
80
+
1. The degree to which the real dataset is recovered under hard labels (hard label recovery): $\text{HLR}=\text{Acc.}{\text{real-hard}}-\text{Acc.}{\text{syn-hard}}$.
81
81
82
82
2. The improvement over random selection when using personalized evaluation methods (improvement over random): $\text{IOR}=\text{Acc.}{\text{syn-any}}-\text{Acc.}{\text{rdm-any}}$.
83
83
$\text{Acc.}$ is the accuracy of models trained on different samples. Samples' marks are as follows:
84
-
- $\text{full-hard}$: Full dataset with hard labels;
84
+
- $\text{real-hard}$: Real dataset with hard labels;
85
85
- $\text{syn-hard}$: Synthetic dataset with hard labels;
86
86
- $\text{syn-any}$: Synthetic dataset with personalized evaluation methods (hard or soft labels);
87
87
- $\text{rdm-any}$: Randomly selected dataset (under the same compression ratio) with the same personalized evaluation methods.
@@ -99,8 +99,6 @@ By default, we set $w = 0.5$ on the leaderboard, meaning that both $\text{IOR}$
99
99
## Overview
100
100
101
101
DD-Ranking is integrated with:
102
-
<!-- Uniform Fair Labels: loss on soft label -->
103
-
<!-- Data Aug. -->
104
102
- Multiple [strategies](https://github.com/NUS-HPC-AI-Lab/DD-Ranking/tree/main/dd_ranking/loss) of using soft labels in existing works;
105
103
- Commonly used [data augmentation](https://github.com/NUS-HPC-AI-Lab/DD-Ranking/tree/main/dd_ranking/aug) methods in existing works;
106
104
- Commonly used [model architectures](https://github.com/NUS-HPC-AI-Lab/DD-Ranking/blob/main/dd_ranking/utils/networks.py) in existing works.
@@ -222,8 +220,6 @@ The following results will be returned to you:
222
220
-`HLR std`: The standard deviation of hard label recovery over `num_eval` runs.
223
221
-`IOR mean`: The mean of improvement over random over `num_eval` runs.
224
222
-`IOR std`: The standard deviation of improvement over random over `num_eval` runs.
225
-
<!-- - `IOR/HLR mean`: The mean of IOR/HLR over `num_eval` runs.
226
-
- `IOR/HLR std`: The standard deviation of IOR/HLR over `num_eval` runs. -->
227
223
228
224
Check out our <spanstyle="color: #ff0000;">[documentation](https://nus-hpc-ai-lab.github.io/DD-Ranking/)</span> to learn more.
229
225
@@ -240,7 +236,7 @@ Feel free to submit grades to update the DD-Ranking list. We welcome and value a
240
236
Please check out [CONTRIBUTING.md](./CONTRIBUTING.md) for how to get involved.
241
237
242
238
243
-
<!--## Team
239
+
## Team
244
240
245
241
### Developers:
246
242
@@ -260,32 +256,35 @@ Please check out [CONTRIBUTING.md](./CONTRIBUTING.md) for how to get involved.
Copy file name to clipboardExpand all lines: doc/introduction.md
+2-2Lines changed: 2 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -45,13 +45,13 @@ Revisit the original goal of dataset distillation:
45
45
>
46
46
47
47
The evaluation method for DD-Ranking is grounded in the essence of dataset distillation, aiming to better reflect the information content of the synthesized data by assessing the following two aspects:
48
-
1. The degree to which the original dataset is recovered under hard labels (hard label recovery): \\( \text{HLR} = \text{Acc.} \text{full-hard} - \text{Acc.} \text{syn-hard} \\)
48
+
1. The degree to which the real dataset is recovered under hard labels (hard label recovery): \\( \text{HLR} = \text{Acc.} \text{real-hard} - \text{Acc.} \text{syn-hard} \\)
49
49
50
50
2. The improvement over random selection when using personalized evaluation methods (improvement over random): \\( \text{IOR} = \text{Acc.} \text{syn-any} - \text{Acc.} \text{rdm-any} \\)
51
51
52
52
\\(\text{Acc.}\\) is the accuracy of models trained on different samples. Samples' marks are as follows:
53
53
54
-
-\\(\text{full-hard}\\): Full dataset with hard labels;
54
+
-\\(\text{real-hard}\\): Real dataset with hard labels;
55
55
-\\(\text{syn-hard}\\): Synthetic dataset with hard labels;
56
56
-\\(\text{syn-any}\\): Synthetic dataset with personalized evaluation methods (hard or soft labels);
57
57
-\\(\text{rdm-any}\\): Randomly selected dataset (under the same compression ratio) with the same personalized evaluation methods.
0 commit comments