About the evaluation metric of Forget Quality #1
Comments
|
Any help is appreciated ! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
{{ message }}
Hi, thanks for the nice work.
I find that the way you calculate the Forget Quality is different from TOFU.
In TOFU, they test whether the Truth Ratio distribution of the unlearned and the retrained models on the Forget Set are indistinguishable. The more indistinguishable, the better the Forget Quality.
However, in this paper, you test whether the Truth Ratio distribution of the unlearned model on the Forget Set and the Retain Set are indistinguishable. The more distinguishable, the better the Forget Quality.
Intuitively, the calculation method in TOFU seems to be more reasonable ?
More importantly, when I used the calculation method in TOFU to evaluate the model obtained from this repository, all unlearned models have poor Forget Quality (i.e.,the p-value is very small).
Did you get similar results or give some reasonable explanation?
Thank you so much.
I use the default parameters in this repo to get the unlearned model (llama2-7b/Tofu_forget10). When using the evaluation metrics in this repo, similar results to those in the paper can be obtained.
When using the evaluation metrics in TOFU, the following results are obtained:
FO-GradDiff
SO-GradDiff
FO_PO
SO_PO
The text was updated successfully, but these errors were encountered: