Getting truth ratio always 1

[shaswati1]

I am trying to reproduce the results in Figure 8 of the paper and getting the truth ratio as 1.0 for forget set (5%) for all the unlearning steps until 30. Can you please help me with this? @zhilif and @pratyushmaini

P.S. I'm using the new eval pipeline to generate the aggreate_stat.

[zhilif]

@shaswati1 Hi, can you post your a snippet of your aggregated_stat?

[shaswati1]

Do you mean the aggreated_stat values? Sure, please take a look at the attached file. This is for 30th unlearning step!
aggregate_stat.csv

[zhilif]

Can you send me the command that you use for unlearning and evaluation?

[shaswati1]

I used the same commands that you provided in this link!

[shaswati1]

@zhilif just to make sure, by command did you mean the configurations used for unlearning and evaluation?

[zhilif]

@zhilif just to make sure, by command did you mean the configurations used for unlearning and evaluation?

Right. So you used the default config and the command in README without modification?

[shaswati1]

Yes, except for the precision. I'm using fp16 instead of bf16.

[zhilif]

Yes, except for the precision. I'm using fp16 instead of bf16.

Sorry for the late response, I was busy with some other stuff. The command in the current README looks like this python aggregate_eval_stat.py retain_result=${path_to_aggregated_retain_result} ckpt_result=${path_to_aggregated_retain_result} \ method_name=${method_name} save_file=${save_filename}, and I notice this typo that ${path_to_aggregated_retain_result} shows up twice. ckpt_result should take the path of ${path_to_aggregated_ckpt_result}. I apologize for this error. Can you try again and let me know if it works? Thanks!

[shaswati1]

@zhilif, which json file should I refer to forckpt_result? This is not defined in the readme.

[Carol-gutianle]

I also have this problem and forget quality always 1.

[Carol-gutianle]

Ok, I checked the function, perhaps ckpt_path refers to the retain version under the same settings.

[jackwwy]

waht is "path_to_aggregated_ckpt_result"

[jackwwy]

Ok, I checked the function, perhaps ckpt_path refers to the retain version under the same settings.

so，Have you solved this problem