Wrong results using checkpoints for Bart Program baseline

[andreabac3]

Hi (@shijx12 @ShulinCao @teacherpeterpan @Flitternie @stellarkey),
thank you for your work.

I have some issue using the code base, can you kindly help me?

I don't know if the training works, but when I try to restore the checkpoint the predict.txt is full of "no".

My requirements.txt is the following:

transformers==4.0.0
torch==1.6.0
sentencepiece==0.1.96
nltk==3.7
numpy==1.19.2

The steps that I perform (from the KQAPro_Baseline/ folder [the root one]) are the following:

python3 -m Bart_Program.preprocess --input_dir data/ --output_dir preproc_data/ --model_name_or_path KQAPro_ckpt/program_ckpt/ 
cp data/kb.json preproc_data/
python3 -m Bart_Program.predict --input_dir preproc_data/ --save_dir log_folder --ckpt preproc_data/

Thank you in advance.

Kind regards,
Andrea

[stellarkey]

How is you training process?

I recall there is a validation step while train. Are those statistical measures normal?

When I am running Bart_Program I also had a accuracy problem (the acc is very low). It turns out to be kb.json's problem.

Take a look of https://github.com/shijx12/KQAPro_Baselines/blob/master/utils/load_kb.py#L9:

"""
knowledge json format:
    'concepts':
    {
        'id':
        {
            'name': '',
            'instanceOf': ['<concept_id>'],
        }
    },
    'entities': # exclude concepts
    {
        'id': 
        {
            'name': '<entity_name>',
            'instanceOf': ['<concept_id>'],
            'attributes':
            [
                {
                    'key': '<key>',
                    'value': 
                    {
                        'type': 'string'/'quantity'/'date'/'year'
                        'value':  # float or int for quantity, int for year, 'yyyy/mm/dd' for date
                        'unit':   # for quantity
                    },
                    'qualifiers':
                    {
                        '<qk>': 
                        [
                            <qv>, # each qv is a dictionary like value, including keys type,value,unit
                        ]
                    }
                }
            ]
            'relations':
            [
                {
                    'predicate': '<predicate>',
                    'object': '<object_id>', # NOTE: it may be a concept id
                    'direction': 'forward' or 'backward',
                    'qualifiers':
                    {
                        '<qk>': 
                        [
                            <qv>, # each qv is a dictionary like value
                        ]
                    }
                }
            ]
        }
    }
"""

Be sure your kb.json follows the exact statements above. (concept instanceOf, relation predicate, ...)

[andreabac3]

Hi @stellarkey,
Thank you for your quick response.

The kb.json file I am using is the one that can be downloaded from here (http://thukeg.gitee.io/kqa-pro/index.html).

This is a portion of that file (the file is too long to copy and paste, sorry for that ahah)
{"concepts": {"Q7270": {"name": "republic", "instanceOf": ["Q7174"]}, .... "entities": {"Q786": {"name": ..., "instanceOf": ..., "attributes": ...

I have not modified the file in any way.
I cloned the repository, downloaded data.zip and checkpoints, and tried the commands I mentioned above.

Can you share the right kb.json?

Thank you for your support.

Kind regards,
Andrea

[stellarkey]

@andreabac3 Sorry for the delay.

I uploaded it to google drive.

[ShulinCao]

The newly updated version fixed the problem. Please check the new checkpoint file and the change log.

[andreabac3]

Thank you for your help.

Please can you also update the bart-base checkpoint? (The non fine tuned version)?

[ShulinCao]

We did not change the bart-base model, which is downloaded from huggingface.