Missing file 'get_all_titles_from_spoken_wikipedia.py'

[thomaschhh]

I am currently looking into the building of the training dataset but it seems like the referenced file is nowhere to be found:

nemo_compatible/scripts/nlp/en_spellmapper/dataset_preparation/build_training_data.sh

Line 23 in 27bce6d

## To generate this file use ${NEMO_PATH}/examples/nlp/spellchecking_asr_customization/evaluation/get_all_titles_from_spoken_wikipedia.py --input_folder en/en/english --output_file spoken_wiki_titles.txt

./build_training_data.sh: 25: /nemo_compatible/scripts/nlp/en_spellmapper/dataset_preparation/NeMo/examples/nlp/spellchecking_asr_customization/evaluation/get_all_titles_from_spoken_wikipedia.py: not found

[bene-ges]

it's in /nemo_compatible/scripts/nlp/en_spellmapper/evaluation/get_all_titles_from_spoken_wikipedia.py
fixed the comment

[thomaschhh]

That's working, thanks.

I am wondering though where the input_folder is supposed to be.

nemo_compatible/scripts/nlp/en_spellmapper/dataset_preparation/build_training_data.sh

Line 23 in 45fdcea

## To generate this file use ${NEMO_COMPATIBLE_PATH}/scripts/nlp/en_spellmapper/evaluation/get_all_titles_from_spoken_wikipedia.py --input_folder en/en/english --output_file spoken_wiki_titles.txt

[bene-ges]

oh, it's the spoken wikipedia folder
it should appear after downloading and unzipping spoken_wikipedia, see this
code

[thomaschhh]

Looks like the dataset is no longer available:

WARNING: cannot verify corpora.uni-hamburg.de's certificate, issued by ‘CN=GEANT OV RSA CA 4,O=GEANT Vereniging,C=NL’:
  Issued certificate has expired.
HTTP request sent, awaiting response... 500 Service unavailable (with message)
2023-11-14 11:24:59 ERROR 500: Service unavailable (with message).

[bene-ges]

Ok, I put spoken_wiki_titles.txt to the repo, should be sufficient for training
if you later need the full spoken_wikipedia for evaluation, and it still is unavailable, tell me, I will upload my copy to huggingface