Metric Chrf

[i55code]

Hi, BLEU, Bertscore and COMET runs very well, however Chrf constantly has problem. I made sure the two system predictions have no empty lines, and checked thoroughly. Please take a look and let me know. Thanks!

++++++
Unable to init server: Could not connect: Connection refused
Unable to init server: Could not connect: Connection refused

(telescope:77231): Gdk-CRITICAL **: 18:45:57.927: gdk_cursor_new_for_display: assertion 'GDK_IS_DISPLAY (display)' failed
2021-11-15 18:45:59.449 INFO comet.download_utils: wmt20-comet-da is already in cache.
Some weights of the model checkpoint at xlm-roberta-large were not used when initializing XLMRobertaModel: ['lm_head.bias', 'lm_head.layer_norm.bias', 'roberta.pooler.dense.weight', 'roberta.pooler.dense.bias', 'lm_head.decoder.weight', 'lm_head.dense.weight', 'lm_head.layer_norm.weight', 'lm_head.dense.bias']

• This IS expected if you are initializing XLMRobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
• This IS NOT expected if you are initializing XLMRobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
  2021-11-15 18:46:08.371 INFO comet.models.base: Encoder model frozen.
  2021-11-15 18:46:09.273 INFO pytorch_lightning.utilities.distributed: GPU available: True, used: True
  2021-11-15 18:46:09.273 INFO pytorch_lightning.utilities.distributed: TPU available: False, using: 0 TPU cores
  2021-11-15 18:46:09.274 INFO pytorch_lightning.accelerators.gpu: LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]
  2021-11-15 18:53:13.018 INFO pytorch_lightning.utilities.distributed: GPU available: True, used: True
  2021-11-15 18:53:13.018 INFO pytorch_lightning.utilities.distributed: TPU available: False, using: 0 TPU cores
  2021-11-15 18:53:13.027 INFO pytorch_lightning.accelerators.gpu: LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]
  Traceback (most recent call last):
  File "/home/.local/bin/telescope", line 8, in
  sys.exit(telescope())
  File "/home/.local/lib/python3.6/site-packages/click/core.py", line 829, in call
  return self.main(*args, **kwargs)
  File "/home/.local/lib/python3.6/site-packages/click/core.py", line 782, in main
  rv = self.invoke(ctx)
  File "/home/.local/lib/python3.6/site-packages/click/core.py", line 1259, in invoke
  return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/.local/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
  return ctx.invoke(self.callback, **ctx.params)
  File "/home/.local/lib/python3.6/site-packages/click/core.py", line 610, in invoke
  return callback(*args, **kwargs)
  File "/home/.local/lib/python3.6/site-packages/telescope/cli.py", line 194, in compare
  for m in metric
  File "/home/.local/lib/python3.6/site-packages/telescope/cli.py", line 194, in
  for m in metric
  File "/home/.local/lib/python3.6/site-packages/telescope/metrics/metric.py", line 47, in pairwise_comparison
  x_result = self.score(testset.src, testset.system_x, testset.ref)
  File "/home/.local/lib/python3.6/site-packages/telescope/metrics/chrf/metric.py", line 28, in score
  chrf = sacrebleu.corpus_chrf(cand, [ref])
  File "/home/.local/lib/python3.6/site-packages/sacrebleu/sacrebleu.py", line 745, in corpus_chrf
  corpus_statistics = get_corpus_statistics(hypotheses, references, order=order, remove_whitespace=remove_whitespace)
  File "/home/.local/lib/python3.6/site-packages/sacrebleu/sacrebleu.py", line 697, in get_corpus_statistics
  statistics = get_sentence_statistics(hypothesis, reference, order=order, remove_whitespace=remove_whitespace)
  File "/home/.local/lib/python3.6/site-packages/sacrebleu/sacrebleu.py", line 678, in get_sentence_statistics
  reference = delete_whitespace(reference) if remove_whitespace else reference
  File "/home/.local/lib/python3.6/site-packages/sacrebleu/sacrebleu.py", line 670, in delete_whitespace
  return re.sub(r'\s+', '', text).strip()
  File "/usr/lib/python3.6/re.py", line 191, in sub
  return _compile(pattern, flags).sub(repl, string, count)
  TypeError: expected string or bytes-like object

[ricardorei]

Could you share the files you used?