Resolution success highly sensitive to Unicode character choice

[pkgw]

In the process of working on the new Arxiv reference extraction pipeline (adsabs/arxiv-reference-extractor), I've been using the reference resolution microservice to test out its success. It seems that the results that I'm getting out of the resolver can depend sensitively on different forms of Unicode punctuation.

Here is a file with some examples: examples.txt. It can be processed with:

python test_reference_service.py -i examples.txt

It has a bunch of pairs of nearly-identical references. I find that for each pair, the first one resolves but the second one doesn't. In some cases, the only difference is a nearly-invisible change of an ASCII hyphen - to a Unicode en-dash –, or vice versa. In some bulk tests, I find that pure-ASCII references resolve the most successfully on average, but sometimes Unicode resolves when a plain-ASCII version doesn't.

[pkgw]

Here's another interesting example. In this file:

2017SoPh__292__117L.txt

about half of the refstrings (24 out of 50) fail to resolve as-is. If I simply replace Alfvén with Alfven everywhere, nearly all of the problems are fixed — only 8 out of 50 fail. There are 17 occurrences of accented "Alfvén" in this particular file, which means that its presence turns a resolvable refstring into an unresolvable one virtually every time, here.

[aaccomazzi]

This is still an issue in production at least, for example:

Hasegawa, A., Chen, L.: 1975, Kinetic process of plasma heating due to Alfvén wave excitation. Phys. Rev. Lett. 35, 370. DOI. ADS.

I believe there is a fix, any reason not to put it in production @golnazads?

[golnazads]

Alberto, the latest release is v2.0.17 and is running everywhere. However, last week, I did address what Peter brought up here and it is now fixed locally: { "resolved": [ { "refstring": "Hasegawa, A., Chen, L.: 1975, Kinetic process of plasma heating due to Alfvén wave excitation. Phys. Rev. Lett. 35, 370. DOI. ADS.", "score": "1.0", "bibcode": "1975PhRvL..35..370H" } ] } I have not released it yet, since I am still trying to find one other issue that Jenny has reported. Sorry. I shall release it on Monday.

…

On Fri, Oct 14, 2022 at 5:18 PM Alberto Accomazzi ***@***.***> wrote: This is still an issue in production at least, for example: Hasegawa, A., Chen, L.: 1975, Kinetic process of plasma heating due to Alfvén wave excitation. Phys. Rev. Lett. 35, 370. DOI. ADS. I believe there is a fix, any reason not to put it in production @golnazads <https://github.com/golnazads>? — Reply to this email directly, view it on GitHub <#52 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AG3M4CCAQPQD4D4TTK24FCDWDHEZHANCNFSM5PRNZPPQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>