Links to office and PLR in static extract are not always correct

[MaxHoeferGEOINFO]

Hi all

The swisstopo has found an issue in the static extract with the links to the office and with the links for the public law restrictions for the contaminated military sites.
Sometimes when clicking on a link, it opens an incorrect webpage.

I have looked deeper into the problem, and it seems, as if the links in the PDF are not always complete. It does not appear very often, but sometimes the mouse-over shows already an incomplete URL. When the URL is opened then, it goes to an incorrect page.
Please see the images attached from the oereb portal AR and from BS, I hope this makes the issue clearer.

Since I can reproduce the same issue for our static extract and the one from BS, I assume it's a problem with the generation of the link, somewhere within pyramid_oereb or the MFP settings?

Would it be possible to have a look at this while implementing the changes for the new "Weisungen"?
Thanks & Cheers
Max

[romefi]

Did you also check the DataExtract? Is the link in the DataExtract right one and just in the static extract wrong? In that case it would point the problem to MFP. Otherwise to pyramid_oereb. Think we should first find out if the problem occurs in pyramid_oereb or in MFP.

[MaxHoeferGEOINFO]

Sorry, I was not clear enough:
This happens in the same PDF file - it looks like it depends on what position you mouse-over on the same link.
All the examples are from the same PDF files, and where reproduced while moving the mouse away from the link and on the link again.
Maybe it depends on where you position the mouse on the link, but I have not found a pattern so far.

And, I have seen it only on the PDF, not in the dynamic extract - the data extract I did not test, but I assume it's only in the static extract.

[romefi]

So it seems to be an MFP problem. I generated a static extract from Heiden and could reproduce the issue. It's really weird, sometimes the link works, sometimes not. I opened the same PDF with Firefox and the links seemed to always work. Still it should work in Adobe Reader as well. Just hope it's not a bug in Adobe :)

As we don't use MFP in SH, I'm afraid I can't give more help on this, sorry.

[voisardf]

As far as I remember it, it is a general issue with multiline URL links in the PDF.
I do not know if it is a problem with the MFP/JasperReport implementation or a general issue with pdf generating libraries.

[MaxHoeferGEOINFO]

Hi - I tried to reproduce the issue with the PDF Print from SH, since they do not use MFP.
So far, I was not able to reproduce, but this could also either be, that I did not try long enough, or enough different links.
I can not reproduce it on all of our links, only on some certain, so maybe I just was not "lucky" on the SH static extract.
It seems to be reproducable more or less reliably on KBS military data, but SH does not have any of those inside of the area of the canton.

[romefi]

We have a tiny area of KBS military, see 204a64a7-5b89-4570-9814-bb9821466ae8.pdf

Still couldn't reproduce on a quick test.

[voisardf]

As far as I can see, the URL covers the whole zone for your PDF @romefi (click on the link and keep the button pressed > the whole area goes black in Adobe PDF Reader).
While with pyramid_oereb only the text seems to be active, in fact even only the selected line (click on the link and keep the button pressed >only one line goes black).

you might want to compare the two examples to have a look if you see the same difference?

20210728035337_extract.pdf

This could also hint to a possible solution: rather to put an href on the text, we might want to put it on the area for MFP, if possible? @jwkaltz @marionb ?

[jwkaltz]

@voisardf @MaxHoeferGEOINFO I found the issue we had in the past about this matter:
openoereb/pyramid_oereb_mfp#38

@MaxHoeferGEOINFO can you please check whether the print templates in your project are up to date? As far as I remember, you did update them, but please double-check to be sure.

[MaxHoeferGEOINFO]

@jwkaltz Where do I see the version of my MFP templates again? Or what file should I compare?

[jwkaltz]

@MaxHoeferGEOINFO these are the most current templates: https://github.com/openoereb/pyramid_oereb_mfp/tree/v1.9.0/print-apps/oereb

[MaxHoeferGEOINFO]

Thanks, I thought there is a version number hidden somewhere, so I would not have to compare files all the.
We actually were not on the latest version of the MFP templates, but I am able to reproduce the issue even after updating.
One of our links seem to be more prone to get the problem - would it help to look why that link has the problem more often?
Or how do we continue?

[jwkaltz]

One of our links seem to be more prone to get the problem - would it help to look why that link has the problem more often?

Is this the link https://www.ar.ch/verwaltung/departement-bau-und-volkswirtschaft/amt-fuer-raum-und-wald/ ?
We can add that to the project test data if this is the problematic link. Did I understand correctly that the problem occurs only in Adobe Acrobat, or do you know if other PDF readers also have a problem with this link?

[MaxHoeferGEOINFO]

This is the link, correct. With that link, it only happens in Adobe, not when opening the extract in the broswer for example. I have no other PDF program right now, and therefore did not test any other.
But, the same (maybe not?) problem happens also with the links to the KBS military, and that is reproducible when opening the PDF within google chrome.
Here is an example Link for those:
https://map.geo.admin.ch/?zoom=12&E=2739650&N=1250525&bgLayer=ch.swisstopo.
swissimage&layers=ch.vbs.kataster-belasteter-standorte-militaer&layers_opacity=0.
75&lang=de

[jwkaltz]

@MaxHoeferGEOINFO I have integrated both links in the test data of the project, and tested the resulting static extract with Adobe Acrobat Reader DC 2021.005.20058, I can not reproduce a problem klicking on the links, this is the file:
20210802063148_extract.pdf
(I added the links you mentioned to the bottom of the last Oereb theme)
When you open this file, do you have problems with the links, and with which PDF reader? Can you also please provide a URL to a static extract on your site where you observe the problem?

[MaxHoeferGEOINFO]

@jwkaltz I was able to reproduce with the PDF file you attached, when opening it in Chrome and opening the KBS link a lot of times. Sometimes the following link is opened then https://map.geo.admin.ch/?zoom=12&E=2739650&N=1250525&bgLayer=ch.swisstopo.pixelkarte-farbe&lang=de&topic=ech&layers=ch.swisstopo.zeitreihen,ch.bfs.gebaeude_wohnungs_register,ch.bav.haltestellen-oev,ch.swisstopo.swisstlm3d-wanderwege,ch.astra.wanderland-sperrungen_umleitungen&layers_opacity=1,1,1,0.8,0.8&layers_visibility=false,false,false,false,false&layers_timestamp=18641231,,,,

Here is the link to our PDF, where you can reproduce the issue with the Link for the "Amt für Raum & Wald" for the topics "statische Walgrenzen", when opening the file in Adobe.

http://oereb.ar.ch/ktar/wsgi/oereb/extract/reduced/pdf/CH345449778476
Please note, that is now not yet generated with the latest MFP templates, those I only have activated so far on our internal system. But, as mentioned before, the issue is also there with latest MFP files, so I assume that's okay?

[jwkaltz]

@MaxHoeferGEOINFO when looking at the data extract (https://oereb.ar.ch/ktar/wsgi/oereb/extract/reduced/json/CH345449778476), I see that the last two links contain a space at the end, so, strictly speaking, the URLs are not correct:
"https://www.ar.ch/verwaltung/departement-bau-und-volkswirtschaft/amt-fuer-raum-und-wald/ "
I assume that Adobe Reader is not tolerant about this.

Can you please repair the links in the project, and try again?

[MaxHoeferGEOINFO]

@jwkaltz Thanks for that - I changed this and now it seems to work reliably. I think we can forget for now the question, why the link worked sometimes and sometimes it didn't?
Anyway, the issue is still there with the BS link I assume.
And, I can still reproduce the problem in the PDF you attached with the KBS military link.

[remyguillaume]

@MaxHoeferGEOINFO Thanks for the message.

I can only confirm that, even if this ticket was closed, the bug described in openoereb/pyramid_oereb_mfp#38 is still here: when a Link is on many lines, there's a small pixel area between the lines where the link is incomplete.

The reason given was "Please note that while the hyperlinks work now, JasperReports does not provide a box around the whole link, as some tools such as Word do; therefore the user click will work only when the user clicks on the text within the link."

@jwkaltz This ticket was in 2019, but now in 2021 with a new version of Jasper, is there still nothing we can do to fix this?

@svamaa FYI

[jwkaltz]

@remyguillaume Regarding the description in openoereb/pyramid_oereb_mfp#38 , to my knowledge the bug as such was solved, meaning, there are no longer any incomplete links that are presented to the user. As least, we could not reproduce that in various PDF viewers after the fix. Do you observe a different behaviour?

What is still the case, and what I can reproduce in the development setup, is that in the current solution there are gaps between the lines, meaning: the link will only open when the user clicks on the link text, but nothing will happen when the user clicks between the text lines. I agree this can be confusing to the user. However, I see no new functionality about the hyperlink rendering in JasperReports in the newest version, which is 6.17.0 (whereas the MapFish Print used for oereb development tests, which is 3.28, is using JasperReports 6.16.0).

[MaxHoeferGEOINFO]

@jwkaltz The issue discussed here was reproducible with the PDF you generated and attached, see my comment on the 2nd of August - or is that something different now?

[remyguillaume]

@jwkaltz I confirm that the bug is still here.

Can be reproduced with the default integrated PDF viewer in Chrome and with Adobe Acrobat DC.
However, I cannot reproduce it with the application Kofax Power PDF Advanced.

We are using pyramid_oereb 1.8.1 with the corresponding templates.

The example file: e68f6413-88ba-4606-8d33-081077499096.pdf

Adobe Version:

[jwkaltz]

@remyguillaume I can reproduce the bug with Adobe Acrobat Reader on your file, but not on the test PDF generated in the development environment; in both cases, when hovering between the lines of the multi-line link, the cursor changes, but in one case it is clickable (with a wrong link) and in the other case it is not. In both cases, when hovering between the lines, the cursor looks like something clickable, which it should not be (other PDF viewers show it differently, meaning, as something not clickable). But in the second case, since it is actually not clickable, the user can not open a bad link.

I bet the difference in behavior is due to the WMS backend behavior,

• when wms.oereb.bs.ch is called with an incomplete link, such as https://wms.oereb.bs.ch?version=1.3.0&SERVICE=WMS&SLD_VERSION=1.1.0&FORMAT=image/png, the result is not really an error, the answer is of type image/svg+xml and a response is displayed in the browser (to the user of course, it appears to be an error)
• when something similar is done to www.geo.admin.ch with an incomplete bad link such as https://wms.geo.admin.ch/?SERVICE=WMS&REQUEST=GetLegendGraphic&VERSION=1.1.1&FORMAT=image/jpg, it is a real error.

Therefore it seems that Acrobat Reader is doing something "smart" about the link URLs (pre-load?) and knows when one is in error, and therefore does not allow the user to follow the link. Other PDF readers that I have checked do not do that.

Now that we know how we can reproduce the bug, we can reinvestigate if other templating options exist which will work also in Acrobat Reader.

However, if you want a quick win, I think changing the behavior of wms.oereb.bs.ch to raise a real error when being called with a bad URL will avoid the problem in your PDFs (in the sense that Acrobat Reader should then appear to behave the same as other PDF viewers). If you get a change to try that, please let us know.

[remyguillaume]

@jwkaltz It seems this difference is within MapServer itself, and is bound to the WMS query Version= parameter.

• When calling the BS URL (Version=1.3.0), I get text/xml as response content-type.
• When calling the Swisstopo URL (Version=1.1.1), I get application/vnd.ogc.se_xml as response content-type.
• When changing the Swisstopo version to 1.3.0, I get text/xml as response content-type as well.

In all cases, the HTTP response code is 200, and thus is identify as success.

In version 1.1.1, the content-type application/vnd.ogc.se_xml is returned by default with all errors. (http://cite.opengeospatial.org/OGCTestData/wms/1.1.1/spec/wms1.1.1.html)
But it seems that with the version 1.3.0, the default content-type for errors was changed to text/xml, which actually isn't a right error type.
However, there is a possibility to force the right content-type by adding the EXCEPTIONS=application/vnd.ogc.se_xmlparameter.

So when calling https://wms.oereb.bs.ch/?version=1.3.0&SERVICE=WMS&SLD_VERSION=1.1.0&FORMAT=image/png&EXCEPTIONS=application/vnd.ogc.se_xml, I can get the right content-type.

I tried to add this parameter to the vollständige Legende link, but it didn't solve anything. Could you evlt try with the WMS 1.3.0 from Swisstopo instead on the 1.1.1, to see if you can repoduce it with the URL of Swisstopo too?

Vollständige Legende page 5 : with_exception.pdf.

When I think about it, I suppose it should really be a report problem (Jasper, MFP or the templates), because the content of the wrong link is exactly what is written in the first line of the multiple-line link. The link is just truncated, without any logic:

[remyguillaume]

Actually we are using the WMS-Services version 1.3.0 from Swisstopo, and I can repoduce the bug (page 6) :
swisstopo_130.pdf

[jwkaltz]

Thanks for the inputs @remyguillaume , but it seems we were on the wooden path yesterday. I can not find any correlation between WMS backend behavior and whether the link is working or not in Acrobat Reader.

To summarize,

1. this is the PDF produced by current test data
   oereb-extract_with_1_1_1_WMS_link.pdf
2. this is the PDF produced by test data from Move test URL to WMS 1.3.0 usage #1295:
   oereb_extract_with_130_links.pdf
3. this is the PDF that you provided:
   oereb-extract-example-BS.pdf

In the first two, I can never open a bad link in Acrobat Reader, in the third, clicking on any multi-line link between the first and second line opens a bad link. Note that the second PDF contains an identical link as the third PDF, compare "Vollständige Legende" on page in the second document with the "Vollständige Legend" on page 4 in the Baselstadt document. Interestingly, even though the link is identical, the line break is not at the same place.

Do you observe the same, maybe I am missing something?

In summary, right now, the only difference I see is the environment which produces the PDF. In the case of the PDFs I produced in test environment, they are produced by MapFish Print 3.28.1, run via docker-compose, and using these templates: https://github.com/openoereb/pyramid_oereb_mfp/tree/V1

[jwkaltz]

After some more tests regarding the previous comment, I see that I can reproduce it sometimes in the first two documents, when I click quickly between the links about ten times, then the (wrong) link is opened. I guess this is the phenomenon that @MaxHoeferGEOINFO also observed.

Whereas in the third document, the one from Baselstadt, I can reproduce it on every single click. So there really seems to be some difference coming from the environment where the PDF is produced.

[remyguillaume]

Hi @jwkaltz

I think I finally found the problem.

The first 2 PDFs are created with the PDF/A conformance.
The third (from BS) is created without PDF/A conformance.

At BS we cannot activate PDF/A conformance because of this issue: #876

When you open a PDF/A conform file with Acrobat Reader, a warning is displayed :

And the problem cannot be reproduced, even if the wrong link is present when you just do a mouse-over :

But if you click on "Bearbeitung aktivieren" in Acrobat, then you can repoduce the bug with the 2 first documents as well.

Furthermore, if you open directly the document in the PDF-Reader in the Browser (chrome), the problem can be reproduced on the first 2 documents.

So in summary I would say that the wrong link is present in all 3 documents, but as the first 2 are PDF/A compliant, the behavior is different in Acrobat Reader.

[jwkaltz]

Yes, thanks for the further investigations @remyguillaume , indeed I can always reproduce it in Acrobat Reader when PDF/A is turned off... what a strange combination of effects.
I therefore close this issue, and reopen and update the print template issue instead (openoereb/pyramid_oereb_mfp#38).