Consider changing the behavior of <meta name="page-ranges"> in Print Reftests

[jwatt]

In the RFC that introduced Print Reftests, a new page-ranges meta value was introduced.

What I would like to be able to do is have some complicated page styling in a test, then test different page ranges against multiple references. So, for example, I might test pages 1-2 against a different two page reference, then page 3 against a single page reference, then pages 4-6 against a different 3 page reference. This would really help wih testing that the print output of a document can contain a range of page sizes, orientations, margins, etc., and that @page rules are being applied correctly to the correct pages.

Unfortunately that isn't supported the way the RFC is written. You can specify multiple meta tags that specify a page range, and each tag's page range is applied either to the test or a reference.

First, it's not clear to me why a page range should be applicable to a reference. Surely references should be as simple as possible, and references should just be crafted more simply so as to not require a page range?

Besides that, being able to apply a page range to references seems a bit weird, and of very limited utility. I say that because if you can only specify a single range for the test itself, then it only seems to make sense to specify at most a single reference, regardless of whether or not you can specify a page range for it. As specified in the RCF, you could in principal specify multiple references with different ranges...but then what are you comparing? You're checking that multiple "references" all look the same as the "test" (or range of pages from it). ... You've now kinda inverts the "test"-"reference" relationship, where the test is now really acting as the reference to multiple "references".

There are currently only 4 WPT tests that are using page-ranges. Could we change the behavior so that ranges are always applicable to the test, and what you're doing when specifying a range in a meta tag is saying "this range from the test should look the same as this entire references, the reference having the same number of pages as the range from the test"?

@jgraham @gsnedders

[jwatt]

@mstensho

[mstensho]

I found more than 4 WPT tests, FWIW, but still not a lot. :)

$ grep -nr reftest-page * | grep -v ^docs | grep -v ^tools
css/css-break/overflowing-block-print-ref.html:2:
css/css-break/zero-height-page-break-001-print.html:9:
css/printing/subpixel-page-size-001-print.html:5:
css/printing/subpixel-page-size-002-print.html:5:
css/printing/fragmented-inline-block-001-print.html:7:
infrastructure/reftest/reftest_match-print-ref.html:2:
infrastructure/reftest/reftest_mismatch-print.html:3:
infrastructure/reftest/reftest_mismatch_fail-print.html:3:

The RFC says "Ranges specified in reference files are an error that are ignored.", so I don't understand what the problem is. If the test specifies a page-ranges META that points to a ref, the page ranges shouldn't be applied to the ref as well, should it? I agree that page ranges in refs doesn't seem very useful.

Also: It's named "page-ranges" in the RFC, but in https://web-platform-tests.org/writing-tests/print-reftests.html it's "reftest-pages", which is what all the WPTs use. The latter doesn't allow multiple refs, as far as I can tell.

[jwatt]

To provide a bit of context, let me back up for a minute. I should have said that my interpretation on the intent of the RFC is reflected by the code that implements it in the WPT/Mozilla trees, and both the RFC and that code was written by @jgraham I think. Regarding the code:

Things start in WPT tools code where the meta tags are gathered into a list then their values are parsed and stored in a Dict where the key for each range is the specified reference's filename, else the test's filename.

Switching to the Mozilla side of the harness code, the range table is passed to the test instance, then through to compareUrls. That then passed the range table through to the same function -- screenshotPaginated -- twice, once passing the test URL and once passing the reference URL. screenshotPaginated then uses getPages to turn the range into a set of page numbers based on the current file's name -- be that test or reference -- and then passes that set to renderPages to render the pages in the set to canvases.

[jwatt]

I found more than 4 WPT tests, FWIW, but still not a lot. :)

Ah, yes, I suppose I wasn't counting the infrastructure files and we don't have one of the others in our tree yet. :)

The RFC says "Ranges specified in reference files are an error that are ignored.", so I don't understand what the problem is.

Right. But that just means that the meta tags all need to be specified in the test. From my reading and from the code, you should be able to do:

  <meta name="reftest-pages" content="1-2">
  <meta name="reftest-pages" content="first-ref.html:5-6">
  <meta name="reftest-pages" content="second-ref.html:8-9">

In that case pages 5 and 6 of first-ref.html would be compared against page 1 and 2 of the test, and pages 8 and 9 of second-ref.html would be compared against page 1 and 2 of the test.

If the test specifies a page-ranges META that points to a ref, the page ranges shouldn't be applied to the ref as well, should it?

In that case it seems that currently what's supposed to happen is that the range specified by that meta tag is ONLY applied to the ref. I'd like it to only apply to the TEST.

I agree that page ranges in refs doesn't seem very useful.

Cool. :)