statusandheaders: if setting a header fails on Headers object, retry with encoded header

[ikreymer]

instead of skipping invalid headers:

• encode header value as latin-1 for use with Headers object
• encode back to original when serializing
  fixes Headers with non-ASCII characters silently disappear #81

[ikreymer]

Trying to decide if we want to go with encodeURI or with latin-1 encoding here... I think latin-1 is technically more 'correct', though encodeURI() more portable / easier to understand. As mentioned in #81, fetch() does the equivalent of latin-1 / iso-8859-1, so maybe we do go with that?

[tw4l]

Trying to decide if we want to go with encodeURI or with latin-1 encoding here... I think latin-1 is technically more 'correct', though encodeURI() more portable / easier to understand. As mentioned in #81, fetch() does the equivalent of latin-1 / iso-8859-1, so maybe we do go with that?

I can see the arguments either way but my vote would be for ISO-8859-1, since that's what the HTTP spec seems to have originally supported and is consistent with fetch.

[ikreymer]

Trying to decide if we want to go with encodeURI or with latin-1 encoding here... I think latin-1 is technically more 'correct', though encodeURI() more portable / easier to understand. As mentioned in #81, fetch() does the equivalent of latin-1 / iso-8859-1, so maybe we do go with that?

I can see the arguments either way but my vote would be for ISO-8859-1, since that's what the HTTP spec seems to have originally supported and is consistent with fetch.

Updated it to the latin-1 encoding again.

[tw4l]

One suggestion to take or leave re: encoding, otherwise looks good and thanks for the tests

[tw4l]

I wonder if using buffer.transcode() might be a good idea here, as there might be edge cases where TextEncoder().encode(value) might encode certain characters as two bytes?

[ikreymer]

Hmm, yeah, though buffer is not available in the browser, so would have to use the equivalent.

[tw4l]

Ah true, so used to working with node I forgot! It is also an edge case within an edge case, so I trust how much attention you want to give this

[ikreymer]

I guess if we change this, it will break this assertion from the test case:

function decodeLatin1(buf) {
  let str = "";
  for (let i = 0; i < buf.length; i++) {
    str += String.fromCharCode(buf[i]);
  }
  return str;
}
d = decodeLatin1(new TextEncoder().encode("https://wiki.archlinux.jp/index.php/Arch_User_Repository?rdfrom=https%3A%2F%2Fwiki.archlinux.org%2Findex.php%3Ftitle%3DAUR_Metadata_%28%25E6%2597%25A5%25E6%259C%25AC%25E8%25AA%259E%29%26redirect%3Dno#AUR_メタデータ"));

// this matches the headers from node fetch:
a = await fetch("https://wiki.archlinux.org/title/AUR_Metadata_(%E6%97%A5%E6%9C%AC%E8%AA%9E)", {redirect: "manual"})
assert(a.headers.get("location") === d)

If we do: decodeLatin1(Buffer.from("https://wiki.archlinux.jp/index.php/Arch_User_Repository?rdfrom=https%3A%2F%2Fwiki.archlinux.org%2Findex.php%3Ftitle%3DAUR_Metadata_%28%25E6%2597%25A5%25E6%259C%25AC%25E8%25AA%259E%29%26redirect%3Dno#AUR_メタデータ", "ascii")); it will give a different result than what node fetch is doing..

[ikreymer]

Resolved this in the most correct way, I think. Still encoding utf8-latin1 to store in Headers object, but then reencoding back when serializing