fixes ingydotnet/io-all-pm#93: -utf8 not applied everywhere.

[sesc]

This pull request ensures correct encoding by first opening the file normally, with all encoding layers being applied, before its file handle is tied to an array reference.

[shlomif]

Hi!

Sorry it took us so long but the relevant commit seems to do with Tie::File and is not related here - see commit 21e63fa . Can you file a new pull-req?

[sesc]

Hi!

I think everything fits, doesn't it? This commit 21e63fa is meant to be related to the array tie bug #93.

The pull-req 0a0392bd that you already accepted (thanks, glad it was of use!) was related to the file name bug #95.

[shlomif]

Hi!

Sorry for the late reply but in the branch I am getting test failures:

Test Summary Report
-------------------
test/in-place.t       (Wstat: 2304 Tests: 0 Failed: 0)
  Non-zero exit status: 9
  Parse errors: Bad plan.  You planned 5 tests but ran 0.
test/os.t             (Wstat: 0 Tests: 14 Failed: 0)
  TODO passed:   5-6
test/tie_file.t       (Wstat: 2304 Tests: 2 Failed: 0)
  Non-zero exit status: 9
Files=56, Tests=373,  6 wallclock secs ( 0.19 usr  0.04 sys +  2.99 cusr  0.34 csys =  3.56 CPU)
Result: FAIL
/home/shlomif/apps/perl/brew/perls/perl-5.24.0/lib/site_perl/5.24.0/auto/share/dist/Zilla-Dist/Makefile:75: recipe for target 'test' failed
make: *** [test] Error 1
'make -f /home/shlomif/apps/perl/brew/perls/perl-5.24.0/lib/site_perl/5.24.0/auto/share/dist/Zilla-Dist/Makefile test' failed: No such file or directory

can you look into fixing them?

[sesc]

Ah, sorry! I can reproduce your issue and have found a working alternative. Will update my branch and let you know.

[coveralls]

Coverage decreased (-0.02%) to 90.106% when pulling 8f61010 on sesc:93-file-utf8-problem2 into e62731b on ingydotnet:master.

[coveralls]

Coverage decreased (-0.05%) to 90.078% when pulling 5a2e273 on sesc:93-file-utf8-problem2 into e62731b on ingydotnet:master.

[coveralls]

Coverage decreased (-0.05%) to 90.078% when pulling 5a2e273 on sesc:93-file-utf8-problem2 into e62731b on ingydotnet:master.

[coveralls]

Coverage decreased (-0.02%) to 90.106% when pulling 5a2e273 on sesc:93-file-utf8-problem2 into e62731b on ingydotnet:master.

[coveralls]

Coverage decreased (-0.02%) to 90.106% when pulling 5a2e273 on sesc:93-file-utf8-problem2 into e62731b on ingydotnet:master.

[coveralls]

Coverage increased (+0.009%) to 90.138% when pulling f6f459b on sesc:93-file-utf8-problem2 into e62731b on ingydotnet:master.

[sesc]

Okay, I'm done.

My new approach has Tie::File open the file itself and set the binmode, using that module's (unfortunately undocumented) 'discipline' option (see perldoc Tie::File L112ff).

I have also expanded the tie_file test to verify the utf8 functionality a bit.

[akarelas]

Should we politely ask the owner of Tie::File to document the discipline option now?

[sesc]

I went looking for a place to execute your suggestion, @akarelas, and I think I now know why they didn't mention it... UTF-8 support in Tie::File is seriously buggy! (However, I think they ought to have warned about the bug instead, since the problem same no matter if one tie()s with my 'discipline' approach here or via an utf8-enabled filehandle as in the bug report above.)

My test happened to exercise the one way that works: Appending multi-byte data to the end of the file.
But if you change data in the tied array, the lengths of the old and replacement strings, and thus the range within the file to be updated, are calculated in bytes, corrupting the file.

For instance, if line 1 has an "ä" (two-byte UTF-8 char) on it and you replace it with "a" (one byte), then there is an extra empty line before the old line 2. On the other hand, if there is an "a" and you replace it with "ä", the first character of line 2 is clobbered by the newline that now has moved to the right.

The bug has been known and acknowledged but unfixed for 2 1/2 years, which implies that it isn't trivial.

@shlomif, I'm not quite sure how we are to handle this problem in our module, until Tie::File is fixed. Maybe we should add a bold warning of our own that tie()ing in UTF-8 mode only is safe for reading?