Julia 1.11: IOBuffer does not update size of a user specified array

[haberdashPI]

In julia 1.10:

julia> bytes = Vector{Int8}(undef, 10)
10-element Vector{Int8}:
 0
 0
 0
 0
 0
 0
 0
 0
 0
 0

julia> bytes = Vector{UInt8}(undef, 10)
10-element Vector{UInt8}:
 0x00
 0x00
 0x00
 0x00
 0x00
 0x00
 0x00
 0x00
 0x00
 0x00

julia> io = IOBuffer(bytes; write=true, read=false)
IOBuffer(data=UInt8[...], readable=false, writable=true, seekable=true, append=false, size=0, maxsize=Inf, ptr=1, mark=-1)

julia> write(io, 10)
8

julia> write(io, 26)
8

julia> bytes
16-element Vector{UInt8}:
 0x0a
 0x00
 0x00
 0x00
 0x00
 0x00
 0x00
 0x00
 0x1a
 0x00
 0x00
 0x00
 0x00
 0x00
 0x00
 0x00

But with the current Julia 1.11 release candidate:

               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.11.0-rc4 (2024-09-25)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> bytes = Vector{Int8}(undef, 10)
10-element Vector{Int8}:
 -96
 -52
  89
  24
   1
   0
   0
   0
   1
   0

julia> bytes = Vector{UInt8}(undef, 10)
10-element Vector{UInt8}:
 0x60
 0x82
 0xaf
 0x19
 0x01
 0x00
 0x00
 0x00
 0x01
 0x00

julia> io = IOBuffer(bytes; write=true, read=false)
IOBuffer(data=UInt8[...], readable=false, writable=true, seekable=true, append=false, size=0, maxsize=Inf, ptr=1, mark=-1)

julia> write(io, 10)
8

julia> write(io, 26)
8

julia> bytes
10-element Vector{UInt8}:
 0x0a
 0x00
 0x00
 0x00
 0x00
 0x00
 0x00
 0x00
 0x01
 0x00

[PallHaraldsson]

Is this a bug? I mean then in 1.10, not 1.11? Are the bytes passed in supposed to be changed? I would then have expected IOBuffer!( but maybe write=true is for that? Was a bug fixed, and you rely on the old semantics?

[haberdashPI]

Based on the documentation I would interpret this as a bug (but maybe the documentation should be changed instead?)

Create an in-memory I/O stream, which may optionally operate on a pre-existing array.

If it operates on the array, presumably writing past the end of the array should either error (happens with no julia version that I know of) or expand the size of the array (the behavior from prior Julia versions).

What is happening in the above example? Is it writing to a different buffer, or is it writing to invalid memory that occurs past the end of the array?

[haberdashPI]

One guess: the new Memory object from the array is stored in the IOBuffer and that is updated, but the array still contains the pointer to the old Memory object.

Definitely seems like a change in behavior to me.

This is easy to fix in my case, I think, but this seems like it could easily lead to silent bugs, if users of IOBuffer are assuming the user-specified array gets updated from calls to write. And if I recall I copied this pattern from another repository (I'm sorry I can't remember where), so it probably does exist out in the wild. The bug wouldn't show up in tests necessarily, because you have to write enough data to require the underlying memory in the array to be reallocated.

[vtjnash]

The correct way to get back a buffer was always to call take! at the end of the usages. Otherwise there was never really any guarantees about where the data was located in the array, so you end up with random uninitialized data in past versions.