Add TextDecoderStream and TextEncoderStream #5648
Comments
const reader = response.body.pipeThrough(new TextDecoderStream()).getReader();
^
ReferenceError: Can't find variable: TextDecoderStreambun version |
|
Workaround: copy the following ponyfill into a // Copyright 2016 Google Inc.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
// Polyfill for TextEncoderStream and TextDecoderStream
// Modified by Sukka (https://skk.moe) to increase compatibility and performance with Bun.
export class PolyfillTextDecoderStream extends TransformStream<Uint8Array, string> {
readonly encoding: string;
readonly fatal: boolean;
readonly ignoreBOM: boolean;
constructor(
encoding: Encoding = 'utf-8',
{
fatal = false,
ignoreBOM = false,
}: ConstructorParameters<typeof TextDecoder>[1] = {},
) {
const decoder = new TextDecoder(encoding, { fatal, ignoreBOM });
super({
transform(chunk: Uint8Array, controller: TransformStreamDefaultController<string>) {
const decoded = decoder.decode(chunk, { stream: true });
if (decoded.length > 0) {
controller.enqueue(decoded);
}
},
flush(controller: TransformStreamDefaultController<string>) {
// If {fatal: false} is in options (the default), then the final call to
// decode() can produce extra output (usually the unicode replacement
// character 0xFFFD). When fatal is true, this call is just used for its
// side-effect of throwing a TypeError exception if the input is
// incomplete.
const output = decoder.decode();
if (output.length > 0) {
controller.enqueue(output);
}
}
});
this.encoding = encoding;
this.fatal = fatal;
this.ignoreBOM = ignoreBOM;
}
}Then |
bun does not suport TextDecoderStream oven-sh/bun#5648
|
Bummer. Can't use |
|
It's not possible to use Vercel AI SDK because of this issue |
|
Here are polyfills you can use based off the Node.js' implementation while you waiting for Bun to catch up: /**
* TextEncoderStream polyfill based on Node.js' implementation https://github.com/nodejs/node/blob/3f3226c8e363a5f06c1e6a37abd59b6b8c1923f1/lib/internal/webstreams/encoding.js#L38-L119 (MIT License)
*/
export class TextEncoderStream {
#pendingHighSurrogate: string | null = null
#handle = new TextEncoder()
#transform = new TransformStream<string, Uint8Array>({
transform: (chunk, controller) => {
// https://encoding.spec.whatwg.org/#encode-and-enqueue-a-chunk
chunk = String(chunk)
let finalChunk = ""
for (let i = 0; i < chunk.length; i++) {
const item = chunk[i]
const codeUnit = item.charCodeAt(0)
if (this.#pendingHighSurrogate !== null) {
const highSurrogate = this.#pendingHighSurrogate
this.#pendingHighSurrogate = null
if (0xdc00 <= codeUnit && codeUnit <= 0xdfff) {
finalChunk += highSurrogate + item
continue
}
finalChunk += "\uFFFD"
}
if (0xd800 <= codeUnit && codeUnit <= 0xdbff) {
this.#pendingHighSurrogate = item
continue
}
if (0xdc00 <= codeUnit && codeUnit <= 0xdfff) {
finalChunk += "\uFFFD"
continue
}
finalChunk += item
}
if (finalChunk) {
controller.enqueue(this.#handle.encode(finalChunk))
}
},
flush: (controller) => {
// https://encoding.spec.whatwg.org/#encode-and-flush
if (this.#pendingHighSurrogate !== null) {
controller.enqueue(new Uint8Array([0xef, 0xbf, 0xbd]))
}
},
});
get encoding() {
return this.#handle.encoding
}
get readable() {
return this.#transform.readable
}
get writable() {
return this.#transform.writable
}
get [Symbol.toStringTag]() {
return 'TextEncoderStream'
}
}
/**
* TextDecoderStream polyfill based on Node.js' implementation https://github.com/nodejs/node/blob/3f3226c8e363a5f06c1e6a37abd59b6b8c1923f1/lib/internal/webstreams/encoding.js#L121-L200 (MIT License)
*/
export class TextDecoderStream {
#handle: TextDecoder
#transform = new TransformStream({
transform: (chunk, controller) => {
const value = this.#handle.decode(chunk, {stream: true})
if (value) {
controller.enqueue(value)
}
},
flush: controller => {
const value = this.#handle.decode()
if (value) {
controller.enqueue(value)
}
controller.terminate()
}
})
constructor(encoding = "utf-8", options: TextDecoderOptions = {}) {
this.#handle = new TextDecoder(encoding, options)
}
get encoding() {
return this.#handle.encoding
}
get fatal() {
return this.#handle.fatal
}
get ignoreBOM() {
return this.#handle.ignoreBOM
}
get readable() {
return this.#transform.readable
}
get writable() {
return this.#transform.writable
}
get [Symbol.toStringTag]() {
return "TextDecoderStream"
}
}Both basically just TS port I use in my projects with Bun. |
|
How will an external package use the pollyfill?
…On Sat, Jun 1 2024 at 05:26, Nick K. < ***@***.*** > wrote:
Here are polyfills you can use based off the Node.js' implementation while
you waiting for Bun to catch up:
/**
* TextEncoderStream polyfill based on Node.js' implementation
https://github.com/nodejs/node/blob/3f3226c8e363a5f06c1e6a37abd59b6b8c1923f1/lib/internal/webstreams/encoding.js#L38-L119
(MIT License)
*/
export class TextEncoderStream {
#pendingHighSurrogate: string | null = null
#handle = new TextEncoder ( )
#transform = new TransformStream < string , Uint8Array > ( {
transform : ( chunk , controller ) => {
// https://encoding.spec.whatwg.org/#encode-and-enqueue-a-chunk
chunk = String ( chunk )
let finalChunk = ""
for ( const item of chunk ) {
const codeUnit = item. charCodeAt ( 0 )
if ( this. #pendingHighSurrogate !== null ) {
const highSurrogate = this. #pendingHighSurrogate
this. #pendingHighSurrogate = null
if ( codeUnit >= 0xDC00 && codeUnit <= 0xDFFF ) {
finalChunk += highSurrogate + item
continue
}
finalChunk += "\uFFFD"
}
if ( codeUnit >= 0xD800 && codeUnit <= 0xDBFF ) {
this. #pendingHighSurrogate = item
continue
}
if ( codeUnit >= 0xDC00 && codeUnit <= 0xDFFF ) {
finalChunk += "\uFFFD"
continue
}
finalChunk += item
}
if ( finalChunk ) {
controller. enqueue ( this. #handle. encode ( finalChunk ) )
}
} ,
flush : controller => {
// https://encoding.spec.whatwg.org/#encode-and-flush
if ( this. #pendingHighSurrogate !== null ) {
controller. enqueue ( new Uint8Array ( [ 0xEF , 0xBF , 0xBD ] ) )
}
}
} )
get encoding ( ) {
return this. #handle. encoding
}
get readable ( ) {
return this. #transform. readable
}
get writable ( ) {
return this. #transform. writable
}
get [ Symbol. toStringTag ] ( ) {
return "TextEncoderStream"
}
}
/**
* TextDecoderStream polyfill based on Node.js' implementation
https://github.com/nodejs/node/blob/3f3226c8e363a5f06c1e6a37abd59b6b8c1923f1/lib/internal/webstreams/encoding.js#L121-L200
(MIT License)
*/
export class TextDecoderStream {
#handle: TextDecoder
#transform = new TransformStream ( {
transform : ( chunk , controller ) => {
const value = this. #handle. decode ( chunk , { stream : true } )
if ( value ) {
controller. enqueue ( value )
}
} ,
flush : controller => {
const value = this. #handle. decode ( )
if ( value ) {
controller. enqueue ( value )
}
controller. terminate ( )
}
} )
constructor ( encoding = "utf-8" , options : TextDecoderOptions = { } )
{
this. #handle = new TextDecoder ( encoding , options )
}
get encoding ( ) {
return this. #handle. encoding
}
get fatal ( ) {
return this. #handle. fatal
}
get ignoreBOM ( ) {
return this. #handle. ignoreBOM
}
get readable ( ) {
return this. #transform. readable
}
get writable ( ) {
return this. #transform. writable
}
get [ Symbol. toStringTag ] ( ) {
return "TextDecoderStream"
}
}
Both basically just TS port I use in my projects with Bun.
—
Reply to this email directly, view it on GitHub (
#5648 (comment) ) , or unsubscribe
(
https://github.com/notifications/unsubscribe-auth/AAA6OA4EDI5TVH2CU4MUBVTZFGVXHAVCNFSM6AAAAAA4357CGSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNBTGQYTGNJWHE
).
You are receiving this because you commented. Message ID: <oven-sh/bun/issues/5648/2143413569
@ github. com>
|
|
I mean, you can use it in your app, can't you? It probably relies on // Add those polyfills to globalThis before you import `@google/generative-ai`
globalThis.TextEncoderStream ||= TextEncoderStream
globalThis.TextDecoderStream ||= TextDecoderStreamBut for library this might not be a good option. |
You can simplify the implementation by simply using |
|
If you read the spec, you'll see TextDecoderStream and TextEncoderStream are not subclasses of TransformStream. |
IMO, the spec only defines what a correct implementation should look like, and it demonstrates that "Correct TextEncoderStream and TextDecoderStream could be implemented like these", but the runtime doesn't have to implement them in the exact same way. That's to say, the spec doesn't enforce what you must do to implement this. IMHO, as long as the implementation exposes the required APIs and fields, and the behaviors are the same, then the implementation is spec-compliant. Also, take a look at the spec:
The spec requires |
Also, my implementation is based on Google, Inc.'s work. And it is not only Google does that, QuickJS also uses |
|
Tried to use the polyfill proposed by @octet-stream above with TRPC client: Getting this error: |
|
@Jarred-Sumner what is the priority on this? Just to have an idea to when expect for this to be released. |
|
@trpc/server 11.0.0-rc.361 onwards is broken on bun because of this. This makes bun unusable for trpc backends with react-query, since it relies on trpc 11+. Workaround is using version 11.0.0-rc.359 EDIT: Only with httpBatchStreamLink |
You can use this polyfill to make it work comment |
|
petition for adding the streams |
|
The (As a workaround I used tsx to run the script in Node.js instead of Bun for now.) |
|
i signed the petition |
|
if you had this issue with tests you need to create a setup.ts/js file // polyfills here...
if (typeof globalThis.TextDecoderStream === 'undefined') {
// @ts-ignore
globalThis.TextDecoderStream = TextDecoderStream;
}
// Ensure the polyfill is applied
const ensureTextDecoderStream = () => {
if (typeof globalThis.TextDecoderStream === 'undefined') {
throw new Error('TextDecoderStream is not defined after polyfill');
}
};
ensureTextDecoderStream();
export { };bun.config.json {
"test": {
"preload": [
"./setup.ts"
],
"include": [
"src/**/*.test.ts"
]
}
} |
|
Jarred has confirmed that if the petition gets 100 signatures, they'll implement this. Please sign!!! Here is the link: |
|
We reached 100 signatures 🎉 |
|
Will @Jarred-Sumner deliver? |
no, but @dylan-conway will :) |
|
Thanks to @dylan-conway for implementing this! Am I correct in that, once #13115 would be merged, bun can now be used for running the Next.js dev server (using this command: |
I am having the same issue |
|
For anyone wondering why reopened, see #13151 |
|
@dylan-conway @Jarred-Sumner I'm still seeing Nix even added bun 1.1.29 support so wondering if others have complained about this. |
|
Same issue as @farezv. Did you find a fix or is there another issue here tracking this? |
and others
TextDecoderStreamandTextEncoderStreamare missingThe text was updated successfully, but these errors were encountered: