Defining new data type to represent ByteString

[irakledibm]

It is common to transport some data, such as payloads etc in form of byte array, without converting it. To cover those use cases we need to add new data type "bytestring_t" to represent immutable sequence of bytes. We can add it as subset of String with base64 encoding or as a base type.

[mavam]

Raw bytes as dedicated type makes a lot of sense to me. How to represent it properly is almost orthogonal, i.e., forcing Base64 just because the default medium is ASCII doesn't do justice to the type system.

[rroupski]

BTW, to use schema conventions, the type name should be byte_string_t

[irakledibm]

Example: datetime_t, hostname_t, timestamp_t, username_t. ByteString as a data type exists in few technologies (Protobuf for example)

[irakledibm]

Any suggestions around encoding, besides base64?

[davaya]

Some people prefer hex because they think it is more "readable". I'd choose URL-safe Base64 by default, but there is no reason to lock in a single encoding since it is orthogonal to the type.

Base64 is more confusable with ASCII than hex when examining chosen serialized data; any text is valid whereas only special hex values ("DEADBEEF") can be mistaken for text. Strong typing (byte_string_t vs string) prevents misinterpreting the deserialized value of "OpenCybersecuritySchemaFramework".

[mavam]

Some people prefer hex because they think it is more "readable". I'd choose URL-safe Base64 by default, but there is no reason to lock in a single encoding since it is orthogonal to the type.

100% on encoding vs. type.

The consequence is that the standard must either prescribe one specific encoding (implicit), or expose a mechanism to let a user choose one (explicit).

There's also a utility spectrum in the choice of encoding. If we just hex-encode bytes, then we get basically a strings-like behavior for free, i.e., can understand the ASCII in there without decoding. For example, just encoding non-ASCII bytes would make it this string mostly comprehensible: "I am a \x2a\x2b string". When using Base64, this opportunity is lost.

[davaya]

Subtypes are a mechanism to explicitly choose a text representation of binary data. A 4 byte IPv4 address normally uses decimal with dots but could use base64 or hex, a 16 byte IPv6 address normally uses hex with colons plus a zero-defaulting scheme but could be base64 or undelimited hex, a 16 byte UUID normally uses hex with hyphens but could be base64, a 256 byte public key can use base64 or hex-upper or hex-lower or ascii-escapes while a 256 byte hash could use a different choice (fingerprints are usually hex-lower).

Implicit signaling doesn't work with hex: "deadbeef" is still ambiguous if hex and ascii-escapes are both possible.

I suggest that the default text representation of byte strings be base64, with subtypes explicitly signaling idiomatic and generic alternatives.

[irakledibm]

So, basically bytestring_t by default is base64, then we may have sub-types if needed as base128_t, leb128_t, hex_t, etc?

[davaya]

Yes.

It works like JSON Schema format that not only constrains an unrestricted "string" type to support "semantic identification of certain kinds of string values", but also includes a semantic-specific bytestring_t to string conversion.

So you could define generic base128_t etc types, and could also define a semantic type "fingerprint_t" that uses hex_t string format or "mac_addr_t" that displays 6 MAC bytes as a colon-separated hex string: "2f:31:49:cc:04:92".

MAC addresses (EUIs) have to be subtypes of bytestring_t, not subtypes of string, because IEEE defines them in terms of bytes, and the same bytes can be displayed as different strings. 2f:31:49:cc:04:92 must have the same value as 2f3.149.cc0.492, which is not not true if they are text string types.