Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ak.forms.form.index_to_dtype is probably wrong: should probably be native, not little-endian #3356

Open
jpivarski opened this issue Dec 19, 2024 · 0 comments
Labels
bug (unverified) The problem described would be a bug, but needs to be triaged

Comments

@jpivarski
Copy link
Member

Version of Awkward Array

HEAD

Description and code to reproduce

Compare

_primitive_to_dtype_dict = {
"bool": np.dtype(np.bool_),
"int8": np.dtype(np.int8),
"uint8": np.dtype(np.uint8),
"int16": np.dtype(np.int16),
"uint16": np.dtype(np.uint16),
"int32": np.dtype(np.int32),
"uint32": np.dtype(np.uint32),
"int64": np.dtype(np.int64),
"uint64": np.dtype(np.uint64),
"float32": np.dtype(np.float32),
"float64": np.dtype(np.float64),
"complex64": np.dtype(np.complex64),
"complex128": np.dtype(np.complex128),
"datetime64": np.dtype(np.datetime64),
"timedelta64": np.dtype(np.timedelta64),
}

which sets the dtype for each primitive category to the native-endianness for the machine (np.dtype(np.float64) will be big-endian on big-endian machines and little-endian on little-endian machines) with

index_to_dtype: Final[dict[str, DType]] = {
"i8": np.dtype("<i1"),
"u8": np.dtype("<u1"),
"i32": np.dtype("<i4"),
"u32": np.dtype("<u4"),
"i64": np.dtype("<i8"),
}

which sets the dtype for each index category to little-endian. This should probably be native-endian, too. I'm 90% sure of it.

The problem is that we'd like to test it before we change it. Is there any way we can test Awkward on a big-endian machine? Maybe in a qemu emulation? (Such a test would probably reveal a lot.)

On the flip side, the fact that it's so hard to find a big-endian machine these days means that this issue would rarely be observed. (Similar to 32-bit testing...)

@jpivarski jpivarski added the bug (unverified) The problem described would be a bug, but needs to be triaged label Dec 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug (unverified) The problem described would be a bug, but needs to be triaged
Projects
None yet
Development

No branches or pull requests

1 participant