Errors for segments with size > 2147483647 bytes

[lblatchford]

When attempting to parse a NITF 2.1 file with a large image segment the following error occurred "Parse Error: Failed to populate ImageSegment[1]. Cause Parse Error: Cannot convert '2516582400' from Long type to Int (Value '2516582400' is out of range for xs:int"

The PayloadLength is xs:int. When the schema file is changed to make PayloadLength xs:unsignedLong, another error occurs:

"Failed to convert: Parse Error: Failed to populate ImageSegment[1]. Cause: Parse Error: Choice dispatch branch failed: List( Parse Error: Length for xs:hexBinary exceeds maximum of 2147483647 bytes: 2516582400".

I believe the xs:hexBinary type should have an unbounded maximum, so the reason for this error is unclear.

The test data would have to be approved for release by DISA. The test data request form and information about the files to be requested can be provided upon request.

[stevedlawrence]

Under the hood, Daffodli uses a Java array to store hex binary bytes which has a maximum size of 2³¹-1 bytes (i.e. 2 GB).

If hexBinary larger than 2GB is needed, it's probably better to use the Daffodil Binary Large Object extension. To use this, you can change the type from xs:hexBinary to xs:anyUri and add the property dfdlx:objectKind="bytes". This will cause Daffodil to write the raw bytes to a temporary file and put the path to that file in the infoset instead of hex binary. This is much more efficient compared xs:hexBinary and has a much larger size limit, I believe 2⁶³-1 bits which is something like 10^18 bytes.

[lblatchford]

Steve - thanks for the quick response. While I was able to get the extension to work, on larger files I'm now getting an Abort: Usage error: startBitAddress0b.>=(0) I can't provide the NITF file in question but I've seen the same error with various large image files. I'll log a Daffodil bug for this. Lara From: Steve Lawrence ***@***.***> Sent: Thursday, February 2, 2023 10:13 AM To: DFDLSchemas/NITF ***@***.***> Cc: Lara Blatchford ***@***.***>; Author ***@***.***> Subject: Re: [DFDLSchemas/NITF] Errors for segments with size > 2147483647 bytes (Issue #16) Under the hood, Daffodli uses a Java array to store hex binary bytes which has a maximum size of 231-1 bytes (i.e. 2 GB). If hexBinary larger than 2GB is needed, it's probably better to use the Daffodil Binary Large Object extension<https://usg02.safelinks.protection.office365.us/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FDAFFODIL%2FProposal%253A%2BBinary%2BLarge%2BObjects&data=05%7C01%7Clara.blatchford%40nteligen.com%7Ce174f0e4c9b745e919c808db05300352%7C379c214c5c944e86a6062d047675f02a%7C0%7C0%7C638109476009598619%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Wr0rgVp39YvYWV0OZFYbf%2B0yd0%2BSBXfUSeWnGVTflLo%3D&reserved=0>. To use this, you can change the type from xs:hexBinary to xs:anyUri and add the property dfdlx:objectKind="bytes". This will cause Daffodil to write the raw bytes to a temporary file and put the path to that file in the infoset instead of hex binary. This is much more efficient compared xs:hexBinary and has a much larger size limit, I believe 263-1 bits which is something like 10^18 bytes. - Reply to this email directly, view it on GitHub<https://usg02.safelinks.protection.office365.us/?url=https%3A%2F%2Fgithub.com%2FDFDLSchemas%2FNITF%2Fissues%2F16%23issuecomment-1413903059&data=05%7C01%7Clara.blatchford%40nteligen.com%7Ce174f0e4c9b745e919c808db05300352%7C379c214c5c944e86a6062d047675f02a%7C0%7C0%7C638109476009598619%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=%2Fh6TF7uCI6NjaQF1zr55B69dM%2BdQFeF1CV8Dmc2O4VE%3D&reserved=0>, or unsubscribe<https://usg02.safelinks.protection.office365.us/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAQ3OWUQ3KYFON6QI76PM7L3WVPFIZANCNFSM6AAAAAAUPECUFU&data=05%7C01%7Clara.blatchford%40nteligen.com%7Ce174f0e4c9b745e919c808db05300352%7C379c214c5c944e86a6062d047675f02a%7C0%7C0%7C638109476009598619%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=xWvOP88DyPHSKCzfge%2Fv%2B%2FkX2uUQGGdwsOCRl5baFpo%3D&reserved=0>. You are receiving this because you authored the thread.Message ID: ***@***.******@***.***>>