FIX: Set file.checksum_md5 non optional

[tnatt]

PR to set file.checksum_md5 non optional. It was non optional in the legacy schema, and was introduced in https://github.com/equinor/fmu-dataio/pull/512/files, probably by mistake.

Closes #835

[mferrera]

I think making this field required is important, but I'm having second thoughts about an empty string being a valid checksum. I think there is also a place for stricter checks here.

A valid md5 checksum usually has 32 characters, 0-9a-f. We should probably make this a validation check, as an invalid checksum compromises data quality

Maybe for reminding me, and for posterity, under what circumstances are we giving an empty string checksum?

[tnatt]

I think making this field required is important, but I'm having second thoughts about an empty string being a valid checksum. I think there is also a place for stricter checks here.

A valid md5 checksum usually has 32 characters, 0-9a-f. We should probably make this a validation check, as an invalid checksum compromises data quality

Maybe for reminding me, and for posterity, under what circumstances are we giving an empty string checksum?

The empty string will only occur in the case that a user directly uses generate_metadata() and explicitly sets compute_md5=False (default is True). And then dumps this metadata themselves to disk. Most users will use this function indirectly through the export function. The export function does not have this argument, and we hardcode it to be True. Meaning all exports will have checksum_md5 in the metadata.

fmu-dataio/src/fmu/dataio/dataio.py

Line 930 in acb2f25

metadata = self.generate_metadata(obj, compute_md5=True, **kwargs)

Before we can really ensure we don't create an empty string in the metadata we need to deprecate the compute_md5 argument in generate_metadata(). ..and I don't really see any big problem to make the change to always compute a checksum, and then add a deprecation warning saying the functionality is no longer supported and that the argument will be removed in the future. I don't think any users actively want to keep the checksum out of the metadata.

What do you think 🙂 ?

[mferrera]

Yes, let's do it 👍 there is no good reason to not generate a checksum. And in 8 months time if we change our mind about that and land back on this PR, at least we will have some context to our decision 😄

[perolavsvendsen]

Could be that this is needed when no files are actually materialized to disk? I.e. in our aggregation service (which at the moment does not use fmu-dataio but absolutely should). @equinor-ruaj may have more context?

[mferrera]

@tnatt pointed out on Slack that we have #552 which touches on this issue. Resolving that would also probably result in a reasonable wall-time performance boost.