Support for writing integer, float and other type fields

[suvarchal]

From api, it seems multio currently supports only writing fields of type double(?) :

multio/src/multio/api/multio_c.cc

Lines 299 to 308 in dc5ec7e

	int multio_write_field(multio_handle_t* mio, multio_metadata_t* md, const double* data, int size) {
	return wrapApiFunction([mio, md, data, size]() {
	ASSERT(mio);
	ASSERT(md);
	eckit::Buffer field_vals{reinterpret_cast<const char*>(data), size * sizeof(double)};
	mio->dispatch(*md, std::move(field_vals), Message::Tag::Field);
	});
	}

This corroborates with examples like nemo-replay that all fields written seem to be of type double.

I can imagine a need (not just imagine i want :) ) to store fields which are say float/singlep or integer type, which seems missing right now but i am sure multio can be easily extended for that. But nevertheless I wonder if there is any grib thing/magic that facilitates using doubles but data types are automatically converted? Grib is still a territory i have very little understanding, but the issue is indeed true for raw encoding.

While it is easy to add more functions for this purpose with signature like int multio_write_field_integer(multio_handle_t* mio, multio_metadata_t* md, const int* data, int size) in api/multo_c.cc but number of functions for all types a user desires can be long and more over can have fragmentation over names and need to write corresponding bindings in Fortran. So i thought why not use a generic write in c-api like (note size is now sizeof instead of size):

int multio_write_field_generic(multio_handle_t* mio, multio_metadata_t* md, const void* data, size_t size) {
    return wrapApiFunction([mio, md, data, size]() {
        ASSERT(mio);
        ASSERT(md);

        eckit::Buffer field_vals{reinterpret_cast<const char*>(data), size};

        mio->dispatch(*md, std::move(field_vals), Message::Tag::Field);
    });
}

the advantage of such generic function is that I used this to bind for multiple types in Fortran api (and in corresponding wrapping interfaces) for instance:

                bind(c, name='multio_write_field_generic')
            use, intrinsic :: iso_c_binding
            implicit none
            type(c_ptr), intent(in), value :: handle
            type(c_ptr), intent(in), value :: metadata
            integer, dimension(*), intent(in) :: data
            integer(8), intent(in), value :: size
            integer(c_int) :: err
        end function
    function multio_write_field_integer(handle, metadata, data, size) result(err)
        class(multio_handle), intent(inout) :: handle
        class(multio_metadata), intent(in) :: metadata
        integer :: err

        integer, dimension(*), intent(in) :: data
        integer(8), intent(in), value :: size
        err = c_multio_write_field_integer(handle%impl, metadata%impl, data, size)
    end function

and this works.

multio_fapi.f90 can have these bindings for commonly used types like single-precision or integers but it can also be delegated to the users of the library how they may chose to bind, for instance it enables Fortran binding to custom derived types.

I would like to know your take on this and if you find it a fit I could happily author a PR based on what you think, but wouldn't mind you doing it as well.

[suvarchal]

btw i am curious if there is any reason to use int size instead of size_t or unsigned int for arguments in relevant functions in c-api: multio_c.cc ?

[dsarmany]

Support for single-precision (32-bit) floating-point numbers is indeed important and it is currently being developed. It will be merged to the develop merge at the beginning (January-February) of 2023. Support for other types, such as half-precision (16-bit) floating-point numbers may also be considered in the future.

multio indeed stores fields in a generic way internally in a message, using eckit::Buffer. However, there are a number of multio actions that rely on the knowledge of the field's underlying type. One way or another, that information need be communicated to multio via the interface. It can either be done by separate interface functions, or by calling the same function and specifying the type in the metadata. In the case of the multio_write_field_<type> functions, we have chosen to express that information explicitly in the interface. In any case, in fortran, it is still possible to create a single interface function and overload on type to simplify user code (in C it isn't).

Without any type information, the number of actions available would be quite small, mainly limited to transport and output.