Import runtime Call / Storage metadata without import the whole runtime

[xlc]

In order to use XCM::Transact, we need to know how to encode the call on destination chain. The best way should be just import the Call enum from runtime. But we cannot do it because last time we tried, it tries to import the whole Rococo runtime into our parachain runtime, which obviously wouldn't work.

We should have some way to allow others to import some generated types from runtime, without import the whole thing.

Not exactly sure how it can be implemented though, all the generic makes thing harder. One potential solution is reconstruct the enum from metadata using polkadot.js.

[bkchr]

Cc @ascjones @tomusdrw

[shaunxw]

I'll post more details here.

We'd like to implement helpers in orml-xcm-utility, to help init/accept HRMP open channel request (It's helpful to all parachains who want to send Xcm). Like this:

#[pallet::call]
impl<T: Config> Pallet<T> {
    //TODO: weight
    #[pallet::weight(10)]
    #[transactional]
    pub fn hrmp_init_open_channel(
        origin: OriginFor<T>,
        recipient: ParaId,
        proposed_max_capacity: u32,
        proposed_max_message_size: u32,
    ) -> DispatchResultWithPostInfo {
        ensure_root(origin)?;

        let call = rococo_runtime::Call::Hrmp(HrmpCall::hrmp_init_open_channel(
            recipient,
            proposed_max_capacity,
            proposed_max_message_size,
        )).encode();

        let msg = Xcm::Transact {
            origin_type: OriginKind::Native,
            call,
        };

        T::XcmSender::send_xcm(Parent.into(), msg).map_err(|_|
        Error::<T>::SendXcmFailed)?;

        Ok(().into())
    }
}

After integrated this pallet into Mandala parachain runtime, build fails with error:

warning: Linking globals named 'Core_version': symbol multiply defined!
error: failed to load bc of "rococo_runtime-ddaa0aa381fdea09.rococo_runtime.504vydv1-cgu.10.rcgu.o":
error: aborting due to previous error; 1 warning emitted
error: could not compile mandala-runtime-wasm

It seems rococo runtime would be included into our runtime. Even if this error was fixable, the build wasm would be too large.

We're currently doing the encoding via PolkadotJS (I wrote a guide), though it doesn't return the exact encoded call and needs workaround.

I guess this issue could be fixed in the similar way how PolkadotJS do it with metadata.

[ascjones]

We are currently working on producing rich type metadata using scale-info for substrate runtimes, which will allow generation of SCALE compatible types for interacting with a chain.

However this won't be available in the short term, but what you could do in the meantime is to manually construct a Call enum with just the pallets/calls you want to use, making sure the types match up, and the index of the variant matches with the target runtime e.g.

#[derive(Encode, Decode)]
pub enum Call {
    #[codec(index = "10")] // the index should match the position of the module in `construct_runtime!`
    Hrmp(HrmpCall),
}

#[derive(Encode, Decode)]
pub enum HrmpCall {
    #[codec(index = "1")] // the index should match the position of the dispatchable in the target pallet
    hrmp_init_open_channel(u32, u32, u32),
}

This obviously would become cumbersome the more calls and pallets you use, but I think it should allow you to make progress while waiting for the new metadata format.

[tomusdrw]

We were also discussing generating such standalone (no dependencies) enum as part of runtime compilation. Perhaps with frame-v2 macros this would be easy enough to implement? CC @thiolliere ?

[ascjones]

My thinking is that the metadata should contain enough information to generate SCALE compatible runtime types in any language, e.g. like in this (out of date) prototype for generating Rust types.

[tomusdrw]

My thinking is that the metadata should contain enough information to generate SCALE compatible runtime types in any language, e.g. like in this (out of date) prototype for generating Rust types.

Yes, that would be awesome! I was thinking of a temporary solution before we add scale-info to the metadata, but indeed chameleon approach is the way to go long term.

[olanod]

@ascjones what are the main blockers to not have the type info available in the short term? I saw the frame-metadata crate already has the V13 version of the metadata that includes the type registry and was happy it could land soon to use in a project that would rely on it. I thought it would only be matter of updating the macros to include the Derive(TypeInfo) for all runtime types. Could there be a workaround? Runtimes manually implementing/deriving the type info and expose it via RPC or something?

[gui1117]

We were also discussing generating such standalone (no dependencies) enum as part of runtime compilation. Perhaps with frame-v2 macros this would be easy enough to implement? CC @thiolliere ?

I think the scale-info is the only solution I see. Call enum has field with types defined outside of the pallet macro.

[ascjones]

@ascjones what are the main blockers to not have the type info available in the short term?

On the scale-info side we still need paritytech/scale-info#44, which is done now just needs some testing and reviewing from my side.

I thought it would only be matter of updating the macros to include the Derive(TypeInfo) for all runtime types.

There is that and adding various bounds to traits, and also producing the new metadata format which I have a prototype of in https://github.com/paritytech/substrate/compare/aj-metadata-vnext. However that is dependent on converting all the pallets to the new framev2 macros, see #7882. If you run that branch then you should be able to produce the v13 metadata for all the modules which have been migrated already.

It would be possible of course to produce the new style metadata with the old macros too, but I don't currently plan to do so.

[tomusdrw]

I think the scale-info is the only solution I see. Call enum has field with types defined outside of the pallet macro.

The question is how much do we want to cover - obviously scale-info will produce all the typing, but I'd argue that the main use case is to be able to call into an external runtime and majority of the types would be quite simple. So as a middle-ground solution we could just accept generic parameters T: Encode for any non-standard type.

If full scale-info is around the corner already, then probably it's best to focus on this though, instead of wasting efforts on some workarounds.

[ascjones]

I'll spend some time this afternoon thinking and experimenting to determine if an interim solution is workable.

[gavofyork]

In order to use XCM::Transact, we need to know how to encode the call on destination chain

Transact is meant for cases where the sender already knows the format of the destination chain. The point of (non-Transact) XCM is to abstract over the chain's capabilities and encodings to allow inter-chain interaction without the need to know its metadata.

[xlc]

polkadot-fellows/xcm-format#22 will address the module index issue.

[stale]

Hey, is anyone still working on this? Due to the inactivity this issue has been automatically marked as stale. It will be closed if no further activity occurs. Thank you for your contributions.