Distributive schemas

[frenchy64]

:merge distributes over :multi in the following manner:

(m/deref
 [:merge
  [:map [:x :int]]
  [:multi {:dispatch :y}
   [1 [:map [:y [:= 1]]]]
   [2 [:map [:y [:= 2]]]]]]
 {:registry registry})
; => [:multi {:dispatch :y}
;     [1 [:map [:x :int] [:y [:= 1]]]]
;     [2 [:map [:x :int] [:y [:= 2]]]]]

(m/deref
 [:merge
  [:multi {:dispatch :y}
   [1 [:map [:y [:= 1]]]]
   [2 [:map [:y [:= 2]]]]]
  [:map [:x :int]]]
 {:registry registry})
; => [:multi {:dispatch :y}
;     [1 [:map [:y [:= 1]] [:x :int]]]
;     [2 [:map [:y [:= 2]] [:x :int]]]]

[ikitommi]

I really like the approach here. What is the reasoning for this working with :or and :orn too?

[ikitommi]

what happens if there is non-map here? gets replaced? I would not expect it to happen.

[ikitommi]

maybe this should work only with :multi 🤔

[frenchy64]

I really like the approach here.

Seems powerful! I'm still grokking the implications and applicability myself...see below.

This is something Typed Clojure does to convert

(t/Merge '{:a t/Int} (t/U '{:b t/Int} '{:c t/Int})

to

(t/U '{:a t/Int :b t/Int} '{:a t/Int :c t/Int})

but such transformations are not so obviously applicable to Malli because of m/parse and error messages relying on named paths.

What is the reasoning for this working with :or and :orn too?

I think all 3 are kinds of disjunctions so it seemed natural to at least try it. I also tried :and which might also be useful in some (non-map) contexts.

:or + :map is not performant but I was also curious if there are other schemas analogous to :merge for which [<COMBINE> X [:or Y Z]] => [:or [<COMBINE> X Y] [<COMBINE> X Z]] makes sense.

what happens if there is non-map here? gets replaced? I would not expect it to happen.

:merge essentially wraps the distributed schema, so it inherits the semantics of merge in that case.

:merge does some weird things if you give it non-maps.

(m/deref
  [:merge
   int?
   [:map [:x :int]]]
  options)
;=> [:map [:x :int]]

(m/deref
  [:merge
   [:map [:x :int]]
   int?]
  options)
;=> int?

Perhaps it should error in these cases? FWIW Typed Clojure implements t/Merge via a protocol AssocableType which knows how to assoc a particular entry onto a map. Merging then happens in an analogous way to clojure.core/merge, but on the type level (pour entries from right-to-left one at a time).

My first impression of mu/merge is that its implementation is too concrete and could be extensible.

Anyway, because of :merge's behavior the distributive rule drops the schema on the left if only one of is a map:

(m/deref
  [:merge
   [:or
    int?
    boolean?]
   [:map [:x :int]]]
  options)
=> 
[:or [:map [:x :int]] [:map [:x :int]]]

(m/deref
  [:merge
   [:map [:x :int]]
   [:or
    int?
    boolean?]]
  options)
=>
[:or int? boolean?]

The idea behind this PR is to implement these distributive rules as abstractly as possible to open the door for other non-map applications, i.e.:

[<*> X [<+> Y Z]]
=>
[<+> [<*> X Y] [<*> X Z]]

[<*> [<+> Y Z] X]
=>
[<+> [<*> Y X] [<*> Z X]]

Going with that theme, here :merge is a kind of <*> schema that doesn't quite fit in the abstraction because of its own inconsistent semantics around non-maps.

maybe this should work only with :multi 🤔

For maps it seems like the most obvious winning combination. Still looking for other applications, but the way it's implemented we can always add others later.

We could also support wrapper schemas like :schema such as

[:merge
 X
 [:schema [:or Y Z]]]

using the same approach as m/-function-schema?.

[ikitommi]

but such transformations are not so obviously applicable to Malli because of m/parse and error messages relying on named paths.

You had a simplifying protocol suggestion for the generators earlier, if such a thing would be in the core of malli, this would be ok I guess. "here's my schema, transform it something else", e.g. user would not expect paths to be the same.

:merge does some weird things if you give it non-maps.

I was following the meta-merge functionality here:

(require '[meta-merge.core :as mm])

(mm/meta-merge 1 {:x 1})
; => {:x 1}

(mm/meta-merge {:x 1} 1)
; => 1

It would be a breaking change to start failing with. non-entry schemas.

My first impression of mu/merge is that its implementation is too concrete and could be extensible.

Indeed, it's concrete. But adding support for :multi is a great first step.

PR Looks good to me, one FIXME I se.

[frenchy64]

It would be a breaking change to start failing with. non-entry schemas.

Agreed. A non-starter.

I was following the meta-merge functionality here:

It seems like it's all working as intended then. Applying the distributive rule would only be surprising if you expected :merge to have different semantics in the first place.

[ikitommi]

About different registries in merge, e.g. distribute-registry-test I thought this as "last one wins" here (like meta-merge):

(def schema1
  (m/schema
   [:map ::x ::y]
   {:registry (mr/composite-registry
               (m/default-schemas)
               {::x :int
                ::y :int})}))

(def schema2
  (m/schema
   [:map ::x ::z]
   {:registry (mr/composite-registry
               (m/default-schemas)
               {::x :string
                ::z :string})}))

; looks odd
(mu/merge schema1 schema2)
;[:map
; [:user/x :string]
; [:user/y :user/y]
; [:user/z :user/z]]

(m/validate *1 {::x "1", ::y 1, ::z "1"})
; => true

Not sure what would be a better way to do this, ideas most welcome. maybe a separate issue as this already solves the :multi merge thing, which is big.

[frenchy64]

About different registries in merge, e.g. distribute-registry-test I thought this as "last one wins" here (like meta-merge):

Yes it works perfectly fine until you deserialize the schema at that point.

maybe a separate issue as this already solves the :multi merge thing, which is big.

#1088

[frenchy64]

Yes it works perfectly fine until you deserialize the schema at that point.

I take that back, this schema cannot be dereferenced because ::y is not in scope for some reason.

[:merge
 [:map
  {::y boolean?}
  [:y ::y]]
 [:map]]

This can be duplicated via the new distributive rule, also saying ::y is not in scope:

[:merge
 [:map
  {::y boolean?}
  [:y ::y]]
 [:multi {:dispatch :z}
  [1 [:map]]]]

[frenchy64]

I documented the order of operations for distributing merge. I pushed in the middle of some work and I to fix the tests. Otherwise it's feeling pretty good.

[frenchy64]

I don't know which order to apply the left/right distributive rules makes sense or is the most intuitive. Currently it's right then left. Here's the difference.

;; right-distributive then left-distributive
[:merge
 [:multi {:dispatch :y}
  [1 [:map [:y [:= 1]]]]
  [2 [:map [:y [:= 2]]]]]
 [:multi {:dispatch :z}
  [3 [:map [:z [:= 3]]]]
  [4 [:map [:z [:= 4]]]]]]
=>
[:multi {:dispatch :y}
 [1 [:multi {:dispatch :z}
     [3 [:map [:y [:= 1]] [:z [:= 3]]]]
     [4 [:map [:y [:= 1]] [:z [:= 4]]]]]]
 [2 [:multi {:dispatch :z}
     [3 [:map [:y [:= 2]] [:z [:= 3]]]]
     [4 [:map [:y [:= 2]] [:z [:= 4]]]]]]]

;; left-distributive then right-distributive
[:merge
 [:multi {:dispatch :y}
  [1 [:map [:y [:= 1]]]]
  [2 [:map [:y [:= 2]]]]]
 [:multi {:dispatch :z}
  [3 [:map [:z [:= 3]]]]
  [4 [:map [:z [:= 4]]]]]]
=>
[:multi {:dispatch :z}
 [3 [:multi {:dispatch :y}
     [1 [:map [:y [:= 1]] [:z [:= 3]]]]
     [2 [:map [:y [:= 2]] [:z [:= 3]]]]]]
 [4 [:multi {:dispatch :y}
     [1 [:map [:y [:= 1]] [:z [:= 4]]]]
     [2 [:map [:y [:= 2]] [:z [:= 4]]]]]]]

I think it mainly affects m/parse:

;; right-distributive then left-distributive
(m/parse
  [:merge
   [:multi {:dispatch :y}
    [1 [:map [:y [:= 1]]]]
    [2 [:map [:y [:= 2]]]]]
   [:multi {:dispatch :z}
    [3 [:map [:z [:= 3]]]]
    [4 [:map [:z [:= 4]]]]]]
  {:y 1 :z 3}
  options)
=> [1 [3 {:y 1, :z 3}]]

;; left-distributive then right-distributive
(m/parse
  [:merge
   [:multi {:dispatch :y}
    [1 [:map [:y [:= 1]]]]
    [2 [:map [:y [:= 2]]]]]
   [:multi {:dispatch :z}
    [3 [:map [:z [:= 3]]]]
    [4 [:map [:z [:= 4]]]]]]
  {:y 1 :z 3}
  options)
=> [3 [1 {:y 1, :z 3}]]

[ikitommi]

this in really, really good. I'll fix the on exception. 🥇

[ikitommi]

TODO.

[ikitommi]

I don't know which order to apply the left/right distributive rules makes sense or is the most intuitive. Currently it's right then left. Here's the difference.

The current makes more sense to me.

[ikitommi]

Actually, the error was not needed, -children of :multi always returns size 3 vector of key properties value. The old code would have failed with merging with entry properties:

[:multi {:dispatch :y}
  [1 {:entry "properties"} [:map [:y [:= 1]]]]
  [2 [:map [:y [:= 2]]]]]

[ikitommi]

be2c2be

[ikitommi]

Run some sample tests with this, really happy with it, thanks!