Skip to content
This repository has been archived by the owner on Jan 12, 2019. It is now read-only.

ES automatically creates mapping for each new object #178

Open
manbradcalf opened this issue Dec 10, 2017 · 0 comments
Open

ES automatically creates mapping for each new object #178

manbradcalf opened this issue Dec 10, 2017 · 0 comments

Comments

@manbradcalf
Copy link

manbradcalf commented Dec 10, 2017

I have an object stored like so in FB:

"users" : {
      "0" : {
      "bio" : "Consectetur ut consectetur eiusmod sunt ipsum. Laboris velit ad et magna tempor deserunt. Sint commodo voluptate aliquip...",
      "citystate" : "Driftwood,Wisconsin",
      "email" : "[email protected]",
      "events" : [ {
        "date" : "2017-02-11",
        "eventname" : "do",
        "id" : 12
      }, {
        "date" : "2014-01-08",
        "eventname" : "esse",
        "id" : 14
      }, {
        "date" : "2017-03-24",
        "eventname" : "commodo",
        "id" : 71
      }, {
        "date" : "2017-10-18",
        "eventname" : "ad",
        "id" : 12
      }, {
        "date" : "2014-12-10",
        "eventname" : "id",
        "id" : 4
      }, {
        "date" : "2014-08-11",
        "eventname" : "dolor",
        "id" : 46
      }, {
        "date" : "2014-07-17",
        "eventname" : "nulla",
        "id" : 21
      } ],
      "phone" : "(811) 459-2045",
      "picture" : "https://loremflickr.com/320/240/music",
      "tags" : [ "occaecat", "tempor", "ex", "adipisicing", "nulla", "ea", "culpa" ],
      "username" : "eu"
    }...

When this object is initially created wholesale, firebase renders the "events" property as an array, because each key in the array is an int. However, if I were to add an event to the events "array" via a POST to myfirebaseinstance.com/users/0/events.json, the unique key autogenerated by firebase causes ES to now index each child as its own object in the mapping.

So when I created my mapping in ES, I had

"events" : {
                "properties" : {
                  "date" : {
                    "type" : "date"
                  },
                  "eventname" : {
                    "type" : "text",
                    "fields" : {
                      "keyword" : {
                        "type" : "keyword",
                        "ignore_above" : 256
                      }
                    }
                  },
                  "id" : {
                    "type" : "long"
                  }
                }
              }

but once that POST goes through to add an event to the user's events array, FB stops rendering it as an array and now renders it as an object because the unique key (ex: "-K-ASDAIS"). Now that this is rendered as an object instead of an array, ES updates the mapping to accommodate, and now my mapping went from looking like the above to looking something like this:

"events" : {
            "properties" : {
              "-Kz_wVObvfW3ZE2vkzgF" : {
                "properties" : {
                  "date" : {
                    "type" : "date"
                  },
                  "eventname" : {
                    "type" : "text",
                    "fields" : {
                      "keyword" : {
                        "type" : "keyword",
                        "ignore_above" : 256
                      }
                    }
                  },
                  "id" : {
                    "type" : "long"
                  }
                }
              },
              "-L-xyscEkIxgrH7JbuWK" : {
                "properties" : {
                  "date" : {
                    "type" : "date"
                  },
                  "eventname" : {
                    "type" : "text",
                    "fields" : {
                      "keyword" : {
                        "type" : "keyword",
                        "ignore_above" : 256
                      }
                    }
                  },
                  "id" : {
                    "type" : "long"
                  }
                }
              },
              "-L-xz1RF_y_6D0MYu-12" : {
                "properties" : {
                  "date" : {
                    "type" : "date"
                  },
                  "eventname" : {
                    "type" : "text",
                    "fields" : {
                      "keyword" : {
                        "type" : "keyword",
                        "ignore_above" : 256
                      }
                    }
                  },
                  "id" : {
                    "type" : "long"
                  }
                }
              },
              "0" : {
                "properties" : {
                  "date" : {
                    "type" : "date"
                  },
                  "eventname" : {
                    "type" : "text",
                    "fields" : {
                      "keyword" : {
                        "type" : "keyword",
                        "ignore_above" : 256
                      }
                    }
                  },
                  "id" : {
                    "type" : "long"
                  }
                }
              },
              "1" : {
                "properties" : {
                  "date" : {
                    "type" : "date"
                  },
                  "eventname" : {
                    "type" : "text",
                    "fields" : {
                      "keyword" : {
                        "type" : "keyword",
                        "ignore_above" : 256
                      }
                    }
                  },
                  "id" : {
                    "type" : "long"
                  }
                }
              },
              "2" : {
                "properties" : {
                  "date" : {
                    "type" : "date"
                  },
                  "eventname" : {
                    "type" : "text",
                    "fields" : {
                      "keyword" : {
                        "type" : "keyword",
                        "ignore_above" : 256
                      }
                    }
                  },
                  "id" : {
                    "type" : "long"
                  }
                }
              },
              "3" : {
                "properties" : {
                  "date" : {
                    "type" : "date"
                  },
                  "eventname" : {
                    "type" : "text",
                    "fields" : {
                      "keyword" : {
                        "type" : "keyword",
                        "ignore_above" : 256
                      }
                    }
                  },
                  "id" : {
                    "type" : "long"
                  }
                }
              },
              "4" : {
                "properties" : {
                  "date" : {
                    "type" : "date"
                  },
                  "eventname" : {
                    "type" : "text",
                    "fields" : {
                      "keyword" : {
                        "type" : "keyword",
                        "ignore_above" : 256
                      }
                    }
                  },
                  "id" : {
                    "type" : "long"
                  }
                }
              },
              "5" : {
                "properties" : {
                  "date" : {
                    "type" : "date"
                  },
                  "eventname" : {
                    "type" : "text",
                    "fields" : {
                      "keyword" : {
                        "type" : "keyword",
                        "ignore_above" : 256
                      }
                    }
                  }

Obviously its not scalable to have ES update my mapping every time I add an event to a user's event's array. Also, it breaks my Android client's serialization. The client is expecting an array wheras now it's getting an object.

Is there any way around this? Am I missing something obvious? Or should I not have used Flashlight/Firebase/ES for this particular use case? I'll admit I didn't know about FB's lack of array support until tonight, so part of this is on me. I'm just hoping I can find a workaround rather than having to reimplement my backend from scratch.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant