Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Let hot_swap work also when there are documents with errors #65

Open
silviurosu opened this issue Feb 19, 2019 · 0 comments
Open

Let hot_swap work also when there are documents with errors #65

silviurosu opened this issue Feb 19, 2019 · 0 comments

Comments

@silviurosu
Copy link

There may be documents that do not pass ES validation. For example self intersecting polygons.
This leads hot_swap failying since every step needs to pass successfully:

with :ok <- create_from_file(config, name, settings_file),
         :ok <- Bulk.upload(config, name, index_config),
         :ok <- __MODULE__.alias(config, name, alias),
         :ok <- clean_starting_with(config, alias, 2),
         :ok <- refresh(config, name) do
      :ok

I can not guarantee that all the documents will always be valid but I would want to reindex the others and inform at the end that some documents could not be inserted.
I had to rewrite bulk upload in my code to pass this limitation:

with :ok <- Index.create_from_file(config, name, settings_file),
         bulk_upload(config, name, index_config),
         :ok <- Index.alias(config, name, alias),
         :ok <- Index.clean_starting_with(config, alias, 2),
         :ok <- Index.refresh(config, name) do
      :ok
.....
defp bulk_upload(config, name, index_config) do
    case Bulk.upload(config, name, index_config) do
      :ok ->
        :ok

      {:error, errors} = err ->
        Bugsnag.report(
          ElasticsearchError.exception("Errors encountered indexing restaurants"),
          severity: "warn",
          metadata: %{errors: errors}
        )

        err
    end
  end

My question is can we make this bulk upload step not failing in case there are some errors?

@silviurosu silviurosu changed the title Let hot_swap work also when there documents with errors Let hot_swap work also when there are documents with errors Feb 19, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants