Upgrade elasticsearch

[chigby]

Fixes #1716

Note: will require upgrading ES on Heroku as this is deployed.

[nmorduch]

How do we upgrade on the server? Please write up how

[chigby]

To upgrade Elasticsearch on the server, you must remove the addon and then re-create it at a different version. Then, the data must be re-indexed.

To upgrade the version, use these commands. Insert the appropriate app name for which environment you want this upgrade to affect.

heroku addons:destroy searchbox --app na-develop
heroku addons:create searchbox --es_version=7 --app na-develop

Then, reindex the data:

heroku run python manage.py update_index --app na-develop

Changes were made, now needs review

[nmorduch]

Looks like we should try 7.10.2 for searchbox's sake 🙃

[nmorduch]

heroku run python manage.py update_index --app na-develop

Running python manage.py update_index on ⬢ na-develop... up, run.6843 (Hobby)
/app/.heroku/python/lib/python3.8/site-packages/wagtail/admin/rich_text/editors/draftail/__init__.py:42: RuntimeWarning: Draftail received an unknown feature 'undo'.
  warnings.warn(
/app/.heroku/python/lib/python3.8/site-packages/wagtail/admin/rich_text/editors/draftail/__init__.py:42: RuntimeWarning: Draftail received an unknown feature 'redo'.
  warnings.warn(
Updating backend: default
default: Rebuilding index elasticsearch__home_customimage
Traceback (most recent call last):
  File "manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/app/.heroku/python/lib/python3.8/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File "/app/.heroku/python/lib/python3.8/site-packages/django/core/management/__init__.py", line 413, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/app/.heroku/python/lib/python3.8/site-packages/django/core/management/base.py", line 354, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/app/.heroku/python/lib/python3.8/site-packages/django/core/management/base.py", line 398, in execute
    output = self.handle(*args, **options)
  File "/app/.heroku/python/lib/python3.8/site-packages/wagtail/search/management/commands/update_index.py", line 124, in handle
    self.update_backend(
  File "/app/.heroku/python/lib/python3.8/site-packages/wagtail/search/management/commands/update_index.py", line 74, in update_backend
    index = rebuilder.start()
  File "/app/.heroku/python/lib/python3.8/site-packages/wagtail/search/backends/elasticsearch5.py", line 949, in start
    self.reset_index()
  File "/app/.heroku/python/lib/python3.8/site-packages/wagtail/search/backends/elasticsearch5.py", line 945, in reset_index
    self.index.reset()
  File "/app/.heroku/python/lib/python3.8/site-packages/wagtail/search/backends/elasticsearch5.py", line 934, in reset
    self.delete()
  File "/app/.heroku/python/lib/python3.8/site-packages/wagtail/search/backends/elasticsearch5.py", line 833, in delete
    self.es.indices.delete(self.name)
  File "/app/.heroku/python/lib/python3.8/site-packages/elasticsearch/client/utils.py", line 152, in _wrapped
    return func(*args, params=params, headers=headers, **kwargs)
  File "/app/.heroku/python/lib/python3.8/site-packages/elasticsearch/client/indices.py", line 294, in delete
    return self.transport.perform_request(
  File "/app/.heroku/python/lib/python3.8/site-packages/elasticsearch/transport.py", line 392, in perform_request
    raise e
  File "/app/.heroku/python/lib/python3.8/site-packages/elasticsearch/transport.py", line 358, in perform_request
    status, headers_response, data = connection.perform_request(
  File "/app/.heroku/python/lib/python3.8/site-packages/elasticsearch/connection/http_urllib3.py", line 269, in perform_request
    self._raise_error(response.status, raw_data)
  File "/app/.heroku/python/lib/python3.8/site-packages/elasticsearch/connection/base.py", line 315, in _raise_error
    raise HTTP_EXCEPTIONS.get(status_code, TransportError)(
elasticsearch.exceptions.AuthenticationException: AuthenticationException(401, '{"status":"error", "message":"access denied"}')
Sentry is attempting to send 0 pending error messages
Waiting up to 2 seconds
Press Ctrl-C to quit

[nmorduch]

You're also welcome to do the attempting this directly on na-develop!

[chigby]

This is working for me now.. I had to specify that we wanted the "small" plan for searchbox, e.g.

heroku addons:create searchbox:small --es_version=7 --app na-develop

[chigby]

Well, the update_index command worked, anyway. The actual search function seems to be having problems now.

[chigby]

Continuing to look into this, and this is what error I'm seeing on the develop server:

>>> from wagtail.core.models import Page
>>> Page.objects.public().search('vermont')
Traceback (most recent call last):
  File "/app/.heroku/python/lib/python3.8/code.py", line 90, in runcode
    exec(code, self.locals)
  File "<console>", line 1, in <module>
  File "/app/.heroku/python/lib/python3.8/site-packages/wagtail/search/backends/base.py", line 258, in __repr__
    data = list(self[:21])
  File "/app/.heroku/python/lib/python3.8/site-packages/wagtail/search/backends/base.py", line 255, in __len__
    return len(self.results())
  File "/app/.heroku/python/lib/python3.8/site-packages/wagtail/search/backends/base.py", line 218, in results
    self._results_cache = list(self._do_search())
  File "/app/.heroku/python/lib/python3.8/site-packages/wagtail/search/backends/elasticsearch5.py", line 800, in _do_search
    hits = self.backend.es.search(**params)['hits']['hits']
  File "/app/.heroku/python/lib/python3.8/site-packages/elasticsearch/client/utils.py", line 152, in _wrapped
    return func(*args, params=params, headers=headers, **kwargs)
  File "/app/.heroku/python/lib/python3.8/site-packages/elasticsearch/client/__init__.py", line 1658, in search
    return self.transport.perform_request(
  File "/app/.heroku/python/lib/python3.8/site-packages/elasticsearch/transport.py", line 392, in perform_request
    raise e
  File "/app/.heroku/python/lib/python3.8/site-packages/elasticsearch/transport.py", line 358, in perform_request
    status, headers_response, data = connection.perform_request(
  File "/app/.heroku/python/lib/python3.8/site-packages/elasticsearch/connection/http_urllib3.py", line 269, in perform_request
    self._raise_error(response.status, raw_data)
  File "/app/.heroku/python/lib/python3.8/site-packages/elasticsearch/connection/base.py", line 315, in _raise_error
    raise HTTP_EXCEPTIONS.get(status_code, TransportError)(
elasticsearch.exceptions.TransportError: TransportError(500, '{"error_id":"f1883ad8-1200-4ced-9e81-61b244cc395a","errors":[{"code":10,"message":"An error occurred while fulfilling the request"}]}')

This is not what I'm seeing locally, with the same version of the ES server (7.10.2). I've tried to figure out what's going on here (the error message is zero help) and the closest thing I can find is that possibly the query we're sending to the server is too large? When I remove all the PageViewRestriction objects (which are what determines which pages appear in the public() query) then the query succeeds. It's possible there's something else going on that this is merely a consequence of.