Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: update index after deploy #308

Closed
wants to merge 1 commit into from
Closed

feat: update index after deploy #308

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 3 additions & 10 deletions basedosdados_api/api/v1/admin.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -167,16 +167,8 @@ def rebuild_search_index(modeladmin, request, queryset):


def update_search_index(modeladmin, request, queryset):
for instance in queryset:
try:
search_backend = connections["default"].get_backend()
search_index = search_backend.get_index(
"basedosdados_api.api.v1.models.Dataset"
)
search_index.update_object(instance, using="default")
messages.success(request, "Search index updated successfully")
except ObjectDoesNotExist:
messages.error(request, f"Search index for {instance} update failed")
call_command("update_index", interactive=False, batchsize=100, workers=4)
messages.success(request, "Search index rebuilt successfully")


update_search_index.short_description = "Update search index"
Expand Down Expand Up @@ -423,6 +415,7 @@ class TagAdmin(TabbedTranslationAdmin):
class DatasetAdmin(OrderedInlineModelAdminMixin, TabbedTranslationAdmin):
actions = [
reorder_tables,
update_search_index,
rebuild_search_index,
]

Expand Down
24 changes: 14 additions & 10 deletions docker-compose.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,10 +16,14 @@ services:
volumes:
- esdata:/usr/share/elasticsearch/data
ports:
- "127.0.0.1:9200:9200"
- "127.0.0.1:9300:9300"
networks:
- esnet
- 9200:9200
- 9300:9300
healthcheck:
test: ["CMD", "curl --silent --fail http://localhost:9200/_cluster/health"]
interval: 1m
timeout: 20s
retries: 5
start_period: 30s

database:
image: postgres:14
Expand All @@ -29,7 +33,7 @@ services:
ports:
- 5432:5432
volumes:
- vol_database:/var/lib/postgresql/data
- pgdata:/var/lib/postgresql/data
healthcheck:
test: ["CMD", "pg_isready", "-U", "${POSTGRES_USER}"]
interval: 1m
Expand All @@ -48,7 +52,10 @@ services:
ports:
- 8080:80
depends_on:
- database
database:
condition: service_healthy
elasticsearch:
condition: service_healthy
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost/healthcheck/"]
interval: 1m
Expand All @@ -58,8 +65,5 @@ services:
restart: unless-stopped

volumes:
vol_database:
pgdata:
esdata:

networks:
esnet:
8 changes: 4 additions & 4 deletions start-server.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
#!/usr/bin/env bash
# start-server.sh
python manage.py makemigrations && python manage.py migrate
if [ -n "$DJANGO_SUPERUSER_USERNAME" ] && [ -n "$DJANGO_SUPERUSER_PASSWORD" ] ; then
(cd /app; python manage.py createsuperuser --no-input)
fi
(cd /app; python manage.py makemigrations && python manage.py migrate)
(cd /app; gunicorn basedosdados_api.wsgi --user www-data --bind 0.0.0.0:8000 --workers 3 --log-level debug --timeout 180) &
nginx -g "daemon off;"
(cd /app; python manage.py makemigrations)
(cd /app; python manage.py migrate)
(cd /app; python manage.py update_index --remove)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • O ideal seria que isso fosse feito apenas quando uma label específica fosse adicionada ao deploy.
  • Nos testes que fiz, o update_index --remove era bem mais lento que o rebuild_index
  • Quando faço o rebuild na mão uso os parâmentros: python manage.py rebuild_index --batch-size 50 --workers 4 --verbosity 2 --noinput para evitar timeout ou falta de memória.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pelo que li o update index deveria ser mais rápido, vou dar uma olhada sobre. Você testou no ambiente de development/staging ou localmente?

(cd /app; gunicorn basedosdados_api.wsgi --user www-data --bind 0.0.0.0:8000 --workers 3 --log-level debug --timeout 180) & nginx -g "daemon off;"
Loading