Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(opentelemetry): migration exception when upgrading from below version 3.3 to 3.7 #13391

Merged
merged 2 commits into from
Jul 26, 2024

Conversation

ms2008
Copy link
Contributor

@ms2008 ms2008 commented Jul 18, 2024

Summary

The migration job try to reset config.queue.max_batch_size of opentelemetry plugin. However the config.queue field has only been introduced since 3.3.

This means that if users are upgrading from a version below 3.3 to 3.7, they will encounter kong migrations finish terminates unsuccessfully with the error:

kong@943321e5c877:/$ kong migrations finish
migrating opentelemetry on database 'kong'...
Error: [PostgreSQL error] cluster_mutex callback threw an error: /usr/local/share/lua/5.1/kong/cmd/utils/migrations.lua:174: [PostgreSQL error] failed to run migration '001_331_to_332' teardown: ./opentelemetry/migrations/001_331_to_332.lua:14: attempt to index field 'queue' (a nil value)
stack traceback:
	./opentelemetry/migrations/001_331_to_332.lua:14: in function 'fixup_fn'
	...are/lua/5.1/kong/db/migrations/operations/331_to_332.lua:48: in function <...are/lua/5.1/kong/db/migrations/operations/331_to_332.lua:35>
	[C]: in function 'xpcall'
	/usr/local/share/lua/5.1/kong/db/init.lua:595: in function 'run_migrations'
	/usr/local/share/lua/5.1/kong/cmd/utils/migrations.lua:174: in function </usr/local/share/lua/5.1/kong/cmd/utils/migrations.lua:158>
	[C]: in function 'xpcall'
	/usr/local/share/lua/5.1/kong/db/init.lua:408: in function </usr/local/share/lua/5.1/kong/db/init.lua:358>
	[C]: in function 'pcall'
	/usr/local/share/lua/5.1/kong/concurrency.lua:66: in function 'cluster_mutex'
	/usr/local/share/lua/5.1/kong/cmd/utils/migrations.lua:158: in function 'finish'
	/usr/local/share/lua/5.1/kong/cmd/migrations.lua:332: in function 'cmd_exec'
	/usr/local/share/lua/5.1/kong/cmd/init.lua:38: in function </usr/local/share/lua/5.1/kong/cmd/init.lua:38>
	[C]: in function 'xpcall'
	/usr/local/share/lua/5.1/kong/cmd/init.lua:38: in function </usr/local/share/lua/5.1/kong/cmd/init.lua:22>
	(command line -e):7: in function 'inline_gen'
	init_worker_by_lua(nginx.conf:157):44: in function <init_worker_by_lua(nginx.conf:157):43>
	[C]: in function 'xpcall'
	init_worker_by_lua(nginx.conf:157):52: in function <init_worker_by_lua(nginx.conf:157):50>
stack traceback:
	[C]: in function 'assert'
	/usr/local/share/lua/5.1/kong/cmd/utils/migrations.lua:174: in function </usr/local/share/lua/5.1/kong/cmd/utils/migrations.lua:158>
	[C]: in function 'xpcall'
	/usr/local/share/lua/5.1/kong/db/init.lua:408: in function </usr/local/share/lua/5.1/kong/db/init.lua:358>
	[C]: in function 'pcall'
	/usr/local/share/lua/5.1/kong/concurrency.lua:66: in function 'cluster_mutex'
	/usr/local/share/lua/5.1/kong/cmd/utils/migrations.lua:158: in function 'finish'
	/usr/local/share/lua/5.1/kong/cmd/migrations.lua:332: in function 'cmd_exec'
	/usr/local/share/lua/5.1/kong/cmd/init.lua:38: in function </usr/local/share/lua/5.1/kong/cmd/init.lua:38>
	[C]: in function 'xpcall'
	/usr/local/share/lua/5.1/kong/cmd/init.lua:38: in function </usr/local/share/lua/5.1/kong/cmd/init.lua:22>
	(command line -e):7: in function 'inline_gen'
	init_worker_by_lua(nginx.conf:157):44: in function <init_worker_by_lua(nginx.conf:157):43>
	[C]: in function 'xpcall'
	init_worker_by_lua(nginx.conf:157):52: in function <init_worker_by_lua(nginx.conf:157):50>

  Run with --v (verbose) or --vv (debug) for more details

Checklist

  • The Pull Request has tests
  • A changelog file has been created under changelog/unreleased/kong or skip-changelog label added on PR if changelog is unnecessary. README.md
  • There is a user-facing docs PR against https://github.com/Kong/docs.konghq.com - PUT DOCS PR HERE

Issue reference

Fix FTI-6109

@github-actions github-actions bot added plugins/opentelemetry cherry-pick kong-ee schedule this PR for cherry-picking to kong/kong-ee labels Jul 18, 2024
@ms2008 ms2008 force-pushed the fix/otel-migrations-exception branch 2 times, most recently from 0226ee4 to 101a861 Compare July 18, 2024 09:08
@ms2008 ms2008 marked this pull request as ready for review July 18, 2024 09:31
@ms2008 ms2008 requested a review from samugi July 18, 2024 09:31
@ms2008 ms2008 merged commit 40c86fa into master Jul 26, 2024
29 checks passed
@ms2008 ms2008 deleted the fix/otel-migrations-exception branch July 26, 2024 03:14
@team-gateway-bot
Copy link
Collaborator

Cherry-pick failed for master, because it was unable to cherry-pick the commit(s).

Please cherry-pick the changes locally.

git remote add upstream https://github.com/kong/kong-ee
git fetch upstream master
git worktree add -d .worktree/cherry-pick-13391-to-master-to-upstream upstream/master
cd .worktree/cherry-pick-13391-to-master-to-upstream
git checkout -b cherry-pick-13391-to-master-to-upstream
ancref=$(git merge-base 5576c64e19fe617ea327db6a0436174d77b0ed36 101a86134c5f9563b7f0c1eddc38c023ab0417e1)
git cherry-pick -x $ancref..101a86134c5f9563b7f0c1eddc38c023ab0417e1

@github-actions github-actions bot added the incomplete-cherry-pick A cherry-pick was incomplete and needs manual intervention label Jul 26, 2024
@ms2008 ms2008 removed the incomplete-cherry-pick A cherry-pick was incomplete and needs manual intervention label Jul 26, 2024
oowl pushed a commit that referenced this pull request Aug 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cherry-pick kong-ee schedule this PR for cherry-picking to kong/kong-ee plugins/opentelemetry size/M
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants