Test case failure with Pongo run

[ghost]

When I use the command pongo run I get the following error

pongo-INFO] auto-starting the test environment, use the 'pongo down' action to stop it
[+] Running 2/2
 ⠿ Network pongo-b7d479e8               Created                                                                                                                                                            0.0s
 ⠿ Container pongo-b7d479e8-postgres-1  Started                                                                                                                                                            0.2s
[pongo-INFO] Waiting for postgres
[pongo-INFO] image 'kong-pongo-test:3.0.0' not found, auto-building it
[pongo-INFO] starting build of image 'kong-pongo-test:3.0.0'
[+] Building 63.9s (18/18) FINISHED                                                                                                                                                                             
 => [internal] load build definition from Dockerfile                                                                                                                                                       0.0s
 => => transferring dockerfile: 2.19kB                                                                                                                                                                     0.0s
 => [internal] load .dockerignore                                                                                                                                                                          0.0s
 => => transferring context: 2B                                                                                                                                                                            0.0s
 => [internal] load metadata for docker.io/library/kong:3.0.0-alpine                                                                                                                                       0.0s
 => [ 1/13] FROM docker.io/library/kong:3.0.0-alpine                                                                                                                                                       0.0s
 => [internal] load build context                                                                                                                                                                          0.0s
 => => transferring context: 1.75MB                                                                                                                                                                        0.0s
 => [ 2/13] COPY ./kong-versions/3.0.0/kong /kong                                                                                                                                                          0.0s
 => [ 3/13] COPY assets/busted_helper.lua        /pongo/busted_helper.lua                                                                                                                                  0.0s
 => [ 4/13] COPY assets/pongo_entrypoint.sh      /pongo/pongo_entrypoint.sh                                                                                                                                0.0s
 => [ 5/13] COPY assets/default-pongo-setup.sh   /pongo/default-pongo-setup.sh                                                                                                                             0.0s
 => [ 6/13] COPY assets/pongo_pack.lua           /pongo/pongo_pack.lua                                                                                                                                     0.0s
 => [ 7/13] COPY assets/kong_migrations_start.sh /pongo/kong_migrations_start.sh                                                                                                                           0.0s
 => [ 8/13] COPY assets/kong_start_dbless.sh     /pongo/kong_start_dbless.sh                                                                                                                               0.0s
 => [ 9/13] COPY assets/kong_export.sh           /pongo/kong_export.sh                                                                                                                                     0.0s
 => [10/13] COPY assets/parse_git_branch.sh      /pongo/parse_git_branch.sh                                                                                                                                0.0s
 => [11/13] COPY assets/pongo_profile.sh         /etc/profile.d/pongo_profile.sh                                                                                                                           0.0s
 => [12/13] RUN apk update     && apk add zip unzip make g++ py-pip jq git bsd-compat-headers m4 openssl-dev curl wget python3-dev shadow     && curl -k -s -S -L https://github.com/fullstorydev/grpcur  62.8s
 => [13/13] WORKDIR /kong                                                                                                                                                                                  0.0s
 => exporting to image                                                                                                                                                                                     0.9s
 => => exporting layers                                                                                                                                                                                    0.9s
 => => writing image sha256:e0eb99568537d3c46383fd51bcd8f75ac52e79f324c0240f7716ec9e1cd628bf                                                                                                               0.0s 
 => => naming to docker.io/library/kong-pongo-test:3.0.0                                                                                                                                                   0.0s 
[pongo-INFO] image 'kong-pongo-test:3.0.0' successfully build                                                                                                                                                   

Stopping after installing dependencies for kong-plugin-myplugin 0.1.0-1

Kong version: 3.0.0
[==========] Running tests from scanned files.
[----------] Global test environment setup.
[----------] Running tests from /kong-plugin/spec/api-version/01-unit_spec.lua
[ RUN      ] /kong-plugin/spec/api-version/01-unit_spec.lua @ 18: api-version: (schema) accepts distinct request_header and response_header
[       OK ] /kong-plugin/spec/api-version/01-unit_spec.lua @ 18: api-version: (schema) accepts distinct request_header and response_header (0.38 ms)
[ RUN      ] /kong-plugin/spec/api-version/01-unit_spec.lua @ 28: api-version: (schema) does not accept identical request_header and response_header
[       OK ] /kong-plugin/spec/api-version/01-unit_spec.lua @ 28: api-version: (schema) does not accept identical request_header and response_header (0.34 ms)
[----------] 2 tests from /kong-plugin/spec/api-version/01-unit_spec.lua (347.98 ms total)

[----------] Running tests from /kong-plugin/spec/api-version/02-integration_spec.lua
/kong-plugin/spec/api-version/02-integration_spec.lua:28: 2022/09/29 21:56:11 [warn] the configuration value 'strict' for configuration property 'worker_consistency' is deprecated
2022/09/29 21:56:11 [warn] the 'worker_consistency' configuration property is deprecated
Error: nginx configuration is invalid (exit code 1):
nginx: the configuration file /kong-plugin/servroot/nginx.conf syntax is ok
nginx: [emerg] bind() to unix:/kong-plugin/servroot/worker_events.sock failed (22: Invalid argument)
nginx: configuration file /kong-plugin/servroot/nginx.conf test failed


  Run with --v (verbose) or --vv (debug) for more details


stack traceback:
	/kong-plugin/spec/api-version/02-integration_spec.lua:28: in function </kong-plugin/spec/api-version/02-integration_spec.lua:11>

/kong-plugin/spec/api-version/02-integration_spec.lua:28: 2022/09/29 21:56:12 [warn] the configuration value 'strict' for configuration property 'worker_consistency' is deprecated
2022/09/29 21:56:12 [warn] the 'worker_consistency' configuration property is deprecated
Error: nginx configuration is invalid (exit code 1):
nginx: the configuration file /kong-plugin/servroot/nginx.conf syntax is ok
nginx: [emerg] bind() to unix:/kong-plugin/servroot/worker_events.sock failed (22: Invalid argument)
nginx: configuration file /kong-plugin/servroot/nginx.conf test failed


  Run with --v (verbose) or --vv (debug) for more details


stack traceback:
	/kong-plugin/spec/api-version/02-integration_spec.lua:28: in function </kong-plugin/spec/api-version/02-integration_spec.lua:11>

[----------] 0 tests from /kong-plugin/spec/api-version/02-integration_spec.lua (2415.31 ms total)

[----------] Global test environment teardown.
[==========] 2 tests from 2 test files ran. (2763.81 ms total)
[  PASSED  ] 2 tests.
[  ERROR   ] 2 errors, listed below:
[  ERROR   ] /kong-plugin/spec/api-version/02-integration_spec.lua @ 11: api-version: (access) [#postgres] lazy_setup
[  ERROR   ] /kong-plugin/spec/api-version/02-integration_spec.lua @ 11: api-version: (access) [#off] lazy_setup

The following command KONG_VERSION=2.3.2 pongo run the expected outout would be something similar to this.

[pongo-INFO] auto-starting the test environment, use the 'pongo down' action to stop it
[+] Running 2/2
 ⠿ Network pongo-b7d479e8               Created                                                                                                                                                            0.0s
 ⠿ Container pongo-b7d479e8-postgres-1  Started                                                                                                                                                            0.2s
[pongo-INFO] Waiting for postgres
[pongo-INFO] image 'kong-pongo-test:2.3.2' not found, auto-building it
[pongo-INFO] starting build of image 'kong-pongo-test:2.3.2'
[+] Building 61.9s (18/18) FINISHED                                                                                                                                                                             
 => [internal] load build definition from Dockerfile                                                                                                                                                       0.0s
 => => transferring dockerfile: 2.19kB                                                                                                                                                                     0.0s
 => [internal] load .dockerignore                                                                                                                                                                          0.0s
 => => transferring context: 2B                                                                                                                                                                            0.0s
 => [internal] load metadata for docker.io/library/kong:2.3.2-alpine                                                                                                                                       0.0s
 => [ 1/13] FROM docker.io/library/kong:2.3.2-alpine                                                                                                                                                       0.0s
 => [internal] load build context                                                                                                                                                                          0.0s
 => => transferring context: 827.98kB                                                                                                                                                                      0.0s
 => [ 2/13] COPY ./kong-versions/2.3.2/kong /kong                                                                                                                                                          0.0s
 => [ 3/13] COPY assets/busted_helper.lua        /pongo/busted_helper.lua                                                                                                                                  0.0s
 => [ 4/13] COPY assets/pongo_entrypoint.sh      /pongo/pongo_entrypoint.sh                                                                                                                                0.0s
 => [ 5/13] COPY assets/default-pongo-setup.sh   /pongo/default-pongo-setup.sh                                                                                                                             0.0s
 => [ 6/13] COPY assets/pongo_pack.lua           /pongo/pongo_pack.lua                                                                                                                                     0.0s
 => [ 7/13] COPY assets/kong_migrations_start.sh /pongo/kong_migrations_start.sh                                                                                                                           0.0s
 => [ 8/13] COPY assets/kong_start_dbless.sh     /pongo/kong_start_dbless.sh                                                                                                                               0.0s
 => [ 9/13] COPY assets/kong_export.sh           /pongo/kong_export.sh                                                                                                                                     0.0s
 => [10/13] COPY assets/parse_git_branch.sh      /pongo/parse_git_branch.sh                                                                                                                                0.0s
 => [11/13] COPY assets/pongo_profile.sh         /etc/profile.d/pongo_profile.sh                                                                                                                           0.0s
 => [12/13] RUN apk update     && apk add zip unzip make g++ py-pip jq git bsd-compat-headers m4 openssl-dev curl wget python3-dev shadow     && curl -k -s -S -L https://github.com/fullstorydev/grpcur  60.7s
 => [13/13] WORKDIR /kong                                                                                                                                                                                  0.0s 
 => exporting to image                                                                                                                                                                                     1.0s 
 => => exporting layers                                                                                                                                                                                    1.0s 
 => => writing image sha256:ba7deff59921bc97f0591b1fd1ad96e68a857a7d7ec5277fcb2f391f739c0e8e                                                                                                               0.0s 
 => => naming to docker.io/library/kong-pongo-test:2.3.2                                                                                                                                                   0.0s 
[pongo-INFO] image 'kong-pongo-test:2.3.2' successfully build                                                                                                                                                   
groupmod: group 'kong' does not exist

Stopping after installing dependencies for kong-plugin-myplugin 0.1.0-1

Kong version: 2.3.2
[==========] Running tests from scanned files.
[----------] Global test environment setup.
[----------] Running tests from /kong-plugin/spec/api-version/01-unit_spec.lua
[ RUN      ] /kong-plugin/spec/api-version/01-unit_spec.lua @ 18: api-version: (schema) accepts distinct request_header and response_header
[       OK ] /kong-plugin/spec/api-version/01-unit_spec.lua @ 18: api-version: (schema) accepts distinct request_header and response_header (0.31 ms)
[ RUN      ] /kong-plugin/spec/api-version/01-unit_spec.lua @ 28: api-version: (schema) does not accept identical request_header and response_header
[       OK ] /kong-plugin/spec/api-version/01-unit_spec.lua @ 28: api-version: (schema) does not accept identical request_header and response_header (0.24 ms)
[----------] 2 tests from /kong-plugin/spec/api-version/01-unit_spec.lua (308.12 ms total)

[----------] Running tests from /kong-plugin/spec/api-version/02-integration_spec.lua
[ RUN      ] /kong-plugin/spec/api-version/02-integration_spec.lua @ 55: api-version: (access) [#postgres] request gets a 'hello-world' header
[       OK ] /kong-plugin/spec/api-version/02-integration_spec.lua @ 55: api-version: (access) [#postgres] request gets a 'hello-world' header (4.72 ms)
[ RUN      ] /kong-plugin/spec/api-version/02-integration_spec.lua @ 73: api-version: (access) [#postgres] response gets a 'bye-world' header
[       OK ] /kong-plugin/spec/api-version/02-integration_spec.lua @ 73: api-version: (access) [#postgres] response gets a 'bye-world' header (3.59 ms)
[ RUN      ] /kong-plugin/spec/api-version/02-integration_spec.lua @ 55: api-version: (access) [#off] request gets a 'hello-world' header
[       OK ] /kong-plugin/spec/api-version/02-integration_spec.lua @ 55: api-version: (access) [#off] request gets a 'hello-world' header (5.11 ms)
[ RUN      ] /kong-plugin/spec/api-version/02-integration_spec.lua @ 73: api-version: (access) [#off] response gets a 'bye-world' header
[       OK ] /kong-plugin/spec/api-version/02-integration_spec.lua @ 73: api-version: (access) [#off] response gets a 'bye-world' header (4.57 ms)
[----------] 4 tests from /kong-plugin/spec/api-version/02-integration_spec.lua (3738.45 ms total)

[----------] Global test environment teardown.
[==========] 6 tests from 2 test files ran. (4046.76 ms total)
[  PASSED  ] 6 tests.

[Tieske]

Know issue with Kong 3.0.0, see Kong/kong#9337 (comment)

[dcominottim]

Hi, @Tieske. Thanks for the update (I opened the duplicate #368).

I am trying to form a picture of where we are regarding this issue, since #9337 (the supposed fix?) was merged a few months back but the issue still happens with the latest code from Pongo's master.

[Tieske]

that PR has 2 fixes in it, and after some internal discussion the 2nd fix was removed to keep it clean. But that is exactly the fix we need here. I am pushing to get this fixed though.

[dol]

As a workaround place your plugin files inside WSL home folder instead of accessing them on the Windows mount /mnt/c .

[nitinthewiz]

Since this issue is still open, I'd like add this here - I'm seeing this behavior in Kong 3.2.2 on macOS 13 using colima. This includes not seeing the issue on Kong 2.3.2. I understand that there's a fix in place since Oct 2022, but has anyone got this working on macOS with a non-Docker Desktop setup?

[Tieske]

@nitinthewiz just to verify, are you seeing this line in your logs;

nginx: [emerg] bind() to unix:/kong-plugin/servroot/worker_events.sock failed (22: Invalid argument)

That would indeed indicate it is the same issue (which hasn't been fixed yet, see my comment above)

[nitinthewiz]

@Tieske Thanks for asking for a clarification. I was tracking multiple issues to find a resolution and "Invalid argument" is not the error I'm seeing.

I'm seeing the following error -
nginx: [emerg] bind() to unix:/kong-plugin/servroot/worker_events.sock failed (1: Operation not permitted)

This seems to have been mentioned here, and by you here. I suppose it is not related to this issue. However, I don't see an issue tracking my error specifically.

I suppose just like Vagrant, this needs an absolute path solution for colima on macOS (or perhaps a more generalized solution for macOS?)

[Tieske]

The solution would be to make the Unix socket location configurable, or move it to a more appropriate system based location. (that's a Kong change however, not a Pongo change)

[antoineauger]

I was experiencing the same issue on Ubuntu 22.04 with rancher-desktop. This is kind of known, see rancher-sandbox/rancher-desktop#1209 for example. The fix mentioned here did not work for me.

As a temporary workaround, I managed to run pongo run -v by editing the volumes section of assets/docker-compose.yml:

  kong:
    image: ${KONG_TEST_IMAGE:-ignore_if_not_provided}
   ...
    volumes:
-      - ${PONGO_WD}:/kong-plugin
+      - /tmp/kong-plugin:/kong-plugin/servroot
+      - ${PONGO_WD}/spec:/kong-plugin/spec
+      - ${PONGO_WD}/kong:/kong-plugin/kong
+      - ${PONGO_WD}/.pongo:/kong-plugin/.pongo
+      - ${PONGO_WD}/.busted:/kong-plugin/.busted
+      - ${PONGO_WD}/.license-scripts:/kong-plugin/.license-scripts
+      - ${PONGO_WD}/kong-plugin-myplugin-0.1.0-1.rockspec:/kong-plugin/kong-plugin-myplugin-0.1.0-1.rockspec

[mfbmina]

Based on @antoineauger solution, I've tested some configuration and I only needed the following:

kong:
    image: ${KONG_TEST_IMAGE:-ignore_if_not_provided}
   ...
    volumes:
      - ${PONGO_WD}:/kong-plugin
      - /tmp/kong-plugin:/kong-plugin/servroot

[antoineauger]

@mfbmina Unfortunately, your simpler configuration did not work for my use case/tests. 😿

Some fixtures were not found:

2024/04/17 15:29:13 [error] 780#0: *1302 lua entry thread aborted: runtime error: content_by_lua(...ng_test_custom_inject_http.test.conf:141):2: module 'spec.fixtures.mock_upstream' not found:No LuaRocks module found for spec.fixtures.mock_upstream
	no field package.preload['spec.fixtures.mock_upstream']
...

[ilozano2]

As @mfbmina said, adding kong-plugin/servroot volume worked fine in Apple Silicon using Rancher-desktop.
Thanks