Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
fix(wasm): disable JIT for proxy_wasm launch
This prevents triggering a LuaJIT issue when attempting to call an FFI callback with an ongoing trace further down the stack; attempting to do so can trigger a "bad callback" assertion. Stack trace demonstrating the issue in question: ``` from /home/zhongweiyao/projects/kong/bazel-bin/build/kong-dev/openresty/nginx/sbin/../modules/ngx_wasm_module.so at /home/zhongweiyao/.cache/bazel/_bazel_zhongweiyao/7df4419a5ca351a16fa75df771d28bc8/execroot/kong/external/ngx_wasm_module/src/wasm/wrt/ngx_wrt_wasmtime.c:657 at /home/zhongweiyao/.cache/bazel/_bazel_zhongweiyao/7df4419a5ca351a16fa75df771d28bc8/execroot/kong/external/ngx_wasm_module/src/wasm/vm/ngx_wavm.c:1107 at /home/zhongweiyao/.cache/bazel/_bazel_zhongweiyao/7df4419a5ca351a16fa75df771d28bc8/execroot/kong/external/ngx_wasm_module/src/wasm/vm/ngx_wavm.c:1184 at /home/zhongweiyao/.cache/bazel/_bazel_zhongweiyao/7df4419a5ca351a16fa75df771d28bc8/execroot/kong/external/ngx_wasm_module/src/wasm/vm/ngx_wavm.c:1287 at /home/zhongweiyao/.cache/bazel/_bazel_zhongweiyao/7df4419a5ca351a16fa75df771d28bc8/execroot/kong/external/ngx_wasm_module/src/http/proxy_wasm/ngx_http_proxy_wasm.c:40 at /home/zhongweiyao/.cache/bazel/_bazel_zhongweiyao/7df4419a5ca351a16fa75df771d28bc8/execroot/kong/external/ngx_wasm_module/src/http/proxy_wasm/ngx_http_proxy_wasm.c:411 at /home/zhongweiyao/.cache/bazel/_bazel_zhongweiyao/7df4419a5ca351a16fa75df771d28bc8/execroot/kong/external/ngx_wasm_module/src/common/proxy_wasm/ngx_proxy_wasm.c:658 at /home/zhongweiyao/.cache/bazel/_bazel_zhongweiyao/7df4419a5ca351a16fa75df771d28bc8/execroot/kong/external/ngx_wasm_module/src/common/proxy_wasm/ngx_proxy_wasm.c:783 at /home/zhongweiyao/.cache/bazel/_bazel_zhongweiyao/7df4419a5ca351a16fa75df771d28bc8/execroot/kong/external/ngx_wasm_module/src/wasm/ngx_wasm_ops.c:417 at /home/zhongweiyao/.cache/bazel/_bazel_zhongweiyao/7df4419a5ca351a16fa75df771d28bc8/execroot/kong/external/ngx_wasm_module/src/wasm/ngx_wasm_ops.c:290 at /home/zhongweiyao/.cache/bazel/_bazel_zhongweiyao/7df4419a5ca351a16fa75df771d28bc8/execroot/kong/external/ngx_wasm_module/src/common/lua/ngx_wasm_lua_ffi.c:164 at ../ngx_lua-0.10.25/src/ngx_http_lua_util.c:1184 respawn=-3) at src/os/unix/ngx_process.c:199 ``` The problem arises when Wasm code eventually calls the FFI callback which triggers Lua code while having an ongoing trace in the stack (see frame 12, `TRACE_1054`, in the example above). Eventually the LuaJIT callback crashes like this: ``` at /home/zhongweiyao/.cache/bazel/_bazel_zhongweiyao/7df4419a5ca351a16fa75df771d28bc8/execroot/kong/external/ngx_wasm_module/src/common/proxy_wasm/ngx_proxy_wasm_properties.c:1058 at /home/zhongweiyao/.cache/bazel/_bazel_zhongweiyao/7df4419a5ca351a16fa75df771d28bc8/execroot/kong/external/ngx_wasm_module/src/common/proxy_wasm/ngx_proxy_wasm_host.c:780 at /home/zhongweiyao/.cache/bazel/_bazel_zhongweiyao/7df4419a5ca351a16fa75df771d28bc8/execroot/kong/external/ngx_wasm_module/src/wasm/vm/ngx_wavm_host.c:265 from /home/zhongweiyao/projects/kong/bazel-bin/build/kong-dev/openresty/nginx/sbin/../modules/ngx_wasm_module.so from /home/zhongweiyao/projects/kong/bazel-bin/build/kong-dev/openresty/nginx/sbin/../modules/ngx_wasm_module.so from /home/zhongweiyao/projects/kong/bazel-bin/build/kong-dev/openresty/nginx/sbin/../modules/ngx_wasm_module.so from /home/zhongweiyao/projects/kong/bazel-bin/build/kong-dev/openresty/nginx/sbin/../modules/ngx_wasm_module.so ... ``` Here's some sample minimal code to reproduce the LuaJIT issue outside of the Gateway: ```lua -- Lua code local ffi = require("ffi") local C = ffi.C ffi.cdef [[ typedef int (*my_fn_t)(int, int, int); int f2(); void setup(my_fn_t f, int, int, int); ]] local lib = ffi.load("test") function setup(cb, a, b, c) lib.setup(cb, a, b, c) end function f0() return lib.f2() + 1 end do local cb = ffi.cast("my_fn_t", function(a, b, c) return a+b+c end) setup(cb, 10, 99, 13) print(f0()) for i=1,300 do if i > 60 then f0() end end end ``` ```c /* C code */ typedef int (*my_fn_t)(int, int, int); my_fn_t gf = 0; int ga; int gb; int gc; void setup(my_fn_t f, int a, int b, int c) { gf = f; ga = a; gb = b; gc = c; } int f2() { return gf(ga, gb, gc) + 1; } ``` The issue in question has been a known for a long time. See: https://luajit.freelists.narkive.com/sdhSLJSr/how-to-make-bad-callback-more-deterministic ``` The bad callback error happens because some JIT-compiled Lua code calls a C function which in turn calls an FFI callback. ``` https://lua-l.lua.narkive.com/qXJrNlpP/luajit-ffi-windows-bad-callback-error-in-msgwaitformultipleobjects-proof-of-concept From Mike Pall: ``` The problem is that a FFI callback cannot safely be called from a C function which is itself called via the FFI from JIT-compiled code. In your case this is the call to MsgWaitForMultipleObjects. I've put in a lot of heuristics to detect this, and it usually succeeds in disabling compilation for such a function. However in your case the loop is compiled before the callback is ever called, so the detection fails. The straighforward solution is to put the message loop into an extra Lua function and use jit.off(func) ``` Signed-off-by: Hisham Muhammad <[email protected]>
- Loading branch information