Skip to content

Commit

Permalink
fix(wasm): disable JIT for proxy_wasm launch
Browse files Browse the repository at this point in the history
This prevents triggering a LuaJIT issue when attempting to call an FFI
callback with an ongoing trace further down the stack; attempting to do so can
trigger a "bad callback" assertion.

Stack trace demonstrating the issue in question:

```
   from /home/zhongweiyao/projects/kong/bazel-bin/build/kong-dev/openresty/nginx/sbin/../modules/ngx_wasm_module.so
    at /home/zhongweiyao/.cache/bazel/_bazel_zhongweiyao/7df4419a5ca351a16fa75df771d28bc8/execroot/kong/external/ngx_wasm_module/src/wasm/wrt/ngx_wrt_wasmtime.c:657
    at /home/zhongweiyao/.cache/bazel/_bazel_zhongweiyao/7df4419a5ca351a16fa75df771d28bc8/execroot/kong/external/ngx_wasm_module/src/wasm/vm/ngx_wavm.c:1107
    at /home/zhongweiyao/.cache/bazel/_bazel_zhongweiyao/7df4419a5ca351a16fa75df771d28bc8/execroot/kong/external/ngx_wasm_module/src/wasm/vm/ngx_wavm.c:1184
    at /home/zhongweiyao/.cache/bazel/_bazel_zhongweiyao/7df4419a5ca351a16fa75df771d28bc8/execroot/kong/external/ngx_wasm_module/src/wasm/vm/ngx_wavm.c:1287
    at /home/zhongweiyao/.cache/bazel/_bazel_zhongweiyao/7df4419a5ca351a16fa75df771d28bc8/execroot/kong/external/ngx_wasm_module/src/http/proxy_wasm/ngx_http_proxy_wasm.c:40
    at /home/zhongweiyao/.cache/bazel/_bazel_zhongweiyao/7df4419a5ca351a16fa75df771d28bc8/execroot/kong/external/ngx_wasm_module/src/http/proxy_wasm/ngx_http_proxy_wasm.c:411
    at /home/zhongweiyao/.cache/bazel/_bazel_zhongweiyao/7df4419a5ca351a16fa75df771d28bc8/execroot/kong/external/ngx_wasm_module/src/common/proxy_wasm/ngx_proxy_wasm.c:658
    at /home/zhongweiyao/.cache/bazel/_bazel_zhongweiyao/7df4419a5ca351a16fa75df771d28bc8/execroot/kong/external/ngx_wasm_module/src/common/proxy_wasm/ngx_proxy_wasm.c:783
    at /home/zhongweiyao/.cache/bazel/_bazel_zhongweiyao/7df4419a5ca351a16fa75df771d28bc8/execroot/kong/external/ngx_wasm_module/src/wasm/ngx_wasm_ops.c:417
    at /home/zhongweiyao/.cache/bazel/_bazel_zhongweiyao/7df4419a5ca351a16fa75df771d28bc8/execroot/kong/external/ngx_wasm_module/src/wasm/ngx_wasm_ops.c:290
    at /home/zhongweiyao/.cache/bazel/_bazel_zhongweiyao/7df4419a5ca351a16fa75df771d28bc8/execroot/kong/external/ngx_wasm_module/src/common/lua/ngx_wasm_lua_ffi.c:164
    at ../ngx_lua-0.10.25/src/ngx_http_lua_util.c:1184
    respawn=-3) at src/os/unix/ngx_process.c:199
```

The problem arises when Wasm code eventually calls the FFI callback which triggers Lua code while having an ongoing
trace in the stack (see frame 12, `TRACE_1054`, in the example above).

Eventually the LuaJIT callback crashes like this:

```
    at /home/zhongweiyao/.cache/bazel/_bazel_zhongweiyao/7df4419a5ca351a16fa75df771d28bc8/execroot/kong/external/ngx_wasm_module/src/common/proxy_wasm/ngx_proxy_wasm_properties.c:1058
    at /home/zhongweiyao/.cache/bazel/_bazel_zhongweiyao/7df4419a5ca351a16fa75df771d28bc8/execroot/kong/external/ngx_wasm_module/src/common/proxy_wasm/ngx_proxy_wasm_host.c:780
    at /home/zhongweiyao/.cache/bazel/_bazel_zhongweiyao/7df4419a5ca351a16fa75df771d28bc8/execroot/kong/external/ngx_wasm_module/src/wasm/vm/ngx_wavm_host.c:265
   from /home/zhongweiyao/projects/kong/bazel-bin/build/kong-dev/openresty/nginx/sbin/../modules/ngx_wasm_module.so
   from /home/zhongweiyao/projects/kong/bazel-bin/build/kong-dev/openresty/nginx/sbin/../modules/ngx_wasm_module.so
   from /home/zhongweiyao/projects/kong/bazel-bin/build/kong-dev/openresty/nginx/sbin/../modules/ngx_wasm_module.so
   from /home/zhongweiyao/projects/kong/bazel-bin/build/kong-dev/openresty/nginx/sbin/../modules/ngx_wasm_module.so
      ...
```

Here's some sample minimal code to reproduce the LuaJIT issue outside of the Gateway:

```lua
-- Lua code
local ffi = require("ffi")
local C = ffi.C

ffi.cdef [[
    typedef int (*my_fn_t)(int, int, int);
    int f2();
    void setup(my_fn_t f, int, int, int);
]]

local lib = ffi.load("test")

function setup(cb, a, b, c)
  lib.setup(cb, a, b, c)
end

function f0()
  return lib.f2() + 1
end

do
  local cb = ffi.cast("my_fn_t", function(a, b, c)
    return a+b+c
  end)
  setup(cb, 10, 99, 13)
  print(f0())

  for i=1,300 do
    if i > 60 then f0() end
  end
end
```

```c
/* C code */
typedef int (*my_fn_t)(int, int, int);

my_fn_t gf = 0;
int ga;
int gb;
int gc;

void setup(my_fn_t f, int a, int b, int c) {
  gf = f;
  ga = a;
  gb = b;
  gc = c;
}

int f2() {
  return gf(ga, gb, gc) + 1;
}
```

The issue in question has been a known for a long time. See:

https://luajit.freelists.narkive.com/sdhSLJSr/how-to-make-bad-callback-more-deterministic

```
The bad callback error happens because some JIT-compiled Lua code calls a C
function which in turn calls an FFI callback.
```

https://lua-l.lua.narkive.com/qXJrNlpP/luajit-ffi-windows-bad-callback-error-in-msgwaitformultipleobjects-proof-of-concept

From Mike Pall:

```
The problem is that a FFI callback cannot safely be called from a
C function which is itself called via the FFI from JIT-compiled
code. In your case this is the call to MsgWaitForMultipleObjects.

I've put in a lot of heuristics to detect this, and it usually
succeeds in disabling compilation for such a function. However in
your case the loop is compiled before the callback is ever called,
so the detection fails.

The straighforward solution is to put the message loop into an
extra Lua function and use jit.off(func)
```

Signed-off-by: Hisham Muhammad <[email protected]>
  • Loading branch information
zhongweiy authored and locao committed Dec 5, 2023
1 parent a0a0be5 commit a796ac6
Showing 1 changed file with 1 addition and 0 deletions.
1 change: 1 addition & 0 deletions kong/runloop/wasm.lua
Original file line number Diff line number Diff line change
Expand Up @@ -935,6 +935,7 @@ function _M.attach(ctx)
return kong.response.error(500)
end

jit.off(proxy_wasm.start)
ok, err = proxy_wasm.start()
if not ok then
log(CRIT, "failed to execute ", chain.label, " filter chain for request: ", err)
Expand Down

0 comments on commit a796ac6

Please sign in to comment.