-
Notifications
You must be signed in to change notification settings - Fork 267
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add support for local Wasm plugins #3349
Conversation
The latest Buf updates on your PR. Results from workflow Buf CI / buf (pull_request).
|
@@ -12,8 +12,7 @@ | |||
// See the License for the specific language governing permissions and | |||
// limitations under the License. | |||
|
|||
//go:build aix || darwin || dragonfly || freebsd || (js && wasm) || linux || netbsd || openbsd || solaris | |||
// +build aix darwin dragonfly freebsd js,wasm linux netbsd openbsd solaris | |||
//go:build unix || wasip1 || js |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Use the unix build constraint and drop support for +build
as that is required for go versions 1.16 and earlier. Fixes builds for GOOS=wasip1
private/buf/bufcli/cache.go
Outdated
@@ -135,6 +139,17 @@ func NewCommitProvider(container appext.Container) (bufmodule.CommitProvider, er | |||
) | |||
} | |||
|
|||
// CreatePluginCacheDir creates the cache directory for plugins. | |||
// | |||
// This is used by the [bufwasm.WithLocalCacheDir] option. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is the []
something that Godoc picks up? If not, we don't use it anywhere else. If so, good to know!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Removed []
use. It is supported in Godoc as a link but must be imported, so here it didn't link correctly.
private/buf/bufcli/cache.go
Outdated
@@ -103,6 +103,10 @@ var ( | |||
// | |||
// Normalized. | |||
v3CacheModuleLockRelDirPath = normalpath.Join("v3", "modulelocks") | |||
// v3CachePluginsRelDirPath is the relative path to the plugins cache directory in its newest iteration. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This documentation isn't really accurate. If you dive down into the code, it appears you are passing this to the bufwasm.Runtime
, which then happens to use this as a cache for some WASM compilation that is happening. Given the naming, and the documentation here, I would expect this to be a cache of downloaded plugins from the BSR, which it is not. This is effectively an implementation-specific cache.
This should be v3/wasmruntime
or the like, whatever makes sense, and the documentation should reflect what this is specifically used for.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Renamed and updated the docs to the v3/wasmruntime
dir.
private/buf/bufcli/cache.go
Outdated
@@ -135,6 +139,17 @@ func NewCommitProvider(container appext.Container) (bufmodule.CommitProvider, er | |||
) | |||
} | |||
|
|||
// CreatePluginCacheDir creates the cache directory for plugins. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This doesn't seem correct. See above.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Renamed to CreateWasmRuntimeCacheDir
.
@@ -94,6 +95,7 @@ func (c *multiClient) Check(ctx context.Context, request check.Request) ([]*anno | |||
jobs = append( | |||
jobs, | |||
func(ctx context.Context) error { | |||
start := time.Now() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Such timings are the responsibilities of tracing.Tracers
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Switched to tracing.
"pluginrpc.com/pluginrpc" | ||
) | ||
|
||
// RunnerProvider provides pluginrpc.Runners for a given plugin config. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
PluginConfig
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed.
} | ||
|
||
// wasmRunner is a runner that loads a Wasm plugin. | ||
type wasmRunner struct { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why is this in this file?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Moved to pluginrpcutil/wasm_runner.go
private/bufpkg/bufwasm/bufwasm.go
Outdated
@@ -0,0 +1,81 @@ | |||
// Copyright 2020-2024 Buf Technologies, Inc. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why is this in bufpkg
? It uses nothing buf-specific, and therefore should be in pkg/wasm
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Moved to pkg/wasm
private/bufpkg/bufwasm/bufwasm.go
Outdated
"pluginrpc.com/pluginrpc" | ||
) | ||
|
||
// Plugin is a Wasm module. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If Plugin is a WASM module, why is it named Plugin, especially given that plugins have a specific meaning in buf?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Originally thought that Wasm Plugin as its a Wasm module that implements the pluginrpc spec. Have renamed to CompiledModule.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Module also has specific meaning in Buf.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nevermind on this - CompiledModule
seems to be a thing for Wasm too
private/bufpkg/bufwasm/bufwasm.go
Outdated
} | ||
|
||
// NewUnimplementedRuntime returns a new unimplemented Runtime. | ||
func NewUnimplementedRuntime() Runtime { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This can be var UnimplementedRuntime = unimplementedRuntime{}
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed.
if r.wasmRuntime == nil { | ||
return nil, syserror.Newf("wasm runtime is required for local wasm plugins") | ||
} | ||
return newWasmRunner( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's weird that this is implemented in bufpluginrunner
while the command.Runner
-based Runner
is implemented in pluginrpcutil
. Why the split? I don't see any reason for one, both Runners
should be implemented in pluginrpcutil
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
They can't be implemented in pluginrpcutil as the pkg now depends on bufconfig for the bufconfig.PluginConfig
. Could move pluginrpcutil under bufpkg?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
the wasmRunner
in no way depends on bufconfig.PluginConfig
, I'm not sure what you are talking about.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Referring to the pluginrpcutil
. All good on moving the wasm runner.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Moved the Wasm pkg to pkg/wasm
. The pluginrpcutil
pkg has a new function NewWasmRunner
that takes a wasm.Runtime
similar to the command.Runner
. The wasm.Runtime
is not implemented as a Run
method to allow the caller to manage the caching of plugins.
private/pkg/wasm/runtime.go
Outdated
} | ||
runtimeConfig = runtimeConfig.WithCompilationCache(cache) | ||
} | ||
r := wazero.NewRuntimeWithConfig(ctx, runtimeConfig) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We don't use single character variable names in this codebase beyond method receivers
private/pkg/wasm/wasm.go
Outdated
} | ||
|
||
// UnimplementedRuntime is an unimplemented Runtime. | ||
var UnimplementedRuntime = unimplementedRuntime{} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
global vars always go at the top, just below global consts
I just merged a PR that will conflict with CHANGELOG.md, you'll need to merge |
Add support for running custom plugins compiled with WebAssembly. The plugins must conform to the WASI 0.1 specification. This system interface is supported by many languages. For bufplugin-go based plugins see the Go WASI docs. Supporting plugins with the WASI specification allows the running of plugins using the pluginrpc protocol without any code modifications required by the user.
Declare a plugin with the suffix
.wasm
to use the Wasm runtime to invoke the plugin. For example the plugin buf-plugin-field-option-safe-for-ml can be compiled withGOOS=wasip1 GOARCH=wasm
and added to yourbuf.yaml
config: