Provide a REST API for Vegeta commands

[nitishm]

Has anyone considered creating REST endpoints for all the Vegeta commands, that would allow you to run a master server that a client can connect to, to initiate load tests.
This could also help with a more robust distributed test framework capable of being deployed on kubernetes, #336

[fenollp]

I’d like that! I’d use it as a cache warmer with a few requests per second.

I have something running based on vegeta already. It generates the next requests to run (goes through all paginantes endpoints) but there are issues such as sometimes the attacker sends a nil object in the response channel...

[waghanza]

This is a very good idea, turning vegeta into a API, this could help some developers create some continuous benchmarking workflow

I like this feature

[nitishm]

I am just not sure if it should be part of this repo or a separate one. Something like vegeta-server. I am starting to create and Swagger spec for the attack command using OAS 3.0.
@tsenart WDYT?

[nitishm]

@waghanza @fenollp @tsenart Would you prefer the REST server to use the library or use exec and invoke the vegeta CLI ? I cooked up a simple swagger spec, and generated the server code in go using `go-swagger, but am stuck at this cross-road.

Let me know what you guys think ?

[fenollp]

Use the lib ;)

[tsenart]

This is a great idea and something I have played around with before, but didn't have the time to pursue further. I think this should be part of vegeta, but a separate sub-command like vegeta server.

[nitishm]

@tsenart I have started working on this. IMO it might be better suited for a separate repository, that uses the vegeta library.

I am creating a Swagger spec for this, but I think we will need to support a more async model if we want to run mutliple attacks via a single server.
I can elaborate a little more, but as a quick example, if the user wishes to check on the status of a submitted attack (POST /attack) job, we could use an endpoint like GET attack/{uuid}. (Adds the need for a data store for the reports, as well. In memory map to begin with and then graduate to options to integrate with redis or similar DBs).
Also it would be great to provide a gRPC server option as well along with HTTP.

So the separate repository can be used for the swagger and protobuf specs and also two server implementations, one using HTTP and the other using gRPC.

WDYT?

[tsenart]

IMO it might be better suited for a separate repository, that uses the vegeta library.

To make it clear to everyone in the future about this decision, can we elaborate in writing about the pro / cons of each approach?

[tsenart]

BTW, just found this on Docker hub. Might be worth contacting the author to share learnings: https://hub.docker.com/r/endianogino/vegeta-server

[nitishm]

IMO it might be better suited for a separate repository, that uses the vegeta library.

To make it clear to everyone in the future about this decision, can we elaborate in writing about the pro / cons of each approach?

Pros:
Creating the server package, and wrappers that utilize the vegeta library, as a separate entity makes it easier to maintain the special handing logic of the API request/response, disjoint from the main library (and repo) and also prevents code bloat in the main repo (since swagger and gRPC servers will add a lot of code via codegen).

Cons:
The version of the library used will depend on the lock file, so it could fall behind and will manually have to maintained. Additionally fixes and improvements in the CLI code will have to be manually introduced into the server code (since it will be based on the CLI code for the most part). The latter goes both ways because the CLI code is tightly coupled with the command line flags, and cannot be reference directly by the server code.

[tsenart]

Creating the server package, and wrappers that utilize the vegeta library, as a separate entity makes it easier to maintain the special handing logic of the API request/response

Can you elaborate a bit more on what would make it easier?

also prevents code bloat in the main repo (since swagger and gRPC servers will add a lot of code via codegen).

This is a non-issue in my view.

[nitishm]

Creating the server package, and wrappers that utilize the vegeta library, as a separate entity makes it easier to maintain the special handing logic of the API request/response

Can you elaborate a bit more on what would make it easier?

This was a more generic statement. A separate entity is always going to be easier to maintain rather than a large project.

But with that said my main concern with the keeping everything in this repo is having the CLI code in the root directory. I completely understand the choice since it is the sole utilizer of the lib, but if we were to add the server (swagger codegen and grpc codegen) it would make sense to refactor this command line logic into a cli package of sorts and then have the server as its own package.
Refactoring and conforming the repo to the standard https://github.com/golang-standards/project-layout/blob/master/cmd/README.md would make it easier to abstract reusable segments out and also encapsulate various components (be it the spec files, helper scripts, k8s resources) under the same parent repo.
This is just my 2 cents.

[tsenart]

But with that said my main concern with the keeping everything in this repo is having the CLI code in the root directory.

It's perfectly reasonable to create the server code in lib/ or lib/server and then import it in the CLI.

[nitishm]

Cool. Let me see if I can figure out how to get go-swagger to generate a more flat directory structure.

[domleb]

I added my comment to #336 regarding creating distributed tests. I think creating a server that starts tests isn't required and will make this tool overly complicated. How would you manage resources if multiple clients try to run tests at the same time? Better to do one thing well and use other tools for orchestration.

[nitishm]

I have created a project that implements the REST API for vegeta - https://github.com/nitishm/vegeta-server
Personally it was easier to spin this off into its own project rather than integrate it into the vegeta CLI repository.