Decouple runtime and tools: brainstorm and roadmap

[pfiaux]

I'm opening this issue to outline possible next steps based on the discussions started in #74 and #80.

There's some very good points about splitting the runtime elements from the tools, and I want to outline a plan here to move forward. What I'd like to discuss here is what changes need to be made short term to enable a flexible and more salable architecture, the mid to long term roadmap will be discussed in a separate issue.

Roadmap

1. Decouple tools from runtime into separate repositories/modules

• neeo-sdk would contain driver & server components (what you need to write a driver)
  • @neeo/sdk-driver
  • @neeo/sdk-server
• neeo-toolkit would contain the CLI, config/settings tools, and the driver manger for example (what you need to run a driver or develop)
  • @neeo/cli

2. Update the sdk examples and documentation to showcase the new setup

Since breaking changes are annoying we probably also should plan a transition over 1 or 2 versions to the new setup (first putting it in place and add deprecation warnings, then schedule removal of the old calls).

Background

Mid/Long term roadmap:

For example:

• Driver capability updates Device Update: Add driver versioning and updating of capabilities #18
• Favorites [Feature Request] Support Favorites for SDK devices #52
• Running drivers on the Brain
• Driver registry

For now these should be discussed in separate issues. I would like to focus on the short term discussion in this issue.

Short term roadmap:

We have several different components and they would be better served more loosely coupled. The general idea is to move away from the current setup to meet the following goals:

• loosely coupled components, with clearer responsibilities
• better defined best practices and driver standards (documentation and neeo-sdk-examples)
• decouple sdk driver/runtime from tools
• make the tools more community driven and focused
• allow the tools to evolve based on community needs and more independent of the NEEO firmware release cycle

Building on the suggestions from @Shepless in #80 and some of our internal discussions (names not final) here's an example breakdown:

• neeo-sdk/driver
• neeo-sdk/server
• neeo-sdk-tools/cli
• neeo-toolkit/manager (NEEO - Driver Manager #74)
• neeo-toolkit/config (feat(user-settings): Allow Custom Settings #80)
• neeo-toolkit/*

neeo-sdk/*

We might keep this as one repo and publish it as 2 modules.

driver: tools needed to write a driver, handlers, callbacks, wiring
server: allows loading and running drivers

There's some discussion on whether a server should run multiple drivers or not. I think if we make both of these separate entities there's no reason it can't do both leaving it up to the users of the server to decide whether to start one per device or one to rule them all.

Since these 2 modules are also closely tied to the Brain firmware, their development will be officially done by NEEO in most cases. But as currently we'll use the issue tracker to discuss bugs, features and more.

Part of this will be consolidating the best practices for writing drivers, coding style and such so we'll update our sdk examples at the same time to reflect that.

neeo-toolkit/*

We'd move everything that's not server/driver here (including the CLI). The code here would not be tied closely to the Brain firmware can more free to evolve at a different pace.

I'll check with @neophob but for brainstorming reasons I'll suggest we create in a mono repo style as well with multiple modules as mentioned above.

An alternative would be to have several repositories not necessarily NEEO owned, we can discuss that as well. Advantages I see to having one tools repo at least for the official tools under NEEO:

• It would keep the dicussions centralized (1 issue tracker/repo)
• Probably easier to maintain in the long run with our support behind it
• Easier to keep a unified look and feel between different tools

While we would be holding the repository, I think we would fill a support role and let it be more community driven when it comes to development.

That means:

• CLI - would be moved here, as a developer tool
  • At first we would keep it's current run server form
  • Similar to @Shepless's efforts in https://github.com/shepless/neeo-cli scaffolding tools are part of the obvious next steps
  • Would use the new neeo-sdk/server
• Driver Manager - Could be merged into this tools repo
  • Allowing a managed sdk server setup (NEEO - Driver Manager #74)
  • Would use the new neeo-sdk/server
  • It could also remain as is but would then only require neeo-sdk/server to run drivers.
• Config utilities - Allowing a shared settings format (feat(user-settings): Allow Custom Settings #80)
  • To keep it flexible I think this would be a better fit here than part of the neeo-sdk/driver
  • provide easy file config that can be done outside of version control
  • using the schema the Driver Manager (and/or CLI) could be use to update these settings

Misc naming conventions

Since I know naming can cause troubles down the road I'll open the discussion here.

If we look at the code we're curently building devices and we actually called part of the server the expressBrainDriver. That puts us at odds with the terms I've seen used on github and planet NEEO which are usually driver and server respectively.

Personally I'm not set on one or the other, but I believe it will be easier if we do avoid having confusing meetings. So IMHO either the expressBrainDriver is renamed or we call drivers devices. Thoughts and feedback welcome.

[pfiaux]

As @bdauria mentioned we're also considering a switch to typescript for the neeo-sdk (driver/server). I would also put that in the short/mid term plan.

[pfiaux]

Another element of the current NEEO SDK that should be split off to the tools repository is the recipe API component which gets some recipes and lists power states.

[bdauria]

@pfiaux regarding the naming, I think it makes sense to keep both:
a driver encapsulates one or multiple drivers. Of course this makes sense only if we allow multiple devices per module, otherwise a device is just a device.
In that case, we can rename the expressBrainDriver to expressBrainAdapter or something like that.

[Shepless]

Really awesome @pfiaux @bdauria

I agree with everything mentioned above and will definitely love to help contribute in anyway I can. I've been busy all this week and will be on holiday soon but I will be spending some time in the next couple of days fleshing out some new repo's and npm packages I have been working on locally for you guys. Hopefully they will be of some use.

@bdauria my honest opinion is a single device export per package as it makes the ecosystem around the devices much more predictable and easier to build around.

[pfiaux]

@Shepless regarding your comment in #100 (comment) on the best way to run multiple drivers (single vs multiple servers and process forking), if you don't mind I think it fits better to discuss it here.

Currently the plan is to decouple the server and the driver, then single and multiple server is possible. The way I see it if both are possible it's an implementation detail so should be left to the implementer (CLI, driver manager, ...). Do you see some arguments against allowing both and leaving it to the implementation?

[pfiaux]

Small update we've published some SDK examples update in the next branch (we'll updated them again when we decouple driver/server components of the SDK). We're also working on splitting off the CLI to it's own repository/module then we'll come back to this repo and split the driver/server into separate modules.

[pfiaux]

The new @neeo/cli is now available here:
https://www.npmjs.com/package/@neeo/cli
https://github.com/NEEOInc/neeo-sdk-toolkit/tree/master/cli