howto_create external plugin

Table of Contents

• Introduction to CobiGen external plug-ins
  • General intuition
  • How to create new external plug-in
    • Server (external process)
    • CobiGen plug-in
  • Testing phase
  • Releasing
    • Release the server
    • Releasing CobiGen plug-in
  • Templates creation
    • Testing templates

Introduction to CobiGen external plug-ins

Since September of 2019, a major change on CobiGen has taken place. CobiGen is written in Java code and previously, it was very hard for developers to create new plug-ins in other languages.

Creating a new plug-in means:

• Being able to parse a file in that language.

• Create a human readable model that can be used to generate templates (by retrieving properties from the model).

• Enable merging files, so that user’s code does not get removed.

For the Java plug-in it was relatively easy. As you are inside the Java world, you can use multiple utilities or libraries in order to get the AST or to merge Java code. With this new feature, we wanted that behaviour to be possible for any programming language.

General intuition

Below you will find a very high level description of how CobiGen worked in previous versions:

Basically, when a new input file was sent to CobiGen, it called the input reader to create a model of it (see here an example of a model). That model was sent to the template engine.

Afterwards, the template engine generated a new file which had to be merged with the original one. All this code was implemented in Java.

On the new version, we have implemented a handler (ExternalProcessHandler) which connects through TCP/IP connection to a server (normally on localhost:5000). This server can be implemented in any language (.Net, Node.js, Python…​) it just needs to implement a REST API defined here. The most important services are the input reading and merging:

CobiGen acts as a client that sends requests to the server in order to read the input file and create a model. The model is returned to the template engine so that it generates a new file. Finally, it is sent back to get merged with the original file.

How to create new external plug-in

The creation of a new plug-in consists mainly in three steps:

• Creation of the server (external process).

• Creation of a CobiGen plug-in.

• Creation of templates.

Server (external process)

The server can be programmed in any language that is able to implement REST services endpoints. The API that needs to implement is defined with this contract. You can paste the content to https://editor.swagger.io/ for a better look.

We have already created a NestJS server that implements the API defined above. You can find the code here which you can use as an example.

As you can see, the endpoints have the following naming convention: processmanagement/todoplugin/nameOfService where you will have to change todo to your plug-in name (e.g. rustplugin, pyplugin, goplugin…​)

When implementing service getInputModel which returns a model from the input file there are only two restrictions:

• A path key must be added. Its value can be the full path of the input file or just the file name. It is needed because in CobiGen there is a batch mode, in which you can have multiple input objects inside the same input file. You do not need to worry about batch mode for now.

• On the root of your model, for each found key that is an object (defined with brackets [{}]), CobiGen will try to use it as an input object. For example, this could be a valid model:

  {
    "path": "example/path/employee.entity.ts"
    "classes": [
      {
        "identifier": "Employee",
        "modifiers": [
          "export"
        ],
        "decorators": [
          {
            "identifier": {
              "name": "Entity",
              "module": "typeorm"
            },
            "isCallExpression": true
          }
        ],
        "properties": [
          {
            "identifier": "id",
      ...
      ...
      ...
      }]
      "interfaces": [{
          ...
      }]
  }

For this model, CobiGen would use as input objects all the classes and interfaces defined. On the templates we would be able to do model.classes[0].identifier to get the class name. These input objects depend on the language, therefore you can use any key.

In order to test the server, you will have to deploy it on your local machine (localhost), default port is 5000. If that port is already in use, you can deploy it on higher port values (5001, 5002…​). Nevertheless, we explain later the testing process as you need to complete the next step before.

Important

Your server must accept one argument when running it. The argument will be the port number (as an integer). This will be used for CobiGen in order to handle blocked ports when deploying your server. Check this code to see how we implemented that argument on our NestJS server.

CobiGen plug-in

You will have to create a new CobiGen plug-in that connects to the server. But do not worry, you will not have to implement anything new. We have a CobiGen plug-in template available, the only changes needed are renaming files and setting some properties on the pom.xml. Please follow these steps:

• Get the CobiGen plug-in template from here. It is a template repository (new GitHub feature), so you can click on "Use this template" as shown below:

  

• Name your repo as cobigen-name-plugin where name can be python, rust, go…​ In our case we will create a nest plug-in. It will create a repo with only one commit which contains all the needed files.

• Clone your just created repo and import folder cobigen-todoplugin as a Maven project on any Java IDE, though we recommend you devonfw ;)

  

• Rename all the todoplugin folders, files and class names to nameplugin. In our case nestplugin. In Eclipse you can easily rename by right clicking and then refactor → rename:

Note	We recommend you to select all the checkboxes

• Remember to change in src/main/java and src/test/java all the package, files and class names to use your plug-in name. The final result would be:

  

• Now we just need to change some strings, this is needed for CobiGen to register all the different plugins (they need unique names). In class TodoPluginActivator (in our case NestPluginActivator), change all the todo to your plug-in name. See below the 3 strings that need to be changed:

  

• Finally, we will change some properties from the pom.xml of the project. These properties define the server (external process) that is going to be used:

  1. Inside pom.xml, press Ctrl + F to perform a find and replace operation. Replace all todo with your plugin name:

     

  2. We are going to explain the server properties:

     1. artifactId: This is the name of your plug-in, that will be used for a future release on Maven Central.

     2. plugin.name: does not need to be changed as it uses the property from the artifactId. When connecting to the server, it will send a request to localhost:5000/{plugin.name}plugin/isConnectionReady, that is why it is important to use an unique name for the plug-in.

     3. server.name: This defines how the server executable (.exe) file will be named. This .exe file contains all the needed resources for deploying the server. You can use any name you want.

     4. server.version: You will specify here the server version that needs to be used. The .exe file will be named as {server.name}-{server.version}.exe.

     5. server.url: This will define from where to download the server. We really recommend you using NPM which is a package manager we know it works well. We explain here how to release the server on NPM. This will download the .exe file for Windows.

     6. server.url.linux: Same as before, but this should download the .exe file for Linux systems. If you do not want to implement a Linux version of the plug-in, just use the same URL from Windows or MacOS.

     7. server.url.macos: Same as before, but this should download the .exe file for MacOS systems. If you do not want to implement a MacOS version of the plug-in, just use the same URL from Linux or Windows.

Testing phase

Now that you have finished with the implementation of the server and the creation of a new CobiGen plug-in, we are going to explain how you can test that everything works fine:

1. Deploy the server on port 5000.

2. Run mvn clean test on the CobiGen-plugin or run the JUnit tests directly on Eclipse.

   1. If the server and the plug-in are working properly, some tests will pass and other will fail (we need to tweak them).

   2. If every test fails, something is wrong in your code.

3. In order to fix the failing tests, go to src/test/java. The failing tests make use of sample input files that we added in sake of example:

   

Replace those files (on src/test/resources/testadata/unittest/files/…​) with the correct input files for your server.

Releasing

Now that you have already tested that everything works fine, we are going to explain how to release the server and the plug-in.

Release the server

We are going to use NPM to store the executable of our server. Even though NPM is a package manager for JavaScript, it can be used for our purpose.

• Get the CobiGen server template from here. It is a template repository (new GitHub feature), so you can click on "Use this template" as shown below:

  

• Name your repo as cobigen-name-server where name can be python, rust, go…​ In our case we will create a nest plug-in. It will create a repo with only one commit which contains all the needed files.

• Clone your just created repo and go to folder cobigen-todo-server. It will just contain two files: ExternalProcessContract.yml is the OpenAPI definition which you can modify with your own server definition (this step is optional), and package.json is a file needed for NPM in order to define where to publish this package:

  {
    "name": "@devonfw/cobigen-todo-server",
    "version": "1.0.0",
    "description": "Todo server to implement the input reader and merger for CobiGen",
    "author": "CobiGen Team",
    "license": "Apache"
  }

Those are the default properties. This would push a new package cobigen-todo-server on the devonfw organization, with version 1.0.0. We have no restrictions here, you can use any organization, though we always recommend devonfw.

Note	Remember to change all the todo to your server name.

• Add your executable file into the cobigen-todo-server folder, just like below. As we said previously, this .exe is the server ready to be deployed.

  cobigen-template-server/
   |- cobigen-todo-server/
     |- ExternalProcessContract.yml
     |- package.json
     |- todoserver-1.0.0.exe

• Finally, we have to publish to NPM. If you have never done it, you can follow this tutorial. Basically you need to login into NPM and run:

  cd cobigen-todo-server/
  npm publish --access=public

Note	To release Linux and MacOS versions of your plug-in, just add the suffix into the package name (e.g. @devonfw/cobigen-todo-server-linux)

That’s it! You have published the first version of your server. Now you just need to modify the properties defined on the pom of your CobiGen plug-in. Please see next section for more information.

Releasing CobiGen plug-in

• Change the pom.xml to define all the properties. You can see below a final example for nest:

  ...
     <groupId>com.devonfw.cobigen</groupId>
     <artifactId>nestplugin</artifactId>
     <name>CobiGen - Nest Plug-in</name>
     <version>1.0.0</version>
     <packaging>jar</packaging>
     <description>CobiGen - nest Plug-in</description>
     
     <properties>
        <!-- External server properties -->
        <plugin.name>${project.artifactId}</plugin.name>
        <server.name>nestserver</server.name>
        <server.version>1.0.0</server.version>
        <server.url>https\://registry.npmjs.org/@devonfw/cobigen-nest-server/-/cobigen-nest-server-${server.version}.tgz</server.url>
        <server.url.linux>https\://registry.npmjs.org/@devonfw/cobigen-nest-server-linux/-/cobigen-nest-server-linux-${server.version}.tgz</server.url.linux>
        <server.url.macos>https\://registry.npmjs.org/@devonfw/cobigen-nest-server-macos/-/cobigen-nest-server-macos-${server.version}.tgz</server.url.macos>
  ...

• Deploy to Maven Central.

Templates creation

After following above steps, we now have a CobiGen plug-in that connects to a server (external process) which reads your input files, returns a model and is able to merge files.

However, we need a key component for our plug-in to be useful. We need to define templates:

• Fork our CobiGen main repository, from here and clone it into your PC. Stay in the master branch and import into your IDE cobigen-templates\templates-devon4j. Set the Java version of the project to 1.8 if needed.

• Create a new folder on src/main/templates, this will contain all your templates. You can use any name, but please use underscores as separators. In our case, we created a folder crud_typescript_angular_client_app to generate an Angular client from a TypeORM entity (NodeJS entity).

  

• Inside your folder, create a templates folder. As you can see below, the folder structure of the generated files starts here (the sources). Also we need a configuration file templates.xml that should be on the same level as templates/ folder. For now, copy and paste a templates.xml file from any of the templates folder.

  

• Start creating your own templates. Our default templates language is Freemarker, but you can also use Velocity. Add the extension to the file (.ftl) and start developing templates! You can find useful documentation here.

• After creating all the templates, you need to modify context.xml which is located on the root of src/main/templates. There you need to define a trigger, which is used for CobiGen to know when to trigger a plug-in. I recommend you to copy and paste the following trigger:

  <trigger id="crud_typescript_angular_client_app" type="nest" templateFolder="crud_typescript_angular_client_app">
    <matcher type="fqn" value="([^\.]+).entity.ts">
      <variableAssignment type="regex" key="entityName" value="1"/>
      <variableAssignment type="regex" key="component" value="1"/>
      <variableAssignment type="constant" key="domain" value="demo"/>
    </matcher>
  </trigger>

• Change templateFolder to your templates folder name. id you can use any, but it is recommendable to use the same as the template folder name. type is the TRIGGER_TYPE we defined above on the NestPluginActivator class. On matcher just change the value: ([^\.]+).entity.ts means that we will only accept input files that contain "anyString.entity.ts". This improves usability, so that users only generate using the correct input files. You will find more info about variableAssignment here.

• Finally, is time to configure templates.xml. It is needed for organizing templates into increments, please take a look into this documentation.

Testing templates

• When you have finished your templates you will like to test them. On the templates-devon4j pom.xml remove the SNAPSHOT from the version (in our case the version will be 3.1.8). Run mvn clean install -DskipTests on the project. We skip tests because you need special permissions to download artifacts from our Nexus. Remember the version that has just been installed:

  

Note	We always recommend using the devonfw console, which already contains a working Maven version.

• Now we have your last version of the templates ready to be used. We need to use that latest version in CobiGen. We will use the CobiGen CLI that you will find in your cloned repo, at cobigen-cli/. Import the project into your IDE.

• Inside the project, go to src/main/resources/pom.xml. This pom.xml is used on runtime in order to install all the CobiGen plug-ins and templates. Add there your latest templates version and the previously created plug-in:

  

• Afterwards, run mvn clean install -DskipTests and CobiGen will get your plug-ins. Now you have two options to test templates:

  1. Using Eclipse run as:

     1. Inside Eclipse, you can run the CobiGen-CLI as a Java application. Right click class CobiGenCLI.java → run as → run configurations…​ and create a new Java application as shown below:

        

     2. That will create a CobiGenCLI configuration where we can set arguments to the CLI. Let’s first begin with showing the CLI version, which should print a list of all plug-ins, including ours.

        

         ...
         name:= propertyplugin version = 2.0.0
         name:= jsonplugin version = 2.0.0
         name:= templates-devon4j version = 3.1.8
         name:= nestplugin version = 1.0.0
         ...

     3. If that worked, now you can send any arguments to the CLI in order to generate with your templates. Please follow this guide that explains all the CLI commands.

  2. Modify the already present JUnit tests: They test the generation of templates from multiple plug-ins, you can add your own tests and input files.