[Proposal]: Data binding annotations for custom and actual names in Ballerina fields and resource/remote parameters

[lnash94]

Updated with review : 2024/09/02

Summary

This proposal suggests adding annotation features to improve Ballerina’s usability and compatibility with modern web standards. Seamless JSON serialization and deserialization are crucial for its adoption in modern software development, especially for web APIs and services.

Goals

• Add annotation features to Ballerina for better usability and compatibility with modern web standards, facilitating clear and maintainable code for web APIs and services.

Motivation

[1] Referring to the OAS sample, since this annotation is required to be part of this implementation #6867

openapi: "3.0.0"
info:
  version: v3
  title: Karbon API 3.0
servers:
  - url: http://localhost:9090/v1
paths:
  /customer/groups:
    get:
      …
      parameters:
        - name: "customer-group" \\query param
          in: query
          schema:
            type: string
      responses:
        '200':
          description: successful
          content:
            "application/json":
              schema:
                $ref: "#/components/schemas/Person"
components:
  schemas:
   Person
      type: object
      required:
         - full-name
         - last-name
         - table
      properties:
        full-name:  \\property-name
          type: string
        last-name:  \\property-name
          type: string
        age: 
          type: integer
        table: 
          type: string

In the above OAS example, it has REST API get with query parameter name, customer-group which is not aligned with Ballerina naming conventions. when we writing the ballerina code we were supposed to use escape for the query parameter name. ex: customer\-group

[2] Referring to the JSON sample

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "title": "Person",
  "type": "object",
  "properties": {
    "first-name": {
      "type": "string",
    },
    "last-name": {
      "type": "string",
    },
    "age": {
      "type": "integer"
    },
    "table": {
      "type": "string"
    }
  "required": ["first-name", "last-name", "table"]
}

Given JSON sample[2] has fields with names "full-name", "last-name" and "table", which are not aligned with Ballerina naming conventions. In addition to that some scenarios can have a ballerina reserved key as a field name like table.
Therefore if the user needs to use this JSON in the Ballerina record we have to escape special characters like the below way

type Person record { 
    string full\-name; 
    string last\-name; 
    string 'table;
    int age?;
}

However, when using this generated record and the query parameters, users may face code readability issues. The escaped characters can make the code harder to read and understand, leading to confusion and potential mistakes during development.

Description

Through this proposal, we are suggesting having an annotation for users to override the record field name according to the preferred name.

Proposed designed

Annotation field for parameters

We propose introducing a new annotating field @http:Query parameters with metadata. The field will be specified using a syntax similar to the following:

public type HttpQuery record {|
   string name?;
|}

This HttpQuery annotation has one field for storing that parameter name.

Annotation for record fields

We propose introducing a new syntax for annotating record fields with JSON metadata. The annotations will be specified using a syntax similar to the following:

public type HttpRecordField record {|
   string name?;
|}

const annotation HttpRecordField Field on record field;

This RecordField annotation enriches name for the given record field name. The field format for the media for serialization which is optional can be added later.

This suggested approach will,

• Improved Interoperability: Enhances the Ballerina's ability to work with external systems and APIs that use JSON.
• Allow users to define custom names for fields when they are serialized to JSON

type Person record {
        @http:Field {| name: "age"|}
	int personAge;
}

• Cleaner Code: Reduces the need for manual JSON manipulation, leading to more maintainable and readable code.
• Handling Reserved Words and Unconventional Names
  ex: Here table is a reserved word

type Person record {
       @http:Field {| name: "table" |}
       string tableName;
}

User experience with proposed annotations (@http:Query , @http:Field):

[3] User experience for HTTP service type

import ballerina/http;
type CustomerServiceType service object {
    *http:Service;
    # Annotation using the query parameter name
    resource function get customer/groups(@http:Query {name: "customer-group"} string customerGroup) returns Person|error;
   };

# Annotation using in-record field
type Person record {
   @http:Field {| name: "first-name"|}
   string firstName;
   int age;
}

[4] User experience for HTTP service implementation

import ballerina/http;

service /v1 on new http:Listener(9090) {

    # Annotation using the query parameter name
    resource isolated function get customer/groups(@http:Query { name: "customer-group"} customerGroup ) returns Person|error {
        //logic here    ...
    }
}

# Annotation using in-record field
type Person record {
   @http:Field {| name: "first-name"|}
   string firstName;
   int age;
}

[5] User experience for HTTP client

[5.1] HTTP client with resource function

Currently, there is no way to define the attachment point for function arguments. Therefore, we decided to have the user provide a record parameter included in the query parameters and enable @http:Query within the record fields to capture query parameter-related meta information.

 http:Client httpClient = check new ("https://www.example.com");

# Represents the Queries record for the operation: getCustomerDetials 
public type GetCustomerGroupQueries record {
    @http:Query { name: "customer-group" }
    string? customerGroup?;
    //other remaining query parameters
};
GetCustomerGroupQueries queries = {
    customerGroup: "G11"
};
string response = check httpClient->/customer/group(params = queries);

[5.2] HTTP client with remote function

public client class Client {
    // The HTTP client to access the HTTP services.
    private final http:Client httpClient;

    function init(string url) returns error? {
        self.httpClient = check new (url);
    }

    remote function getCustomerGroups(GetCustomerGroupQueries customerGroup) returns Person|error {
        ...
        Person res = check httpClient->/customer/group(params = customerGroup);
       ...
    }
}
# annotation for query parameters in the client remote functions
public type GetCustomerGroupQueries record {
    @http:Query { name: "customer-group" }
    string? customerGroup?;
    //other remaining query parameters
};

Annotation usage for OpenAPI Tool generated client and service

The OpenAPI tool encountered an issue while generating Ballerina service and client code for a given OAS reference proposal.

• Service code generation: the tool included the @http:Query and @http:Field annotations in the generated service, as described here[3].

• Service contract type generation: the tool included the @http:Query and @http:Field annotations in the generated service contract type, as described here

import ballerina/http;
@http:ServiceConfig {basePath: "/v1"}
type OASServiceType service object {
    *http:ServiceContract;
    # Annotation using the query parameter name
    resource function get customer/groups(@http:Query {name: "customer-group"} string customerGroup) returns Person|error;
   };

# Annotation using in-record field
type Person record {
   @http:Field {| name: "first-name"|}
   string firstName;
   int age;
}

• Client Generation
  However, when generating the client, the tool observed an issue where the generated client treated the query parameters as included record parameters. Therefore we need to enable @http:Query annotation within those record fields.
  The rest of the payload, and response data-binding used @http:Field annotations for record fields.

User experience with generated code

import ballerina/http;

public client class Client {

    resource isolated function get customer/groups(map<string|string[]> headers = {}, *GetCustomerGroupQueries queries) returns Person|error {
        string resourcePath = string `/customer/groups`;
        // Need to support query generation here with openapi query annotation
        resourcePath = resourcePath + check getPathForQueryParam(queries);
        return self.clientEp->get(resourcePath, headers);
    }
}

# Represents the Queries record for the operation
public type GetCustomerGroupQueries record {
    #query annotation for the parameter 
    @http:Query{ name: "customer-group" }
    string? customerGroup?;
    //other remaining query parameters
};

# Annotation using the in-record field for payload binding
type Person record {
   @http:Field {| name: "first-name"|}
   string firstName;
   int age;
}

Annotation usage for OpenAPI Tool generated OpenAPI Contracts for particular service

The given @http:Field and @http:Query annotation names are mapped under the x-ballerina-name extension in the particular sections.

Given ballerina service with annotations

import ballerina/http;

service /v1 on new http:Listener(9090) {

    # Annotation using the query parameter name
    resource isolated function get customer/groups(@http:Query { name: "customer-group"} customerGroup ) returns Person{
        //logic here    ...
    }
}

# Annotation using in-record field
type Person record {
   @http:Field {| name: "first-name"|}
   string firstName;
   int age;
}

Expected generated OpenAPI contract

openapi: "3.0.0"
info:
 ...
paths:
  /customer/groups:
    get:
      …
      parameters:
        - name: "customer-group"
          x-ballerina-name: “customerGroup”   <--- query name extension
          in: query
          schema:
            type: string
      responses:
        '200':
          description: successful
          content:
            "application/json":
              schema:
                $ref: "#/components/schemas/Person"
...
components:
 schemas:
   Person:
     type: object
     properties:
       first-name: //property name
         type: string
         x-ballerina-name: firstName          <--property name extension 
       age:
         type: integer
       

[TharmiganK]

How about introducing a single annotation to represent the mapping between OpenAPI and Ballerina?

Annotation definition:

public type MappingInfo record {|
   string name;
|};

public const annotation MappingInfo Mapping on record field, parameter;

Sample usage:

type Person record {|
    @openapi:Mapping { name: "full-name" }
    string fullName;
    @openapi:Mapping { name: "phone-number" }
    string phoneNumber;
|};

[daneshk]

+1 for having a single annotation.

In xmldata, persist, we used theName annotation for both the record fields and parameters. Shall we use the same here

# Defines the schema name which matches the record field. The default value is the record field name. 
public type NameConfig record {|
    # The schema  fields and parameters value
    string value;
|};

# The Annotation used to map schema fields and param to record and record fields
public annotation NameConfig Name on record field, record;

[TharmiganK]

+1 for the openapi:Name annotation if we only need the name for mapping. But the proposal also includes format. @lnash94 Can you elaborate the requirement for this?

We need to have a section on how are we going to map this with the data-binding features provided by the Ballerina http package.

[lnash94]

@daneshk, @TharmiganK Following are some points to have two separate annotations for data binding

• An optional format field to the record field to capture whether the format is XML or JSON when performing data binding. This field requires discussion with the HTTP module regarding data binding (we can remove this field if the record fields always support JSON data binding).

• Upon further observation, the parameter may have specific fields beyond the record fields, such as parameter serialization options. Refer to Swagger's serialization documentation for more details.
  Currently, we have no way to represent these serialization details in query parameters. Therefore, introducing a specific annotation for parameters could allow room for future expansion of these details.

type ParamAnnotation record {|
   string name;
   string explode;
   string style;
|}

• We need to discuss whether this annotation should be included within the OpenAPI module or another module (HTTP or language level), as this proposal is based on general user requirements. If the user writes any client or service code then the user has no requirement to attach openapi module to their code.

[lnash94]

Attendees Meeting: @shafreenAnfar , @daneshk , @lnash94 , @TharmiganK
Proposal Meeting Notes:

1. Provide two separate annotations for the data binding with the name since these two areas have the potential to expand with different area-specific attributes
2. Implementing this annotation within the HTTP module since this annotation engages with data binding within the HTTP module
3. Use already existing @http:Query annotation for query parameter details mapping
4. Use one oas extension to map annotation details into OAS generation. (ex: x-bal-modified-name)

Attendees : @daneshk , @lnash94 , @TharmiganK
Annotation Name finalise

1. Provide @http:Field as the annotation name consistent with the query annotation
2. Provide @openapi:Query as a separate annotation for handling the OpenAPI tool's client generation. Since we have designed the query parameters to be passed as included record parameters in the function, they will be represented as record fields to the client connector. We need this annotation solely to concatenate the name, and we've decided to implement it on the OpenAPI tool side.

[lnash94]

2024/09/02 : @http:Query Annotations support for record fields

Attendees: @daneshk , @TharmiganK , @lnash94

• When trying to support @http:Query for native HTTP client , we observed that we can not attach attachment points into functions arguments, therefore we specialized this client query parameter support via the record parameter included type.
• Therefore we don't need to have separate openapi query parameter that discussed above comments
  

[lnash94]

Implement HTTP client-related task :

• Query annotation: [master] Add Query Parameter Annotation Mapping  module-ballerina-http#2143
• Data binding with toJson method [master] Use toJson method from data.jsondata module in http client and service module-ballerina-http#2204
• Annotation for record name to overwrite the JSON field implementation: Add support for overwriting field name in ToJson method module-ballerina-data.jsondata#35

[lnash94]

We can close this proposal since we provided all fixes and tested the change against the depended ballerina packages[1]
[1]
ballerina-platform/module-ballerina-websub#666
ballerina-platform/module-ballerina-websocket#1462
ballerina-platform/module-ballerina-grpc#1653
ballerina-platform/module-ballerina-websubhub#1039
ballerina-platform/module-ballerina-graphql#2088
ballerina-platform/module-ballerina-sql#748
ballerina-platform/module-ballerinax-kafka#1214
ballerina-platform/module-ballerinax-rabbitmq#1001