Skip to content

Tasks

Jump to bottom
John Gamboa edited this page Sep 18, 2017 · 27 revisions

Introduction

At OSA, we are aiming at solving five important tasks in Conversational Intelligence. These tasks have been developed keeping in mind their business utility and scientific contribution. The following are the tasks

1. Core Components of Ovation-CI: Develop the building blocks of a Conversational Intelligence framework, which are,

        a. Intent Classification

        b. Entity Recognition

        c. Sentiment Analysis

        d. Integration with rasa-NLU

2. Business Use-cases: Develop demos for three business use-cases that will be presented at OSA using some existing open-source frameworks like Rasa.

3. Ovation Voice: Develop demos for the same use-cases (as mentioned above) using the Ovation Voice Interface.

The following sections will describe each of these tasks in detail.

1.a Intent Classification

Intent classification is usually the first component of any Conversational Intelligence framework. The second task that we are going to work on at OSA is Intent Classification. We do not differentiate the task of Semantic Text Similarity (STS) and Intent Classification. In Semantic Text Similarity, one generates a similarity score of two given texts. This idea can be extended and a new model can be trained to perform Intent Classification. For this task, the following is expected of the participants,

  1. Create a set of intents using the rasa-nlu-trainer. Keep in mind that our goal is to train our models with as less data as possible.

  2. Train an Intent Classifier using the intents that you created.

  3. Deploy your model as a server

If you want to refer to some template STS models that you can use out of the box, then you can find them here. If you want to know how to use existing templates then you can refer to the Using Templates page.

Required Endpoints

The Intent Classification Server that you will deploy should have the following endpoints

  1. /preinit: This should contain all the boilerplate code for loading the model, building the data structures, etc. It should send a {"status": 1} on success and {"status": 0} on failure as a response.

  2. /train: This should contain the code to train a new model given input data in the following format:

{
  "data": [
    {
      "text": "yes",
      "intent": "affirm",
      "entities": []
    },
    {
      "text": "yep",
      "intent": "affirm",
      "entities": []
    },
    {
      "text": "yeah",
      "intent": "affirm",
      "entities": []
    },
    {
        "text": "Techniker Krankenkasse offices in Kaiserslautern",
        "intent": "inquiry",
        "entities": [
          {
            "start": 34,
            "end": 48,
            "value": "Kaiserslautern",
            "entity": "location"
          },
          {
            "start": 0,
            "end": 22,
            "value": "Techniker Krankenkasse",
            "entity": "organisation"
          }
        ]
      }
    ]
}

This endpoint should send {"status": 1} as a response when the training is over, and {"status": 0} if some error occurred or the model could not be trained for some reason. You should ideally add a status message as well. We leave everything else to your creativity.

  1. /save: This endpoint should save the current model that is being used. Similar to /train, it should also send {"status": 1} on success and {"status": 0} on failure as a response message.

  2. /cleanup: This method should implement releasing of resources like, cleaning the memory, deleting files, persisting data structures, etc. It should send {"status": 1} on success and {"status": 0} on failure as a response message.

  3. /infer: This should implement the model inference code. E.g., given data as input like {"data": "hello Ovation"} it should return all possible intents with their confidence scores. E.g.,

{
  "query": "hello Ovation",
  "intents": [
    {
      "intent": "greetings",
      "intent_id": 1,
      "confidence": 0.85
    },
    {
      "intent": "welcome",
      "intent_id": 2,
      "confidence": 0.15
    }
    ],
  "status": 1,
  "message": "success"
}

On failure, this should return {"status": 0}

1.b Entity Recognition

Given a user's query we want to extract the Named Entities from them. This task has the following requirements,

  1. Train a Named Entity Recognition (NER) model using a NER dataset (English/German). You will find more information about the datasets supported by the Ovation framework here.

  2. Deploy your model as a server

If you want to refer to some template NER models that you can use out of the box, then you can find them here. If you want to know how to use existing templates then you can refer to the Using Templates page.

Required Endpoints

The Entity Recognition Server that you will deploy should have the following endpoints,

  1. /preinit: This should contain all the boilerplate code like loading the model, building the data structures, etc. It should send a {"status": 1} on success and {"status": 0} on failure as a response.

  2. /train: This should contain the code to train a new model given input data in the following format:

{
  "data": [
    {
        "text": "Techniker Krankenkasse offices in Kaiserslautern",
        "intent": "inquiry",
        "entities": [
          {
            "start": 34,
            "end": 48,
            "value": "Kaiserslautern",
            "entity": "location"
          },
          {
            "start": 0,
            "end": 22,
            "value": "Techniker Krankenkasse",
            "entity": "organisation"
          }
        ]
      }
    ]
}

This endpoint should send {"status": 1} on success and {"status": 0} on failure as a response. You should ideally add a status message as well. We leave everything else to your creativity.

  1. /save: This endpoint should save the current model that is being used. Similar to /train, it should also send {"status": 1} on success and {"status": 0} on failure as a response message.

  2. /cleanup: This method should implement releasing of resources like, cleaning the memory, deleting files, persisting datastructures, etc. It should send {"status": 1} on success and {"status": 0} on failure as a response message.

  3. /infer: This should implement the model inference code. E.g., given data as input like {"data": "hello Ovation"} it should return all possible entities, their location (start and end indices) in the query and their confidence scores. E.g.,

{
  "query": "hello Ovation",
  "entities": [
        {
          "start": 6,
          "end": 12,
          "value": "Ovation",
          "entity": "organisation",
          "confidence": 0.76
        }
    ]
}

On failure, this should return {"status": 0}

1.c Sentiment Analysis

The third core component that we will focus on in OSA is Sentiment Analysis. For this task, we not just want to generate a sentiment score given an input query. What we are also interested in is to give a cause or give a reason for why did the model give this score. In this task, you have the following requirements,

  1. Train a Sentiment Analysis model using a Sentiment Analysis dataset in English/German. You will find more information about the datasets supported by the Ovation framework here.

  2. Deploy your model as a server

If you want to refer to some template Sentiment Analysis models that you can use out of the box, then you can find them here. If you want to know how to use existing templates then you can refer to the Using Templates page.

Required Endpoints

The Sentiment Analysis Server that you will deploy should have the following endpoints,

  1. /preinit: This should contain all the boilerplate code like loading the model, building the data structures, etc. It should send a {"status": 1} on success and {"status": 0} on failure as a response.

  2. /train: This should contain the code to train a new model given input data in the following format:

{
  "data": [
    {
        "text": "Thanks a lot. Good Job Ovation",
        "sentiment": 1,
    }
  ]
}

The sentiment parameter should be a value between 0 to 1.

This endpoint should send {"status": 1} on success and {"status": 0} on failure as a response. You should ideally add a status message as well. We leave everything else to your creativity.

  1. /save: This endpoint should save the current model that is being used. Similar to /train, it should also send {"status": 1} on success and {"status": 0} on failure as a response message.

  2. /cleanup: This method should implement releasing of resources like, cleaning the memory, deleting files, persisting datastructures, etc. It should send {"status": 1} on success and {"status": 0} on failure as a response message.

  3. /infer: This should implement the model inference code. E.g., given data as input like {"data": "I do not like your suggestion"} it should return a sentiment score between 0 and 1, a coarse sentiment value ("neutral", "positive", "negative"), cause/reason for generating this score ("not like" in the example) and the location in the text of the causes (if applicable). E.g.,

{
  "query": "I do not like your suggestion",
  "sentiment": {
    "score": 0.1,
    "coarse": "neutral",
    "cause": [
      {
        "text": "do not like your suggestion",
        "start": 2,
        "end": 28
      }
    ]
  }
}

On failure, this should return {"status": 0}

1.d Integration with rasa-NLU

Once we have servers for any of the previous tasks, we would like to try to integrate them with Rasa. Rasa uses a pipeline for processing its input, and we could exploit this pipeline by adding new modules in the pipeline or modifying some of the existing modules according to our needs. You can find more about the pipeline here.

For more information on how exactly to do that, refer to the page about the Rasa integration, or, alternatively, talk to Shan, who has had some successes in this direction.

2. Business Use-cases

Three Business use cases will be presented at OSA and the task will be to develop a demo for each of the three use cases. Demos for three business use-cases can be developed using some existing open-source frameworks like Rasa. The goal is to integrate Ovation-CI in the end with the demos. Keep in mind that if you integrate with Rasa then you might want to have a module that makes it easier to integrate with Ovation-CI too.

3. Ovation Voice

Implement the above three use cases as demos by using the Ovation-voice API. These demos should be pure voice based. The details of the use cases will be shared at OSA

Sponsored By

Sponsors

Clone this wiki locally