The aim of this experiment is to gain an understanding of the productivity gains that are possible when using generative AI technologies to aid with development and testing work.
A cloned repository will default to the main branch. This branch contains a development solution in three languages, C#, Java and JavaScript, a suite of Cypress API tests and a suite of Postman API tests.
If you wish to develop the web server you should checkout the start-here-dev branch. This contains a skeleton web service implementation in three languages, C#, Java and JavaScript, a suite of Cypress API tests and a suite of Postman API tests. If you wish to do the code race version of the development task then you need the start-here-code-race branch.
If you wish to implement the API tests you should checkout the start-here-test branch. This contains a development solution in three languages, C#, Java and JavaScript, a skeleton suite of Cypress API tests and a suite of Postman API tests.
git checkout <start-here-dev|start-here-test|start-here-code-race>
You should create your own branch to work on, commit your code and push it to the repository on your branch. We may wish to analyse the code that the generative AI helped produce as part of the outcomes of this experiment.
git checkout -b <new-branch-name>
The development task is to implement a web service that provides endpoints to manage a to do list. The choice of implementation language is at your discretion. Skeleton implementations have been provided in C#, Java and JavaScript on the start-here-dev branch under the src directory. It is also valid to start completely from scratch if you wish.
No guidance is provided as to the use of generative AI tools in completing the implementation of the web server. It is valid to implement a solution using generative AI and consider the experience and productivity gains as a standalone activity. It is also valid to implement a solution without using generative AI and then revert to the skeleton code and re-implement a solution using generative AI to compare the experience and the time taken for each implementation.
We are interested in all aspects of the development experience, not just the amount of time saved by using code generated by the AI tool.
The server data will be initialised with a list containing three tasks:
[
{
"uuid": "f360ba09-4682-448b-b32f-0a9e538502fa",
"name": "Walk the dog",
"description": "Walk the dog for forty five minutes",
"created": "2023-06-23T09:30:00Z",
"completed": null,
"complete": false
},
{
"uuid": "fd5ff9df-f194-4c6e-966a-71b38f95e14f",
"name": "Mow the lawn",
"description": "Mow the lawn in the back garden",
"created": "2023-06-23T09:00:00Z",
"completed": null,
"complete": false
},
{
"uuid": "5c3ec8bc-6099-4cd5-b6da-8e2956db3a34",
"name": "Test generative AI",
"description": "Use generative AI technology to write a simple web service",
"created": "2023-06-23T09:00:00Z",
"completed": null,
"complete": false
}
]
The intial GET endpoint returns the list of tasks held by the server. If you are starting from scratch you need to implement this endpoint before continuing.
The data above is stored in the file quantifying-ai/static_data/ToDoList.json within the project. Feel free to copy and paste it into the implementation, or load the data from the file.
Change the existing GET endpoint to accept an optional boolean parameter named complete. The returned list of tasks should be filtered based on the value given for the parameter, if supplied.
http://localhost:8080/todo{?complete=true}
A GET endpoint that takes an optional boolean parameter complete. If the parameter is given then the endpoint returns a list of tasks that have been filtered based on the value supplied for the parameter:
| URI | Required behaviour |
|---|---|
| http://localhost:8080/todo | Return a list of all tasks with HTTP status 200. |
| http://localhost:8080/todo?complete=true | Return a list only containing completed tasks with HTTP status 200. |
| http://localhost:8080/todo?complete=false | Return a list only containing incomplete tasks with HTTP status 200. |
Add a new GET endpoint that uses a uuid as a path parameter to return a specific task from the list of tasks.
http://localhost:8080/todo/{uuid}
A GET endpoint that uses a uuid as a path parameter to return the task with the supplied uuid from the list of tasks. The endpoint returns the task with the given uuid if it exists, otherwise a fixed UNKNOWN_TASK is returned. If an invalid uuid is supplied the endpoint will return a bad request error.
| URI | Required behaviour |
|---|---|
| http://localhost:8080/todo/{uuid} | Return the task with supplied uuid with HTTP status 200. |
Given the static data data above:
http://localhost:8080/todo/5c3ec8bc-6099-4cd5-b6da-8e2956db3a34 returns
{
"uuid": "5c3ec8bc-6099-4cd5-b6da-8e2956db3a34",
"name": "Test generative AI",
"description": "Use generative AI technology to write a simple web service",
"created": "2023-06-23T09:00:00Z",
"completed": null,
"complete": false
}
with HTTP status 200.
http://localhost:8080/todo/5c3ec8bc-6099-1a2b-b6da-8e2956db3a34 returns
{
"uuid": "00000000-0000-0000-0000-000000000000",
"name": "Unknown Task",
"description": "Unknown Task",
"created": "1970-01-01T00:00:00.000Z",
"completed": null,
"complete": false
}
with HTTP status 200.
http://localhost:8080/todo/invalid-uuid returns HTTP status 400 Bad Request. A meaningful response may also be returned, e.g.
{
"timestamp": "2023-06-27T12:32:05.590Z",
"status": 400,
"error": "Bad Request",
"path": "/todo/invalid-uuid"
}
Add a new PUT endpoint that uses a uuid as a path parameter to mark a specific task from the list of tasks as complete. To mark the task as complete the completed field should be set to the current time and the complete boolean should be set to true. The body of the response should indicate if the request was successful and should contain a boolean success flag and a string message.
{
"success": true,
"message": "This task has now been completed."
}
If the task is already marked as completed, or the task is not found, then no change is made and the problem should be indicated in the response. If an invalid uuid is supplied the endpoint will return a bad request error.
| URI | Required behaviour |
|---|---|
| http://localhost:8080/todo/completed/{uuid} | Mark the task with supplied uuid as complete and return a meaningful response with HTTP status 200. To mark a task as complete the completed fields should be set to the current time and the complete boolean value should be set to true. |
Given the static data above:
http://localhost:8080/todo/completed/5c3ec8bc-6099-4cd5-b6da-8e2956db3a34 returns
{
"success": true,
"message": "This task has now been completed."
}
with HTTP status 200.
A further call to http://localhost:8080/todo/completed/5c3ec8bc-6099-4cd5-b6da-8e2956db3a34 returns
{
"success": false,
"message": "Task already marked complete."
}
with HTTP status 200.
http://localhost:8080/todo/completed/5c3ec8bc-6099-1a2b-b6da-8e2956db3a34 returns
{
"success": false,
"message": "Task not found."
}
with HTTP status 200.
http://localhost:8080/todo/completed/invalid-uuid returns HTTP status 400 Bad Request. A meaningful response may also be returned, e.g.
{
"timestamp": "2023-06-27T12:32:05.590Z",
"status": 400,
"error": "Bad Request",
"path": "/todo/completed/invalid-uuid"
}
Add a new POST endpoint that takes two parameters, task name and task description, that creates a new task item with the given name and description. The uuid of the new task will be assigned by the server as a random uuid and the created timestamp should be set to the current time. The new item will have no value for the completed timestamp and a value of false for the complete flag. The body of the response should include the uuid of the new task and a string message.
{
"taskId": "13f8e57c-49dc-4301-afe9-0bcf2e840056",
"message": "Task {task name} added successfully."
}
The status of the response should be 201 (CREATED) for a successful operation. If both name and description parameters are not supplied the endpoint will return a bad request error.
| URI | Required behaviour |
|---|---|
| http://localhost:8080/todo/addTask{?name=TaskName&description=Description} | Create a new task with the given name and description, add it to the list of tasks and return HTTP status 201. |
http://localhost:8080/todo/addTask?name=TaskName&description=Description returns:
{
"taskId": "13f8e57c-49dc-4301-afe9-0bcf2e840056",
"message": "Task TaskName added successfully."
}
with HTTP status 201 (CREATED). Note that the uuid will be randomly generated.
http://localhost:8080/todo/addTask?name=Name returns HTTP status 400 Bad Request. A meaningful response may also be returned, e.g.
{
"timestamp": "2023-06-27T12:32:05.590Z",
"status": 400,
"error": "Bad Request",
"path": "/todo/addTask?name=Name"
}
You can test your web service implementation using Postman and Cypress test suites.
To test using the Postman collection, first download and install Postman. After opening Postman, sign-in or register an account. You can then import the Postman collection by clicking Import. The postmanCollection.json file can be found under the test\postman folder.
The Postman suite is intended as an aid for development and testing.
The Cypress test suite contains fifteen tests that should all pass when the web service implementation is complete. To use Cypress first download and install nodejs. Once installed, open a new terminal, change directory into the test directory and run:
npm install
npx cypress open
Once Cypress opens, click on E2E Testing and select a browser (eg. Chrome). On the next screen you can run the tests.
The tests can be also run from the command line:
npx cypress run
The testing task is to implement API tests using any generative AI model, such as ChatGPT, Google Bard, etc. A skeleton framework has been provided in Cypress on the start-here-test branch under the test/cypress/e2e/api-tests directory. It is also valid to start completely from scratch if you wish. See the above for details of the four web service API endpoints.
Start the web service in the language of your choice and make sure it is running on localhost:8080. Instructions about starting each of the web service implementations is in the README files in the respective source folders, e.g. quantifying-ai/src/javascript/README.md.
To use Cypress first download and install nodejs. To start Cypress, open a new terminal, change directory into the quantifying-ai/test directory, and type:
npm install
npx cypress open
This will open a Cypress browser. Select 'E2E testing', choose a browser (e.g. Chrome) and 'Start E2E Testing...'
The tests found in test/cypress/e2e will appear. To run, click on your test suite, e.g. todoAppAPI.cy.js
You can add tests to this file, or create a new test suite.
You can also run tests headlessly:
npx cypress run
For more detailed information check the Quantifying Generative AI pages on Confluence.