Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(rest) : Modifications to improve API GET result speed for name search #2096

Merged
merged 1 commit into from
Oct 25, 2023

Conversation

KoukiHama
Copy link
Member

@KoukiHama KoukiHama commented Aug 22, 2023

Please provide a summary of your changes here.

  • Which issue is this pull request belonging to and how is it solving it? (Refer to issue here)
  • Did you add or update any new dependencies that are required for your change?

Issue:  #2078

Performance Issues with Name Search via API

Suggest Reviewer

You can suggest reviewers here with an @ag4ums

How to test

Using the Rest API of Release, "Acquisition of all records" and "Filtering by name" are executed, and confirm that there is no difference in the response JSON before and after the modification.

Preparation

Use the script below to register 10,000 Release and Component records for testing.
make_dummy_data.zip

1. Retrieving the response before modification

Start the system before merging the changes in this pull request and execute the following commands.

  • Acquisition of all records
    Command line
    curl --location 'https://sw360.org/resource/api/releases' \
    --header 'Authorization: Bearer *****' \
    Expected Output
    {
        "_embedded": {
            "sw360:releases": [
                {
                    "name": "APIPerformanceCheck_A_2758",
                    "version": "1.00",
                    "_links": {
                        "self": {
                            "href": "https://sw360.org/resource/api/releases/0004585d3d634a5589b196221f169f66"
                        }
                    }
                },
                {
                    "name": "APIPerformanceCheck_A_8970",
                    "version": "1.00",
                    "_links": {
                        "self": {
                            "href": "https://sw360.org/resource/api/releases/00045947d9ec4f9aa58d02c3a4e8d2e1"
                        }
                    }
                },
                .
                .
                .
  • Filtering by name
    Command line
    curl --location 'https://sw360.org/resource/api/releases?name=APIPerformanceCheck_A_9999' \
    --header 'Authorization: Bearer *****' \
    Expected Output
    {
        "_embedded": {
            "sw360:releases": [
                {
                    "name": "APIPerformanceCheck_A_9999",
                    "version": "1.00",
                    "_links": {
                        "self": {
                            "href": "https://sw360.org/resource/api/releases/0004585d3d634a5589b196221f169f66"
                        }
                    }
                }
            ]
        },
        "_links": {
            "curies": [
                {
                    "href": "https://sw360.org/resource/docs/{rel}.html",
                    "name": "sw360",
                    "templated": true
                }
            ]
        }
    }

2. Retrieving the response after modification

Start the system after merging the changes in this pull request and execute the following commands.

  • Acquisition of all records
    Command line

    curl --location 'https://sw360.org/resource/api/releases' \
    --header 'Authorization: Bearer *****' \

    Expected Output

    {
        "_embedded": {
            "sw360:releases": [
                {
                    "name": "APIPerformanceCheck_A_2758",
                    "version": "1.00",
                    "_links": {
                        "self": {
                            "href": "https://sw360.org/resource/api/releases/0004585d3d634a5589b196221f169f66"
                        }
                    }
                },
                {
                    "name": "APIPerformanceCheck_A_8970",
                    "version": "1.00",
                    "_links": {
                        "self": {
                            "href": "https://sw360.org/resource/api/releases/00045947d9ec4f9aa58d02c3a4e8d2e1"
                        }
                    }
                },
                .
                .
                .
  • Filtering by name
    Command line

    curl --location 'https://sw360.org/resource/api/releases?name=APIPerformanceCheck_A_9999' \
    --header 'Authorization: Bearer *****' \

    Expected Output

    {
        "_embedded": {
            "sw360:releases": [
                {
                    "name": "APIPerformanceCheck_A_9999",
                    "version": "1.00",
                    "_links": {
                        "self": {
                            "href": "https://sw360.org/resource/api/releases/0004585d3d634a5589b196221f169f66"
                        }
                    }
                }
            ]
        },
        "_links": {
            "curies": [
                {
                    "href": "https://sw360.org/resource/docs/{rel}.html",
                    "name": "sw360",
                    "templated": true
                }
            ]
        }
    }

3. Comparison of response JSON before and after changes

Confirm that there is no difference in the response JSON before and after applying the changes in this pull request.

fix(rest) : Modifications to improve GET result speed

Purpose

Fix bottlenecks in Release API performance issues.

  • The related issue
    Performance Issues with Name Search via API Performance Issues with Name Search via API #2078
    Performance Issues with Name Search via API #2078

  • Bottlenecks
    In the Rest API for Relaese, when "Acquisition of all records" or "Filtering by name" is executed, the process of retrieving the ComponentType accounts for about 92% of the processing time.

    ReleaseController.getReleasesForUser method:

    for(Release release: sw360Releases) {
        releaseService.setComponentDependentFieldsInRelease(release, sw360User);
    }
  • Improvement effect of this modification
    Comparison of processing times (unit: second)
    DB has 10000 Release and Component records

    Endpoint Method Before modification (average) After modification (average) Remarks
    /releases GET 70.900 13.61 Acquisition of all records
    /releases?name=*** GET 69.343 5.55 Filtering by name (1 record is retrieved)

The outline of changes

  • Replacing processing order
    The order of execution of the filtering process by name of Release and the acquisition process of ComponentType is switched.
    This reduces unnecessary Component acquisition processing, which is performed for Release records excluded by filter processing.
  • Add batch acquisition of Component
    If the "sha1" or "name" filter is not specified in the request, it is treated as a request to acquire all Release records.
    In this case, the process is changed so that all Components records are retrieved in a batch. and the ComponentType is acquired.
    If the number of Release records is sufficiently narrowed down by filter conditions, it is more efficient to acquire Component individually, so the conventional processing is performed.

The detail of changes

ReleaseController.java

rest/resource-server/src/main/java/org/eclipse/sw360/rest/resourceserver/release/ReleaseController.java

  • Replacing the processing order
  • Added batch acquisition of Component

Sw360ReleaseService.java

rest/resource-server/src/main/java/org/eclipse/sw360/rest/resourceserver/release/Sw360ReleaseService.java

  • Added setComponentDependentFieldsInRelease method for batch acquisition of Component

Limitations

The following Rest API changes may cause performance issues similar to the ComponentType acquisition process.

feat(rest): Add Information Vendor to response Get all Releases detail #2089
#2089

However, since this change is limited to cases where "allDetails" is enabled and also skips Release that do not have a vendor id, we believe the impact is limited.
This pull request does not consider the impact of #2089.

@KoukiHama KoukiHama self-assigned this Aug 22, 2023
@KoukiHama KoukiHama added WIP work in progress enhancement New feature or request REST do not merge - нет! needs code review needs general test This is general testing, meaning that there is no org specific issue to check for and removed WIP work in progress do not merge - нет! labels Aug 22, 2023
@KoukiHama KoukiHama requested a review from ag4ums August 23, 2023 04:45
@KoukiHama KoukiHama changed the title fix(rest) : Modifications to improve GET result speed fix(rest) : Modifications to improve API GET result speed Aug 23, 2023
@KoukiHama KoukiHama changed the title fix(rest) : Modifications to improve API GET result speed fix(rest) : Modifications to improve API GET result speed for name search Aug 23, 2023
@ag4ums ag4ums self-assigned this Aug 23, 2023
Copy link
Contributor

@smrutis1 smrutis1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

code looks good to me.

@ag4ums ag4ums added ready ready to merge and removed needs general test This is general testing, meaning that there is no org specific issue to check for labels Oct 25, 2023
@ag4ums ag4ums merged commit e95dc4f into eclipse-sw360:main Oct 25, 2023
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request ready ready to merge REST
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants