Skip to content

Commit

Permalink
tests(api): dynamically set delay in eventually() assertion helper (#…
Browse files Browse the repository at this point in the history 
…10929)

This updates the logic of `eventually()` to start with a very small initial delay of 100ms. On each failure, the delay is increased by 25% (clamped to the remaining time) + 1-10ms of jitter.

### other changes

  1. Time spent within the assertion function passed to `eventually()` will now count towards the timeout. It's possible these behavioral changes could break due to assumptions of some callers, but this doesn't seem _too_ likely.
  2. There were several functions with their own implementation of this pattern. They've all been refactored to wrap `eventually()` and should reap the same benefits: `retryRequest()`, `waitForCacheInvalidation()`, and `waitForTargetStatus()`
 
### rationale

This _should_ translate to faster test runs. Here are some example timings from the `Execute Gateway API Tests` step in various jobs:

* `API Test Packages - ubuntu-24.04 / Package API Tests - using ubuntu-24.04 artifact, hybrid mode`
  * [this branch](https://github.com/Kong/kong-ee/actions/runs/12168109558/job/33938264421?pr=10929#logs) => 1m43s
  * [master](https://github.com/Kong/kong-ee/actions/runs/12130570459/job/33821902232#logs) => 5m8s
* `API Test Packages - ubuntu-24.04-arm64 / Package API Tests - using ubuntu-24.04-arm64 artifact, classic mode`
  * [this branch](https://github.com/Kong/kong-ee/actions/runs/12168109558/job/33938263450#logs) => 1m59s
  * [master](https://github.com/Kong/kong-ee/actions/runs/12130570459/job/33821899862#logs) => 3m58s
* `E2E Tests - ubuntu / non-smoke API test - classic mode on ubuntu-latest-kong for kong/kong-gateway-dev:<tag>`
  * [this branch](https://github.com/Kong/kong-ee/actions/runs/12168109558/job/33938557043?pr=10929#logs) => 15m30s
  * [master](https://github.com/Kong/kong-ee/actions/runs/12130570459/job/33822233896#logs) => 31m36s + 1m48s in `Retry Failed E2E API Tests`
 
### links

KAG-5953
  • Loading branch information
@flrgh
flrgh authored Dec 5, 2024
1 parent a8fc322 commit 872215f
Show file tree
Hide file tree
Showing 9 changed files with 148 additions and 214 deletions.
82 changes: 21 additions & 61 deletions spec-ee/kong-api-tests/support/utilities/entities-gateway.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,7 @@ import axios, { AxiosPromise, AxiosResponse } from 'axios';
import { expect } from '../assert/chai-expect';
import { Environment, getBasePath, isGateway, isKoko } from '../config/environment';
import { logResponse } from './logging';
import { randomString, wait } from './random';
import { retryRequest } from './retry-axios';
import { randomString } from './random';
import { getNegative } from './negative-axios';
import { isKongOSS, eventually } from '@support';

Expand Down Expand Up @@ -812,8 +811,8 @@ export const getRouterFlavor = async () => {
* after getting 200, delete the service/route, send request again to the route until it is 404
* This triggers router rebuild making sure all configuration updates have been propagated in kong
* @param {object} options
* @property {number} timeout - retryRequest timeout
* @property {number} interval - retryRequest interval
* @property {number} timeout - retry timeout
* @property {number} delay - retry delay
* @property {object} proxyReqHeader - custom proxy request header e.g. key-auth key
*/
export const waitForConfigRebuild = async (options: any = {}) => {
Expand Down Expand Up @@ -849,44 +848,21 @@ export const waitForConfigRebuild = async (options: any = {}) => {
const pluginId = plugin.id;

// send request to route until response is 401
const reqSuccess = () =>
getNegative(`${proxyUrl}${routePath}`, options?.proxyReqHeader);
const assertionsSuccess = (resp) => {
expect(
resp.status,
'waitForConfigRebuild - route should return 401'
).to.equal(401);
};

await retryRequest(
reqSuccess,
assertionsSuccess,
options?.timeout,
options?.interval,
options?.verbose,
);
await eventually(async () => {
const resp = await getNegative(`${proxyUrl}${routePath}`, options?.proxyReqHeader);
expect(resp.status, 'waitForConfigRebuild - expecting new entities to be active').to.equal(401);
}, options?.timeout, options?.delay, options?.verbose);

// removing the entities
await deletePlugin(pluginId);
await deleteGatewayRoute(routeId);
await deleteGatewayService(serviceId);

// send request to route until response is 404
const reqFail = () =>
getNegative(`${proxyUrl}${routePath}`, options?.proxyReqHeader);
const assertionsFail = (resp) => {
expect(
resp.status,
'waitForConfigRebuild - route should return 404'
).to.equal(404);
};

await retryRequest(
reqFail,
assertionsFail,
options?.timeout,
options?.interval
);
await eventually(async () => {
const resp = await getNegative(`${proxyUrl}${routePath}`, options?.proxyReqHeader);
expect(resp.status, 'waitForConfigRebuild - expecting 404 after deleting entities').to.equal(404);
}, options?.timeout, options?.delay, options?.verbose);

return true
};
Expand Down Expand Up @@ -977,32 +953,16 @@ export const clearKongResource = async (endpoint: string, workspaceNameorId?: st
}
};


/**
* Wait for /status/ready to return given status
* @param {string} cacheKey - cache key to wait for
* @param {number} timeout - timeout in ms
* Wait for /cache/${cacheKey} to return a 404
* @param cacheKey - cache key to wait for
* @param timeout - timeout in ms
*/
export const waitForCacheInvalidation = async (
cacheKey,
timeout,
) => {
let response;
let wantedTimeout
while (timeout > 0) {
const response = await getNegative(`${getUrl('cache')}/${cacheKey}`);
if (response.status === 404) {
// log final response
logResponse(response);
return true;
}
await wait(1000); // eslint-disable-line no-restricted-syntax
timeout -= 1000;
}
// log last response received
logResponse(response);

// throw
expect(false, `${wantedTimeout}ms exceeded waiting for "${cacheKey}" to invalidate from cache`).to.equal(true);
return false;
export const waitForCacheInvalidation = async (cacheKey: string, timeout: number) => {
await eventually(async () => {
const res = await getNegative(`${getUrl('cache')}/${cacheKey}`);
expect(res.status, `cache API endpoint for ${cacheKey} should return 404 when item is invalidated`).to.equal(404);
},
timeout
);
};
56 changes: 33 additions & 23 deletions spec-ee/kong-api-tests/support/utilities/eventually.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,42 +4,52 @@ import { isCI } from '@support';
* Wait for the `assertions` does not throw any exceptions.
* @param assertions - The assertions to be executed.
* @param timeout - The timeout in milliseconds.
* @param interval - The interval in milliseconds.
* @param delay - initial delay between retries
* @param verbose - Verbose logs in case of error
* @returns {Promise<void>} - Asnyc void promise.
*/
export const eventually = async (
assertions: () => Promise<void>,
export const eventually = async <T = void>(
assertions: () => Promise<T>,
timeout = 120000,
interval = 3000,
delay = 100,
verbose = false,
): Promise<void> => {
let errorMsg = '';
): Promise<T> => {
// enable verbose logs in GH Actions for debugability
verbose = isCI() ? true : verbose
verbose = isCI() ? true : verbose;

let errorMsg = '';
let elapsed = 0;
let remaining = timeout;

while (timeout >= 0) {
const start = Date.now();
const start = Date.now();

while (remaining >= 0) {
try {
await assertions();
return;
return await assertions();
} catch (error: any) {
errorMsg = error.message;
elapsed = Date.now() - start;
remaining = timeout - elapsed;

if (remaining > 0) {
const jitter = Math.random() * 10;
delay = Math.min(delay * 1.25 + jitter, remaining);

if (verbose) { // Inside CI environment, do exponential backoff
interval *= 2; // Double the delay for the next attempt
interval += Math.random() * 1000; // Add jitter
if (verbose) {
console.log(errorMsg);
console.log(`** Assertion(s) Failed -- Retrying in ${delay / 1000} seconds **`);
}

errorMsg = error.message;
console.log(errorMsg);
console.log(
`** Assertion(s) Failed -- Retrying in ${interval / 1000} seconds **`
);
await new Promise(resolve => setTimeout(resolve, delay));
}
await new Promise((resolve) => setTimeout(resolve, interval));
const end = Date.now();
timeout -= interval + (end - start);
console.log(`remaining timeout: ${timeout}`);
}
}

errorMsg = `** Timed Out (after ${elapsed / 1000} seconds) -- Last error: '${errorMsg}' **`;

if (verbose) {
console.log(errorMsg);
}

throw new Error(errorMsg);
};
60 changes: 10 additions & 50 deletions spec-ee/kong-api-tests/support/utilities/retry-axios.ts
View file
Original file line number Diff line number Diff line change
@@ -1,68 +1,28 @@
import { AxiosResponse } from 'axios';
import { logResponse } from './logging';
import { isCI } from '@support';
import { eventually } from '@support';

/**
* Retry the given Axios request to match the status code using a timeout and interval
* @param {Promise<AxiosResponse>} axiosRequest request to perform
* @param {(response: AxiosResponse) => void} assertions assertions to match
* @param {number} timeout timeout of retry loop
* @param {number} interval interval between tries
* @param axiosRequest - request to perform
* @param assertions - assertions to match
* @param timeout - max amount of time before raising an assertion error
* @param delay - initial delay between request retries
* @returns {Promise<AxiosResponse>} axios response object
*/
export const retryRequest = async (
axiosRequest: () => Promise<AxiosResponse>,
assertions: (response: AxiosResponse) => void,
timeout = 30000,
interval = 3000,
delay = 100,
verbose = false,
): Promise<AxiosResponse> => {
// enable verbose logs in GH Actions for debugability
verbose = isCI() ? true : verbose
let response: AxiosResponse = {} as any;
let errorMsg = '';
while (timeout >= 0) {
response = await axiosRequest();
const wrapper: () => Promise<AxiosResponse> = async () => {
const response = await axiosRequest();
logResponse(response);
try {
assertions(response);
return response;
} catch (error: any) {
if (verbose) {
errorMsg = error.message;
console.log(errorMsg);
console.log(
`** Assertion(s) Failed -- Retrying in ${interval / 1000} seconds **`
);
}
await new Promise((resolve) => setTimeout(resolve, interval));
timeout -= interval;
}
}

/*
* The last try.
*
* If we get here, we've timed out,
* but we might miss a try in some cases,
* For example, if the timeout is 10 seconds,
* and the interval is 3 seconds,
* we'll try 3 times, the last try will be at 9 seconds.
* But the condition might be true at 10 seconds,
* and the timeout not is less than 0,
* so we'll exit the above loop,
* and we'll miss the last try.
*/
try {
assertions(response);
return response;
} catch (error: any) {
errorMsg = error.message;
console.log(errorMsg);
console.log(
`** Assertion(s) Failed -- Retrying in ${interval / 1000} seconds **`
);
}
};

throw new Error(errorMsg);
return await eventually(wrapper, timeout, delay, verbose);
};
35 changes: 13 additions & 22 deletions spec-ee/kong-api-tests/support/utilities/status-endpoint.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,10 +3,10 @@ import https from 'https';
import {
logResponse,
getGatewayHost,
wait,
expect,
getBasePath,
Environment,
eventually,
} from '@support';

const defaultPort = 8100;
Expand Down Expand Up @@ -64,29 +64,20 @@ export const expectStatusReadyEndpoint503 = async (

/**
* Wait for /status/ready to return given status
* @param {number} returnStatus - status to wait for
* @param {number} timeout - timeout in ms
* @param {number} port - port to use
* @param returnStatus - status to wait for
* @param timeout - timeout in ms
* @param port - port to use
*/
export const waitForTargetStatus = async (
returnStatus,
timeout,
port = defaultPort
returnStatus: number,
timeout: number,
port: number = defaultPort
) => {
let response;
while (timeout > 0) {
response = await getStatusReadyEndpointResponse(port);
if (response.status === returnStatus) {
// log final response
logResponse(response);
return true;
}
await wait(1000); // eslint-disable-line no-restricted-syntax
timeout -= 1000;
}
// log last response received
logResponse(response);
return false;
await eventually(async () => {
const response = await getStatusReadyEndpointResponse(port);
logResponse(response);
expect(response.status).to.equal(returnStatus);
}, timeout);
};

export const getClusteringDataPlanes = async () => {
Expand All @@ -95,4 +86,4 @@ export const getClusteringDataPlanes = async () => {
})

return resp.data
}
}
6 changes: 3 additions & 3 deletions spec-ee/kong-api-tests/test/gateway/admin-api/websocket/websocket-size-limit-plugin.spec.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ describe('Websocket Size Limit Plugin Tests', function () {
},
};

await waitForConfigRebuild({ interval: 1000, timeout: 120000 });
await waitForConfigRebuild();
});

it('should be able to add websocket size limit plugin', async function () {
Expand All @@ -73,7 +73,7 @@ describe('Websocket Size Limit Plugin Tests', function () {

expect(resp.status, 'Status should be 201').to.equal(201);
pluginId = resp.data.id;
await waitForConfigRebuild({ interval: 1000 });
await waitForConfigRebuild();
});

it('should send message when the size is below the limit', async function () {
Expand Down Expand Up @@ -111,7 +111,7 @@ describe('Websocket Size Limit Plugin Tests', function () {
});
expect(checkResponse.data.config.client_max_payload).to.equal(client_max_payload);

await waitForConfigRebuild({ interval: 1000 });
await waitForConfigRebuild();
});

it('should not send data when size is limited', async function () {
Expand Down
6 changes: 3 additions & 3 deletions spec-ee/kong-api-tests/test/gateway/admin-api/websocket/websocket.spec.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -83,7 +83,7 @@ describe('Gateway Websocket Tests', function () {
url: 'https://websocket-echo-server:9443',
});

await waitForConfigRebuild({ interval: 1000 });
await waitForConfigRebuild();
});

it('should not create http route when service is websocket service', async function () {
Expand Down Expand Up @@ -193,7 +193,7 @@ describe('Gateway Websocket Tests', function () {
paths: ['/.httpswss'],
protocols: ['https'],
});
await waitForConfigRebuild({ interval: 1000 });
await waitForConfigRebuild();
});

it('should route ws traffic when service and route is http', async function () {
Expand All @@ -219,6 +219,6 @@ describe('Gateway Websocket Tests', function () {
after(async function () {
await clearAllKongResources()
});

});
});
Loading

0 comments on commit 872215f

Please sign in to comment.