Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add datasource field in accelerations cache #1525

Merged
merged 2 commits into from
Mar 13, 2024
Merged

Add datasource field in accelerations cache #1525

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
43d85ed
add datasource in accelerations cache
ps48 Mar 12, 2024
47f0768
fixed nits
ps48 Mar 12, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 11 additions & 6 deletions common/types/data_connections.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@
interface AsyncApiDataResponse {
status: string;
schema?: Array<{ name: string; type: string }>;
datarows?: any;

Check warning on line 54 in common/types/data_connections.ts

View workflow job for this annotation

GitHub Actions / Lint 

Unexpected any. Specify a different type
total?: number;
size?: number;
error?: string;
Expand Down Expand Up @@ -83,19 +83,19 @@

export interface CachedTable {
name: string;
columns: CachedColumn[];
columns?: CachedColumn[];
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why optional here?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cause columns are lazy loaded for tables. These will be loaded in cache when users open table flyout or use a table in create acceleration flyout.

}

export interface CachedDatabase {
name: string;
tables: CachedTable[];
lastUpdated: string; // Assuming date string in UTC format
lastUpdated: string; // date string in UTC format
status: CachedDataSourceStatus;
}

export interface CachedDataSource {
name: string;
lastUpdated: string; // Assuming date string in UTC format
lastUpdated: string; // date string in UTC format
status: CachedDataSourceStatus;
databases: CachedDatabase[];
}
Expand All @@ -115,13 +115,18 @@
status: string;
}

export interface AccelerationsCacheData {
version: string;
export interface CachedAcclerationByDataSource {
name: string;
accelerations: CachedAccelerations[];
lastUpdated: string; // Assuming date string in UTC format
lastUpdated: string; // date string in UTC format
status: CachedDataSourceStatus;
}

export interface AccelerationsCacheData {
version: string;
dataSources: CachedAcclerationByDataSource[];
}

export interface PollingSuccessResult {
schema: Array<{ name: string; type: string }>;
datarows: Array<Array<string | number | boolean>>;
Expand Down
181 changes: 147 additions & 34 deletions public/framework/catalog_cache/cache_loader.test.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,14 +11,16 @@
mockShowTablesPollingResult,
} from '../../../test/datasources';
import {
createLoadQuery,
updateAccelerationsToCache,
updateDatabasesToCache,
updateTablesToCache,
updateToCache,
} from './cache_loader';
import { CatalogCacheManager } from './cache_manager';

interface LooseObject {
[key: string]: any;

Check warning on line 23 in public/framework/catalog_cache/cache_loader.test.tsx

View workflow job for this annotation

GitHub Actions / Lint 

Unexpected any. Specify a different type
}

// Mock localStorage
Expand Down Expand Up @@ -146,10 +148,7 @@
dataSourceName,
expect.objectContaining({
name: databaseName,
tables: [
{ name: 'Table1', columns: [] },
{ name: 'Table2', columns: [] },
],
tables: [{ name: 'Table1' }, { name: 'Table2' }],
lastUpdated: expect.any(String),
status: CachedDataSourceStatus.Updated,
})
Expand All @@ -166,55 +165,169 @@
it('should save empty accelerations cache and status failed when polling result is null', () => {
const pollingResult = null;

updateAccelerationsToCache(pollingResult);
updateAccelerationsToCache('sampleDS', pollingResult);

// Verify that saveAccelerationsCache is called with the correct parameters
expect(CatalogCacheManager.saveAccelerationsCache).toHaveBeenCalledWith({
version: CATALOG_CACHE_VERSION,
accelerations: [],
lastUpdated: expect.any(String),
status: CachedDataSourceStatus.Failed,
dataSources: [
{
name: 'sampleDS',
accelerations: [],
lastUpdated: expect.any(String),
status: CachedDataSourceStatus.Failed,
},
],
});
});

it('should save new accelerations cache when polling result is not null', () => {
updateAccelerationsToCache(mockShowIndexesPollingResult);
updateAccelerationsToCache('sampleDS', mockShowIndexesPollingResult);

// Verify that saveAccelerationsCache is called with the correct parameters
expect(CatalogCacheManager.saveAccelerationsCache).toHaveBeenCalledWith({
version: CATALOG_CACHE_VERSION,
accelerations: [
{
flintIndexName: 'flint_mys3_default_http_logs_skipping_index',
type: 'skipping',
database: 'default',
table: 'http_logs',
indexName: 'skipping_index',
autoRefresh: false,
status: 'Active',
},
dataSources: [
{
flintIndexName: 'flint_mys3_default_http_logs_status_clientip_and_day_index',
type: 'covering',
database: 'default',
table: 'http_logs',
indexName: 'status_clientip_and_day',
autoRefresh: true,
status: 'Active',
name: 'sampleDS',
accelerations: [
{
flintIndexName: 'flint_mys3_default_http_logs_skipping_index',
type: 'skipping',
database: 'default',
table: 'http_logs',
indexName: 'skipping_index',
autoRefresh: false,
status: 'Active',
},
{
flintIndexName: 'flint_mys3_default_http_logs_status_clientip_and_day_index',
type: 'covering',
database: 'default',
table: 'http_logs',
indexName: 'status_clientip_and_day',
autoRefresh: true,
status: 'Active',
},
{
flintIndexName: 'flint_mys3_default_http_count_view',
type: 'materialized',
database: 'default',
table: '',
indexName: 'http_count_view',
autoRefresh: true,
status: 'Active',
},
],
lastUpdated: expect.any(String),
status: CachedDataSourceStatus.Updated,
},
],
});
});
});

describe('updateToCache', () => {
it('should call updateDatabasesToCache when loadCacheType is "databases"', () => {
const loadCacheType = 'databases';
const dataSourceName = 'TestDataSource';

updateToCache(mockShowDatabasesPollingResult, loadCacheType, dataSourceName);

// Verify that addOrUpdateDataSource is called
expect(CatalogCacheManager.addOrUpdateDataSource).toHaveBeenCalled();
expect(CatalogCacheManager.updateDatabase).not.toHaveBeenCalled();
expect(CatalogCacheManager.saveAccelerationsCache).not.toHaveBeenCalled();
});

it('should call updateTablesToCache when loadCacheType is "tables"', () => {
const loadCacheType = 'tables';
const dataSourceName = 'TestDataSource';
const databaseName = 'TestDatabase';

CatalogCacheManager.addOrUpdateDataSource({
databases: [
{
flintIndexName: 'flint_mys3_default_http_count_view',
type: 'materialized',
database: 'default',
table: '',
indexName: 'http_count_view',
autoRefresh: true,
status: 'Active',
name: databaseName,
lastUpdated: '',
status: CachedDataSourceStatus.Empty,
tables: [],
},
],
lastUpdated: expect.any(String),
name: dataSourceName,
lastUpdated: new Date().toUTCString(),
status: CachedDataSourceStatus.Updated,
});

updateToCache(mockShowTablesPollingResult, loadCacheType, dataSourceName, databaseName);

// Verify that updateDatabase is called
expect(CatalogCacheManager.addOrUpdateDataSource).toHaveBeenCalled();
expect(CatalogCacheManager.updateDatabase).toHaveBeenCalled();
expect(CatalogCacheManager.saveAccelerationsCache).not.toHaveBeenCalled();
});

it('should call updateAccelerationsToCache when loadCacheType is "accelerations"', () => {
const loadCacheType = 'accelerations';
const dataSourceName = 'TestDataSource';

updateToCache(mockShowIndexesPollingResult, loadCacheType, dataSourceName);

// Verify that saveAccelerationsCache is called
expect(CatalogCacheManager.addOrUpdateDataSource).not.toHaveBeenCalled();
expect(CatalogCacheManager.updateDatabase).not.toHaveBeenCalled();
expect(CatalogCacheManager.saveAccelerationsCache).toHaveBeenCalled();
});

it('should not call any update function when loadCacheType is not recognized', () => {
const pollResults = {};
const loadCacheType = '';
const dataSourceName = 'TestDataSource';

updateToCache(pollResults, loadCacheType, dataSourceName);

// Verify that no update function is called
expect(CatalogCacheManager.addOrUpdateDataSource).not.toHaveBeenCalled();
expect(CatalogCacheManager.updateDatabase).not.toHaveBeenCalled();
expect(CatalogCacheManager.saveAccelerationsCache).not.toHaveBeenCalled();
});
});

describe('createLoadQuery', () => {
it('should create a query for loading databases', () => {
const loadCacheType = 'databases';
const dataSourceName = 'example';
const expectedQuery = 'SHOW SCHEMAS IN `example`';
expect(createLoadQuery(loadCacheType, dataSourceName)).toEqual(expectedQuery);
});

it('should create a query for loading tables', () => {
const loadCacheType = 'tables';
const dataSourceName = 'example';
const databaseName = 'test';
const expectedQuery = 'SHOW TABLES IN `example`.`test`';
expect(createLoadQuery(loadCacheType, dataSourceName, databaseName)).toEqual(expectedQuery);
});

it('should create a query for loading accelerations', () => {
const loadCacheType = 'accelerations';
const dataSourceName = 'example';
const expectedQuery = 'SHOW FLINT INDEX in `example`';
expect(createLoadQuery(loadCacheType, dataSourceName)).toEqual(expectedQuery);
});

it('should return an empty string for unknown loadCacheType', () => {
const loadCacheType = 'unknownType';
const dataSourceName = 'example';
expect(createLoadQuery(loadCacheType, dataSourceName)).toEqual('');
});

it('should properly handle backticks in database name', () => {
const loadCacheType = 'tables';
const dataSourceName = 'example';
const databaseName = '`sample`';
const expectedQuery = 'SHOW TABLES IN `example`.`sample`';
expect(createLoadQuery(loadCacheType, dataSourceName, databaseName)).toEqual(expectedQuery);
});
});
});
24 changes: 12 additions & 12 deletions public/framework/catalog_cache/cache_loader.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,7 @@
*/

import { useEffect, useState } from 'react';
import {
ASYNC_POLLING_INTERVAL,
CATALOG_CACHE_VERSION,
} from '../../../common/constants/data_sources';
import { ASYNC_POLLING_INTERVAL } from '../../../common/constants/data_sources';
import {
AsyncPollingResult,
CachedDataSourceStatus,
Expand Down Expand Up @@ -44,7 +41,7 @@
}

const combinedData = combineSchemaAndDatarows(pollingResult.schema, pollingResult.datarows);
const newDatabases = combinedData.map((row: any) => ({

Check warning on line 44 in public/framework/catalog_cache/cache_loader.tsx

View workflow job for this annotation

GitHub Actions / Lint 

Unexpected any. Specify a different type
name: row.namespace,
tables: [],
lastUpdated: '',
Expand Down Expand Up @@ -78,9 +75,8 @@
}

const combinedData = combineSchemaAndDatarows(pollingResult.schema, pollingResult.datarows);
const newTables = combinedData.map((row: any) => ({

Check warning on line 78 in public/framework/catalog_cache/cache_loader.tsx

View workflow job for this annotation

GitHub Actions / Lint 

Unexpected any. Specify a different type
name: row.tableName,
columns: [],
}));

CatalogCacheManager.updateDatabase(dataSourceName, {
Expand All @@ -91,12 +87,15 @@
});
};

export const updateAccelerationsToCache = (pollingResult: AsyncPollingResult) => {
export const updateAccelerationsToCache = (
dataSourceName: string,
pollingResult: AsyncPollingResult
) => {
const currentTime = new Date().toUTCString();

if (!pollingResult) {
CatalogCacheManager.saveAccelerationsCache({
version: CATALOG_CACHE_VERSION,
CatalogCacheManager.addOrUpdateAccelerationsByDataSource({
name: dataSourceName,
accelerations: [],
lastUpdated: currentTime,
status: CachedDataSourceStatus.Failed,
Expand All @@ -106,7 +105,7 @@

const combinedData = combineSchemaAndDatarows(pollingResult.schema, pollingResult.datarows);

const newAccelerations = combinedData.map((row: any) => ({

Check warning on line 108 in public/framework/catalog_cache/cache_loader.tsx

View workflow job for this annotation

GitHub Actions / Lint 

Unexpected any. Specify a different type
flintIndexName: row.flint_index_name,
type: row.kind === 'mv' ? 'materialized' : row.kind,
database: row.database,
Expand All @@ -116,8 +115,8 @@
status: row.status,
}));

CatalogCacheManager.saveAccelerationsCache({
version: CATALOG_CACHE_VERSION,
CatalogCacheManager.addOrUpdateAccelerationsByDataSource({
name: dataSourceName,
accelerations: newAccelerations,
lastUpdated: currentTime,
status: CachedDataSourceStatus.Updated,
Expand All @@ -125,7 +124,7 @@
};

export const updateToCache = (
pollResults: any,

Check warning on line 127 in public/framework/catalog_cache/cache_loader.tsx

View workflow job for this annotation

GitHub Actions / Lint 

Unexpected any. Specify a different type
loadCacheType: LoadCacheType,
dataSourceName: string,
databaseName?: string
Expand All @@ -138,7 +137,7 @@
updateTablesToCache(dataSourceName, databaseName!, pollResults);
break;
case 'accelerations':
updateAccelerationsToCache(pollResults);
updateAccelerationsToCache(dataSourceName, pollResults);
break;
default:
break;
Expand Down Expand Up @@ -184,11 +183,12 @@
error: pollingError,
startPolling,
stopPolling: stopLoading,
} = usePolling<any, any>((params) => {

Check warning on line 186 in public/framework/catalog_cache/cache_loader.tsx

View workflow job for this annotation

GitHub Actions / Lint 

Unexpected any. Specify a different type

Check warning on line 186 in public/framework/catalog_cache/cache_loader.tsx

View workflow job for this annotation

GitHub Actions / Lint 

Unexpected any. Specify a different type
return sqlService.fetchWithJobId(params);
}, ASYNC_POLLING_INTERVAL);

const startLoading = (dataSourceName: string, databaseName?: string) => {
setLoadStatus(DirectQueryLoadingStatus.SCHEDULED);

Check warning on line 191 in public/framework/catalog_cache/cache_loader.tsx

View check run for this annotation

Codecov / codecov/patch

public/framework/catalog_cache/cache_loader.tsx#L191 

Added line #L191 was not covered by tests
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this meaning that, SCHEDULED means loading? For example, instead of checking if it is SUCCESS, FAILED, or CANCELLED, I can use this as an indicator to see if it is at a loading/refreshing stage?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is added as we want to start with status as scheduled whenever the startpolling is called. Otherwise, it keep the old calls status.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SCHEDULED is the initial state, it may not necessarily be loading.

const { loadStatus, startLoading, stopLoading } = useLoadAccelerationsToCache();

If startLoading isn't called, then loadStatus will remain as SCHEDULED.

setCurrentDataSourceName(dataSourceName);
setCurrentDatabaseName(databaseName);

Expand Down Expand Up @@ -257,7 +257,7 @@
} else {
setLoadStatus(status);
}
}, [pollingResult, pollingError]);

Check warning on line 260 in public/framework/catalog_cache/cache_loader.tsx

View workflow job for this annotation

GitHub Actions / Lint 

React Hook useEffect has missing dependencies: 'currentDataSourceName', 'currentDatabaseName', 'loadCacheType', and 'stopLoading'. Either include them or remove the dependency array

return { loadStatus, startLoading, stopLoading };
};
Expand All @@ -272,7 +272,7 @@
return { loadStatus, startLoading, stopLoading };
};

export const useAccelerationsToCache = () => {
export const useLoadAccelerationsToCache = () => {
const { loadStatus, startLoading, stopLoading } = useLoadToCache('accelerations');
return { loadStatus, startLoading, stopLoading };
};
Loading
Loading