Skip to content

Commit

Permalink
[Security Solution] Fixes Related Integrations showing as not install…
Browse files Browse the repository at this point in the history 
…ed or enabled when they actually are (elastic#152055)

## Summary

Resolves: elastic#142081
elastic#149970
elastic#150968

By adding an initial query for installed integrations and augments the
existing `InstalledIntegrationArray` constructed using
`PackagePolicy`'s. Also removes `version` from the `packageKey` when
calculating installed integrations as there can be mis-matches between
different policy versions and the integration itself, and I believe the
intended behavior here is to not have multiple `relatedIntegrations`
returned for different versions. We may want to expand the response here
to include all the different policy versions that exist (and perhaps #
of agents assigned the policy).

Lastly, updates `getIntegrationsInfoFromPolicy()` to also pull the base
`package` details in addition to the policy_template details, as this is
what ensure base packages show as `Installed: enabled` if they have an
integration policy assigned (vs just showing as `Installed` like when
there isn't an integration policy).

Note: This PR also adds the `getPackages()` method to the
`PackageClient` as it didn't currently exist, and was only available via
the fleet API via the `/api/fleet/epm/packages` route.

### Before:
<p align="center">
<img width="500"
src="https://user-images.githubusercontent.com/2946766/221066781-be7aa1c6-1728-4200-98b2-d40946e48bbe.png"
/>
</p>

### After
<p align="center">
<img width="500"
src="https://user-images.githubusercontent.com/2946766/221323469-e24081f9-0741-41fd-8227-9e319c98b0d3.png"
/>
</p>

---------

Co-authored-by: Georgii Gorbachev <[email protected]>
(cherry picked from commit b833b10)
  • Loading branch information
@spong
spong committed Mar 1, 2023
1 parent 03aea9d commit f4f8418
Show file tree
Hide file tree
Showing 5 changed files with 102 additions and 63 deletions.
1 change: 1 addition & 0 deletions x-pack/plugins/fleet/server/services/epm/package_service.mock.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ const createClientMock = (): jest.Mocked<PackageClient> => ({
ensureInstalledPackage: jest.fn(),
fetchFindLatestPackage: jest.fn(),
getPackage: jest.fn(),
getPackages: jest.fn(),
reinstallEsAssets: jest.fn(),
});

Expand Down
26 changes: 25 additions & 1 deletion x-pack/plugins/fleet/server/services/epm/package_service.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,10 @@ import type {
Logger,
} from '@kbn/core/server';

import type { PackageList } from '../../../common';

import type {
CategoryId,
EsAssetReference,
InstallablePackage,
Installation,
Expand All @@ -28,7 +31,7 @@ import { FleetUnauthorizedError } from '../../errors';
import { installTransforms, isTransform } from './elasticsearch/transform/install';
import type { FetchFindLatestPackageOptions } from './registry';
import { fetchFindLatestPackageOrThrow, getPackage } from './registry';
import { ensureInstalledPackage, getInstallation } from './packages';
import { ensureInstalledPackage, getInstallation, getPackages } from './packages';

export type InstalledAssetType = EsAssetReference;

Expand Down Expand Up @@ -56,6 +59,12 @@ export interface PackageClient {
packageVersion: string
): Promise<{ packageInfo: ArchivePackage; paths: string[] }>;

getPackages(params?: {
excludeInstallStatus?: false;
category?: CategoryId;
prerelease?: false;
}): Promise<PackageList>;

reinstallEsAssets(
packageInfo: InstallablePackage,
assetPaths: string[]
Expand Down Expand Up @@ -137,6 +146,21 @@ class PackageClientImpl implements PackageClient {
return getPackage(packageName, packageVersion, options);
}

public async getPackages(params?: {
excludeInstallStatus?: false;
category?: CategoryId;
prerelease?: false;
}) {
const { excludeInstallStatus, category, prerelease } = params || {};
await this.#runPreflight();
return getPackages({
savedObjectsClient: this.internalSoClient,
excludeInstallStatus,
category,
prerelease,
});
}

public async reinstallEsAssets(
packageInfo: InstallablePackage,
assetPaths: string[]
Expand Down
2 changes: 1 addition & 1 deletion x-pack/plugins/security_solution/cypress/e2e/detection_rules/related_integrations.cy.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -117,7 +117,7 @@ describe('Related integrations', () => {
const rule = {
name: 'Related integrations rule',
integrations: [
{ name: 'Amazon CloudFront', installed: true, enabled: true },
{ name: 'AWS Cloudfront', installed: true, enabled: true },
{ name: 'AWS CloudTrail', installed: true, enabled: false },
{ name: 'Aws Unknown', installed: false, enabled: false },
{ name: 'System', installed: true, enabled: true },
Expand Down
89 changes: 68 additions & 21 deletions ...ion_engine/fleet_integrations/api/get_installed_integrations/installed_integration_set.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,8 @@
* 2.0.
*/

import type { PackageListItem, PackagePolicy } from '@kbn/fleet-plugin/common';
import { capitalize, flatten } from 'lodash';
import type { PackagePolicy, ArchivePackage } from '@kbn/fleet-plugin/common';
import type {
InstalledIntegration,
InstalledIntegrationArray,
Expand All @@ -17,8 +17,8 @@ import type {
} from '../../../../../../common/detection_engine/fleet_integrations';

export interface IInstalledIntegrationSet {
addPackage(fleetPackage: PackageListItem): void;
addPackagePolicy(policy: PackagePolicy): void;
addRegistryPackage(registryPackage: ArchivePackage): void;

getPackages(): InstalledPackageArray;
getIntegrations(): InstalledIntegrationArray;
Expand All @@ -33,10 +33,57 @@ interface PackageInfo extends InstalledPackageBasicInfo {
export const createInstalledIntegrationSet = (): IInstalledIntegrationSet => {
const packageMap: PackageMap = new Map<string, PackageInfo>([]);

const addPackage = (fleetPackage: PackageListItem): void => {
if (fleetPackage.type !== 'integration') {
return;
}
if (fleetPackage.status !== 'installed') {
return;
}

const packageKey = `${fleetPackage.name}`;
const existingPackageInfo = packageMap.get(packageKey);

if (existingPackageInfo != null) {
return;
}

// Actual `installed_version` is buried in SO, root `version` is latest package version available
const installedPackageVersion = fleetPackage.savedObject.attributes.install_version;

// Policy templates correspond to package's integrations.
const packagePolicyTemplates = fleetPackage.policy_templates ?? [];

const packageInfo: PackageInfo = {
package_name: fleetPackage.name,
package_title: fleetPackage.title,
package_version: installedPackageVersion,

integrations: new Map<string, InstalledIntegrationBasicInfo>(
packagePolicyTemplates.map((pt) => {
const integrationTitle: string =
packagePolicyTemplates.length === 1 && pt.name === fleetPackage.name
? fleetPackage.title
: pt.title;

const integrationInfo: InstalledIntegrationBasicInfo = {
integration_name: pt.name,
integration_title: integrationTitle,
is_enabled: false, // There might not be an integration policy, so default false and later update in addPackagePolicy()
};

return [integrationInfo.integration_name, integrationInfo];
})
),
};

packageMap.set(packageKey, packageInfo);
};

const addPackagePolicy = (policy: PackagePolicy): void => {
const packageInfo = getPackageInfoFromPolicy(policy);
const integrationsInfo = getIntegrationsInfoFromPolicy(policy, packageInfo);
const packageKey = `${packageInfo.package_name}:${packageInfo.package_version}`;
const packageKey = `${packageInfo.package_name}`;
const existingPackageInfo = packageMap.get(packageKey);

if (existingPackageInfo == null) {
Expand All @@ -56,21 +103,6 @@ export const createInstalledIntegrationSet = (): IInstalledIntegrationSet => {
}
};

const addRegistryPackage = (registryPackage: ArchivePackage): void => {
const policyTemplates = registryPackage.policy_templates ?? [];
const packageKey = `${registryPackage.name}:${registryPackage.version}`;
const existingPackageInfo = packageMap.get(packageKey);

if (existingPackageInfo != null) {
for (const integration of existingPackageInfo.integrations.values()) {
const policyTemplate = policyTemplates.find((t) => t.name === integration.integration_name);
if (policyTemplate != null) {
integration.integration_title = policyTemplate.title;
}
}
}
};

const getPackages = (): InstalledPackageArray => {
const packages = Array.from(packageMap.values());
return packages.map((packageInfo): InstalledPackage => {
Expand Down Expand Up @@ -106,8 +138,8 @@ export const createInstalledIntegrationSet = (): IInstalledIntegrationSet => {
};

return {
addPackage,
addPackagePolicy,
addRegistryPackage,
getPackages,
getIntegrations,
};
Expand All @@ -125,15 +157,30 @@ const getIntegrationsInfoFromPolicy = (
policy: PackagePolicy,
packageInfo: InstalledPackageBasicInfo
): InstalledIntegrationBasicInfo[] => {
return policy.inputs.map((input) => {
// Construct integration info from the available policy_templates
const integrationInfos = policy.inputs.map((input) => {
const integrationName = normalizeString(input.policy_template ?? input.type); // e.g. 'cloudtrail'
const integrationTitle = `${packageInfo.package_title} ${capitalize(integrationName)}`; // e.g. 'AWS Cloudtrail'
return {
integration_name: integrationName,
integration_title: integrationTitle, // title gets re-initialized later in addRegistryPackage()
integration_title: integrationTitle,
is_enabled: input.enabled,
};
});

// Base package may not have policy template, so pull directly from `policy.package` if so
return [
...integrationInfos,
...(policy.package
? [
{
integration_name: policy.package.name,
integration_title: policy.package.title,
is_enabled: true, // Always true if `policy.package` exists since this corresponds to the base package
},
]
: []),
];
};

const normalizeString = (raw: string | null | undefined): string => {
Expand Down
47 changes: 7 additions & 40 deletions ...on/server/lib/detection_engine/fleet_integrations/api/get_installed_integrations/route.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -7,16 +7,13 @@

import type { Logger } from '@kbn/core/server';
import { transformError } from '@kbn/securitysolution-es-utils';
import { initPromisePool } from '../../../../../utils/promise_pool';
import { buildSiemResponse } from '../../../routes/utils';
import type { SecuritySolutionPluginRouter } from '../../../../../types';

import type { GetInstalledIntegrationsResponse } from '../../../../../../common/detection_engine/fleet_integrations';
import { GET_INSTALLED_INTEGRATIONS_URL } from '../../../../../../common/detection_engine/fleet_integrations';
import { createInstalledIntegrationSet } from './installed_integration_set';

const MAX_CONCURRENT_REQUESTS_TO_PACKAGE_REGISTRY = 5;

/**
* Returns an array of installed Fleet integrations and their packages.
*/
Expand All @@ -40,48 +37,18 @@ export const getInstalledIntegrationsRoute = (
const fleet = ctx.securitySolution.getInternalFleetServices();
const set = createInstalledIntegrationSet();

const packagePolicies = await fleet.packagePolicy.list(fleet.internalReadonlySoClient, {});
// Pulls all packages into memory just like the main fleet landing page
// No pagination support currently, so cannot batch this call
const allThePackages = await fleet.packages.getPackages();
allThePackages.forEach((fleetPackage) => {
set.addPackage(fleetPackage);
});

const packagePolicies = await fleet.packagePolicy.list(fleet.internalReadonlySoClient, {});
packagePolicies.items.forEach((policy) => {
set.addPackagePolicy(policy);
});

const registryPackages = await initPromisePool({
concurrency: MAX_CONCURRENT_REQUESTS_TO_PACKAGE_REGISTRY,
items: set.getPackages(),
executor: async (packageInfo) => {
const registryPackage = await fleet.packages.getPackage(
packageInfo.package_name,
packageInfo.package_version
);
return registryPackage;
},
});

if (registryPackages.errors.length > 0) {
const errors = registryPackages.errors.map(({ error, item }) => {
return {
error,
packageId: `${item.package_name}@${item.package_version}`,
};
});

const packages = errors.map((e) => e.packageId).join(', ');
logger.error(
`Unable to retrieve installed integrations. Error fetching packages from registry: ${packages}.`
);

errors.forEach(({ error, packageId }) => {
const logMessage = `Error fetching package info from registry for ${packageId}`;
const logReason = error instanceof Error ? error.message : String(error);
logger.debug(`${logMessage}. ${logReason}`);
});
}

registryPackages.results.forEach(({ result }) => {
set.addRegistryPackage(result.packageInfo);
});

const installedIntegrations = set.getIntegrations();

const body: GetInstalledIntegrationsResponse = {
Expand Down

0 comments on commit f4f8418

Please sign in to comment.