Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce Kibana task to deploy agentless connectors for 9.0 #203973

Merged
merged 36 commits into from
Jan 10, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
4927e05
WIP
artem-shelkovnikov Dec 12, 2024
5716a4e
WIP 2
artem-shelkovnikov Dec 12, 2024
45278cb
More changes, fun fun
artem-shelkovnikov Dec 17, 2024
1aedf35
More WIP + tests
artem-shelkovnikov Dec 23, 2024
e791be2
Revert artifacts/task.ts
artem-shelkovnikov Dec 24, 2024
8c27d06
Minor tweaks
artem-shelkovnikov Dec 24, 2024
8f996be
More tests + refactoring
artem-shelkovnikov Dec 25, 2024
65a50eb
Make methods private again
artem-shelkovnikov Dec 25, 2024
3c294fe
Humanise the creation time
artem-shelkovnikov Dec 25, 2024
3ec5dce
Clean up a comment
artem-shelkovnikov Dec 25, 2024
6d3fea6
Fix error message for scheduling
artem-shelkovnikov Dec 25, 2024
8b19c58
[CI] Auto-commit changed files from 'node scripts/lint_ts_projects --…
kibanamachine Dec 25, 2024
ec75692
Remove a comment in task.ts
artem-shelkovnikov Dec 25, 2024
4c3eda4
Increase interval to 1m
artem-shelkovnikov Dec 25, 2024
9738e86
[CI] Auto-commit changed files from 'node scripts/eslint --no-cache -…
kibanamachine Dec 25, 2024
cd17d94
Fix issues with types/linters
artem-shelkovnikov Dec 25, 2024
294f411
Fix import
artem-shelkovnikov Dec 26, 2024
8351e8a
Remove async from the methods
artem-shelkovnikov Dec 27, 2024
6367d72
Fix tests?
artem-shelkovnikov Dec 27, 2024
bec29db
Fix test for task manager
artem-shelkovnikov Dec 27, 2024
6d02c2a
Rebase fix
artem-shelkovnikov Dec 30, 2024
39c1429
[CI] Auto-commit changed files from 'node scripts/eslint --no-cache -…
kibanamachine Dec 30, 2024
ee087ba
Merge branch 'main' into artem/add-agentless-connectors-task
artem-shelkovnikov Dec 30, 2024
c59f0cb
Merge branch 'main' into artem/add-agentless-connectors-task
artem-shelkovnikov Dec 30, 2024
e6e96b8
Rebase fix
artem-shelkovnikov Dec 30, 2024
f53d034
Apply suggestions from code review
artem-shelkovnikov Jan 2, 2025
898020b
[CI] Auto-commit changed files from 'node scripts/notice'
kibanamachine Jan 2, 2025
0d83497
Improvements here and there
artem-shelkovnikov Jan 3, 2025
b96ec2e
Merge branch 'main' into artem/add-agentless-connectors-task
artem-shelkovnikov Jan 3, 2025
0f4db80
Merge branch 'main' into artem/add-agentless-connectors-task
artem-shelkovnikov Jan 6, 2025
75b22f7
Merge branch 'main' into artem/add-agentless-connectors-task
artem-shelkovnikov Jan 6, 2025
89b18b8
Update interval to 1m
artem-shelkovnikov Jan 9, 2025
3ec5393
Merge branch 'main' into artem/add-agentless-connectors-task
artem-shelkovnikov Jan 9, 2025
189ee5b
Merge branch 'main' into artem/add-agentless-connectors-task
artem-shelkovnikov Jan 9, 2025
9717116
Merge branch 'main' into artem/add-agentless-connectors-task
artem-shelkovnikov Jan 9, 2025
543af73
Merge branch 'main' into artem/add-agentless-connectors-task
artem-shelkovnikov Jan 10, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions x-pack/platform/plugins/shared/fleet/server/mocks/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -228,8 +228,10 @@ export const createPackagePolicyServiceMock = (): jest.Mocked<PackagePolicyClien
*/
export const createMockAgentPolicyService = (): jest.Mocked<AgentPolicyServiceInterface> => {
return {
create: jest.fn().mockReturnValue(Promise.resolve()),
get: jest.fn().mockReturnValue(Promise.resolve()),
list: jest.fn().mockReturnValue(Promise.resolve()),
delete: jest.fn().mockReturnValue(Promise.resolve()),
getFullAgentPolicy: jest.fn().mockReturnValue(Promise.resolve()),
getByIds: jest.fn().mockReturnValue(Promise.resolve()),
turnOffAgentTamperProtections: jest.fn().mockReturnValue(Promise.resolve()),
Expand Down
11 changes: 1 addition & 10 deletions x-pack/platform/plugins/shared/fleet/server/plugin.ts
Original file line number Diff line number Diff line change
Expand Up @@ -816,16 +816,7 @@ export class FleetPlugin
core.elasticsearch.client.asInternalUser,
internalSoClient
),
agentPolicyService: {
get: agentPolicyService.get,
list: agentPolicyService.list,
getFullAgentPolicy: agentPolicyService.getFullAgentPolicy,
getByIds: agentPolicyService.getByIDs,
turnOffAgentTamperProtections:
agentPolicyService.turnOffAgentTamperProtections.bind(agentPolicyService),
fetchAllAgentPolicies: agentPolicyService.fetchAllAgentPolicies,
fetchAllAgentPolicyIds: agentPolicyService.fetchAllAgentPolicyIds,
},
agentPolicyService,
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was done due to the fact that I needed a create method that depends on a lot of other private/internal methods.

I had to either make the methods public + add here; or I could pass the service itself. Potentially there might be other way, but I'm not familiar enough with Kibana development yet to know, please tell me if there's a better way :)

packagePolicyService,
registerExternalCallback: (type: ExternalCallback[0], callback: ExternalCallback[1]) => {
return appContextService.addExternalCallback(type, callback);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -196,7 +196,7 @@ export const bulkGetAgentPoliciesHandler: FleetRequestHandler<
'full query parameter require agent policies read permissions'
);
}
let items = await agentPolicyService.getByIDs(soClient, ids, {
let items = await agentPolicyService.getByIds(soClient, ids, {
Copy link
Member Author

@artem-shelkovnikov artem-shelkovnikov Dec 25, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Side-effect of removing the usage of AgentPolicyServiceInterface: interface had getByIDs and implementation has getByIds. I chose the latter to stay, but it's easy to rename implementation to getByIDs. This was mostly done to avoid pinging other code owners that might have used the interface method name.

withPackagePolicies,
ignoreMissing,
});
Expand Down Expand Up @@ -687,7 +687,7 @@ export const GetListAgentPolicyOutputsHandler: FleetRequestHandler<
body: { items: [] },
});
}
const agentPolicies = await agentPolicyService.getByIDs(soClient, ids, {
const agentPolicies = await agentPolicyService.getByIds(soClient, ids, {
withPackagePolicies: true,
});

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ import {
jest.mock('../../services', () => ({
agentPolicyService: {
get: jest.fn(),
getByIDs: jest.fn(),
getByIds: jest.fn(),
},
appContextService: {
getInternalUserSOClientWithoutSpaceExtension: jest.fn(),
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ jest.mock('../../services', () => ({
},
agentPolicyService: {
get: jest.fn(),
getByIDs: jest.fn(),
getByIds: jest.fn(),
},
}));

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -504,7 +504,7 @@ describe('Agent policy', () => {
});
});

describe('getByIDs', () => {
describe('getByIds', () => {
it('should call audit logger', async () => {
const soClient = savedObjectsClientMock.create();

Expand All @@ -525,7 +525,7 @@ describe('Agent policy', () => {
],
});

await agentPolicyService.getByIDs(soClient, ['test-agent-policy-1', 'test-agent-policy-2']);
await agentPolicyService.getByIds(soClient, ['test-agent-policy-1', 'test-agent-policy-2']);

expect(mockedAuditLoggingService.writeCustomSoAuditLog).toHaveBeenNthCalledWith(1, {
action: 'get',
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -507,7 +507,7 @@ class AgentPolicyService {
return agentPolicy;
}

public async getByIDs(
public async getByIds(
soClient: SavedObjectsClientContract,
ids: Array<string | { id: string; spaceId?: string }>,
options: { fields?: string[]; withPackagePolicies?: boolean; ignoreMissing?: boolean } = {}
Expand Down Expand Up @@ -1345,7 +1345,7 @@ class AgentPolicyService {
});
}

const policies = await agentPolicyService.getByIDs(soClient, agentPolicyIds);
const policies = await agentPolicyService.getByIds(soClient, agentPolicyIds);
const policiesMap = keyBy(policies, 'id');
const fullPolicies = await pMap(
agentPolicyIds,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ import { getHostedPolicies, isHostedAgent } from './hosted_agent';
jest.mock('../agent_policy', () => {
return {
agentPolicyService: {
getByIDs: jest.fn().mockResolvedValue([
getByIds: jest.fn().mockResolvedValue([
{ id: 'hosted-policy', is_managed: true },
{ id: 'regular-policy', is_managed: false },
]),
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ export async function getHostedPolicies(
);

// get the agent policies for those ids
const agentPolicies = await agentPolicyService.getByIDs(soClient, Array.from(policyIdsToGet), {
const agentPolicies = await agentPolicyService.getByIds(soClient, Array.from(policyIdsToGet), {
fields: ['is_managed'],
ignoreMissing: true,
});
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ jest.mock('../agent_policy', () => {
return {
agentPolicyService: {
getInactivityTimeouts: jest.fn().mockResolvedValue([]),
getByIDs: jest.fn().mockResolvedValue([{ id: 'hosted-agent-policy', is_managed: true }]),
getByIds: jest.fn().mockResolvedValue([{ id: 'hosted-agent-policy', is_managed: true }]),
list: jest.fn().mockResolvedValue({ items: [] }),
},
};
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -204,7 +204,7 @@ describe('getFleetServerPolicies', () => {
page: 1,
perPage: mockPackagePolicies.length,
});
(mockedAgentPolicyService.getByIDs as jest.Mock).mockResolvedValueOnce(mockFleetServerPolicies);
(mockedAgentPolicyService.getByIds as jest.Mock).mockResolvedValueOnce(mockFleetServerPolicies);
const result = await getFleetServerPolicies(soClient);
expect(result).toEqual(mockFleetServerPolicies);
});
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ export const getFleetServerPolicies = async (

// Retrieve associated agent policies
const fleetServerAgentPolicies = fleetServerAgentPolicyIds.length
? await agentPolicyService.getByIDs(
? await agentPolicyService.getByIds(
soClient,
uniqBy(fleetServerAgentPolicyIds, (p) => p.id)
)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -15,10 +15,12 @@ export { getRegistryUrl } from './epm/registry/registry_url';
*/

export interface AgentPolicyServiceInterface {
create: (typeof agentPolicyService)['create'];
get: (typeof agentPolicyService)['get'];
list: (typeof agentPolicyService)['list'];
delete: (typeof agentPolicyService)['delete'];
getFullAgentPolicy: (typeof agentPolicyService)['getFullAgentPolicy'];
getByIds: (typeof agentPolicyService)['getByIDs'];
getByIds: (typeof agentPolicyService)['getByIds'];
turnOffAgentTamperProtections: (typeof agentPolicyService)['turnOffAgentTamperProtections'];
fetchAllAgentPolicyIds: (typeof agentPolicyService)['fetchAllAgentPolicyIds'];
fetchAllAgentPolicies: (typeof agentPolicyService)['fetchAllAgentPolicies'];
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -316,7 +316,7 @@ describe('Output Service', () => {
} as unknown as ReturnType<typeof mockedPackagePolicyService.list>;

beforeEach(() => {
mockedAgentPolicyService.getByIDs.mockResolvedValue([]);
mockedAgentPolicyService.getByIds.mockResolvedValue([]);
mockedAgentPolicyService.list.mockClear();
mockedPackagePolicyService.list.mockReset();
mockedAgentPolicyService.hasAPMIntegration.mockClear();
Expand All @@ -334,7 +334,7 @@ describe('Output Service', () => {
});

afterEach(() => {
mockedAgentPolicyService.getByIDs.mockClear();
mockedAgentPolicyService.getByIds.mockClear();
});

describe('create', () => {
Expand Down Expand Up @@ -688,7 +688,7 @@ describe('Output Service', () => {
mockedPackagePolicyService.list.mockResolvedValue(
mockedPackagePolicyWithFleetServerResolvedValue
);
mockedAgentPolicyService.getByIDs.mockResolvedValue(
mockedAgentPolicyService.getByIds.mockResolvedValue(
(await mockedAgentPolicyWithFleetServerResolvedValue).items
);

Expand Down Expand Up @@ -727,7 +727,7 @@ describe('Output Service', () => {
mockedPackagePolicyService.list.mockResolvedValue(
mockedPackagePolicyWithSyntheticsResolvedValue
);
mockedAgentPolicyService.getByIDs.mockResolvedValue(
mockedAgentPolicyService.getByIds.mockResolvedValue(
(await mockedAgentPolicyWithSyntheticsResolvedValue).items
);

Expand Down Expand Up @@ -845,7 +845,7 @@ describe('Output Service', () => {
mockedPackagePolicyService.list.mockResolvedValue(
mockedPackagePolicyWithFleetServerResolvedValue
);
mockedAgentPolicyService.getByIDs.mockResolvedValue(
mockedAgentPolicyService.getByIds.mockResolvedValue(
(await mockedAgentPolicyWithFleetServerResolvedValue).items
);

Expand Down Expand Up @@ -884,7 +884,7 @@ describe('Output Service', () => {
mockedPackagePolicyService.list.mockResolvedValue(
mockedPackagePolicyWithSyntheticsResolvedValue
);
mockedAgentPolicyService.getByIDs.mockResolvedValue(
mockedAgentPolicyService.getByIds.mockResolvedValue(
(await mockedAgentPolicyWithSyntheticsResolvedValue).items
);

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -176,7 +176,7 @@ async function getAgentPoliciesPerOutput(outputId?: string, isDefault?: boolean)
}, [])
),
];
const agentPoliciesFromPackagePolicies = await agentPolicyService.getByIDs(
const agentPoliciesFromPackagePolicies = await agentPolicyService.getByIds(
internalSoClientWithoutSpaceExtension,
agentPolicyIdsFromPackagePolicies
);
Expand Down Expand Up @@ -245,7 +245,7 @@ async function findPoliciesWithFleetServerOrSynthetics(outputId?: string, isDefa
);
const agentPolicyIds = _.uniq(packagePolicies.flatMap((p) => p.policy_ids));
if (agentPolicyIds.length) {
agentPolicies = await agentPolicyService.getByIDs(
agentPolicies = await agentPolicyService.getByIds(
internalSoClientWithoutSpaceExtension,
agentPolicyIds
);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -227,7 +227,7 @@ const mockAgentPolicyGet = (spaceIds: string[] = ['default']) => {
});
}
);
mockAgentPolicyService.getByIDs.mockImplementation(
mockAgentPolicyService.getByIds.mockImplementation(
// @ts-ignore
(_soClient: SavedObjectsClientContract, ids: string[]) => {
return Promise.resolve(
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -518,7 +518,7 @@ class PackagePolicyClientImpl implements PackagePolicyClient {

const agentPolicyIds = new Set(packagePolicies.flatMap((pkgPolicy) => pkgPolicy.policy_ids));

const agentPolicies = await agentPolicyService.getByIDs(soClient, [...agentPolicyIds]);
const agentPolicies = await agentPolicyService.getByIds(soClient, [...agentPolicyIds]);
const agentPoliciesIndexById = indexBy('id', agentPolicies);
for (const agentPolicy of agentPolicies) {
validateIsNotHostedPolicy(agentPolicy, options?.force);
Expand Down Expand Up @@ -1551,7 +1551,7 @@ class PackagePolicyClientImpl implements PackagePolicyClient {
return acc;
}, new Set());

const agentPolicies = await agentPolicyService.getByIDs(soClient, uniquePolicyIdsR);
const agentPolicies = await agentPolicyService.getByIds(soClient, uniquePolicyIdsR);

for (const policyId of uniquePolicyIdsR) {
const agentPolicy = agentPolicies.find((p) => p.id === policyId);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -217,7 +217,7 @@ describe('UninstallTokenService', () => {
agentPolicyService.deployPolicies = jest.fn();

getAgentPoliciesByIDsMock = jest.fn().mockResolvedValue([]);
agentPolicyService.getByIDs = getAgentPoliciesByIDsMock;
agentPolicyService.getByIds = getAgentPoliciesByIDsMock;

if (scoppedInSpace) {
soClientMock.getCurrentNamespace.mockReturnValue(scoppedInSpace);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -305,7 +305,7 @@ export class UninstallTokenService implements UninstallTokenServiceInterface {
}

private async getPolicyIdNameDictionary(policyIds: string[]): Promise<Record<string, string>> {
const agentPolicies = await agentPolicyService.getByIDs(this.soClient, policyIds, {
const agentPolicies = await agentPolicyService.getByIds(this.soClient, policyIds, {
ignoreMissing: true,
});

Expand Down Expand Up @@ -615,7 +615,7 @@ export class UninstallTokenService implements UninstallTokenServiceInterface {
const batchSize = config?.setup?.agentPolicySchemaUpgradeBatchSize ?? 100;

await asyncForEach(chunk(policyIds, batchSize), async (policyIdsBatch) => {
const policies = await agentPolicyService.getByIDs(
const policies = await agentPolicyService.getByIds(
appContextService.getInternalUserSOClientWithoutSpaceExtension(),
policyIds.map((id) => ({ id, spaceId: '*' }))
);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,11 @@
"search",
"connectors"
],
"requiredPlugins": [],
"optionalPlugins": [],
"requiredBundles": []
"requiredPlugins": [
"licensing",
"taskManager",
"fleet"
],
"optionalPlugins": []
}
}
Loading
Loading