Verify Fleet Messaging Service can sign messages in FIPS mode #195282

kc13greiner · 2024-10-07T15:32:02Z

Summary

Verify Fleet message signing service works in FIPS mode

Details

While running the new Jest FIPS pipeline I noticed an error while running x-pack/plugins/fleet/server/services/security/message_signing_service.test.ts

kibana/x-pack/plugins/fleet/server/services/security/message_signing_service.test.ts

Lines 192 to 211 in 17fcaa5

    
           it('can correctly sign messages', async () => { 
        
             mockCreatePointInTimeFinderAsInternalUser([keyPairObj]); 
        
             const message = Buffer.from(JSON.stringify({ message: 'foobar' }), 'utf8'); 
        
             const { data, signature } = await messageSigningService.sign(message); 
        
             const verifier = createVerify('SHA256'); 
        
             verifier.update(data); 
        
             verifier.end(); 
        
             const serializedPublicKey = await messageSigningService.getPublicKey(); 
        
             const publicKey = Buffer.from(serializedPublicKey, 'base64'); 
        
             const isVerified = verifier.verify( 
        
               { key: publicKey, format: 'der', type: 'spki' }, 
        
               signature, 
        
               'base64' 
        
             ); 
        
             expect(isVerified).toBe(true); 
        
             expect(data).toBe(message); 
        
           });

Specifically this code:

kibana/x-pack/plugins/fleet/server/services/security/message_signing_service.ts

Lines 128 to 132 in 17fcaa5

    
           const privateKey = Buffer.from(serializedPrivateKey, 'base64'); 
        
           const signature = signer.sign( 
        
             { key: privateKey, passphrase, format: 'der', type: 'pkcs8' }, 
        
             'base64' 
        
           );

Based on the code, everything looks FIPS compliant, and the Key Pair generation looks good as well.

I "patched" some other errors for this image (related #194944) so I could have this image build.

Once the other errors were patched, the failing jest test above started passing 🤔

I was unable to reproduce the failing test in my local FIPS env, so it was always suspicious.

I would like to verify that this functionality works as expected in FIPS mode, but I am not familiar enough with Fleet to do so.

Setup a FIPS environment locally

Start an ES instance in a method of your choosing, but not using yarn es snapshot. I like to use a 9.0.0-snapshot from the .es/cache directory by running tar -xzvf elasticsearch-9.9.0-SNAPSHOT-darwin-aarch64.tar.gz and cd into the new directory's bin folder to run ./elasticsearch

In a new terminal window, navigate to your the top level of your elasticsearch folder and run:
curl -X POST --cacert config/certs/http_ca.crt -u elastic:YOUR_PASSWORD_HERE "https://localhost:9200/_license/start_trial?acknowledge=true&pretty"

This will enable the trial license for ES.

Ensure you have Docker running locally.

From any command line, run:

docker run --rm -it \
-v "$(pwd)"/config/keyStore.p12:/keyStore.p12:ro \
-e XPACK_SECURITY_FIPSMODE_ENABLED='true' \
-p 5601:5601/tcp docker.elastic.co/kibana-ci/kibana-ubi-fips:9.0.0-SNAPSHOT-412a745685504415e750e90a8815a36ae0327c45

This will start Kibana into Interactive Setup mode, copy and paste the token from the ES startup logs.

AC

Make sure Message Signing Key is still working
Make sure Uninstall token is still working
Make sure Agent tamper protection is still working

The text was updated successfully, but these errors were encountered:

elasticmachine · 2024-10-07T15:32:05Z

Pinging @elastic/fleet (Team:Fleet)

elasticmachine · 2024-10-07T15:32:05Z

Pinging @elastic/kibana-security (Team:Security)

jlind23 · 2024-10-07T15:49:35Z

Hey @ferullo, iirc this service is owned by your team. Would you mind having a look?

elasticmachine · 2024-10-08T07:26:04Z

Pinging @elastic/security-defend-workflows (Team:Defend Workflows)

cmacknz · 2024-10-08T15:58:13Z

The signatures generated here are validated by the Elastic Agent for actions here and policy changes here.

This functionality was added in elastic/elastic-agent#2348. We will have to verify that these code blocks can tolerate a change in the signature key correctly. The key is delivered in the checkin response so this should be the case but I am unsure if this has dedicated tests.

The agent side of this looks to be FIPS compliant because it is only parsing the key, the crypto primitives used to verify the key appear FIPS approved here:

szwarckonrad · 2024-10-10T13:48:01Z

I’ve completed the initial validation on the described environment and can confirm that we are able to:

Enroll Fleet Server
Create Agent Policy with Defend Integration
Enroll an Agent through Fleet Server and assign it to previously created agent policy
Enable Agent Tamper Protection (ATP)
We are not able to uninstall agent with active ATP without uninstall token
We are able to uninstall agent with active ATP with valid uninstall token

kc13greiner · 2024-10-10T14:23:00Z

++ thank you for the quick turnaround!

Just to clarify:

We are not able to uninstall agent with active ATP without uninstall token

This is desired behavior?

szwarckonrad · 2024-10-11T09:49:45Z

++ thank you for the quick turnaround!

Just to clarify:

We are not able to uninstall agent with active ATP without uninstall token

This is desired behavior?

It is! :)
I will be performing more thorough validation in the upcoming week.

szwarckonrad · 2024-10-21T12:18:52Z

@kc13greiner
CC @dasansol92 @ferullo

I was able to validate the behavior with FIPS both disabled and enabled, and I can confirm that this change does not seem to affect any functionalities our team owns that utilize Fleet’s signed messages. Please review the methodology I followed and let me know if anything might have been overlooked in terms of environment setup or validation assumptions.

Elasticsearch:

Started the ES instance using .es/cache/elasticsearch-9.9.0-SNAPSHOT-darwin-aarch64.tar.gz from the latest main branch.
Set the license to trial via the API call:

curl -X POST --cacert config/certs/http_ca.crt -u elastic:YOUR_PASSWORD_HERE "https://localhost:9200/_license/start_trial?acknowledge=true&pretty"

Kibana:

Started the Kibana instance with the following command:

docker run --rm -it \
-v "$(pwd)"/config/keyStore.p12:/keyStore.p12:ro \
-e XPACK_ENCRYPTEDSAVEDOBJECTS_ENCRYPTIONKEY="4ba83274dab90fe88cef7f904dff2327" \
-e XPACK_SECURITY_FIPSMODE_ENABLED='false' \
-p 5601:5601/tcp docker.elastic.co/kibana-ci/kibana-ubi-fips:9.0.0-SNAPSHOT-412a745685504415e750e90a8815a36ae0327c45

Observed logs NOT containing "Kibana is running in FIPS mode"/
Enrolled a Fleet Server.
Created two agent policies, both with Defend integrations.
Enrolled 3 agents—2 in the first agent policy and 1 in the second.
Modified agent policies (set ATP to true), updated package policies (created artifacts, adjusted settings).
Sent several response actions to all endpoints.
Stopped Kibana and relaunched it with FIPS mode enabled:

docker run --rm -it \
-v "$(pwd)"/config/keyStore.p12:/keyStore.p12:ro \
-e XPACK_ENCRYPTEDSAVEDOBJECTS_ENCRYPTIONKEY="4ba83274dab90fe88cef7f904dff2327" \
-e XPACK_SECURITY_FIPSMODE_ENABLED='true' \
-p 5601:5601/tcp docker.elastic.co/kibana-ci/kibana-ubi-fips:9.0.0-SNAPSHOT-412a745685504415e750e90a8815a36ae0327c45

Observed logs confirming that "Kibana is running in FIPS mode".

Validation after upgrading Kibana from NON-FIPS to FIPS:

Able to see all previously applied changes (agent policies with ATP enabled, custom artifact settings, response actions history).
Unable to uninstall an agent without providing an uninstall token.
Able to modify agent/integration policies further—changes were properly saved, and modified artifacts had updated IDs.
Able to perform response actions on enrolled agents, with response action history updating correctly.
Able to create a new agent policy with Defend integration.
Able to enroll a new agent and assign the newly created agent policy.
Able to modify the newly created agent policy, as well as the integration policy, by enabling ATP and modifying artifacts.
Able to perform response actions on the newly created agent.
Able to isolate the newly created host using automated response actions triggered by a rule.
Able to release the agent with response action.
Unable to uninstall the agent without providing an uninstall token.
Able to uninstall the agent when providing the correct uninstall token.

kc13greiner · 2024-10-21T14:37:44Z

Apologies, @szwarckonrad that test is not valid.

I provided the old env var in my instructions, XPACK_SECURITY_FIPSMODE_ENABLED, it should be XPACK_SECURITY_EXPERIMENTAL_FIPSMODE_ENABLED

Setting the new env var to false will not allow this image to start.

We can try to use a "default" image for the non-FIPS portion of this test. I will provide the image soon.

szwarckonrad · 2024-10-22T09:09:08Z

@kc13greiner @dasansol92

I was able to achieve the same validation results as mentioned above by following this setup process:

Elasticsearch:

Started the ES instance using .es/cache/elasticsearch-9.9.0-SNAPSHOT-darwin-aarch64.tar.gz from the latest main branch.
Set the license to trial via the API call:

curl -X POST --cacert config/certs/http_ca.crt -u elastic:YOUR_PASSWORD_HERE "https://localhost:9200/_license/start_trial?acknowledge=true&pretty"

Kibana:

Started the Kibana instance with the following command:

docker run --rm -it \          
-e XPACK_ENCRYPTEDSAVEDOBJECTS_ENCRYPTIONKEY="4ba83274dab90fe88cef7f904dff2327" \
-p 5601:5601/tcp docker.elastic.co/kibana-ci/kibana-cloud:9.0.0-SNAPSHOT-c858162866dd9b2ff145b061f0cfd30f1c064fec

Observed logs NOT containing "Kibana is running in FIPS mode"
Enrolled a Fleet Server.
Created two agent policies, both with Defend integrations.
Enrolled 3 agents—2 in the first agent policy and 1 in the second.
Modified agent policies (set ATP to true), updated package policies (created artifacts, adjusted settings).
Sent several response actions to all endpoints.
Stopped Kibana and relaunched it with FIPS mode enabled using FIPS compliant image:

docker run --rm -it \
-v "$(pwd)"/config/keyStore.p12:/keyStore.p12:ro \
-e XPACK_ENCRYPTEDSAVEDOBJECTS_ENCRYPTIONKEY="4ba83274dab90fe88cef7f904dff2327" \
-e XPACK_SECURITY_EXPERIMENTAL_FIPSMODE_ENABLED="true" \
-p 5601:5601/tcp docker.elastic.co/kibana-ci/kibana-ubi-fips:9.0.0-SNAPSHOT-c858162866dd9b2ff145b061f0cfd30f1c064fec

Observed logs confirming that "Kibana is running in FIPS mode".

@dasansol92
I have also validated that the fleet-message-signing-keys SO remains unchanged after the upgrade and is still a single SO, and that the fleet-uninstall-tokens SO contains the same content as before the upgrade.

kc13greiner · 2024-10-22T12:08:37Z

Excellent @szwarckonrad - that test looks good!

Thank you so much for getting this checked so quickly 🚀

szwarckonrad · 2024-10-25T14:09:42Z

@kc13greiner
Should we close this issue?

kc13greiner · 2024-10-25T14:12:46Z

Yes! I think we are all set with this testing. Thanks again for the fast turnaround 🚀

kc13greiner added Feature:FIPS FIPS mode for Kibana Team:Fleet Team label for Observability Data Collection Fleet team Team:Security Team focused on: Auth, Users, Roles, Spaces, Audit Logging, and more! labels Oct 7, 2024

dasansol92 added the Team:Defend Workflows “EDR Workflows” sub-team of Security Solution label Oct 8, 2024

dasansol92 added the grooming label Oct 8, 2024

kpollich mentioned this issue Oct 8, 2024

Add or change FIPS compliant password hashing algorithm in Fleet #194944

Closed

dasansol92 added planning and removed grooming labels Oct 9, 2024

dasansol92 added OLM Sprint and removed planning labels Oct 10, 2024

dasansol92 assigned szwarckonrad Oct 10, 2024

kc13greiner closed this as completed Oct 25, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Verify Fleet Messaging Service can sign messages in FIPS mode #195282

Verify Fleet Messaging Service can sign messages in FIPS mode #195282

kc13greiner commented Oct 7, 2024 •

edited by dasansol92

Loading

elasticmachine commented Oct 7, 2024

elasticmachine commented Oct 7, 2024

jlind23 commented Oct 7, 2024

elasticmachine commented Oct 8, 2024

cmacknz commented Oct 8, 2024

szwarckonrad commented Oct 10, 2024

kc13greiner commented Oct 10, 2024

szwarckonrad commented Oct 11, 2024

szwarckonrad commented Oct 21, 2024 •

edited

Loading

kc13greiner commented Oct 21, 2024 •

edited

Loading

szwarckonrad commented Oct 22, 2024

kc13greiner commented Oct 22, 2024

szwarckonrad commented Oct 25, 2024

kc13greiner commented Oct 25, 2024

Verify Fleet Messaging Service can sign messages in FIPS mode #195282

Verify Fleet Messaging Service can sign messages in FIPS mode #195282

Comments

kc13greiner commented Oct 7, 2024 • edited by dasansol92 Loading

Summary

Details

Setup a FIPS environment locally

AC

elasticmachine commented Oct 7, 2024

elasticmachine commented Oct 7, 2024

jlind23 commented Oct 7, 2024

elasticmachine commented Oct 8, 2024

cmacknz commented Oct 8, 2024

szwarckonrad commented Oct 10, 2024

kc13greiner commented Oct 10, 2024

szwarckonrad commented Oct 11, 2024

szwarckonrad commented Oct 21, 2024 • edited Loading

Elasticsearch:

Kibana:

Validation after upgrading Kibana from NON-FIPS to FIPS:

kc13greiner commented Oct 21, 2024 • edited Loading

szwarckonrad commented Oct 22, 2024

Elasticsearch:

Kibana:

kc13greiner commented Oct 22, 2024

szwarckonrad commented Oct 25, 2024

kc13greiner commented Oct 25, 2024

kc13greiner commented Oct 7, 2024 •

edited by dasansol92

Loading

szwarckonrad commented Oct 21, 2024 •

edited

Loading

kc13greiner commented Oct 21, 2024 •

edited

Loading