Skip to content

Knox Client

Jump to bottom
Jon Parise edited this page Oct 7, 2019 · 7 revisions

Knox Client

The Knox client is designed to be used on every machine that needs access to keys. It is an important part of the knox infrastructure since it caches the key locally and keeps it up to date during rotation. In the below sections we will discuss the general design of the local caching, the commands available via the client, how to interact with the knox client via code, and a general developer workflow for migrating a secret into knox.

How file system caching works

On each machine using keys, the key is cached locally on the machine to prevent outages that may occur if the knox server is ever unavailable. Knox uses the file system to store these cached keys. In order to do this the knox client must be installed on any machine using knox. The knox client includes a long running command (“knox daemon”) that will update keys stored in “/var/lib/knox/v0/keys/<key_id>” every 10 minutes or whenever the register file is updated. This register file should be updated by running the commands “knox register” and “knox unregister” which take in a key id or list of key ids. The long running command updates the files by reading the registered list, going through all of the files in “/var/lib/knox/v0/keys/”, removing ones not in the registered list, pulling the version hash from those that are already registered, sending the server a list with the current cached version, and then requesting any key that does not match the current cached version and saving them into a key file.

Commands available

$ knox help
Knox is a tool for storing and rotating keys.

Usage:

	knox command [arguments]

The commands are:

    daemon      runs a process to keep keys in sync with server
    register    register keys to cache locally using daemon
    unregister  unregister a key identifier from daemon
    keys        gets keys and associated version hash
    get         get a knox key
    versions    gets the versions for a key
    promote     promotes a key to primary state
    create      creates a new key
    add         adds a new key version to knox
    deactivate  deactivates a key version
    reactivate  Reactivates an inactive key version
    access      access modifies the acl of a key
    delete      deletes an existing key
    login       login as user and save authentication data

Using knox in code

At Pinterest we have knox clients written in many languages, but they all do essentially the same thing. In the source code we include a client in Golang. For clients at Pinterest we recommend that they execute “knox register -k <key_id>” and “KNOX_MACHINE_AUTH=hostname knox get -j <key_id>” and then periodically check the file in /var/lib/knox/v0/keys/<key_id>. This allows the primary and active keys to be cached in memory. We also highly recommend that the functions that use the keys are called at time of use rather than once in a settings file. This allows for rotation to happen on the fly.

Example of Go code using knox secret.

// called in main function and passed through
c := NewFileClient(‘webapp:session_hmac’)
...

func Sign(knox.Client c, data string) ([]byte, error) {
	// GetPrimary is called when the underlying signing operation is called.
	return crypto.sign(c.GetPrimary(), data)
}

func Verify(knox.Client c, data []byte) error {
	for _, k := range c.GetActive() {
		// getactive is called when verify operation is called
		out, err := crypto.verify(k, data)
		If err == nil {
			return nil
		}
	}
	return fmt.Errorf(“no key could verify data”)
}

Migrating a hard-coded secret into Knox

Key migration generally involves 5 steps:

  1. Logging into Knox
  2. Creating a key with your secret data
  3. Telling Knox who can access that key
  4. Talking to Knox from your code
  5. Rotating your key as necessary.

Logging into Knox

Run "knox login" and enter your password on any machine with knox installed (i.e. your devapp). This authenticates you as an employee so you can create and edit keys you have access to.

devinlundberg@dev-devinlundberg:~$ knox login
Please enter your password:
devinlundberg@dev-devinlundberg:~$
### Creating a key with your secret data

The first step is to create a key. "knox create " reads data from stdin and creates a key with that name. The name for your key should follow the format :

# if your key is in a file named "data/super_secure_key_file" you can pipe it in like this. 
cat data/super_secure_key_file | knox create security_service:demo_using_knox_mac
# or if your key is a string that isn't in a file.
printf "SECRETKEY@#*&SDANklaf#sl" | knox create security_service:demo_using_knox_s3_access

Telling Knox who can access that key

By default, the creator and a select group of SRE/security admins are the only ones who can access your key. It is likely that more users and machines will actually need to access the key that you create. You can choose read (-r), write (-w), admin (-a), or none (-n) for the permissions and machine(-M), user(-U), user group(-G), or hostname prefix (-P). To add another user who is able to do key rotation (but not able to delete or add other users/machines):

knox access -U -w security_service:demo_using_knox_mac devinlundberg

You can also use an LDAP group to define a group of users who can modify your key. This is the best way to ensure your team can access and rotate the key:

knox access -G -w security_service:demo_using_knox_mac data-infra

To add a group of machines to be able to read your key, you would use the machine prefix (which allows any machine with a hostname matching that prefix to access the key):

knox access -P -r security_service:demo_using_knox_mac webapp-

To remove access from a user, machine, group, machine prefix you previously added, use the -n option which means no access. This would remove access from the group of webapp- machines:

knox access -P -n security_service:demo_using_knox_mac webapp-

Talking to Knox from your code

Now that you have created your key and given access to the right machines and users, you'll need to actually get your key. To get the key from the command line you can run:

knox get security_service:demo_using_knox_mac

This will return the primary version of that key to stdout. To get all of the active versions of your key, you should run:

knox versions security_service:demo_using_knox_mac
# or
knox versions -s active security_service:demo_using_knox_mac

In order to do rotation properly without having to redeploy your code, you will need to check with knox at time of use (every time your key is used). This begs the questions: do I need to make a network call every time I talk to Knox? No, the knox daemon running on your machine will automatically cache your machines on the file system if you register what keys you need. This command supports several options to register groups of keys. The simplest option will register a single key by its key_id:

knox register -k security_service:demo_using_knox_mac

What I recommend for services is to maintain a list of new line separated keys in a file and send it to knox on deploy with this command. The r flag will remove all existing keys from the file system and stop updating them (if they are not on the list you passed in). This way you if you phase out the usage of a key entirely, it won't take up disk space.

knox register -r -f /mnt/security_service/list_o_keys

Once the keys are registered using this method. The "knox get" command will access them locally for read operations rather than making network calls. If you have a high performance application where executing "knox get" is unacceptable, then you are welcome to make a file watcher on "/var/lib/knox/v0/keys/". This will return an json object which you can decode and use. The format of the json is as follows:

{
  "id": "bob",
  "acl": [
    {
      "type": "User",
      "id": "devinlundberg",
      "access": "Admin"
    }
  ],
  "versions": [
    {
      "id": 5577006791947779000,
      "data": "d29vZgo=",
      "status": "Primary",
      "ts": 1429644160665073000
    }
  ],
  "hash": "1743d464e7fb0a00733b8cd68c63f3bc663118ba444dd98383f68314efa3aa83"
}

For writing a client library to communicate with Knox from your code base, the easiest to use and most performant approach tends to be to call knox register -g -k <key_id> in the initializer which should get called on startup or once per long running process and then do polling every few minutes on /var/lib/knox/v0/keys/<key_id> for updates to the key. The -g flag will register the key (if it is not already registered) and return the JSON formatted key. This is faster than calling both knox register and knox get. To see an example of this see the Go client at https://github.com/pinterest/knox/blob/master/client.go#L70.

Key Rotation

The last thing you should do is rotate your key. Since your key was likely living in github it was exposed all over the place. There are 3 steps for rotation. You should wait a little bit to make sure there are no problems during the process. Problems may occur if you start a subsequent step before your fleet as pulled the latest updates. The first step is to add the new version of the key. This will make it an active version, but not the primary version.

cat ~/super-secret-key-v2 | knox add security_service:demo_using_knox_mac

The second step is to promote the new key version to become the primary version. The key version was returned by the add action but can also be found using versions:

# This lists the current primary version of the key
knox versions -s primary test
# This lists the all active versions of the key
knox versions -s active test

To promote the new key version and make the old version active:

knox promote security_service:demo_using_knox_mac 2138977845343389399

Deactivation of the old primary key is last. You should wait for a little while the update spreads across your fleet and make sure there are no problems. Deactivating your old primary key will mean that it is no longer used as a fallback in the case of old data coming through. In the case of a database this means all of your data needs to be migrated to the new key. In the case of a session cookie, all users with the old cookie will be logged out when this happens. In terms of an api key, this step likely won't cause problems since active keys will likely not be used.

knox deactivate security_service:demo_using_knox_mac 8674665227082153551

Congrats you have finished migrating your keys to knox!

Clone this wiki locally