Keytabs are critical for secure Hadoop clusters, as they allow the services to be launched without prompts for passwords
If your management tools sets up keytabs for you: use it.
kadmin.local
ktadd -k zk.service.keytab -norandkey zookeeper/devix@COTHAM
ktadd -k zk.service.keytab -norandkey zookeeper/devix.cotham.uk@COTHAM
exit
and of course, make it accessible
chgrp hadoop zk.service.keytab
chown zookeeper zk.service.keytab
check that the user can login
# sudo -u zookeeper klist -e -kt zk.service.keytab
# sudo -u zookeeper kinit -kt zk.service.keytab zookeeper/devix.cotham.uk
# sudo -u zookeeper klist
Keytabs expire
That is: entries in them have a limited lifespan (default: 1 year)
This is actually a feature —it limits how long a lost/stolen keytab can have access to the system.
At the same time, it's a major inconvenience as (a) the keytabs expire and (b) it's never immediately obvious why your cluster has stopped working.
Keytabs are sensitive items. They need to be treated as having all the access to the data of that principal