Cobra-policytool is a tool to ease management of Apache Ranger together with tags in Apache Atlas. These tools manage access policies in Hadoop environments. Cobra-policytool makes it easy to apply configuration files direct to Atlas and Ranger at scale.
The advantages are:
- configurations can be version controlled
- changes can be reviewed, audited and tracked
- integrates with existing CI/CD
Cobra-policytool does also add functionality to Atlas and Ranger. Cobra-policytool can manage have row level filtering policies for Apache Hive based on tags. Ranger requires one row level policy per table, but with cobra-policytool one can have one rule per tag. This rule is then expanded by the cobra-policytool to one rule fore each table having the tag.
Most often one want to have the same access rights hive tables and corresponding files and directories on hdfs. Cobra-policytool can automatically convert a policy for a Hive table to policy for hdfs.
This eases the maintenance and reduce risks for errors.
To be able to use the tool you need to have the right permissions in the environment you are using. For Atlas you must be able to read and create tags and to be able to add and delete them from resources. For the Ranger rules you must be admin, unfortunately.
Cobra-policytool is idempotent, that means you can rerun it as much as you want, the result will not change if on have not changed the input.
There is an introduction how to use cobra-policytool tool on Medium
A presentation about how Cobra-policytool is used within Svenska Spel can be found at Slideshare. and Youtube
- Make it easy to manage access policies and metadata within a Apache Hadoop environment that uses Apache Atlas and Apache Ranger.
- Provide an easy way to apply policies from configuration files, that can be version controlled.
- Configuration files shall be easy to generate, for instance from a central metadata management system.
- Handle the security within the Hadoop environment. We rely on Apache Atlas, Atlas Ranger and other tools within the Hadoop ecosystem.
We welcome contributions. In order for us to be able to accept them, please review our contributor guidelines.
This project is released as open source according to the terms laid out in the LICENSE.
- Sync of table and column tags from metadata files to Atlas.
- Keep tags between hive corresponding directory on hdfs in sync (use option --hdfs)
- Audit to show differences between metadata and Atlas.
- New tag definitions are automatically added to Atlas on sync.
- Verbose output to provide changes done.
- Authentication using kerberos ticket.
- Sync policies from metadata file to Ranger.
- Expand tag based row filtering rules to Hive row based filtering.
- Atlas, Ranger, and Hive installed and working.
- Kerberos turned on on the Hadoop cluster, including Atlas and Ranger. Your client do also need to have a valid kerberos ticket.
- Python 2.7.
- We have successfully used it on MS Windows, MacOS and Linux.
pip install cobra-policytool
To get up to date help how to use the tool:
cobra-policy --help
For any use where policytool talks to the Atlas server a kerberos ticket must be available.
Create a configfile matching your environment, see docs/Configfile.md.
Read about the indata files in docs/indata.md.
Policytool takes files in --srcdir directory created according
to indata specification and sync them with the metadata
store in Hadoop called Atlas. To do this run:
$ cobra-policy tags_to_atlas --srcdir src/main/tags/ --environment utv
There is an option --verbose to get more output from cobra-policytool describing what
tables and columns was changed. Note! If you run same cobra-policytool command twice
you will not get any changes the second time since all changes happened the
first round.
Sync Ranger policies works in a similar fashion, though it requires that project-name is provided. Project-name is a name of the project you are working in. It is used to find already existing policies in Ranger and to be able to separate the ranger rules into multiple projects.
$ cobra-policy rules_to_ranger --srcdir src/main/tags/ --environment dev --project-name dimension_out
The package can also be used as a python library. Here is a short example to use the Atlas Client class.
from requests_kerberos import HTTPKerberosAuth
import policytool.atlas
c = policytool.atlas.Client(
'http://atlas.test.my.org:21000/api/atlas',
auth=HTTPKerberosAuth())
c.known_tags()
c.get_tables("hadoop_out_utv")
For details read the Python doc for the code and look how the command line client is implemented.
Beside this document there are more in the docs directory. You can also find a todo list including future plans.
We recommend to read the convention document and indata document before you start.
Copyright 2018 AB SvenskaSpel
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.