Fluentd filter plugin to mask sensitive or privacy records with ******* in place of the original value. This data masking plugin protects data such as name, email, phonenumber, address, and any other field you would like to mask.
| fluent-plugin-masking | fluentd | ruby |
|---|---|---|
| 1.2.x | >= v0.14.0 | >= 2.5 |
Install with gem:
fluent-gem install fluent-plugin-masking
In order to setup this plugin, the parameter fieldsToMaskFilePath needs to be a valid path to a file containing a list of all the fields to mask. The file should have a unique field on each line. These fields are case-sensitive (Name != name). in case you need one (or more) of the fields to be case insensitive, use the /i suffix in your field name. see example below.
-
fieldsToExcludeJSONPaths- this field receives as input a comma separated string of JSON fields that should be excluded in the masking procedure. Nested JSON fields are supported bydot notation(i.e:path.to.excluded.field.in.record.nestedExcludedField) The JSON fields that are excluded are comma separated.
This can be used for logs of registration services or audit log entries which do not need to be masked. -
handleSpecialEscapedJsonCases- a boolean value that try to fix special escaped json cases. this feature is currently on alpha stage (default: false). for more details about thoose special cases see Special Json Cases
An example with optional configuration parameters:
<filter "**">
@type masking
fieldsToMaskFilePath "/path/to/fields-to-mask-file"
fieldsToExcludeJSONPaths "excludedField,exclude.path.nestedExcludedField"
handleSpecialEscapedJsonCases true
</filter>
Example fields-to-mask-file:
name
email
phone/i # the '/i' suffix will make sure phone field will be case insensitive
# fluent.conf
----------------------------------
<source>
@type tail
path /tmp/test.log
pos_file /tmp/test.log.pos
tag maskme
format none
</source>
<filter "**">
@type masking
fieldsToMaskFilePath "/path/to/fields-to-mask-file"
fieldsToExcludeJSONPaths "excludedField,exclude.path.nestedExcludedField"
</filter>
<match "**">
@type stdout
</match>
# /path/to/fields-to-mask-file
----------------------------------
first_name
last_name
address
phone
password
email
To run the above configuration, run the following commands:
fluentd -c fluent.conf
echo '{ :body => "{\"first_name\":\"mickey\", \"type\":\"puggle\", \"last_name\":\"the-dog\", \"password\":\"d0g43u39\"}"}' > /tmp/test.log
This sample result is created from the above configuration file fluent.conf. As expected, the following fields configured to be masked are masked with ******* in the output.
2019-09-15 16:12:50.359191000 +0300 maskme: {"message":"{ :body => \"{\\\"first_name\\\":\\\"*******\\\", \\\"type\\\":\\\"puggle\\\", \\\"last_name\\\":\\\"*******\\\", \\\"password\\\":\\\"*******\\\"}\"}"}
A sample with exclude in use:
fluentd -c fluent.conf
echo '{ :body => "{\"first_name\":\"mickey\", \"type\":\"puggle\", \"last_name\":\"the-dog\", \"password\":\"d0g43u39\"}", "excludeMaskFields"=>"first_name,last_name"}' > /tmp/test.log
2019-12-01 14:25:53.385681000 +0300 maskme: {"message":"{ :body => \"{\\\"first_name\\\":\\\"mickey\\\", \\\"type\\\":\\\"puggle\\\", \\\"last_name\\\":\\\"the-dog\\\", \\\"password\\\":\\\"*******\\\"}\"}"}
gem install bundler
bundle install
ruby -r ./test/*.rb