YAML based file format for sanitizer configuration

[svenjacobs]

I've been thinking about how to implement custom rules in Leon. Also there have been various requests for additional sanitizers. Over time implementing and maintaining all these sanitizers becomes cumbersome 😉

My idea is to implement a YAML based file format for declaring sanitizer configurations. I see the following benefits:

1. No programming knowledge is required to add a new sanitizer to the YAML file, which should simplify contributions to this project
2. Users could easily add their own rules to the file

But also there are the following open questions and challenges:

1. The file needs to be parsed at runtime and possible errors need to be handled. We will lose the compile-time safety of Kotlin files.
2. The file is bundled with the application but it should probably also be possible to update it via a network request? This would allow updating the sanitizer database without an app update. Adds additional complexity because of network requests and update mechanism.
3. How do users update the file with their own custom rules? It should be possible to update the database from a local YAML file on the Android device.
4. We should probably provide a command-line application for verifying the validity of the YAML file?

What do you think? Please provide your feedback and ideas.

[KibaNoOu]

Hi Sven,
You should try another app that is outdated but have an interesting concept to custom rules:
Clean Share

[svenjacobs]

I cannot install this app (it says the app is too old for my device) but according to the screenshots custom rules are added as regular expressions? Yes, this is also something that I thought about. But with the YAML file I see the benefit that everyone will benefit if users, instead of adding their custom rules just for themselves, provide an updated YAML file to this project in terms of a pull request?

[KibaNoOu]

Yes that app use regular expression, but the YAML approch should be better to have a comprehensive list of filter.

[svenjacobs]

The YAML could still use regular expressions. It could look like this:

- name: some_sanitizer
  rules:
    - regex1
    - regex2

[brsysadmin]

Hi @svenjacobs

Sorry for the delay: I was a little bit busy last days.

Personally I work in IT so YAML is a format I know: I have no problem with it.
But I think you should choose a format easy to parse / verify in an Android context: here I can't help you because I'm not developer and worst and I don't know Android.
I don't know if it exists a linter for YAML on Android but I found this for JSON: https://www.tutorialspoint.com/android/android_json_parser.htm
It seems Android supports natively JSON and therefore we could imagine it is able to raise an exception if the file is badly formatted no?
In this context, if it simplify your life, maybe you can use JSON instead of YAML ?!

The file needs to be parsed at runtime and possible errors need to be handled. We will lose the compile-time safety of Kotlin files.

Yes you are right but it is not so rare to "disconnect" the code/the app from the data no?
It is your application so of course you are the boss but personally I don't see the problem :-)
Your code is safe thanks to Kotlin / the build / etc.
And data are parsed when the app run: in case of error you could catch the Exception to debug the data file :)

The file is bundled with the application but it should probably also be possible to update it via a network request? This would allow updating the sanitizer database without an app update. Adds additional complexity because of network requests and update mechanism.

At the beginning I thought about a 'customized' sanitizer in addition of what you did with the embedded sanitizers.
But here I understood you think to the possibility to refactor a lot to have the same mechanism for all sanitizers.
So yes for the complexity I agree. But a lot of applications have this kind of approach :

• there is the app on a side
• data on another side
• when you start, the app downloads the data
• the app has a feature to periodically update the data

The problem is more this one: have you a website on which you can host the definition files?
If yes, we could imagine this:

• 1/ When you build the app, you embed a local copy of the files
• 2/ When the application starts:
  • if the embedded files are more recent than the files in the database (first launch, last update too far compared to the new version of the app): local files are copied in the database
  • periodical check (I don't know the name of the system in Android sorry) to compare the version of the files in the database with the remote server: if there is an update, download

To manage definition files I see many solutions :

• 1/ There is ONE global file for all sanitizers you manage :

  • you hardcode the url to get the file + the url to get the checksum file
  • to know if there is an update, you compare the remote checksum with the checksum of the local file (you have to calculate it of course)

• 2/ The app read a remote general JSON file (or another format) which contain:

  • the URL of each file to have in the database + the URL of each checksum
  • you hardcode the url to get the general file: it can be seen as a reference file
  • the app download the reference file (or read it remotely) + parse it
  • for each file in the reference file, the app compare the local checksum with remote checksum and download if needed (local file missing, checksum different)

    How do users update the file with their own custom rules? It should be possible to update the database from a local YAML file on the Android device.

Maybe you could start with an easy approach for you but complex for an end user.
You embedded / hardcode an empty file which is the "custom" rule file.
You write the code to parse it etc.
On our side (== as end user) we have to edit the file on our smartphone to fill it.
And a native system to edit the file through the app could come later? Step by step style :)

We should probably provide a command-line application for verifying the validity of the YAML file?

It is called a linter from my memory: I don't know if it exists for Android but if you read above I mentioned the potential solution with JSON :)