Add trufflehog for secret detection

[edoardottt]

See #150

[edoardottt]

Hi @hugo-syn , you can find here https://github.com/edoardottt/cariddi/tree/trufflehog a working code for trufflehog secrets detection support.
The problem of trufflehog is that A LOT of false positives are found. See the image below (scanned my website).
None of those are valid.

[hugo-syn]

Hi @edoardottt, that's why I initially add the option to filter some file extensions but I also enabled the secret verification feature of trufflhog. Normally each secret has a verifier that ignore invalid one. This is enabled here:

• https://github.com/hugo-syn/cariddi/blob/32cbfdadf342d6af7b105971855eedf9b8da6001/pkg/crawler/scan.go#L77
• https://github.com/trufflesecurity/trufflehog/blob/8c6f852a9cc98c29e7f3d666328ab45acef65658/pkg/detectors/detectors.go#L21

The detector might be broken try to reproduce the logic with one of the "secret" in your screenshot. For example for Rechargpayment: https://github.com/trufflesecurity/trufflehog/blob/8c6f852a9cc98c29e7f3d666328ab45acef65658/pkg/detectors/rechargepayments/rechargepayments.go#L49

It shouldn't be reported as a secret 🤔

[edoardottt]

Tbh the verify option was set to True, but nothing changed. Seems not working fine (or maybe it's my fault, idk...).
Obviously there should be an option also for this, it's not obvious that cariddi will send HTTP requests to these services.
Then, I scanned my website (quite small, few content) and trufflehog found many false positives, what about big targets? Will it find thousands of false positives? (and so thousands of HTTP requests to verify the findings?)
It would be a huge improvement, but I don't know how to proceeed to be honest...

[hugo-syn]

Hi @edoardottt It was my fault, I fixed it here:

• 00d38c6
  There is no truffelhog documentation so the code is the doc I missed this.

However, it does not work with your -proxy I don't know how trufflehog handle the proxy but if fix the false positive problem. You can still use export HTTPS_PROXY="http://127.0.0.1:8080" and it will work.

What's the problem with the fact that the tool verify the secrets by making HTTP request? By verifying them it will reduce the number of false positive.

[hugo-syn]

Otherwise you could add an option --enable-trufflehog to enable or not the usage of trufflhog and add a warning in the readme stating that it will verify the potential secrets and that it will result in additional request being made to the différent services

[edoardottt]

Hi @hugo-syn !

Yes, I've to admit that would be an interesting option...
My only concern is if that would be useful. Integrating trufflehog means adding a huge amount of dependencies