feat(server): Apply filters for IgnoreCves and PkgsRegexps on server mode #1270

qwexvf · 2021-07-08T06:39:22Z

What did you implement:

Closes #1267

The default settings should be applied to vuls server scans as well.

I applied the filters for ignoreCves and PkgsRegexps to the result from server mode.

Type of change

New feature (non-breaking change which adds functionality)

How Has This Been Tested?

How to reproduce this?

To get the same kind of results, simply run the vuls server command with the same config on listed below on run the same POST command.

Tested with the listed config below

[default]
ignorePkgsRegexp = [
  "^openssh"
]

ignoreCves = [
  "CVE-2004-0230"
]

[servers.via-server]
host = "localhost"
port = "local"

Commands that were used to test

dpkg-query -W -f='${binary:Package},${db:Status-Abbrev},${Version},${Source},${source:Version}\n' |
curl -sS -o - -X POST \
  -H "content-type: text/plain" -H 'X-Vuls-OS-Family: debian' \
  -H "X-Vuls-Server-Name: via-server" \
  -H "X-Vuls-OS-Release: $(cat /etc/debian_version)" \
  -H "X-Vuls-Kernel-Release: $(uname -r)" \
  -H "X-Vuls-Kernel-Version: $(uname -a | awk '{print $7}')" \
  http://127.0.0.1:5515/vuls \
  --data-binary @- | jq . > via-server-without-openssh.json

Then, Checked the result by myself.

Results for `ignoreCves`

before

❯ grep CVE-2004-0230 via-server.json
      "CVE-2004-0230": {
        "cveID": "CVE-2004-0230",
            "cveID": "CVE-2004-0230",
            "sourceLink": "https://security-tracker.debian.org/tracker/CVE-2004-0230",
            "cveID": "CVE-2004-0230",
                "link": "http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2004-0230",
                "link": "http://nvd.nist.gov/nvd.cfm?cvename=CVE-2004-0230",
            "cveID": "CVE-2004-0230",
            "sourceLink": "https://nvd.nist.gov/vuln/detail/CVE-2004-0230",

after

❯ grep CVE-2004-0230 via-server.json
empty

Results for `PkgsRegexps`

CVE-2019-16905 is one of the cves that includes openssh

before

❯ grep CVE-2019-16905 via-server-without-opnessh.json
      "CVE-2019-16905": {
        "cveID": "CVE-2019-16905",
            "cveID": "CVE-2019-16905",
            "sourceLink": "https://security-tracker.debian.org/tracker/CVE-2019-16905",
            "cveID": "CVE-2019-16905",
                "link": "https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-16905",
                "link": "https://nvd.nist.gov/vuln/detail/CVE-2019-16905",
            "cveID": "CVE-2019-16905",
            "sourceLink": "https://nvd.nist.gov/vuln/detail/CVE-2019-16905",

after

❯ grep CVE-2019-16905 via-server-without-opnessh.json
empty

Checklist:

Write tests
Write documentation
Check that there aren't other open pull requests for the same issue/feature
Format your source code by make fmt
Pass the test by make test
Provide verification config / commands
Enable "Allow edits from maintainers" for this PR
Update the messages below

Is this ready for review?: YES

References

docs(server): Add examples for ignorePkgsRegexp vulsdoc/vuls#161

MaineK00n

This is the template for the config.toml generated by the discover command. There is no example of ignorePkgsRegexp here. I would like to have it added.

vuls/subcmds/discover.go

Lines 80 to 238 in f0b3a8b

	# https://vuls.io/docs/en/config.toml.html#database-section
	[cveDict]
	#type = ["sqlite3", "mysql", "postgres", "redis", "http" ]
	#sqlite3Path = "/path/to/cve.sqlite3"
	#url = ""

	[ovalDict]
	#type = ["sqlite3", "mysql", "postgres", "redis", "http" ]
	#sqlite3Path = "/path/to/oval.sqlite3"
	#url = ""

	[gost]
	#type = ["sqlite3", "mysql", "postgres", "redis", "http" ]
	#sqlite3Path = "/path/to/gost.sqlite3"
	#url = ""

	[exploit]
	#type = ["sqlite3", "mysql", "postgres", "redis", "http" ]
	#sqlite3Path = "/path/to/go-exploitdb.sqlite3"
	#url = ""

	[metasploit]
	#type = ["sqlite3", "mysql", "postgres", "redis", "http" ]
	#sqlite3Path = "/path/to/go-msfdb.sqlite3"
	#url = ""

	# https://vuls.io/docs/en/config.toml.html#slack-section
	#[slack]
	#hookURL = "https://hooks.slack.com/services/abc123/defghijklmnopqrstuvwxyz"
	##legacyToken = "xoxp-11111111111-222222222222-3333333333"
	#channel = "#channel-name"
	##channel = "${servername}"
	#iconEmoji = ":ghost:"
	#authUser = "username"
	#notifyUsers = ["@username"]

	# https://vuls.io/docs/en/config.toml.html#email-section
	#[email]
	#smtpAddr = "smtp.example.com"
	#smtpPort = "587"
	#user = "username"
	#password = "password"
	#from = "[email protected]"
	#to = ["[email protected]"]
	#cc = ["[email protected]"]
	#subjectPrefix = "[vuls]"

	# https://vuls.io/docs/en/config.toml.html#http-section
	#[http]
	#url = "http://localhost:11234"

	# https://vuls.io/docs/en/config.toml.html#syslog-section
	#[syslog]
	#protocol = "tcp"
	#host = "localhost"
	#port = "514"
	#tag = "vuls"
	#facility = "local0"
	#severity = "alert"
	#verbose = false

	# https://vuls.io/docs/en/usage-report.html#example-put-results-in-s3-bucket
	#[aws]
	#profile = "default"
	#region = "ap-northeast-1"
	#s3Bucket = "vuls"
	#s3ResultsDir = "/path/to/result"
	#s3ServerSideEncryption = "AES256"

	# https://vuls.io/docs/en/usage-report.html#example-put-results-in-azure-blob-storage<Paste>
	#[azure]
	#accountName = "default"
	#accountKey = "xxxxxxxxxxxxxx"
	#containerName = "vuls"

	# https://vuls.io/docs/en/config.toml.html#chatwork-section
	#[chatwork]
	#room = "xxxxxxxxxxx"
	#apiToken = "xxxxxxxxxxxxxxxxxx"

	# https://vuls.io/docs/en/config.toml.html#googlechat-section
	#[googlechat]
	#webHookURL = "https://chat.googleapis.com/v1/spaces/xxxxxxxxxx/messages?key=yyyyyyyyyy&token=zzzzzzzzzz%3D"
	#skipIfNoCve = false
	#serverNameRegexp = "^(\\[Reboot Required\\] )?((spam\|ham).\|.(egg)$)" # include spamonigiri, hamburger, boiledegg
	#serverNameRegexp = "^(\\[Reboot Required\\] )?(?:(spam\|ham).\|.(?:egg)$)" # exclude spamonigiri, hamburger, boiledegg

	# https://vuls.io/docs/en/config.toml.html#telegram-section
	#[telegram]
	#chatID = "xxxxxxxxxxx"
	#token = "xxxxxxxxxxxxxxxxxx"

	#[wpscan]
	#token = "xxxxxxxxxxx"
	#detectInactive = false

	# https://vuls.io/docs/en/config.toml.html#default-section
	[default]
	#port = "22"
	#user = "username"
	#keyPath = "/home/username/.ssh/id_rsa"
	#scanMode = ["fast", "fast-root", "deep", "offline"]
	#scanModules = ["ospkg", "wordpress", "lockfile", "port"]
	#cpeNames = [
	# "cpe:/a:rubyonrails:ruby_on_rails:4.2.1",
	#]
	#owaspDCXMLPath = "/tmp/dependency-check-report.xml"
	#ignoreCves = ["CVE-2014-6271"]
	#containersOnly = false
	#containerType = "docker" #or "lxd" or "lxc" default: docker
	#containersIncluded = ["${running}"]
	#containersExcluded = ["container_name_a"]

	# https://vuls.io/docs/en/config.toml.html#servers-section
	[servers]
	{{- $names:= .Names}}
	{{range $i, $ip := .IPs}}
	[servers.{{index $names $i}}]
	host = "{{$ip}}"
	#port = "22"
	#user = "root"
	#sshConfigPath = "/home/username/.ssh/config"
	#keyPath = "/home/username/.ssh/id_rsa"
	#scanMode = ["fast", "fast-root", "deep", "offline"]
	#scanModules = ["ospkg", "wordpress", "lockfile", "port"]
	#type = "pseudo"
	#memo = "DB Server"
	#cpeNames = [ "cpe:/a:rubyonrails:ruby_on_rails:4.2.1" ]
	#owaspDCXMLPath = "/path/to/dependency-check-report.xml"
	#ignoreCves = ["CVE-2014-0160"]
	#containersOnly = false
	#containerType = "docker" #or "lxd" or "lxc" default: docker
	#containersIncluded = ["${running}"]
	#containersExcluded = ["container_name_a"]

	#[servers.{{index $names $i}}.containers.container_name_a]
	#cpeNames = [ "cpe:/a:rubyonrails:ruby_on_rails:4.2.1" ]
	#owaspDCXMLPath = "/path/to/dependency-check-report.xml"
	#ignoreCves = ["CVE-2014-0160"]

	#[servers.{{index $names $i}}.githubs."owner/repo"]
	#token = "yourToken"
	#ignoreGitHubDismissed = false

	#[servers.{{index $names $i}}.wordpress]
	#cmdPath = "/usr/local/bin/wp"
	#osUser = "wordpress"
	#docRoot = "/path/to/DocumentRoot/"

	#[servers.{{index $names $i}}.portscan]
	#scannerBinPath = "/usr/bin/nmap"
	#hasPrivileged = true
	#scanTechniques = ["sS"]
	#sourcePort = "65535"

	#[servers.{{index $names $i}}.optional]
	#key = "value1"

	{{end}}

There is little description of ignorePkgsRegexp even in vulsdoc/vuls. In particular, there is no explanation in the config.toml section. I would like to see the explanation added here as well.

Here are the links to the parts that I need clarification on. However, I suspect that these are not all the parts that need to be explained, so please add explanations where you think they are needed.
https://github.com/vulsdoc/vuls/blob/47f6cedb39ef59f823e06ecd5cd1b98d8d91272f/docs/configtoml.md
https://github.com/vulsdoc/vuls/blob/47f6cedb39ef59f823e06ecd5cd1b98d8d91272f/docs/usage-automatic-discovery.md

MaineK00n

If I look at the options for server mode, I see -ignore-unscored-cves and -ignore-unfixed. Can't you also support this?

$ vuls server -h
...
  -ignore-unfixed
    	Don't show the unfixed CVEs
  -ignore-unscored-cves
    	Don't Server the unscored CVEs
...

The report seems to be handled as follows.

vuls/detector/detector.go

Line 136 in f0b3a8b

r.ScannedCves = r.ScannedCves.FilterUnfixed(config.Conf.IgnoreUnfixed)

vuls/detector/detector.go

Lines 156 to 159 in f0b3a8b

    
           // IgnoreUnscored 
        
           if config.Conf.IgnoreUnscoredCves { 
        
           	r.ScannedCves = r.ScannedCves.FindScoredVulns() 
        
           }

qwexvf · 2021-07-21T01:05:46Z

Can't you also support this?

what exactly do you want me to implement here?

MaineK00n · 2021-07-23T10:45:02Z

Can't you also support this?

what exactly do you want me to implement here?

I am sorry that my English is not good enough to convey my intentions.
The following is an additional task.
If you don't want to be included in this PR, or don't feel the need to do it, please express your intention and I will do it for you.
Please don't hesitate to tell me.

There are three tasks that I would like you to do in this PR.

The first is to add an example of ignorePkgsRegexp to the discover command.
The second is to add a description of ignorePkgsRegexp in the section describing config.toml of vulsdoc/vuls.
Third, -ignore-unscored-cves and -ignore-unfixed are available as options of the server command, but they don't work because they are not implemented in the server command. This may be the same as ignorePkgsRegexp.

qwexvf · 2021-07-29T07:53:03Z

Thanks for the reply! <3

I am sorry that my English is not good enough to convey my intentions.

Your English is totally fine!! i just wanted more context which you added for me! thanks!

If you don't want to be included in this PR, or don't feel the need to do it, please express your intention and I will do it for you.

I would love to work on this so ill try to fix the -ignore-unscored-cves and -ignore-unfixed options to work.
Thanksss!!

MaineK00n

First, thanks for the PR to vulsdoc/vuls.

I saw commit:93729c6e73fef787c2d349e1ee1a404a32824d6c.
Thanks for the implementation of -ignore-unscored-cves and -ignore-unfixed.
I commented on the implementation questions.
Also, please write how you verified this feature and the results in the PR section How Has This Been Tested?.

The first is to add an example of ignorePkgsRegexp to the discover command.

Please add #ignorePkgsRegexp = ["^kernel", "^python"] below the link line.

vuls/subcmds/discover.go

Line 187 in 5f4d68c

#ignoreCves = ["CVE-2014-6271"]
vuls/subcmds/discover.go

Line 209 in 5f4d68c

#ignoreCves = ["CVE-2014-0160"]
vuls/subcmds/discover.go

Line 218 in 5f4d68c

#ignoreCves = ["CVE-2014-0160"]

MaineK00n · 2021-07-31T09:56:42Z

subcmds/server.go

-	f.BoolVar(&config.Conf.IgnoreUnscoredCves, "ignore-unscored-cves", false,
-		"Don't Server the unscored CVEs")
-
-	f.BoolVar(&config.Conf.IgnoreUnfixed, "ignore-unfixed", false,
+	f.BoolVar(&p.ignoreUnfixed, "ignore-unfixed", false,
 		"Don't show the unfixed CVEs")

+	f.BoolVar(&p.ignoreUnscoredCves, "ignore-unscored-cves", false,
+		"Don't show the unscored CVEs")
+


I don't understand why you add it to serverCmd to manage option states instead of config.Conf.IgnoreUnfixed and config.Conf.IgnoreUnscoredCves.
https://github.com/qwexvf/vuls/blob/93729c6e73fef787c2d349e1ee1a404a32824d6c/config/config.go#L77-L78

MaineK00n · 2021-07-31T10:00:01Z

server/server.go

+	r.ScannedCves = r.ScannedCves.FilterIgnorePkgs(ignorePkgsRegexps)
+
+	// IgnoreUnfixed
+	r.ScannedCves = r.ScannedCves.FilterUnfixed(h.IgnoreUnfixed)


Isn't it possible to use config.Conf.IgnoreUnfixed?

MaineK00n · 2021-07-31T10:00:11Z

server/server.go

+	r.ScannedCves = r.ScannedCves.FilterUnfixed(h.IgnoreUnfixed)
+
+	// IgnoreUnscoredCves
+	if h.IgnoreUnscoredCves {


Isn't it possible to use config.Conf.IgnoreUnscoredCves?

qwexvf (Marcelo) added 2 commits July 8, 2021 05:35

filter IgnoreCves and PkgsRegexps from result

9616571

remove empty line

574906b

qwexvf mentioned this pull request Jul 8, 2021

ignore CVEs by pattern-matching in Summary field #1269

Open

kotakanbe requested review from kotakanbe and MaineK00n July 8, 2021 07:20

MaineK00n reviewed Jul 8, 2021

View reviewed changes

kotakanbe assigned qwexvf Jul 13, 2021

qwexvf mentioned this pull request Jul 30, 2021

docs(server): Add examples for ignorePkgsRegexp vulsdoc/vuls#161

Merged

implement missing features

93729c6

qwexvf requested a review from MaineK00n July 30, 2021 05:32

MaineK00n reviewed Jul 31, 2021

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(server): Apply filters for IgnoreCves and PkgsRegexps on server mode #1270

feat(server): Apply filters for IgnoreCves and PkgsRegexps on server mode #1270

qwexvf commented Jul 8, 2021 •

edited by MaineK00n

Loading

MaineK00n left a comment

MaineK00n left a comment

qwexvf commented Jul 21, 2021

MaineK00n commented Jul 23, 2021

qwexvf commented Jul 29, 2021

MaineK00n left a comment

MaineK00n Jul 31, 2021

MaineK00n Jul 31, 2021

MaineK00n Jul 31, 2021

	// IgnoreUnscored
	if config.Conf.IgnoreUnscoredCves {
	r.ScannedCves = r.ScannedCves.FindScoredVulns()
	}

feat(server): Apply filters for IgnoreCves and PkgsRegexps on server mode #1270

Are you sure you want to change the base?

feat(server): Apply filters for IgnoreCves and PkgsRegexps on server mode #1270

Conversation

qwexvf commented Jul 8, 2021 • edited by MaineK00n Loading

What did you implement:

Type of change

How Has This Been Tested?

How to reproduce this?

Results for ignoreCves

Results for PkgsRegexps

Checklist:

References

MaineK00n left a comment

Choose a reason for hiding this comment

MaineK00n left a comment

Choose a reason for hiding this comment

qwexvf commented Jul 21, 2021

MaineK00n commented Jul 23, 2021

qwexvf commented Jul 29, 2021

MaineK00n left a comment

Choose a reason for hiding this comment

MaineK00n Jul 31, 2021

Choose a reason for hiding this comment

MaineK00n Jul 31, 2021

Choose a reason for hiding this comment

MaineK00n Jul 31, 2021

Choose a reason for hiding this comment

qwexvf commented Jul 8, 2021 •

edited by MaineK00n

Loading

Results for `ignoreCves`

Results for `PkgsRegexps`