Merge pull request #9 from org-metaeffekt/AEAA-343-vad-detail-levels

AEAA-343: Added documentation for detail levels, new MSRC mirror and version comparator

doc/inventory-enrichment/enrichment/vad-detail-levels.md

@@ -0,0 +1,113 @@
+> [Vulnerability Monitoring](../inventory-enrichment-overview.md) > VAD Detail Levels
+
+# Vulnerability Assessment Dashboard Detail Levels
+
+In some cases, displaying all the information available in a dashboard can lead to the dashboard growing rapidly in
+size, depending on the amount of vulnerabilities the dashboard is generated for.
+
+To at least partially mitigate this, the dashboard can be configured to only display certain information, depending on
+several factors.
+
+## Matchers
+
+Matchers are used to determine whether a certain detail level should be used for a vulnerability. They have the
+following properties:
+
+- `status`: a comma-separated list of vulnerability statuses that the matcher applies to. Matches if the vulnerability
+  has one of the specified statuses.
+- `allCpe`: a comma-separated list of CPEs that the matcher applies to. Matches if the vulnerability has all of the
+  specified CPEs.
+- `anyCpe`: a comma-separated list of CPEs that the matcher applies to. Matches if the vulnerability has at least one of
+  the specified CPEs.
+- `vulnerabilityName`: a comma-separated list of vulnerability names that the matcher applies to. Matches if the
+  vulnerability has at least one of the specified names.
+
+The matcher will only match if all of the specified properties match.
+
+## Detail Levels
+
+The following detail levels are available:
+
+- `timeline`: whether the timeline should be displayed. This attribute is the main reasons the detail levels were
+  introduced, as the timeline can not only take very long to generate, but also take up a lot of space in the dashboard.
+- `references`: whether the vulnerability references should be displayed.
+- `advisoriesGlobal`: whether advisory information should be displayed at all.
+- `advisoriesReferences`: whether references in the advisory information should be displayed.
+- `advisoryByTypes`: what type of advisories should be shown, whilst all others are hidden.
+    - `any` (default): any advisory type.
+    - comma-separated list of advisory types (notice, ...)
+- `advisoryByProviders`: what provider of advisories should be shown, whilst all others are hidden.
+    - `any` (default): any advisory provider.
+    - comma-separated list of advisory providers (CERT-FR, GHSA, ...)
+- `eolDate`: whether the EOL date information should be displayed.
+
+## Default Detail Level
+
+By default, all properties are set to the most detailed level, to ensure that all information is displayed.
+
+## Creating Detail Levels
+
+### Correlation YAML
+
+```yaml
+- affects:
+    Id: linux-kernel
+  append:
+    VAD Detail Level Configurations: |-
+      matcher:
+        status = "in review, insignificant";
+        allCpe = "cpe:/a:linux:linux_kernel, cpe:/o:linux:linux_kernel";
+        anyCpe = "cpe:/a:linux:linux_kernel, cpe:/a:linux_test:linux_kernel";
+        vulnerabilityName = "CVE-2023-35829, CVE-2023-35828, CVE-1999-0431";
+      detail:
+        timeline = "false";
+        advisoriesGlobal = "false";
+      matcher: status = "in review, insignificant"
+      detail: timeline = "false"
+```
+
+The detail level information is appended to artifacts in the `VAD Detail Level Configurations` field. This field
+contains a custom format:
+
+- `matcher`: the matcher information, as described above.
+- `detail`: the detail level information, as described above.
+
+Each of these two sections contains information in the following format: `key = "value";` (a key-value pair), where the
+key is the name of the property, and the value is the value of the property. The value is a string, and must be enclosed
+in double quotes.
+
+The newlines are optional, but make the configuration easier to read. The last semicolon is optional as well.
+
+If multiple detail levels are specified, the individual detail levels must be separated by a newline.
+
+### VAD Configuration
+
+```xml
+
+<detailLevels>
+    <map>
+        <matcher>
+            <status>in review, insignificant</status>
+            <allCpe>cpe:/a:linux:linux_kernel, cpe:/o:linux:linux_kernel</allCpe>
+            <anyCpe>cpe:/a:linux:linux_kernel, cpe:/a:linux_test:linux_kernel</anyCpe>
+            <vulnerabilityName>CVE-2023-35829, CVE-2023-35828, CVE-1999-0431
+            </vulnerabilityName>
+        </matcher>
+        <timeline>false</timeline>
+        <references>true</references>
+        <advisoriesGlobal>true</advisoriesGlobal>
+        <advisoriesReferences>true</advisoriesReferences>
+        <advisoryByTypes>
+            <entry>any</entry>
+        </advisoryByTypes>
+        <advisoryByProviders>
+            <entry>any</entry>
+        </advisoryByProviders>
+        <eolDate>true</eolDate>
+    </map>
+</detailLevels>
+```
+
+The Vulnerability Assessment Dashboard configuration contains a `detailLevels` section, which is a map of detail levels.
+Just like the correlation YAML, each detail level contains a `matcher` section, but the `detail` section is replaced by
+the individual properties.

doc/inventory-enrichment/inventory-enrichment-overview.md

@@ -36,6 +36,7 @@ View the overview page of the [**Inventory Enrichment**](enrichment/inventory-en
 - [Vulnerability Status files](enrichment/vulnerability-status.md)
 - [Vulnerability Keywords files](enrichment/vulnerability-keywords.md)
 - [Vulnerability Filter format](enrichment/vulnerability-filter-format.md)
+- [Dashboard Detail Levels format](enrichment/vad-detail-levels.md)
 - [Custom Vulnerability Data Sources](enrichment/custom-vulnerabilities.md)
 
 ### Data Sources Explained
@@ -50,6 +51,7 @@ View the overview page of the [**Inventory Enrichment**](enrichment/inventory-en
 - [Architecture: Superclasses](enrichment/java-super-classes.md) (useful background information for understanding
   the process)
 - [Inventory CPE data and effective CPE](enrichment/parsing-effective-cpe.md)
+- [Version comparator](other/version-comparator.md)
 
 ## Overview graphs
 

doc/inventory-enrichment/msrc/understanding-data.md

@@ -75,9 +75,11 @@ Table of contents:
 ### Sources
 
 - [Security Update Guide - Microsoft Security Response Center](https://msrc.microsoft.com/update-guide)
-    - can **manually** be downloaded as multiple csv files, one per year. Time frame has to be set manually.
-    - contains `CVE/ADV ↔︎ KB` relations that are not present in the other sources.
-    - does not contain `KB ↔︎ KB` relations.
+    - ~~can **manually** be downloaded as multiple csv files, one per year. Time frame has to be set manually,~~
+    - can **automatically** be mirrored using the underlying API used by the CSV generator on the website and using our
+      process (see below),
+    - contains `CVE/ADV ↔︎ KB` relations that are not present in the other sources,
+    - does not contain `KB ↔︎ KB` relations,
     - A short how-to can be found [here](performing-csv-download.md).
 - [Microsoft Security Updates API | MSRC](https://api.msrc.microsoft.com/cvrf/v2.0/swagger/index)
     - can **automatically** be mirrored using our process (see below),
@@ -91,8 +93,7 @@ Table of contents:
 
 ### Problems with the data sources
 
-- Only one of the three data sources can be retrieved automatically. The one available for mirroring is the most
-  important one, yet it still misses a lot of information.
+- One of the three data sources cannot be mirrored automatically or manually, meaning some data is missing.
 - The data in between the different sources is inconsistent. Every data source either contains `CVE/ADV → KB`
   or `KB → KB`relations that are missing in the other sources. The Update Catalog knows about this information, but is
   inconsistent in itself.  
@@ -105,10 +106,7 @@ Table of contents:
     - **In the API**: the KB replaces (`4487259` and `4487081`) and is replaced by `4507423`
       ![Example](kb-chain-example-1.png)
 - This means, our data will never be complete unless considering all three data sources, which is impossible to
-  automate. it is reasonable to manually download the csv files from the MSRC, but scraping the Update Catalog is not
-  feasible, manually or automated.
-- The MSRC csv has to be downloaded in time segments of one year, which have to be set manually, which takes quite some
-  time
+  automate. Scraping the Update Catalog is not feasible, manually or automated.
 
 ## Data mirror
 
@@ -260,6 +258,49 @@ Table of contents:
 }
 ```
 
+... an entry from the underlying MSRC Security Update Guide CSV API
+
+```json
+{
+  "productFamily": "Windows",
+  "productFamilyId": 100000010,
+  "severity": "Important",
+  "temporalScore": "6.5",
+  "product": "Windows 10 Version 1809 for 32-bit Systems",
+  "productId": 11568,
+  "releaseDate": "2023-01-10T08:00:00Z",
+  "impactId": 100000001,
+  "impact": "Denial of Service",
+  "issuingCna": "Microsoft",
+  "platformId": 0,
+  "baseScore": "7.5",
+  "kbArticles": [
+    {
+      "rebootRequired": "Yes",
+      "articleName": "5022286",
+      "knownIssuesName": "5022286",
+      "affectedBinaries": [],
+      "knownIssuesUrl": "https://support.microsoft.com/help/5022286",
+      "downloadUrl": "https://catalog.update.microsoft.com/v7/site/Search.aspx?q=KB5022286",
+      "downloadName": "Security Update",
+      "articleUrl": "https://support.microsoft.com/help/5022286",
+      "fixedBuildNumber": "10.0.17763.3887",
+      "supercedence": "5021237",
+      "ordinal": 0
+    }
+  ],
+  "initialReleaseDate": "2023-01-10T08:00:00Z",
+  "cveNumber": "CVE-2023-21527",
+  "isMariner": false,
+  "productVersion": "10.0.0",
+  "architectureId": 0,
+  "id": "00000000-0000-0000-302d-00006e4d7c04",
+  "releaseNumber": "2023-Jan",
+  "severityId": 100000001,
+  "vectorString": "CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H/E:U/RL:O/RC:C"
+}
+```
+
 </details>
 
 ### Data inconsistencies