Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Set update interval (1-120 days) manually for NVD API #300

Merged
merged 2 commits into from
Jul 3, 2024

Conversation

oh2fih
Copy link
Collaborator

@oh2fih oh2fih commented Jul 2, 2024

Closes #299

This takes 1-120 days before current date as an integer since it is easier to use than correctly formatted exact datetime.

The NVD API supports downloading data for maximum 120 days. If the period missing CPEs or CVEs is longer than that, full re-population is required.

 - CveXplore.core.nvd_nist.nvd_nist_api - WARNING  - Requested timeframe exceeds the 120 days limit; capping the requested date range to 120 days. End date is now: 2020-08-29T15:40:27.088062
 - CveXplore.core.nvd_nist.nvd_nist_api - INFO     - Getting count for datasource: 2
 - CveXplore.core.nvd_nist.nvd_nist_api - INFO     - urllib3.util.retry did not support configurable backoff_max; falling back to default for urllib3 1.x compatibility
 - CveXplore.core.nvd_nist.nvd_nist_api - ERROR    - Unexpected response type: <Response [404]>
 - CveXplore.core.nvd_nist.nvd_nist_api - ERROR    - Data retrieval error: {'resultsPerPage': 1, 'lastModStartDate': '2020-04-30T15:45:01', 'lastModEndDate': '2020-08-29T15:40:27.088062'}

Usage

>>> from CveXplore import CveXplore
>>> cvx = CveXplore()
>>> cvx.database.update(manual_days=7)
 - CveXplore.core.database_maintenance.main_updater - INFO     - Starting Database update...
 - CveXplore.core.nvd_nist.nvd_nist_api - WARNING  - Could not find a NIST API Key in the environment variable 'NVD_NIST_API_KEY' (e.g. from the '~/.cvexplore/.env' file); you could request one at: https://nvd.nist.gov/developers/request-an-api-key
 - CveXplore.core.database_maintenance.sources_process - INFO     - CPE database update started
 - CveXplore.core.database_maintenance.sources_process - INFO     - Starting download...
 - CveXplore.core.database_maintenance.sources_process - INFO     - Retrieving CPEs starting from 2024-06-25 18:59:04.364340
 - CveXplore.core.nvd_nist.nvd_nist_api - INFO     - Getting count for datasource: 2
 - CveXplore.core.nvd_nist.nvd_nist_api - INFO     - urllib3.util.retry did not support configurable backoff_max; falling back to default for urllib3 1.x compatibility
 - CveXplore.core.database_maintenance.sources_process - INFO     - Preparing to download 2031 CPE entries

Tested with variants:

  • cvx.database.update("cpe", 1)
  • cvx.database.update("cve", 1)
  • cvx.database.update(None, 1)
  • cvx.database.update(manual_days=7)
  • cvx.database.update(["cve", "via4"], 1)

The last one correctly gives:

 - CveXplore.core.database_maintenance.sources_process - INFO     - Retrieving CVEs starting from 2024-07-01 21:41:33.724703

 - CveXplore.core.database_maintenance.main_updater - WARNING  - Update interval in days not supported by source via4; ignoring

@oh2fih oh2fih changed the title Configurable update interval (1-120 days) for CPE & CVE Set update interval (1-120 days) manually for CPE & CVE Jul 2, 2024
@oh2fih oh2fih changed the title Set update interval (1-120 days) manually for CPE & CVE Set update interval (1-120 days) manually for NVD API Jul 2, 2024
@oh2fih oh2fih force-pushed the manual-lastmodified branch 5 times, most recently from 5244c98 to 8d67981 Compare July 2, 2024 21:28
@P-T-I P-T-I merged commit c85a7fa into cve-search:master Jul 3, 2024
3 checks passed
@oh2fih oh2fih deleted the manual-lastmodified branch July 3, 2024 09:04
oh2fih added a commit to oh2fih/cve-search that referenced this pull request Jul 3, 2024
Adds -d (--days) options for passing manual interval to CveXplore.

Requires CveXplore with cve-search/CveXplore#300.
oh2fih added a commit to oh2fih/cve-search that referenced this pull request Jul 3, 2024
Adds -d (--days) options for passing manual interval to CveXplore.

Requires CveXplore with cve-search/CveXplore#300.
oh2fih added a commit to oh2fih/cve-search that referenced this pull request Jul 3, 2024
Adds -d (--days) options for passing manual interval to CveXplore.

Requires CveXplore with cve-search/CveXplore#300.
oh2fih added a commit to oh2fih/cve-search that referenced this pull request Jul 3, 2024
Adds -d (--days) options for passing manual interval to CveXplore.

Requires CveXplore with cve-search/CveXplore#300.
P-T-I pushed a commit to cve-search/cve-search that referenced this pull request Jul 5, 2024
* Set update interval (1-120 days) manually for NVD API

Adds -d (--days) options for passing manual interval to CveXplore.

Requires CveXplore with cve-search/CveXplore#300.

* Add long option strings & reorder help

Makes the code more readable and the short option strings more
understandable and memorable. Related options grouped together.

* Update documentation for Updating the database

Update the documentation to reflect these and some other recent changes.

* Set sources manually for NVD API

Adds -s (--sources) option for overriding the configured sources.

Requires #1112 for EPSS source as it is not currently configured.

* Warn about --sources not available in CveXplore
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Partial repopulation in case NVD API fails
2 participants