Cannot complete initial NVD download - 503 status #6107
Comments
|
Same for me. |
|
It looks like perhaps there is a retry mechanism, but for some reason the connection pool has been closed and it doesn't do anything about it. That repeats for a bit, with the ex-00000... number continuing to increment. |
|
Can you try increasing the delay? For the CLI it would be |
|
By default it is attempting to use |
|
I’m using an API key and still getting a 503 after a while what should I try for the delay value?On Nov 22, 2023, at 19:07, Jeremy Long ***@***.***> wrote:
By default it is attempting to use 8000 without an API key.
—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>
|
|
I am using the gradle plugin with the following: dependencyCheck {
nvd {
apiKey = '<my-api-key>'
delay = 16000
}
}Still result in 503: |
|
yup... apparently the NVD API is under load... maybe they'll delay the retirement of the data feeds... |
|
same is happening for us. |
|
Yeah even with DT and API key I'm getting rate limited to no end.
If that is the outcome then it'd be a win for everyone 😆 DC adopting the NVD REST API must be one of the most brutal load tests you can get! 9.0 was only released today so I'm assuming the situation will not get better as folks proceed with upgrading. |
Not sure if it's relevant. I've tried a 300000ms delay with vulnz (with or without API key), but still getting 503 after a couple requests. |
|
Edit: seems this was an invalid API key somehow, see #6107 (comment) |
|
Yes, I also experiencing 503 issue with or without the NVD API Key. |
|
I have the same problem with an api key |
|
Same situation here with an API Key 😞 |
|
Same here - despite 16k delay. |
|
Same here...404 One or more exceptions occurred during analysis: I have the nvdApiKey set and 16000. |
|
Looks like the Regenerated my key and am now onto the |
|
@chadlwilson Yep. Ran the test curl -H "Accept: application/json" -H "apiKey: ########-####-####-####-############" -v https://services.nvd.nist.gov/rest/json/cves/2.0\?cpeName\=cpe:2.3:o:microsoft:windows_10:1607:\*:\*:\*:\*:\*:\*:\* The key was bad. Requested and new one, and the second one they sent me works. Looks like they have some issues to workout with this new API. |
|
@chadlwilson How are you supposed to list the API key in nvdApiKey? apiKey: ########-####-####-####-############ If I list it the first way I get a 503, if I list it the second way I get a 404, but the test using curl with the new API key it works fine. |
Just the key on its own. The header is populated by the library: https://github.com/jeremylong/Open-Vulnerability-Project/blob/27c72f84943d28448a867f803c699867fcb864d3/open-vulnerability-clients/src/main/java/io/github/jeremylong/openvulnerability/client/nvd/NvdCveClient.java#L235-L237 |
|
The curl command is working fine, but from the plugin it's getting a 503... One or more exceptions occurred during analysis: |
|
I had to add <nvdApiDelay>16000</nvdApiDelay> back, now it's working. There was about a 30 second delay before it finally started to download. |
|
@chadlwilson Have you tested with maven parallel builds before? We build with -T1C and it's looking like it might be attempting to use multiple threads to connect to the API. I'm running with mvn -X now and I can see it downloading, but after about a minute of downloading it got... One or more exceptions occurred during analysis: I'm going to try again and double the nvdApiDelay |
|
Looks like they have some issues to work on. I bumped up <nvdApiDelay>32000</nvdApiDelay>. It got further this time, but eventually failed with a 503. Here are the debug logs showing the failure. |
|
🤞 that results in them continuing the significantly more efficient data downloads |
|
They also seems to have closed th API Key deliverance. |
|
@jeremylong Thx for your work! Do you have any new information about the usability of the API? |
|
@Maxouwell I have just generated an API key, it seems they have enabled that back. Just FYI |
|
If you refresh on the page https://nvd.nist.gov/developers/confirm-api-key?uuid= by mistake it regenerates the key and the old one will no longer work resulting in 404. |
|
Isn't working for me since two days. I got new API key and I get |
|
Can you elaborate as to why this was closed? I see no solution and am experiencing the same issue as mentioned in the ticket. |
|
@carlmolemans have you done it with 9.0.2? For me it's working #6186 |
|
[Update] It's working now for me with |
|
It's failing for me again with 9.0.7 in Azure Devops |
We solved it by creating a "common" pipeline for the dependency checks. This runs each night and our builds verify themselves against this cache. |
Same for me from a Jenkins instance running on AWS. On my machine works fine though... |
|
I obtained the api key and provided an api key in my pom but still getting the below error 15:44:42 [INFO] Building **** - Parent POMs 3-0-SNAPSHOT [1/21] I had earlier tried with version 9.0.7 and was getting this error 14:33:37 [INFO] --- dependency-check-maven:9.0.7:purge (default-cli) @ parent-poms --- I read a comment on another this thread that said 9.0.2 worked but it doesn't seem to be working for me. Is the issue fixed please @jeremylong ? |
|
@somera @nico-arianto How long did it take before 9.0.2 worked for you. Did you provide an api key? Please advise. Thanks |
|
@ifyenuoyibo Version 9.0.7 has been released since then, I'd suggest you give that a try: https://github.com/jeremylong/DependencyCheck/blob/main/CHANGELOG.md |
|
The logs state an API key was NOT provided. |
|
@ifyenuoyibo for me it's working since 9.0.2 and I have an NVD API Key. Current run with 9.0.7: |
|
@ifyenuoyibo I'm using the NVD key and it took me ~30 mins with |
|
I needed ~45 minutes with 9.0.2 for the first download. |
I cant analyze a project after update to 9.0.10. I'm upgrading from 7.4 according to deprecation informed. After a lot of times trying to analyze, it looks like after 95% always get this error. Other times, before 95%, when Ticket taken At: xxxx reach count: 50. Actually I have this properties on pom.xml |
I am seeing the same, as this was working fine yesterday with |
From https://nvd.nist.gov/:
|
It's possible that error is cause by an external server, but, Isn't possible to reuse already downloaded segments? Now i'm running locally, 4 hours at least with a lot of retries, and without download db for first time. |
|
It would be nice to have some kind of configuration option like: |
|
In my case even failOnError=false does not help, as the build task checkAggregate still fails. |
same for me! |
|
Locked conversation on this one as 'me too' comments are unneeded For the people suffering from a 'me too' that came here to weigh in: double-check that you do cache your OWASP DependencyCheck data folder, then you should be able to run (with current cached data) using the "don't update" flag of your tool. If you previously did not cache the intermediate results then make sure to start doing so ASAP in order to not contribute to the DDoS on the NVD API in the future. Preferably set up a dedicated periodic job (e.g. every 4 hours) to just update your cached copy in your infrastructure and configure your builds to run without updating so that you are less impacted by infrastructure overload at the side of the NIST NVD (as a bonus you also are causing less impact on NIST NVD infrastructure by your builds reducing the risk for everyone to run into a 503). |
and others
Describe the bug
The default configuration, with an API key, is either making requests too quickly, or not retrying enough, or both. It always eventually fails with a 503 error from NVD.
Version of dependency-check used
Maven plugin 9.0.0
Log file
The debug log in this case is a nightmare as it logs every raw request and response, 16 bytes at a time, without synchronisation so they're all misordered. I'm not going through that to sanitise my API key.
To Reproduce
mvn dependency-check:checkExpected behavior
Successful completion of the download.
The text was updated successfully, but these errors were encountered: