-
-
Notifications
You must be signed in to change notification settings - Fork 350
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Revise nut-scanner
parallelization of scanning
#2511
Labels
Milestone
Comments
jimklimov
added
enhancement
nut-scanner
portability
We want NUT to build and run everywhere possible
labels
Jul 5, 2024
jimklimov
added a commit
that referenced
this issue
Jul 8, 2024
…scans do not parallelize together [#2244] See also: #2511 Signed-off-by: Jim Klimov <[email protected]>
jimklimov
added a commit
to jimklimov/nut
that referenced
this issue
Jul 8, 2024
Signed-off-by: Jim Klimov <[email protected]>
jimklimov
added a commit
to jimklimov/nut
that referenced
this issue
Jul 8, 2024
…status in this build and run [networkupstools#2511] Signed-off-by: Jim Klimov <[email protected]>
jimklimov
added a commit
to jimklimov/nut
that referenced
this issue
Jul 8, 2024
…r of WIN32 active init calls vs. pthread variable declarations [networkupstools#2511] Signed-off-by: Jim Klimov <[email protected]>
jimklimov
added a commit
to jimklimov/nut
that referenced
this issue
Jul 8, 2024
…upstools#2511] Signed-off-by: Jim Klimov <[email protected]>
jimklimov
added a commit
to jimklimov/nut
that referenced
this issue
Jul 8, 2024
…upstools#2511] Signed-off-by: Jim Klimov <[email protected]>
jimklimov
added a commit
to jimklimov/nut
that referenced
this issue
Jul 8, 2024
…upstools#2511] Signed-off-by: Jim Klimov <[email protected]>
jimklimov
added a commit
to jimklimov/nut
that referenced
this issue
Jul 8, 2024
…upstools#2511] Signed-off-by: Jim Klimov <[email protected]>
jimklimov
added a commit
to jimklimov/nut
that referenced
this issue
Jul 8, 2024
…upstools#2511] Signed-off-by: Jim Klimov <[email protected]>
jimklimov
added a commit
to jimklimov/nut
that referenced
this issue
Jul 8, 2024
Signed-off-by: Jim Klimov <[email protected]>
jimklimov
added a commit
to jimklimov/nut
that referenced
this issue
Jul 8, 2024
…status in this build and run [networkupstools#2511] Signed-off-by: Jim Klimov <[email protected]>
jimklimov
added a commit
to jimklimov/nut
that referenced
this issue
Jul 8, 2024
…r of WIN32 active init calls vs. pthread variable declarations [networkupstools#2511] Signed-off-by: Jim Klimov <[email protected]>
jimklimov
added a commit
to jimklimov/nut
that referenced
this issue
Jul 8, 2024
…upstools#2511] Signed-off-by: Jim Klimov <[email protected]>
jimklimov
added a commit
to jimklimov/nut
that referenced
this issue
Jul 8, 2024
… for ip_ranges[] list and helper metadata, and methods as part of libnutscan [networkupstools#2244, networkupstools#2511] Not bumping library version, because it was recently bumped as part of other PRs about this issue. Technically the scope of the library has been changed by new exported methods and header lines. Signed-off-by: Jim Klimov <[email protected]>
jimklimov
added a commit
to jimklimov/nut
that referenced
this issue
Jul 8, 2024
…orkupstools#2244, networkupstools#2511] Signed-off-by: Jim Klimov <[email protected]>
jimklimov
added a commit
to jimklimov/nut
that referenced
this issue
Jul 8, 2024
…nts [networkupstools#2244, networkupstools#2511] Signed-off-by: Jim Klimov <[email protected]>
jimklimov
added a commit
to jimklimov/nut
that referenced
this issue
Jul 8, 2024
… comments [networkupstools#2244, networkupstools#2511] Signed-off-by: Jim Klimov <[email protected]>
jimklimov
added a commit
to jimklimov/nut
that referenced
this issue
Jul 8, 2024
…orkupstools#2244, networkupstools#2511] Signed-off-by: Jim Klimov <[email protected]>
jimklimov
added a commit
to jimklimov/nut
that referenced
this issue
Jul 8, 2024
…nts [networkupstools#2244, networkupstools#2511] Signed-off-by: Jim Klimov <[email protected]>
jimklimov
added a commit
to jimklimov/nut
that referenced
this issue
Jul 8, 2024
… comments [networkupstools#2244, networkupstools#2511] Signed-off-by: Jim Klimov <[email protected]>
jimklimov
added a commit
to jimklimov/nut
that referenced
this issue
Jul 9, 2024
…ction [networkupstools#2511] Signed-off-by: Jim Klimov <[email protected]>
jimklimov
added a commit
to jimklimov/nut
that referenced
this issue
Jul 9, 2024
…t_ip<=irl.stop_ip in alphanumeric comparison order [networkupstools#2511] Signed-off-by: Jim Klimov <[email protected]>
jimklimov
added a commit
to jimklimov/nut
that referenced
this issue
Jul 9, 2024
…#2244, networkupstools#2511] Signed-off-by: Jim Klimov <[email protected]>
jimklimov
added a commit
to jimklimov/nut
that referenced
this issue
Jul 10, 2024
…FD leak [networkupstools#2511] Signed-off-by: Jim Klimov <[email protected]>
jimklimov
added a commit
to jimklimov/nut
that referenced
this issue
Jul 10, 2024
…rain libnetsnmp leak reports [networkupstools#2511] It seems that although `init_snmp()` is called once, every use of the library initializes some data which it then does not release, at least on the test system here. Signed-off-by: Jim Klimov <[email protected]>
jimklimov
added a commit
to jimklimov/nut
that referenced
this issue
Jul 10, 2024
…s/valgrind/.valgrind.supp: constrain warnings about library methods [networkupstools#2511] Signed-off-by: Jim Klimov <[email protected]>
jimklimov
added a commit
to jimklimov/nut
that referenced
this issue
Jul 10, 2024
…rain warnings about library methods [networkupstools#2511] Signed-off-by: Jim Klimov <[email protected]>
jimklimov
added a commit
to jimklimov/nut
that referenced
this issue
Jul 10, 2024
…age [networkupstools#2511] Signed-off-by: Jim Klimov <[email protected]>
jimklimov
added a commit
to jimklimov/nut
that referenced
this issue
Jul 10, 2024
…() the loaded library modules [networkupstools#2511] Avoid some memory leak reports Signed-off-by: Jim Klimov <[email protected]>
jimklimov
added a commit
to jimklimov/nut
that referenced
this issue
Jul 10, 2024
…t_snmp() once per library load [networkupstools#2511] Signed-off-by: Jim Klimov <[email protected]>
jimklimov
added a commit
to jimklimov/nut
that referenced
this issue
Jul 10, 2024
…tions more reliably [networkupstools#2511] Signed-off-by: Jim Klimov <[email protected]>
jimklimov
added a commit
to jimklimov/nut
that referenced
this issue
Jul 10, 2024
…FD leak [networkupstools#2511] Signed-off-by: Jim Klimov <[email protected]>
jimklimov
added a commit
to jimklimov/nut
that referenced
this issue
Jul 10, 2024
…rain libnetsnmp leak reports [networkupstools#2511] It seems that although `init_snmp()` is called once, every use of the library initializes some data which it then does not release, at least on the test system here. Signed-off-by: Jim Klimov <[email protected]>
jimklimov
added a commit
to jimklimov/nut
that referenced
this issue
Jul 10, 2024
…s/valgrind/.valgrind.supp: constrain warnings about library methods [networkupstools#2511] Signed-off-by: Jim Klimov <[email protected]>
jimklimov
added a commit
to jimklimov/nut
that referenced
this issue
Jul 10, 2024
…rain warnings about library methods [networkupstools#2511] Signed-off-by: Jim Klimov <[email protected]>
jimklimov
added a commit
to jimklimov/nut
that referenced
this issue
Jul 11, 2024
…pstools#2511] * Clearly name "thread-ready" scanning methods that can be used standalone or from pthread_create() constraints * Settle on "thready" methods freeing caller's data (so we do not worry about doing it after the parallel loops, or at a wrong moment before) * Unify code mark-up in the files to facilitate later comparisons * Update some comments Signed-off-by: Jim Klimov <[email protected]>
jimklimov
added a commit
to jimklimov/nut
that referenced
this issue
Jul 11, 2024
…age [networkupstools#2511] Signed-off-by: Jim Klimov <[email protected]>
jimklimov
added a commit
to jimklimov/nut
that referenced
this issue
Jul 11, 2024
…() the loaded library modules [networkupstools#2511] Avoid some memory leak reports Signed-off-by: Jim Klimov <[email protected]>
jimklimov
added a commit
to jimklimov/nut
that referenced
this issue
Jul 11, 2024
…t_snmp() once per library load [networkupstools#2511] Signed-off-by: Jim Klimov <[email protected]>
jimklimov
added a commit
to jimklimov/nut
that referenced
this issue
Jul 11, 2024
…tions more reliably [networkupstools#2511] Signed-off-by: Jim Klimov <[email protected]>
jimklimov
added a commit
to jimklimov/nut
that referenced
this issue
Jul 11, 2024
…FD leak [networkupstools#2511] Signed-off-by: Jim Klimov <[email protected]>
jimklimov
added a commit
to jimklimov/nut
that referenced
this issue
Jul 11, 2024
…rain libnetsnmp leak reports [networkupstools#2511] It seems that although `init_snmp()` is called once, every use of the library initializes some data which it then does not release, at least on the test system here. Signed-off-by: Jim Klimov <[email protected]>
jimklimov
added a commit
to jimklimov/nut
that referenced
this issue
Jul 11, 2024
…s/valgrind/.valgrind.supp: constrain warnings about library methods [networkupstools#2511] Signed-off-by: Jim Klimov <[email protected]>
jimklimov
added a commit
to jimklimov/nut
that referenced
this issue
Jul 11, 2024
…rain warnings about library methods [networkupstools#2511] Signed-off-by: Jim Klimov <[email protected]>
jimklimov
added a commit
to jimklimov/nut
that referenced
this issue
Jul 11, 2024
…evise trace-logging and comment markup [networkupstools#2511] Signed-off-by: Jim Klimov <[email protected]>
jimklimov
added a commit
to jimklimov/nut
that referenced
this issue
Jul 11, 2024
…upstools#2511] Signed-off-by: Jim Klimov <[email protected]>
jimklimov
added a commit
to jimklimov/nut
that referenced
this issue
Jul 11, 2024
…ementations [networkupstools#2511] Per https://forums.freebsd.org/threads/named-semaphore-uninitialized-bytes.84850/ apparently BSD libc and GLibc at least do suffer Signed-off-by: Jim Klimov <[email protected]>
jimklimov
added a commit
to jimklimov/nut
that referenced
this issue
Jul 15, 2024
…nt [networkupstools#2511, networkupstools#2431, networkupstools#2450 et al] Earlier bump went to "patch" component, but here we actually have an API expansion (and more exported symbols), so the more important component should be bumped. Signed-off-by: Jim Klimov <[email protected]>
jimklimov
added a commit
to jimklimov/nut
that referenced
this issue
Jul 15, 2024
…nt [networkupstools#2511, networkupstools#2431, networkupstools#2450 et al] Earlier bump went to "patch" component, but here we actually have an API expansion (and more exported symbols), so the more important component should be bumped. Signed-off-by: Jim Klimov <[email protected]>
jimklimov
added a commit
to jimklimov/nut
that referenced
this issue
Jul 17, 2024
…e of files [networkupstools#2511] Signed-off-by: Jim Klimov <[email protected]>
jimklimov
added a commit
to jimklimov/nut
that referenced
this issue
Jul 17, 2024
…e of files [networkupstools#2511] Signed-off-by: Jim Klimov <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
Earlier work (~2017 for 42ITy) saw addition of pthreads support for networked scans, so that large subnets could be scanned in finite time. IIRC, this used separate thread pools per "bus", so scans involving e.g. NetXML and SNMP and Old NUT would use 2x-3x the requested limit of threads.
With work on issue #2244 (PR #2509) allowing several IP address ranges to be requested for scanning, it was discovered that each range-scan request, even if for just one IP address, suffers the timeout (5 sec by default) if some addresses do not reply. Then the fanned-out threads from one IP address range request are all reaped and another range for the same bus is inspected.
This issue proposes to separate the thread-pool creation and reaping from the scanning method, so that many range scans can be requested and get reaped only afterwards. As an option, maybe converge to one thread pool shared by all buses (so limiting the amount of threads to what the user allowed/requested).
It could be nice to come up with a way to reduce copy-paste in the scanning methods about different threading implementations. It could in fact be part of moving to some single shared thread pool implemented in whatever somewhat abstracted manner (e.g. like we have IP address iterator methods, we could have some thread iterator - or block on it waiting for a free slot).
On a related note, it can help to update the IP address iteration method to handle not one range, but the array of such ranges (introduced with PR #2509), so we would call the scanning method once and avoid the problem of waiting for a timeout of one range before we can start the next.
Also note that
sem_init()
is not implemented on MacOS; some other methods should be used. See e.g. https://stackoverflow.com/questions/1413785/sem-init-on-os-xnut-scanner
scanning support on MacOS #2522Also, IPMI scanning is not parallelized at all
nut-scanner
scanning support for IPMI #2523The text was updated successfully, but these errors were encountered: