Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Revise nut-scanner parallelization of scanning #2511

Open
jimklimov opened this issue Jul 5, 2024 · 1 comment
Open

Revise nut-scanner parallelization of scanning #2511

jimklimov opened this issue Jul 5, 2024 · 1 comment
Labels
enhancement macOS nut-scanner portability We want NUT to build and run everywhere possible
Milestone

Comments

@jimklimov
Copy link
Member

jimklimov commented Jul 5, 2024

Earlier work (~2017 for 42ITy) saw addition of pthreads support for networked scans, so that large subnets could be scanned in finite time. IIRC, this used separate thread pools per "bus", so scans involving e.g. NetXML and SNMP and Old NUT would use 2x-3x the requested limit of threads.

With work on issue #2244 (PR #2509) allowing several IP address ranges to be requested for scanning, it was discovered that each range-scan request, even if for just one IP address, suffers the timeout (5 sec by default) if some addresses do not reply. Then the fanned-out threads from one IP address range request are all reaped and another range for the same bus is inspected.

This issue proposes to separate the thread-pool creation and reaping from the scanning method, so that many range scans can be requested and get reaped only afterwards. As an option, maybe converge to one thread pool shared by all buses (so limiting the amount of threads to what the user allowed/requested).

It could be nice to come up with a way to reduce copy-paste in the scanning methods about different threading implementations. It could in fact be part of moving to some single shared thread pool implemented in whatever somewhat abstracted manner (e.g. like we have IP address iterator methods, we could have some thread iterator - or block on it waiting for a free slot).

On a related note, it can help to update the IP address iteration method to handle not one range, but the array of such ranges (introduced with PR #2509), so we would call the scanning method once and avoid the problem of waiting for a timeout of one range before we can start the next.

Also note that sem_init() is not implemented on MacOS; some other methods should be used. See e.g. https://stackoverflow.com/questions/1413785/sem-init-on-os-x

Also, IPMI scanning is not parallelized at all

@jimklimov jimklimov added enhancement nut-scanner portability We want NUT to build and run everywhere possible labels Jul 5, 2024
@jimklimov jimklimov added this to the 2.8.4 milestone Jul 5, 2024
jimklimov added a commit that referenced this issue Jul 8, 2024
…scans do not parallelize together [#2244]

See also: #2511

Signed-off-by: Jim Klimov <[email protected]>
jimklimov added a commit to jimklimov/nut that referenced this issue Jul 8, 2024
jimklimov added a commit to jimklimov/nut that referenced this issue Jul 8, 2024
jimklimov added a commit to jimklimov/nut that referenced this issue Jul 8, 2024
…r of WIN32 active init calls vs. pthread variable declarations [networkupstools#2511]

Signed-off-by: Jim Klimov <[email protected]>
@jimklimov jimklimov added the macOS label Jul 8, 2024
jimklimov added a commit to jimklimov/nut that referenced this issue Jul 8, 2024
jimklimov added a commit to jimklimov/nut that referenced this issue Jul 8, 2024
jimklimov added a commit to jimklimov/nut that referenced this issue Jul 8, 2024
jimklimov added a commit to jimklimov/nut that referenced this issue Jul 8, 2024
jimklimov added a commit to jimklimov/nut that referenced this issue Jul 8, 2024
jimklimov added a commit to jimklimov/nut that referenced this issue Jul 8, 2024
jimklimov added a commit to jimklimov/nut that referenced this issue Jul 8, 2024
jimklimov added a commit to jimklimov/nut that referenced this issue Jul 8, 2024
…r of WIN32 active init calls vs. pthread variable declarations [networkupstools#2511]

Signed-off-by: Jim Klimov <[email protected]>
jimklimov added a commit to jimklimov/nut that referenced this issue Jul 8, 2024
jimklimov added a commit to jimklimov/nut that referenced this issue Jul 8, 2024
… for ip_ranges[] list and helper metadata, and methods as part of libnutscan [networkupstools#2244, networkupstools#2511]

Not bumping library version, because it was recently bumped as
part of other PRs about this issue. Technically the scope of the
library has been changed by new exported methods and header lines.

Signed-off-by: Jim Klimov <[email protected]>
jimklimov added a commit to jimklimov/nut that referenced this issue Jul 8, 2024
jimklimov added a commit to jimklimov/nut that referenced this issue Jul 8, 2024
jimklimov added a commit to jimklimov/nut that referenced this issue Jul 9, 2024
jimklimov added a commit to jimklimov/nut that referenced this issue Jul 9, 2024
…t_ip<=irl.stop_ip in alphanumeric comparison order [networkupstools#2511]

Signed-off-by: Jim Klimov <[email protected]>
jimklimov added a commit to jimklimov/nut that referenced this issue Jul 10, 2024
jimklimov added a commit to jimklimov/nut that referenced this issue Jul 10, 2024
…rain libnetsnmp leak reports [networkupstools#2511]

It seems that although `init_snmp()` is called once, every use of the
library initializes some data which it then does not release, at least
on the test system here.

Signed-off-by: Jim Klimov <[email protected]>
jimklimov added a commit to jimklimov/nut that referenced this issue Jul 10, 2024
…s/valgrind/.valgrind.supp: constrain warnings about library methods [networkupstools#2511]

Signed-off-by: Jim Klimov <[email protected]>
jimklimov added a commit to jimklimov/nut that referenced this issue Jul 10, 2024
jimklimov added a commit to jimklimov/nut that referenced this issue Jul 10, 2024
jimklimov added a commit to jimklimov/nut that referenced this issue Jul 10, 2024
…() the loaded library modules [networkupstools#2511]

Avoid some memory leak reports

Signed-off-by: Jim Klimov <[email protected]>
jimklimov added a commit to jimklimov/nut that referenced this issue Jul 10, 2024
jimklimov added a commit to jimklimov/nut that referenced this issue Jul 10, 2024
jimklimov added a commit to jimklimov/nut that referenced this issue Jul 10, 2024
jimklimov added a commit to jimklimov/nut that referenced this issue Jul 10, 2024
…rain libnetsnmp leak reports [networkupstools#2511]

It seems that although `init_snmp()` is called once, every use of the
library initializes some data which it then does not release, at least
on the test system here.

Signed-off-by: Jim Klimov <[email protected]>
jimklimov added a commit to jimklimov/nut that referenced this issue Jul 10, 2024
…s/valgrind/.valgrind.supp: constrain warnings about library methods [networkupstools#2511]

Signed-off-by: Jim Klimov <[email protected]>
jimklimov added a commit to jimklimov/nut that referenced this issue Jul 10, 2024
jimklimov added a commit to jimklimov/nut that referenced this issue Jul 11, 2024
…pstools#2511]

* Clearly name "thread-ready" scanning methods that can be used
  standalone or from pthread_create() constraints
* Settle on "thready" methods freeing caller's data (so we do not
  worry about doing it after the parallel loops, or at a wrong
  moment before)
* Unify code mark-up in the files to facilitate later comparisons
* Update some comments

Signed-off-by: Jim Klimov <[email protected]>
jimklimov added a commit to jimklimov/nut that referenced this issue Jul 11, 2024
jimklimov added a commit to jimklimov/nut that referenced this issue Jul 11, 2024
…() the loaded library modules [networkupstools#2511]

Avoid some memory leak reports

Signed-off-by: Jim Klimov <[email protected]>
jimklimov added a commit to jimklimov/nut that referenced this issue Jul 11, 2024
jimklimov added a commit to jimklimov/nut that referenced this issue Jul 11, 2024
jimklimov added a commit to jimklimov/nut that referenced this issue Jul 11, 2024
jimklimov added a commit to jimklimov/nut that referenced this issue Jul 11, 2024
…rain libnetsnmp leak reports [networkupstools#2511]

It seems that although `init_snmp()` is called once, every use of the
library initializes some data which it then does not release, at least
on the test system here.

Signed-off-by: Jim Klimov <[email protected]>
jimklimov added a commit to jimklimov/nut that referenced this issue Jul 11, 2024
…s/valgrind/.valgrind.supp: constrain warnings about library methods [networkupstools#2511]

Signed-off-by: Jim Klimov <[email protected]>
jimklimov added a commit to jimklimov/nut that referenced this issue Jul 11, 2024
jimklimov added a commit to jimklimov/nut that referenced this issue Jul 11, 2024
…evise trace-logging and comment markup [networkupstools#2511]

Signed-off-by: Jim Klimov <[email protected]>
jimklimov added a commit to jimklimov/nut that referenced this issue Jul 11, 2024
jimklimov added a commit to jimklimov/nut that referenced this issue Jul 11, 2024
jimklimov added a commit to jimklimov/nut that referenced this issue Jul 15, 2024
…nt [networkupstools#2511, networkupstools#2431, networkupstools#2450 et al]

Earlier bump went to "patch" component, but here we actually have
an API expansion (and more exported symbols), so the more important
component should be bumped.

Signed-off-by: Jim Klimov <[email protected]>
jimklimov added a commit to jimklimov/nut that referenced this issue Jul 15, 2024
…nt [networkupstools#2511, networkupstools#2431, networkupstools#2450 et al]

Earlier bump went to "patch" component, but here we actually have
an API expansion (and more exported symbols), so the more important
component should be bumped.

Signed-off-by: Jim Klimov <[email protected]>
jimklimov added a commit to jimklimov/nut that referenced this issue Jul 17, 2024
jimklimov added a commit to jimklimov/nut that referenced this issue Jul 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement macOS nut-scanner portability We want NUT to build and run everywhere possible
Projects
None yet
Development

No branches or pull requests

1 participant