HPCC-31498 Kafka shared library not constructing/destructing properly

[dcamper]

Use Singleton pattern instead of std::once.

Obscure problems arose with the destruction of a static std::once, when the plugin was loaded and unloaded multiple times in a single job.

Type of change:

• This change is a bug fix (non-breaking change which fixes an issue).
• This change is a new feature (non-breaking change which adds functionality).
• This change improves the code (refactor or other change that does not change the functionality)
• This change fixes warnings (the fix does not alter the functionality or the generated code)
• This change is a breaking change (fix or feature that will cause existing behavior to change).
• This change alters the query API (existing queries will have to be recompiled)

Checklist:

• My code follows the code style of this project.
  • My code does not create any new warnings from compiler, build system, or lint.
• The commit message is properly formatted and free of typos.
  • The commit message title makes sense in a changelog, by itself.
  • The commit is signed.
• My change requires a change to the documentation.
  • I have updated the documentation accordingly, or...
  • I have created a JIRA ticket to update the documentation.
  • Any new interfaces or exported functions are appropriately commented.
• I have read the CONTRIBUTORS document.
• The change has been fully tested:
  • I have added tests to cover my changes.
  • All new and existing tests passed.
  • I have checked that this change does not introduce memory leaks.
  • I have used Valgrind or similar tools to check for potential issues.
• I have given due consideration to all of the following potential concerns:
  • Scalability
  • Performance
  • Security
  • Thread-safety
  • Cloud-compatibility
  • Premature optimization
  • Existing deployed queries will not be broken
  • This change fixes the problem, not just the symptom
  • The target branch of this pull request is appropriate for such a change.
• There are no similar instances of the same problem that should be addressed
  • I have addressed them here
  • I have raised JIRA issues to address them separately
• This is a user interface / front-end modification
  • I have tested my changes in multiple modern browsers
  • The component(s) render as expected

Smoketest:

• Send notifications about my Pull Request position in Smoketest queue.
• Test my draft Pull Request.

Testing:

Manual testing with a local Kafka instance, publishing and consuming messages.

[ghalliday]

@dcamper I don't think this is quite right.

[ghalliday]

Potential race condition - two threads could get here at the same time, and both pass the !isAlive() test.
I think the correct solution (using an auto pattern jake uses) is

 static PublisherCacheExpirerObj & queryPublisherCache()
 {
     auto initFunction = [] ()
    {
        PublisherCacheExpirerObj * publisher = new PublisherCacheExpirerObj;
        KafkaPlugin::publisherCacheExpirer = new KafkaPlugin::PublisherCacheExpirerObj;
        KafkaPlugin::publisherCacheExpirer->start();
        return publisher;
    };
    return *publisherCacheExpirer.query(initFunction);
}

[ghalliday]

Then you can delete the calls to setupPublisherCache() since queryPublisherCache() will handle it for you.

[dcamper]

Due to the circular dependencies between PublisherCacheObj and PublisherCacheExpirerObj I opted to refactor: The expirer class (and its background thread) are now gone. Expiration is done after a new publisher object is created. This simplifies things a great deal and probably increases system performance slightly (one less thread to manage).

[ghalliday]

This still needs to call stop - simplest is in the destructor since nothing is derived from it.

[ghalliday]

The change looks reasonable. One question.

Since the expiry is now handled by the main calling thread, if a publisher is being expired the call to get a publisher could block for pollTimeout before it returns (while it waits to join the thread). Is that going to cause any issues?

[dcamper]

@ghalliday With the current value of poll timeout (1s) it should not be a problem. In practice, a given job will publish to only one Kafka topic on a single broker (and therefore there will be only one publisher object).

[dcamper]

@ghalliday I am retargeting this PR to 9.4.x, as soon as my internal testing succeeds. I will need a final once-over review after I push those changes.

[dcamper]

Retargeted to 9.4.x. Also removed an unnecessary check from within expire() method.

[ghalliday]

Approving assume testing goes as you expect.

[dcamper]

Squashed in preparation for merge, but don't merge yet. I am awaiting customer verification.

[ghalliday]

@dcamper can you also change the commit message so that it matches the normal convention. HPCC-XXXXX ...

[dcamper]

@ghalliday My apologies! Commit message amended.

[dcamper]

@ghalliday Please merge. Thanks!

[github-actions]

Jira Issue: https://hpccsystems.atlassian.net//browse/HPCC-31498

Jirabot Action Result:
Workflow Transition: Merge Pending