Add data source stats by type

[noCharger]

Description

• Add data_counts_by_connector_type metric into the plugins, exposed via /stats API

BUILD SUCCESSFUL in 19m 37s
172 actionable tasks: 115 executed, 57 up-to-date

Usage:

1. Empty state

curl localhost:9200/_plugins/_ppl/stats
{"active_data_sources_count":{} ... }

2. Create several data sources

curl --request POST \
  --url localhost:9200/_plugins/_query/_datasources \
  --header 'content-type: application/x-ndjson' \
  --data '{"name": "mys5","description": "","connector": "SPARK","allowedRoles": [],"properties": {"glue.auth.type": "iam_role","glue.auth.role_arn": "arn:aws:iam::123456:role/flint-opensearch-role","glue.indexstore.opensearch.uri": "http://localhost:9200","glue.indexstore.opensearch.auth": "noauth"}}'

3. Run stats API again

curl localhost:9200/_plugins/_ppl/stats
{"active_data_sources_count":{"OPENSEARCH":1,"S3GLUE":4}

Issues Resolved

[List any issues this PR will resolve]

Check List

• New functionality includes testing.
  • All tests pass, including unit test, integration test and doctest
• New functionality has been documented.
  • New functionality has javadoc added
  • New functionality has user manual doc added
• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (94bd664) 95.43% compared to head (9848a08) 95.44%.

Additional details and impacted files

@@            Coverage Diff            @@
##               main    #2495   +/-   ##
=========================================
  Coverage     95.43%   95.44%           
- Complexity     5024     5029    +5     
=========================================
  Files           483      483           
  Lines         13989    14017   +28     
  Branches        943      945    +2     
=========================================
+ Hits          13350    13378   +28     
  Misses          617      617           
  Partials         22       22           

Flag	Coverage Δ
sql-engine	95.44% <100.00%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[noCharger]

Python compatible issue on MacOS. Not related to the change.

> Task :doctest:doctest FAILED
======================================================================
ERROR: test_docs (unittest.loader._FailedTest.test_docs)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_docs
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/unittest/loader.py", line 137, in loadTestsFromName
    module = __import__(module_name)
             ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/runner/work/sql/sql/doctest/test_docs.py", line 16, in <module>
    from opensearch_sql_cli.opensearch_connection import OpenSearchConnection
  File "/Users/runner/work/sql/sql/doctest/sql-cli/src/opensearch_sql_cli/opensearch_connection.py", line 6, in <module>
    import boto3
  File "/Users/runner/work/sql/sql/doctest/.venv/lib/python3.12/site-packages/boto3/__init__.py", line 16, in <module>
    from boto3.session import Session
  File "/Users/runner/work/sql/sql/doctest/.venv/lib/python3.12/site-packages/boto3/session.py", line 17, in <module>
    import botocore.session
  File "/Users/runner/work/sql/sql/doctest/.venv/lib/python3.12/site-packages/botocore/session.py", line 29, in <module>
    import botocore.configloader
  File "/Users/runner/work/sql/sql/doctest/.venv/lib/python3.12/site-packages/botocore/configloader.py", line 19, in <module>
    from botocore.compat import six
  File "/Users/runner/work/sql/sql/doctest/.venv/lib/python3.12/site-packages/botocore/compat.py", line 33, in <module>
    from botocore.vendored.six.moves import http_client
ModuleNotFoundError: No module named 'botocore.vendored.six.moves'

Passed on Linux

----------------------------------------------------------------------
Ran 44 tests in 440.920s

OK
Doctest Done

Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

For more on this, please refer to https://docs.gradle.org/8.4/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.

BUILD SUCCESSFUL in 8m 2s
12 actionable tasks: 9 executed, 3 up-to-date

[noCharger]

Discussed with @vamsi-amazon offline, there is an opportunity to rework all the schedulers that scan the index.