[BUG] prometheus datasource: Request failed to get to the server (status code: 502)

[rmarops]

Describe the bug
when adding prometheus as a datasource from dev tools, I get Request failed to get to the server (status code: 502)

To Reproduce
Steps to reproduce the behavior:

POST _plugins/_query/_datasources
{
    "name" : "prometheus",
    "connector": "prometheus",
    "properties" : {
        "prometheus.uri" : "http://ec2-xx-xx-xx-xx.compute-1.amazonaws.com:9050"
    }
}

Returns:
Request failed to get to the server (status code: 502)

• i confirm that 9050 is a nonstandard port I am using while testing a fresh docker run of prometheus
• from the opensearch host cli, a curl http://ec2-xx-xx-xx-xx.compute-1.amazonaws.com:9050/metrics returns results

Expected behavior
no error and a working datasource

Plugins
opensearch-alerting 2.9.0.0
opensearch-anomaly-detection 2.9.0.0
opensearch-asynchronous-search 2.9.0.0
opensearch-cross-cluster-replication 2.9.0.0
opensearch-geospatial 2.9.0.0
opensearch-index-management 2.9.0.0
opensearch-job-scheduler 2.9.0.0
opensearch-knn 2.9.0.0
opensearch-ml 2.9.0.0
opensearch-neural-search 2.9.0.0
opensearch-notifications 2.9.0.0
opensearch-notifications-core 2.9.0.0
opensearch-observability 2.9.0.0
opensearch-performance-analyzer 2.9.0.0
opensearch-reports-scheduler 2.9.0.0
opensearch-security 2.9.0.0
opensearch-security-analytics 2.9.0.0
opensearch-sql 2.9.0.0

Host/Environment (please complete the following information):

• OS:
  Amazon Linux
• Version
  2023

Additional context
following https://opensearch.org/docs/latest/observing-your-data/prometheusmetrics/

[rmarops]

Reproduced by extracting and running prometheus locally on the opensearch host, confirmed curl localhost:9090/metrics produced metrics then attempted from devtools:

POST _plugins/_query/_datasources
{
    "name" : "prometheus",
    "connector": "prometheus",
    "properties" : {
        "prometheus.uri" : "http://localhost:9090"
    }
}

[yudhiesh]

Facing the same issue adding an ML connector using POST /_plugins/_ml/connectors/_create, did you find a fix? I am using OpenSearch 2.9.

[scubbx]

I can confirm having the same problem.
But a subsequent GET request shows that the connection was stored in OpenSearch at least.
Not sure if related, but the "allowedRoles": [] field in the GET reqest is always empty, despite having set up the roles according to the documentation (https://github.com/opensearch-project/sql/blob/main/docs/user/ppl/admin/security.rst).

[andrross]

@bbarani Can you transfer this issue to https://github.com/opensearch-project/sql? I believe the POST _plugins/_query/_datasources API in question here is implemented by the SQL plugin.

[penghuo]

@vamsi-amazon could u take a look of this issue.

[AdaptiveStep]

i have this problem too. Same situation: It gives 502 error, but if you try to post it again, it says it already exists. But when you look in the observability/metrics tab, it cannot be found there.

Does the prometheus connector work for anyone at all?