grafana-cli: failed to verify certificate: x509: certificate signed by unknown authority

[xyuan]

I am trying to build the grafana UI locally on my macOS/ubuntu machine, I have always failed with the following message,

_311.0 logger=settings t=2024-09-27T22:57:21.272514922Z level=info msg=Path Logs path=/var/log/grafana
311.0 logger=settings t=2024-09-27T22:57:21.272517381Z level=info msg=Path Plugins path=/var/lib/grafana/plugins
311.0 logger=settings t=2024-09-27T22:57:21.272526131Z level=info msg=Path Provisioning path=/etc/grafana/provisioning
311.0 logger=settings t=2024-09-27T22:57:21.272539506Z level=info msg=App mode production
311.5 Error: ✗ Get  https://grafana.com/api/plugins/michaeldmoore-multistat-panel/versions: tls: failed to verify certificate: x509: certificate signed by unknown authority
------
failed to solve: process /bin/sh -c apt-get install -y tzdata systemd apt-utils npm vim net-tools &&     /usr/sbin/grafana-cli plugins install michaeldmoore-multistat-panel &&     chown root:grafana /etc/grafana &&     cd /var/lib/grafana/plugins/omniperf_plugin &&     npm install &&     npm run build &&     apt-get autoremove -y &&     apt-get autoclean -y &&     sed -i \"s/  bindIp.*/  bindIp: 0.0.0.0/\" /etc/mongod.conf &&     mkdir -p /var/lib/grafana &&     touch /var/lib/grafana/grafana.lib &&     chown grafana:grafana /var/lib/grafana/grafana.lib did not complete successfully: exit code: 1_

I have generated the certificate using openssl, and setup the ini file like,

492;tls_skip_verify_insecure = true
493 ;tls_client_cert =/Work/omniperf/grafana/client.pem
494 ;tls_client_key = /Work/omniperf/grafana/client.key
495 ;tls_client_ca = /Work/omniperf/grafana/ca.pem

after looking into the Dockfile, this error is from,

_48 /usr/sbin/grafana-cli plugins install michaeldmoore-multistat-panel && _

anyone can help to resolve this issue, or help to generate these certificate/key files?

[coleramos425]

Linking related issue #405

[coleramos425]

@njobypet could you please assign to an engineer? Thank you.

[vstempen]

Can you please describe your system configuration? Are you building on macOS or Ubuntu? Which version? Docker container? Which image? VirtualBox? Thanks.

[xyuan]

I am building on macos by following this webpage, https://rocm.docs.amd.com/projects/omniperf/en/latest/install/grafana-setup.html using docker.

some minor changes in the dockfile to use arm distribution instead of the original ubuntum distribution. with this change, I am able to pass most, except the grafana-cli update the apis with tls certification failure.

[vstempen]

It's not a solution but just a workaround, but you can add --no-check-certificate key to wget and --insecure to curl to pass the build. We are still working the issue. It seems docker container cannot find SSL certificates, when running on macos. Even I implicitly dump a certificate from a web-site to a file and then specify it as a parameter. Will update you as soon as we find a proper solution.

[xyuan]

yes, I add -k to curl and wget, and both work fine. but it is difficult for me to figure out how to workaround the grafana-cli command. I tried to download these apps directly, and using ADD in the dockerfile, but it still failed. maybe I did wrong, will try it again.

[vstempen]

Hi, can you attach output of "openssl s_client -connect dl.grafana.com:443 -showcerts" form your macos.

[vstempen]

In my case it was endpoint security service that was not letting openssl to get proper certificate chain from server. It was replacing the proper certificate with some default company certificate during handshake. Disabling the service helped.

[xyuan]

Hi, can you attach output of "openssl s_client -connect dl.grafana.com:443 -showcerts" form your macos.

I wasn't sure whether it is allowed to post this information publically on github, though

[vstempen]

OK, can you make sure there are no security services running on macOS and try again.

[xyuan]

I made changes to the Dockfile to use 'ADD' the pre-download plugin, and disable the wget, and curl ssl security check, and it works,

19 ADD plugins/omniperf_plugin /var/lib/grafana/plugins/omniperf_plugin
20 ADD plugins/aceiot-svg-panel /var/lib/grafana/plugins/aceiot-svg-panel
21 ADD plugins/ae3e-plotly-panel /var/lib/grafana/plugins/ae3e-plotly-panel
22 ADD plugins/grafana-image-renderer /var/lib/grafana/plugins/grafana-image-renderer
23 ADD plugins/michaeldmoore-multistat-panel /var/lib/grafana/plugins/michaeldmoore-multistat-panel
24 ADD plugins/natel-plotly-panel /var/lib/grafana/plugins/natel-plotly-panel

and

29 RUN apt-get update &&
30 apt-get install -y adduser libfontconfig1 musl wget &&
31 wget -q https://dl.grafana.com/enterprise/release/grafana-enterprise_10.4.3_arm64.deb --no-check-certificate &&
32 dpkg -i grafana-enterprise_10.4.3_arm64.deb &&
33 apt-get install -y gnupg curl &&
34 curl -fksSL https://www.mongodb.org/static/pgp/server-7.0.asc | gpg -o /usr/share/keyrings/mongodb-server-7.0.gpg --dearmor &&
35 echo "deb [ arch=amd64,arm64 signed-by=/usr/share/keyrings/mongodb-server-7.0.gpg ] https://repo.mongodb.org/apt/ubuntu jammy/mongodb-org/7.0 multiverse" | tee /etc/apt/sources.list. d/mongodb-org-7.0.list &&
36 apt-get update &&
37 apt-get install -y mongodb-org

these pre-download plugins have to be at the grafana/plugin directory.

[vstempen]

I don't think we can re-distribute Grafana plugins. You solution works, but disabling network proxy services on macOS system should work too. I cannot reproduce the problem when network proxy services are disabled.

[xyuan]

no worries about it, at least it is working for us now, thanks

[vstempen]

Can we close the issue then? Thanks.

[xyuan]

yes, please