Conflict between docker config and docker service with ssh driver #16231

aishwarya-senthilkumar · 2023-04-04T04:58:37Z

Unable to start, stop or delete minikube

minikube start --driver=ssh --ssh-ip-address=10.186.72.184 --native-ssh=false --service-cluster-ip-range='192.168.115.0/24' --extra-config=kubeadm.pod-network-cidr='192.168.114.0/24' --extra-config=kubeadm.pod-network-cidr=192.168.120.0/22 --cni=bridge --extra-config=kubelet.resolv-conf=/run/systemd/resolve/resolv.conf
😄  minikube v1.29.0 on Darwin 12.3.1
    ▪ KUBECONFIG=/Users/sai/.kube/config
✨  Using the ssh driver based on user configuration
👍  Starting control plane node minikube in cluster minikube
🔥  Deleting "minikube" in ssh ...
🤦  StartHost failed, but will try again: creating host: create host timed out in 360.000000 seconds
😿  Failed to start ssh bare metal machine. Running "minikube delete" may fix it: creating host: create host timed out in 360.000000 seconds

❌  Exiting due to DRV_CREATE_TIMEOUT: Failed to start host: creating host: create host timed out in 360.000000 seconds
💡  Suggestion: Try 'minikube delete', and disable any conflicting VPN or firewall software
🍿  Related issue: https://github.com/kubernetes/minikube/issues/7072

sai@saiEMD6M ~ % minikube stop
✋  Stopping node "minikube"  ...
✋  Stopping node "minikube"  ...
✋  Stopping node "minikube"  ...
✋  Stopping node "minikube"  ...
✋  Stopping node "minikube"  ...
✋  Stopping node "minikube"  ...

❌  Exiting due to GUEST_STOP_TIMEOUT: Temporary Error: stop: containers: docker: docker ps -a --filter=name=k8s_ --format=: Process exited with status 1
stdout:

stderr:
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?


╭───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│                                                                                                                       │
│    😿  If the above advice does not help, please let us know:                                                         │
│    👉  https://github.com/kubernetes/minikube/issues/new/choose                                                       │
│                                                                                                                       │
│    Please run `minikube logs --file=logs.txt` and attach logs.txt to the GitHub issue.                                │
│    Please also attach the following file to the GitHub issue:                                                         │
│    - /var/folders/n1/7_bs1z291n9dx8vy62dg873h0000gq/T/minikube_stop_bc7f49f349e822641d09c5fb48f410196b822884_0.log    │
│                                                                                                                       │
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

sai@saiEMD6M ~ % minikube delete
🔄  Uninstalling Kubernetes v1.25.3 using kubeadm ...
E0404 10:17:19.593080   13318 delete.go:516] unpause failed: list paused: docker: docker ps --filter status=paused --filter=name=k8s_ --format={{.ID}}: Process exited with status 1
stdout:

stderr:
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
💣  error deleting profile "minikube": failed to delete cluster: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.25.3:$PATH" kubeadm reset --cri-socket /var/run/cri-dockerd.sock --force": Process exited with status 127
stdout:

stderr:
env: ‘kubeadm’: No such file or directory

DETAILS :

OS : MacOS Monterey
Version 12.3.1 (21E258)

sai@saiEMD6M ~ % minikube version
minikube version: v1.28.0
commit: 986b1ebd987211ed16f8cc10aed7d2c42fc8392f

sai@saiEMD6M ~ % docker version
Client:
 Cloud integration: v1.0.29
 Version:           20.10.22
 API version:       1.41
 Go version:        go1.18.9
 Git commit:        3a2c30b
 Built:             Thu Dec 15 22:28:41 2022
 OS/Arch:           darwin/amd64
 Context:           default
 Experimental:      true

Server: Docker Desktop 4.16.2 (95914)
 Engine:
  Version:          20.10.22
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.18.9
  Git commit:       42c8b31
  Built:            Thu Dec 15 22:26:14 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.14
  GitCommit:        9ba4b250366a5ddde94bb7c9d1def331423aa323
 runc:
  Version:          1.1.4
  GitCommit:        v1.1.4-0-g5fd4c4d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

I managed to login to my VM, docker service is down

root@minikube:~# systemctl status docker
× docker.service - Docker Application Container Engine
     Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/docker.service.d
             └─10-machine.conf
     Active: failed (Result: exit-code) since Mon 2023-04-03 21:00:01 PDT; 52min ago
TriggeredBy: × docker.socket
       Docs: https://docs.docker.com
    Process: 158216 ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --storage-driver overlay2 --tlsverify --tlscacert /etc/docker/ca.pem --tlscert /etc/docker/server.pem --tlskey /etc/docker/server-key.pem --label provider=ssh --insecure-r>
   Main PID: 158216 (code=exited, status=1/FAILURE)
        CPU: 89ms

Apr 03 21:00:01 minikube systemd[1]: docker.service: Scheduled restart job, restart counter is at 3.
Apr 03 21:00:01 minikube systemd[1]: Stopped Docker Application Container Engine.
Apr 03 21:00:01 minikube systemd[1]: docker.service: Start request repeated too quickly.
Apr 03 21:00:01 minikube systemd[1]: docker.service: Failed with result 'exit-code'.
Apr 03 21:00:01 minikube systemd[1]: Failed to start Docker Application Container Engine.
lines 1-16/16 (END)

Is there something wrong with my minikube setup? or is it something simple I’m missing?

The text was updated successfully, but these errors were encountered:

aishwarya-senthilkumar · 2023-04-04T08:28:35Z

root@minikube:~# sudo journalctl -eu docker
~
~
~
~
~
~
~
~
~
~
~
~
~
~
Apr 04 00:06:52 minikube systemd[1]: Starting Docker Application Container Engine...
Apr 04 00:06:52 minikube dockerd[168313]: unable to configure the Docker daemon with file /etc/docker/daemon.json: the following directives are specified both as a flag and in the configuration file: storage-driver: (from flag: overlay2, from file: overlay2)
Apr 04 00:06:52 minikube systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE
Apr 04 00:06:52 minikube systemd[1]: docker.service: Failed with result 'exit-code'.
Apr 04 00:06:52 minikube systemd[1]: Failed to start Docker Application Container Engine.
Apr 04 00:06:52 minikube systemd[1]: docker.service: Scheduled restart job, restart counter is at 1.
Apr 04 00:06:52 minikube systemd[1]: Stopped Docker Application Container Engine.
Apr 04 00:06:52 minikube systemd[1]: Starting Docker Application Container Engine...
Apr 04 00:06:52 minikube dockerd[168324]: unable to configure the Docker daemon with file /etc/docker/daemon.json: the following directives are specified both as a flag and in the configuration file: storage-driver: (from flag: overlay2, from file: overlay2)
Apr 04 00:06:52 minikube systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE
Apr 04 00:06:52 minikube systemd[1]: docker.service: Failed with result 'exit-code'.
Apr 04 00:06:52 minikube systemd[1]: Failed to start Docker Application Container Engine.
Apr 04 00:06:52 minikube systemd[1]: docker.service: Scheduled restart job, restart counter is at 2.
Apr 04 00:06:52 minikube systemd[1]: Stopped Docker Application Container Engine.
Apr 04 00:06:52 minikube systemd[1]: Starting Docker Application Container Engine...
Apr 04 00:06:52 minikube dockerd[168335]: unable to configure the Docker daemon with file /etc/docker/daemon.json: the following directives are specified both as a flag and in the configuration file: storage-driver: (from flag: overlay2, from file: overlay2)
Apr 04 00:06:52 minikube systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE
Apr 04 00:06:52 minikube systemd[1]: docker.service: Failed with result 'exit-code'.
Apr 04 00:06:52 minikube systemd[1]: Failed to start Docker Application Container Engine.
Apr 04 00:06:52 minikube systemd[1]: docker.service: Scheduled restart job, restart counter is at 3.
Apr 04 00:06:52 minikube systemd[1]: Stopped Docker Application Container Engine.
Apr 04 00:06:52 minikube systemd[1]: docker.service: Start request repeated too quickly.
Apr 04 00:06:52 minikube systemd[1]: docker.service: Failed with result 'exit-code'.
Apr 04 00:06:52 minikube systemd[1]: Failed to start Docker Application Container Engine.
Apr 04 00:08:51 minikube systemd[1]: Starting Docker Application Container Engine...
Apr 04 00:08:51 minikube dockerd[168374]: unable to configure the Docker daemon with file /etc/docker/daemon.json: the following directives are specified both as a flag and in the configuration file: storage-driver: (from flag: overlay2, from file: overlay2)
Apr 04 00:08:51 minikube systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE
Apr 04 00:08:51 minikube systemd[1]: docker.service: Failed with result 'exit-code'.
Apr 04 00:08:51 minikube systemd[1]: Failed to start Docker Application Container Engine.
Apr 04 00:08:51 minikube systemd[1]: docker.service: Scheduled restart job, restart counter is at 1.
Apr 04 00:08:51 minikube systemd[1]: Stopped Docker Application Container Engine.
Apr 04 00:08:51 minikube systemd[1]: Starting Docker Application Container Engine...
Apr 04 00:08:51 minikube dockerd[168384]: unable to configure the Docker daemon with file /etc/docker/daemon.json: the following directives are specified both as a flag and in the configuration file: storage-driver: (from flag: overlay2, from file: overlay2)
Apr 04 00:08:51 minikube systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE
Apr 04 00:08:51 minikube systemd[1]: docker.service: Failed with result 'exit-code'.
Apr 04 00:08:51 minikube systemd[1]: Failed to start Docker Application Container Engine.
Apr 04 00:08:51 minikube systemd[1]: docker.service: Scheduled restart job, restart counter is at 2.
Apr 04 00:08:51 minikube systemd[1]: Stopped Docker Application Container Engine.
Apr 04 00:08:51 minikube systemd[1]: Starting Docker Application Container Engine...
Apr 04 00:08:51 minikube dockerd[168393]: unable to configure the Docker daemon with file /etc/docker/daemon.json: the following directives are specified both as a flag and in the configuration file: storage-driver: (from flag: overlay2, from file: overlay2)
Apr 04 00:08:51 minikube systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE
Apr 04 00:08:51 minikube systemd[1]: docker.service: Failed with result 'exit-code'.
Apr 04 00:08:51 minikube systemd[1]: Failed to start Docker Application Container Engine.
Apr 04 00:08:52 minikube systemd[1]: docker.service: Scheduled restart job, restart counter is at 3.
Apr 04 00:08:52 minikube systemd[1]: Stopped Docker Application Container Engine.
Apr 04 00:08:52 minikube systemd[1]: docker.service: Start request repeated too quickly.
Apr 04 00:08:52 minikube systemd[1]: docker.service: Failed with result 'exit-code'.
Apr 04 00:08:52 minikube systemd[1]: Failed to start Docker Application Container Engine.
root@minikube:~#

aishwarya-senthilkumar · 2023-04-04T08:31:28Z

Issue:

unable to configure the Docker daemon with file /etc/docker/daemon.json: the following directives are specified both as a flag and in the configuration file: storage-driver: (from flag: overlay2, from file: overlay2)

There is a directive in /etc/docker/daemon.json, that is a duplicate of the one passed in ExecStart in /etc/systemd/system/docker.service.d/10-machine.conf (storage-driver)

Solution:

Inside vm, Remove --storage-driver overlay2 from /etc/systemd/system/docker.service.d/10-machine.conf
then sudo systemctl daemon-reload
then sudo systemctl restart docker

Then try running the same minikube start --driver=ssh ... on your machine

afbjorklund · 2023-04-04T15:21:45Z

The "ssh" driver is for advanced users, it seems there might be some conflict between the minikube docker config and what was installed on the VM before. I would suggest using another driver, unless your workaround is enough for you.

k8s-triage-robot · 2023-07-03T16:07:33Z

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue as fresh with /remove-lifecycle stale
Close this issue with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

vaibhav2107 · 2023-08-03T12:53:47Z

/remove-lifecycle stale

oxeye-mher · 2023-12-08T08:02:59Z

I have the same issue with minikube v1.32.0, unfortunately the workaround is not working as minikube start rewrites the /etc/systemd/system/docker.service.d/10-machine.conf file and adds the --storage-driver overlay2 after I remove it.

Any idea how to make it work?

chronicc · 2024-01-27T16:11:04Z

This issue really is a problem when using the ssh driver. The problem does not arise from a wrong setup of the target machine but is reintroduced by minikube every time minikube start is run.

k8s-triage-robot · 2024-04-26T16:28:45Z

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue as fresh with /remove-lifecycle stale
Close this issue with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot · 2024-05-26T16:35:17Z

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue as fresh with /remove-lifecycle rotten
Close this issue with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

chronicc · 2024-06-05T15:41:21Z

/remove-lifecycle rotten

k8s-triage-robot · 2024-09-03T15:43:30Z

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue as fresh with /remove-lifecycle stale
Close this issue with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot · 2024-10-03T16:28:53Z

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue as fresh with /remove-lifecycle rotten
Close this issue with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

k8s-triage-robot · 2024-11-02T17:13:55Z

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Reopen this issue with /reopen
Mark this issue as fresh with /remove-lifecycle rotten
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

k8s-ci-robot · 2024-11-02T17:14:01Z

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

After 90d of inactivity, lifecycle/stale is applied

After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied

After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Reopen this issue with /reopen

Mark this issue as fresh with /remove-lifecycle rotten

Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

aishwarya-senthilkumar changed the title ~~Failed to start host: creating host: create host timed out in 360.000000 seconds~~ Unable to start, stop or delete minikube Apr 4, 2023

aishwarya-senthilkumar changed the title ~~Unable to start, stop or delete minikube~~ Exiting due to DRV_CREATE_TIMEOUT: Failed to start host: creating host: create host timed out in 360.000000 secondsUnable to start, stop or delete minikube Apr 4, 2023

aishwarya-senthilkumar closed this as completed Apr 4, 2023

aishwarya-senthilkumar reopened this Apr 4, 2023

afbjorklund added co/generic-driver kind/bug Categorizes issue or PR as related to a bug. priority/awaiting-more-evidence Lowest priority. Possibly useful, but not yet enough support to actually get it done. labels Apr 4, 2023

x7upLime mentioned this issue Apr 4, 2023

Removes storage_driver option from /etc/docker/daemon.json #16235

Closed

afbjorklund mentioned this issue Apr 14, 2023

changes between 1.28 and 1.29+ lead to decrease in docker/daemon.json options (aka daemon.json is being always overwritten) #16323

Closed

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jul 3, 2023

k8s-ci-robot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Aug 3, 2023

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Apr 26, 2024

k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels May 26, 2024

k8s-ci-robot removed the lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. label Jun 5, 2024

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Sep 3, 2024

k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Oct 3, 2024

k8s-ci-robot closed this as not planned Won't fix, can't repro, duplicate, stale Nov 2, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Conflict between docker config and docker service with ssh driver #16231

Conflict between docker config and docker service with ssh driver #16231

aishwarya-senthilkumar commented Apr 4, 2023

aishwarya-senthilkumar commented Apr 4, 2023

aishwarya-senthilkumar commented Apr 4, 2023 •

edited

Loading

afbjorklund commented Apr 4, 2023

k8s-triage-robot commented Jul 3, 2023

vaibhav2107 commented Aug 3, 2023

oxeye-mher commented Dec 8, 2023

chronicc commented Jan 27, 2024

k8s-triage-robot commented Apr 26, 2024

k8s-triage-robot commented May 26, 2024

chronicc commented Jun 5, 2024

k8s-triage-robot commented Sep 3, 2024

k8s-triage-robot commented Oct 3, 2024

k8s-triage-robot commented Nov 2, 2024

k8s-ci-robot commented Nov 2, 2024

Conflict between docker config and docker service with ssh driver #16231

Conflict between docker config and docker service with ssh driver #16231

Comments

aishwarya-senthilkumar commented Apr 4, 2023

aishwarya-senthilkumar commented Apr 4, 2023

aishwarya-senthilkumar commented Apr 4, 2023 • edited Loading

Issue:

Solution:

afbjorklund commented Apr 4, 2023

k8s-triage-robot commented Jul 3, 2023

vaibhav2107 commented Aug 3, 2023

oxeye-mher commented Dec 8, 2023

chronicc commented Jan 27, 2024

k8s-triage-robot commented Apr 26, 2024

k8s-triage-robot commented May 26, 2024

chronicc commented Jun 5, 2024

k8s-triage-robot commented Sep 3, 2024

k8s-triage-robot commented Oct 3, 2024

k8s-triage-robot commented Nov 2, 2024

k8s-ci-robot commented Nov 2, 2024

aishwarya-senthilkumar commented Apr 4, 2023 •

edited

Loading