Failing tests on CentOS-CI + KVM #121

gbraad · 2017-07-20T04:29:38Z

This is a general issue, and I have seen this happen from time to time with the CentOS image:

Error: E0720 04:12:09.674403   23446 start.go:278] Error starting the VM: Error configuring authorization on host: Maximum number of retries (60) exceeded. Retrying.
E0720 04:15:49.210763   23446 start.go:278] Error starting the VM: Error configuring authorization on host: Maximum number of retries (60) exceeded. Retrying.
E0720 04:19:29.286027   23446 start.go:278] Error starting the VM: Error configuring authorization on host: Maximum number of retries (60) exceeded. Retrying.
Error starting the VM: Error configuring authorization on host: Maximum number of retries (60) exceeded
Error configuring authorization on host: Maximum number of retries (60) exceeded
Error configuring authorization on host: Maximum number of retries (60) exceeded

It is unclear why this happens...

The text was updated successfully, but these errors were encountered:

gbraad · 2017-07-20T04:31:33Z

Just a few ...

gbraad · 2017-07-20T06:34:03Z

Added a sleep statement for the minishift start, as the error in this case is consistently happening at the swapspace test

gbraad · 2017-07-20T07:02:24Z

Added sleep (as in #122)... but still failing:

gbraad · 2017-07-24T05:07:23Z

Issue still occasionally occurs as can be seen in the following build for different PRs:

LalatenduMohanty · 2017-07-25T11:53:03Z

@gbraad Just to confirm, you have seen these kind of error with local runs too?

gbraad · 2017-07-25T12:11:19Z

less often, but yes. Although I was in a hurry to test, so didn't pay a lot of attention.

…

On Tue, Jul 25, 2017 at 7:53 PM, Lalatendu Mohanty ***@***.*** > wrote: @gbraad <https://github.com/gbraad> Just to confirm, you have seen these kind of error with local runs too? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#121 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAHZtHnZ2RkATvHJaARECXeOwJI36cLks5sRdcfgaJpZM4Odlv_> .

-- Gerard Braad | http://gbraad.nl [ Doing Open Source Matters ]

coolbrg · 2017-08-22T14:05:20Z

I am able to reproduce the error locally as well for even starting the VM:

Executing command : /home/budhram/redhat/minishift-centos-iso/tests/../build/bin/minishift start --vm-driver kvm --iso-url file:///home/budhram/redhat/minishift-centos-iso/tests/../build/minishift-centos7.iso
*******
2017-08-22 19:33:10,018 test             L0119 DEBUG| Error: E0822 19:26:46.940451   12265 start.go:342] Error starting the VM: Error creating the VM. Error creating machine: Error detecting OS: Too many retries waiting for SSH to be available.  Last error: Maximum number of retries (60) exceeded. Retrying.
E0822 19:29:58.276836   12265 start.go:342] Error starting the VM: Error configuring authorization on host: Too many retries waiting for SSH to be available.  Last error: Maximum number of retries (60) exceeded. Retrying.
E0822 19:33:10.016776   12265 start.go:342] Error starting the VM: Error configuring authorization on host: Too many retries waiting for SSH to be available.  Last error: Maximum number of retries (60) exceeded. Retrying.
Error starting the VM: Error creating the VM. Error creating machine: Error detecting OS: Too many retries waiting for SSH to be available.  Last error: Maximum number of retries (60) exceeded
Error configuring authorization on host: Too many retries waiting for SSH to be available.  Last error: Maximum number of retries (60) exceeded
Error configuring authorization on host: Too many retries waiting for SSH to be available.  Last error: Maximum number of retries (60) exceeded
...

gbraad · 2017-08-22T14:11:59Z

add --show-libmachine-logs ;-) and let's see what happens.... This error happens when there is an issue with the provisioning not able to start... which can be related to the IP address... is it a lease issue?

…

On Tue, Aug 22, 2017 at 10:05 PM, Budh Ram Gurung ***@***.***> wrote: I am able to reproduce the error locally as well for even starting the VM: Executing command : /home/budhram/redhat/minishift-centos-iso/tests/../build/bin/minishift start --vm-driver kvm --iso-url file:///home/budhram/redhat/minishift-centos-iso/tests/../build/minishift-centos7.iso ******* 2017-08-22 19:33:10,018 test L0119 DEBUG| Error: E0822 19:26:46.940451 12265 start.go:342] Error starting the VM: Error creating the VM. Error creating machine: Error detecting OS: Too many retries waiting for SSH to be available. Last error: Maximum number of retries (60) exceeded. Retrying. E0822 19:29:58.276836 12265 start.go:342] Error starting the VM: Error configuring authorization on host: Too many retries waiting for SSH to be available. Last error: Maximum number of retries (60) exceeded. Retrying. E0822 19:33:10.016776 12265 start.go:342] Error starting the VM: Error configuring authorization on host: Too many retries waiting for SSH to be available. Last error: Maximum number of retries (60) exceeded. Retrying. Error starting the VM: Error creating the VM. Error creating machine: Error detecting OS: Too many retries waiting for SSH to be available. Last error: Maximum number of retries (60) exceeded Error configuring authorization on host: Too many retries waiting for SSH to be available. Last error: Maximum number of retries (60) exceeded Error configuring authorization on host: Too many retries waiting for SSH to be available. Last error: Maximum number of retries (60) exceeded ... — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#121 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAHZrYIRhIMv3XEkYlfkgt99iQzGwlyks5sauAggaJpZM4Odlv_> .

-- Gerard Braad | http://gbraad.nl [ Doing Open Source Matters ]

coolbrg · 2017-08-22T14:13:55Z

@gbraad could you add those debugs lines here https://github.com/gbraad/minishift-centos-iso/blob/opt/tests/test.py#L90 then we can see why it is failing in CI.

I am also trying locally.

The upstream released latest ISO passed locally.

coolbrg · 2017-08-22T14:46:46Z

Logs

Logs with upstream releasted latest centos ISO

(minishift) DBG | Starting VM minishift
(minishift) DBG | GetIP called for minishift
(minishift) DBG | Failed to retrieve dnsmasq leases from /var/lib/libvirt/dnsmasq/docker-machines.leases
(minishift) DBG | Unable to locate IP address for MAC 52:54:00:4f:77:aa
(minishift) DBG | Waiting for the VM to come up... 0
[...]
(minishift) DBG | GetIP called for minishift
(minishift) DBG | Failed to retrieve dnsmasq leases from /var/lib/libvirt/dnsmasq/docker-machines.leases
(minishift) DBG | Unable to locate IP address for MAC 52:54:00:4f:77:aa
(minishift) DBG | Waiting for the VM to come up... 14
(minishift) DBG | GetIP called for minishift
(minishift) DBG | Failed to retrieve dnsmasq leases from /var/lib/libvirt/dnsmasq/docker-machines.leases
(minishift) DBG | IP address: 192.168.42.87
(minishift) DBG | Unable to locate IP address for MAC 52:54:00:4f:77:aa
(minishift) Calling .GetConfigRaw
(minishift) Calling .DriverName
(minishift) Calling .DriverName
(minishift) Calling .GetState
(minishift) DBG | Getting current state...
Getting to WaitForSSH function...
(minishift) Calling .GetSSHHostname
(minishift) DBG | GetIP called for minishift
(minishift) DBG | Failed to retrieve dnsmasq leases from /var/lib/libvirt/dnsmasq/docker-machines.leases
(minishift) DBG | IP address: 192.168.42.87
(minishift) DBG | Unable to locate IP address for MAC 52:54:00:4f:77:aa
(minishift) Calling .GetSSHPort
(minishift) Calling .GetSSHKeyPath
(minishift) Calling .GetSSHKeyPath
(minishift) DBG | AK: resolvestorepath: /home/budhram/.minishift
(minishift) DBG | AK: resolvestorepath: /home/budhram/.minishift
(minishift) Calling .GetSSHUsername
Using SSH client type: external
Using SSH private key: /home/budhram/.minishift/machines/minishift/id_rsa (-rw-------)
&{[-F /dev/null -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none [email protected] -o IdentitiesOnly=yes -i /home/budhram/.minishift/machines/minishift/id_rsa -p 22] /usr/bin/ssh <nil>}
About to run SSH command:
exit 0
SSH cmd err, output: exit status 255: 
Error getting ssh command 'exit 0' : ssh command error:
command : exit 0
err     : exit status 255
output  : 
Getting to WaitForSSH function...
(minishift) Calling .GetSSHHostname
(minishift) DBG | GetIP called for minishift
(minishift) DBG | Failed to retrieve dnsmasq leases from /var/lib/libvirt/dnsmasq/docker-machines.leases
(minishift) DBG | IP address: 192.168.42.87
(minishift) Calling .GetSSHPort
(minishift) DBG | Unable to locate IP address for MAC 52:54:00:4f:77:aa
(minishift) Calling .GetSSHKeyPath
(minishift) DBG | AK: resolvestorepath: /home/budhram/.minishift
(minishift) Calling .GetSSHKeyPath
(minishift) Calling .GetSSHUsername
(minishift) DBG | AK: resolvestorepath: /home/budhram/.minishift
Using SSH client type: external
Using SSH private key: /home/budhram/.minishift/machines/minishift/id_rsa (-rw-------)
&{[-F /dev/null -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none [email protected] -o IdentitiesOnly=yes -i /home/budhram/.minishift/machines/minishift/id_rsa -p 22] /usr/bin/ssh <nil>}
About to run SSH command:
exit 0
SSH cmd err, output: <nil>: 
(minishift) Calling .GetSSHHostname
(minishift) DBG | GetIP called for minishift
(minishift) DBG | Failed to retrieve dnsmasq leases from /var/lib/libvirt/dnsmasq/docker-machines.leases
(minishift) DBG | IP address: 192.168.42.87
(minishift) DBG | Unable to locate IP address for MAC 52:54:00:4f:77:aa
(minishift) Calling .GetSSHPort
(minishift) Calling .GetSSHKeyPath
(minishift) Calling .GetSSHKeyPath
(minishift) DBG | AK: resolvestorepath: /home/budhram/.minishift
(minishift) Calling .GetSSHUsername
(minishift) DBG | AK: resolvestorepath: /home/budhram/.minishift
Using SSH client type: external
Using SSH private key: /home/budhram/.minishift/machines/minishift/id_rsa (-rw-------)
&{[-F /dev/null -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none [email protected] -o IdentitiesOnly=yes -i /home/budhram/.minishift/machines/minishift/id_rsa -p 22] /usr/bin/ssh <nil>}
About to run SSH command:
cat /etc/os-release
SSH cmd err, output: <nil>: NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

VARIANT="minishift"
VARIANT_VERSION="1.1.0"
BUILD_ID="ecb5ada-25072017180632-local"
  ...

Logs with Locally built ISO or affected ISO

(minishift) DBG | Starting VM minishift
(minishift) DBG | GetIP called for minishift
(minishift) DBG | Failed to retrieve dnsmasq leases from /var/lib/libvirt/dnsmasq/docker-machines.leases
(minishift) DBG | Unable to locate IP address for MAC 52:54:00:6b:3e:7e
(minishift) DBG | Waiting for the VM to come up... 0
[...]
(minishift) DBG | Failed to retrieve dnsmasq leases from /var/lib/libvirt/dnsmasq/docker-machines.leases
(minishift) DBG | Unable to locate IP address for MAC 52:54:00:6b:3e:7e
(minishift) DBG | Waiting for the VM to come up... 14
(minishift) DBG | GetIP called for minishift
(minishift) DBG | Failed to retrieve dnsmasq leases from /var/lib/libvirt/dnsmasq/docker-machines.leases
(minishift) DBG | IP address: 192.168.42.53
(minishift) DBG | Unable to locate IP address for MAC 52:54:00:6b:3e:7e
(minishift) Calling .GetConfigRaw
(minishift) Calling .DriverName
(minishift) Calling .DriverName
(minishift) Calling .GetState
(minishift) DBG | Getting current state...
Getting to WaitForSSH function...
(minishift) Calling .GetSSHHostname
(minishift) DBG | GetIP called for minishift
(minishift) DBG | Failed to retrieve dnsmasq leases from /var/lib/libvirt/dnsmasq/docker-machines.leases
(minishift) DBG | IP address: 192.168.42.53
(minishift) DBG | Unable to locate IP address for MAC 52:54:00:6b:3e:7e
(minishift) Calling .GetSSHPort
(minishift) Calling .GetSSHKeyPath
(minishift) Calling .GetSSHKeyPath
(minishift) DBG | AK: resolvestorepath: /home/budhram/.minishift
(minishift) DBG | AK: resolvestorepath: /home/budhram/.minishift
(minishift) Calling .GetSSHUsername
Using SSH client type: external
Using SSH private key: /home/budhram/.minishift/machines/minishift/id_rsa (-rw-------)
&{[-F /dev/null -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none [email protected] -o IdentitiesOnly=yes -i /home/budhram/.minishift/machines/minishift/id_rsa -p 22] /usr/bin/ssh <nil>}
About to run SSH command:
exit 0
SSH cmd err, output: exit status 255: 
Error getting ssh command 'exit 0' : ssh command error:
command : exit 0
err     : exit status 255
output  : 
Getting to WaitForSSH function...
(minishift) Calling .GetSSHHostname
(minishift) DBG | GetIP called for minishift
(minishift) DBG | Failed to retrieve dnsmasq leases from /var/lib/libvirt/dnsmasq/docker-machines.leases
(minishift) DBG | IP address: 192.168.42.53
(minishift) DBG | Unable to locate IP address for MAC 52:54:00:6b:3e:7e
(minishift) Calling .GetSSHPort
(minishift) Calling .GetSSHKeyPath
(minishift) Calling .GetSSHKeyPath
(minishift) DBG | AK: resolvestorepath: /home/budhram/.minishift
(minishift) DBG | AK: resolvestorepath: /home/budhram/.minishift
(minishift) Calling .GetSSHUsername
Using SSH client type: external
Using SSH private key: /home/budhram/.minishift/machines/minishift/id_rsa (-rw-------)
&{[-F /dev/null -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none [email protected] -o IdentitiesOnly=yes -i /home/budhram/.minishift/machines/minishift/id_rsa -p 22] /usr/bin/ssh <nil>}
About to run SSH command:
exit 0
SSH cmd err, output: exit status 1: /bin/bash: Permission denied

Error getting ssh command 'exit 0' : ssh command error:
command : exit 0
err     : exit status 1
output  : /bin/bash: Permission denied

Getting to WaitForSSH function...
(minishift) Calling .GetSSHHostname
(minishift) DBG | GetIP called for minishift
(minishift) DBG | Failed to retrieve dnsmasq leases from /var/lib/libvirt/dnsmasq/docker-machines.leases
(minishift) DBG | IP address: 192.168.42.53
(minishift) DBG | Unable to locate IP address for MAC 52:54:00:6b:3e:7e
(minishift) Calling .GetSSHPort
(minishift) Calling .GetSSHKeyPath
(minishift) Calling .GetSSHKeyPath
(minishift) DBG | AK: resolvestorepath: /home/budhram/.minishift
(minishift) Calling .GetSSHUsername
(minishift) DBG | AK: resolvestorepath: /home/budhram/.minishift
Using SSH client type: external
Using SSH private key: /home/budhram/.minishift/machines/minishift/id_rsa (-rw-------)
&{[-F /dev/null -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none [email protected] -o IdentitiesOnly=yes -i /home/budhram/.minishift/machines/minishift/id_rsa -p 22] /usr/bin/ssh <nil>}
About to run SSH command:
exit 0
************************************** This get repeated *******************************************
SSH cmd err, output: exit status 1: /bin/bash: Permission denied

Error getting ssh command 'exit 0' : ssh command error:
command : exit 0
err     : exit status 1
output  : /bin/bash: Permission denied
************************************** This get repeated *******************************************

coolbrg · 2017-09-18T09:56:44Z

Logs of unsuccessful restart - https://gist.github.com/budhrg/1bdc7444d9796082c212f0b165286bfd

LalatenduMohanty · 2017-09-26T12:57:02Z

A sleep of 90s added after the minishift start to work around the issue [1] In case we choose to close this issue. We will create a separate issue to findout why we need a sleep and how we can remove it.

[1] https://github.com/minishift/minishift-centos-iso/blob/master/tests/test.sh#L129

LalatenduMohanty added kind/bug priority/major labels Jul 20, 2017

LalatenduMohanty added this to the v1.2.0 milestone Jul 20, 2017

gbraad added a commit to gbraad-redhat/minishift-centos-iso that referenced this issue Jul 20, 2017

Fixes minishift#121 Adding sleep before (re)start of Minishift

40ff7dd

gbraad mentioned this issue Jul 20, 2017

Fixes #121 Adding sleep before (re)start of Minishift #122

Closed

gbraad added a commit to gbraad-redhat/minishift-centos-iso that referenced this issue Jul 20, 2017

Fixes minishift#121 Adding sleep before (re)start of Minishift

48b76b1

gbraad added a commit to gbraad-redhat/minishift-centos-iso that referenced this issue Jul 20, 2017

Fixes minishift#121 Restart VM using the same arguments

2a24196

gbraad mentioned this issue Jul 20, 2017

Fixes #121 Restart VM using the same arguments #123

Merged

LalatenduMohanty modified the milestones: v1.1.0, v1.2.0 Jul 20, 2017

praveenkumar closed this as completed in #123 Jul 20, 2017

gbraad reopened this Jul 24, 2017

gbraad mentioned this issue Jul 25, 2017

Issue #124 RHSM should be persistent across reboot #126

Merged

LalatenduMohanty added kind/task and removed kind/bug labels Jul 25, 2017

LalatenduMohanty modified the milestones: v1.2.0, v1.1.0 Jul 25, 2017

coolbrg self-assigned this Aug 9, 2017

gbraad added a commit to gbraad-redhat/minishift-centos-iso that referenced this issue Aug 22, 2017

Issue minishift#121 Investigate failures on CI

39023c1

gbraad added a commit to gbraad-redhat/minishift-centos-iso that referenced this issue Aug 22, 2017

Issue minishift#121 Investigate failures on CI

2978318

LalatenduMohanty modified the milestones: v1.2.0, v1.3.0 Sep 20, 2017

coolbrg mentioned this issue Sep 22, 2017

Issue #141 Replace avocado based tests with shell scripts based test #152

Merged

LalatenduMohanty modified the milestones: v1.3.0, v1.4.0, v1.5.0 Oct 9, 2017

gbraad mentioned this issue Nov 1, 2017

Cannot start minishift again after stopping it because of loop back storage issue minishift/minishift#1639

Closed

praveenkumar modified the milestones: v1.4.0, v1.5.0 Dec 11, 2017

LalatenduMohanty modified the milestones: v1.5.0, v1.6.0 Dec 13, 2017

coolbrg modified the milestones: v1.6.0, v1.7.0 Feb 12, 2018

LalatenduMohanty modified the milestones: v1.7.0, v1.8.0 Feb 15, 2018

anjannath modified the milestones: v1.8.0, v1.9.0 Mar 26, 2018

LalatenduMohanty modified the milestones: v1.9.0, v1.10.0 Mar 28, 2018

LalatenduMohanty modified the milestones: v1.10.0, v1.11.0 May 31, 2018

coolbrg modified the milestones: v1.11.0, v1.12.0 Jul 6, 2018

LalatenduMohanty modified the milestones: v1.12.0, v1.13.0 Jul 10, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Failing tests on CentOS-CI + KVM #121

Failing tests on CentOS-CI + KVM #121

gbraad commented Jul 20, 2017 •

edited

Loading

gbraad commented Jul 20, 2017

gbraad commented Jul 20, 2017

gbraad commented Jul 20, 2017 •

edited

Loading

gbraad commented Jul 24, 2017

LalatenduMohanty commented Jul 25, 2017

gbraad commented Jul 25, 2017 via email

coolbrg commented Aug 22, 2017 •

edited

Loading

gbraad commented Aug 22, 2017 via email

coolbrg commented Aug 22, 2017 •

edited

Loading

coolbrg commented Aug 22, 2017

coolbrg commented Sep 18, 2017

LalatenduMohanty commented Sep 26, 2017

Failing tests on CentOS-CI + KVM #121

Failing tests on CentOS-CI + KVM #121

Comments

gbraad commented Jul 20, 2017 • edited Loading

gbraad commented Jul 20, 2017

gbraad commented Jul 20, 2017

gbraad commented Jul 20, 2017 • edited Loading

gbraad commented Jul 24, 2017

LalatenduMohanty commented Jul 25, 2017

gbraad commented Jul 25, 2017 via email

coolbrg commented Aug 22, 2017 • edited Loading

gbraad commented Aug 22, 2017 via email

coolbrg commented Aug 22, 2017 • edited Loading

coolbrg commented Aug 22, 2017

Logs with upstream releasted latest centos ISO

Logs with Locally built ISO or affected ISO

coolbrg commented Sep 18, 2017

LalatenduMohanty commented Sep 26, 2017

gbraad commented Jul 20, 2017 •

edited

Loading

gbraad commented Jul 20, 2017 •

edited

Loading

coolbrg commented Aug 22, 2017 •

edited

Loading

coolbrg commented Aug 22, 2017 •

edited

Loading