make test fails on the PIP upgrade command

[lamchakchan]

Description

The call for make test fails within the docker step for pip install--upgrade pip. TravisCI should be modified to utilize the steps in the Makefile so the testing environment is reproducible in both CI and user workstations.

How to Reproduce

git clone https://github.com/cloudviz/agentless-system-crawler.git
cd agentless-system-crawler
sudo make test

Log Output

...
debconf: falling back to frontend: Readline
update-rc.d: warning: start and stop actions are no longer supported; falling back to defaults
diff: /var/lib/apparmor/profiles/.apparmor.md5sums: No such file or directory
Setting up libcacard0:amd64 (1:2.5.0-2) ...
Setting up qemu-system-x86 (1:2.5+dfsg-5ubuntu10.32) ...
/var/lib/dpkg/info/qemu-system-x86.postinst: 34: /var/lib/dpkg/info/qemu-system-x86.postinst: rmmod: not found
Setting up qemu-kvm (1:2.5+dfsg-5ubuntu10.32) ...
Setting up qemu-utils (1:2.5+dfsg-5ubuntu10.32) ...
Setting up dh-python (2.20151103ubuntu1.1) ...
Processing triggers for libc-bin (2.23-0ubuntu10) ...
Processing triggers for systemd (229-4ubuntu21.4) ...
Processing triggers for sgml-base (1.26+nmu4ubuntu1) ...
Collecting pip
  Downloading https://files.pythonhosted.org/packages/c2/d7/90f34cb0d83a6c5631cf71dfe64cc1054598c843a92b400e55675cc2ac37/pip-18.1-py2.py3-none-any.whl (1.3MB)
Installing collected packages: pip
  Found existing installation: pip 8.1.1
    Not uninstalling pip at /usr/lib/python2.7/dist-packages, outside environment /usr
Successfully installed pip-18.1
Traceback (most recent call last):
  File "/usr/bin/pip", line 9, in <module>
    from pip import main
ImportError: cannot import name main
The command '/bin/sh -c apt-get update &&     echo "y" | apt-get install python     python-dev     python-pip     build-essential     docker-engine     dpkg     dpkg-dev     qemu-kvm   && pip install --upgrade pip   && pip install mock' returned a non-zero code: 1
Makefile:45: recipe for target 'test' failed
make: *** [test] Error 1

Debugging Commands Output

Output of docker version:

Client:
 Version:	17.12.1-ce
 API version:	1.35
 Go version:	go1.10.1
 Git commit:	7390fc6
 Built:	Wed Apr 18 01:23:11 2018
 OS/Arch:	linux/amd64

Server:
 Engine:
  Version:	17.12.1-ce
  API version:	1.35 (minimum version 1.12)
  Go version:	go1.10.1
  Git commit:	7390fc6
  Built:	Wed Feb 28 17:46:05 2018
  OS/Arch:	linux/amd64
  Experimental:	false

Output of docker info:

Containers: 1
 Running: 0
 Paused: 0
 Stopped: 1
Images: 4
Server Version: 17.12.1-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 9b55aab90508bd389d7654c4baf173a981477d55
runc version: 9f9c96235cc97674e935002fc3d78361b696a69e
init version: v0.13.0 (expected: 949e6facb77383876aeff8a6944dde66b3089574)
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.15.0-29-generic
Operating System: Ubuntu 18.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.79GiB
Name: lchan-1
ID: OL2Q:3CKD:Q5EQ:OZJE:ZU7X:AIZU:MNP3:6G4L:RGUB:52OX:L25I:RPHX
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support

Output of python --version:

Python 2.7.15rc1

Output of pip freeze:

asn1crypto==0.24.0
backports.ssl-match-hostname==3.5.0.1
certifi==2018.10.15
chardet==3.0.4
configobj==4.7.0
cryptography==2.1.4
docker-py==1.10.6
docker-pycreds==0.3.0
enum34==1.1.6
fluent-logger==0.4.6
idna==2.6
ipaddress==1.0.17
kafka==1.3.3
kafka-python==1.3.1
kazoo==2.5.0
keyring==10.6.0
keyrings.alt==3.0
morph==0.1.2
msgpack-python==0.5.6
netifaces==0.10.4
psutil==2.2.0
pycrypto==2.6.1
pygobject==3.26.1
pykafka==1.1.0
python-dateutil==2.4.2
pyxdg==0.25
requests==2.19.1
requests-unixsocket==0.1.5
SecretStorage==2.3.1
semantic-version==2.5.0
six==1.11.0
tabulate==0.8.2
urllib3==1.23
websocket-client==0.53.0
Yapsy==1.11.223

[nadgowdas]

I can re-produce this issue on my machine.
after discussing with @sahilsuneja1 this is pip version issue and need more investigation to fix this.

In the meantime, you can get around this issue, by commenting out the pip upgrade command in Dockerfile.test line #17.

     16     qemu-kvm \
     17 # && pip install --upgrade pip \
     18   && pip install mock
     19
     20 COPY requirements.txt /requirements.txt

[canturkisci]

Thank you @nadgowdas for your help with this. Following pretty much what you have done.
This is as far as I get on an Ubuntu 18.04:
Requirements

sudo apt  install docker.io
sudo service docker start

sudo make test fails on the same:

Collecting pip
  Downloading https://files.pythonhosted.org/packages/c2/d7/90f34cb0d83a6c5631cf71dfe64cc1054598c843a92b400e55675cc2ac37/pip-18.1-py2.py3-none-any.whl (1.3MB)
Installing collected packages: pip
  Found existing installation: pip 8.1.1
    Not uninstalling pip at /usr/lib/python2.7/dist-packages, outside environment /usr
Successfully installed pip-18.1
Traceback (most recent call last):
  File "/usr/bin/pip", line 9, in <module>
    from pip import main
ImportError: cannot import name main
The command '/bin/sh -c apt-get update &&     echo "y" | apt-get install python     python-dev     python-pip     build-essential     docker-engine     dpkg     dpkg-dev     qemu-kvm   && pip install --upgrade pip   && pip install mock' returned a non-zero code: 1
Makefile:45: recipe for target 'test' failed
make: *** [test] Error 1

[canturkisci]

I ran the command explicitly from inside the ubuntu container, and end up in the same place.

Also: One more issue is on apt-get install docker-engine; no idea if this matters.

Same error message at the end:

Traceback (most recent call last):
  File "/usr/bin/pip", line 9, in <module>
    from pip import main
ImportError: cannot import name main

Same running the pip upgrade command in container

root@da0c782d74ed:/# pip install --upgrade pip
Traceback (most recent call last):
  File "/usr/bin/pip", line 9, in <module>
    from pip import main
ImportError: cannot import name main

[canturkisci]

@nadgowdas I have looked at this enough, I think...
There are a zillion issues in pip on this. This is a good reference:
pypa/pip#5221 (comment)

The net is: You should NOT pip install --upgrade your system packages, as we do in container.root here. It seems OK use case, when you are doing this in a virtualenv.

No wonder a lot of docker people get hit by the same problem. For system packages, the recommended use is: sudo apt install python-pip, which the Dockerfile is already doing up top.

It seem then OK to comment the pip install --upgrade pip line, and rely on distro packages. I expect when FROM goes to 18.04 we might see the newer pip.

Let me know if it makes sense to you all as well when you get a chance to look at the pip issue above. That and some neighboring issues have bunch of options they discuss.

We can then update the Dockerfile and close.

[canturkisci]

After commenting out pip install --upgrade

This flow works on an Ubuntu 18.04:

sudo apt  install docker.io
sudo service docker start

sudo make test

goes about 20mins of testing. There are some failing tests, which will be a separate issue.

[nadgowdas]

The discussion in the comment above is interesting. I guess, we can wait until ubuntu distro supports newer pip version (installed through apt-get install) and we should avoid upgrading it directly through pip upgrade.

Regarding Ubuntu 18.04, we need to make few more changes to support it.

For now, we should work with workaround by commenting out pip upgrade .
@sahilsuneja1 any thoughts ?

[sahilsuneja1]

I tested with a fresh make test.
The existing Dockerfile.test works with one change (alread in .travis.yml):

-- && pip install --upgrade pip
++ && pip install -I pip==9.0.1

[canturkisci]

That explains why travis builds still work.
The recommendation above is not to install pip via pip for system files.
I expect this works, as the change to pip10+ causes deb tools to break.
This gets us on parity w/ Travis.

[sahilsuneja1]

Yup, pip is to be installed via apt, and then version be locked to 9.0.1
Was addressed in travis here: #365