Crawler sometimes crashes in k8 env while processing container-create events

[sahilsuneja1]

Description

Seems like k8 has several transient containers that come and go too quickly before crawler can do anything about it.

Reported by Stefan Berger.

vagrant@shard1-worker1:~/agentless-system-crawler$ CRAWLER --crawlContainers --crawlmode OUTCONTAINER --features cpu,fprobe --frequency=10 --url kafka://xx:yy --environment kubernetes

Traceback (most recent call last):
File "crawler/crawler.py", line 239, in
main()
File "crawler/crawler.py", line 233, in main
worker.run()
File "/home/vagrant/agentless-system-crawler/crawler/worker.py", line 83, in run
self.iterate(time_to_sleep)
File "/home/vagrant/agentless-system-crawler/crawler/worker.py", line 38, in iterate
frame = self.crawler.polling_crawl(timeout)
File "/home/vagrant/agentless-system-crawler/crawler/containers_crawler.py", line 70, in polling_crawl
host_namespace=self.host_namespace)
File "/home/vagrant/agentless-system-crawler/crawler/containers.py", line 58, in poll_containers
host_namespace=host_namespace)
File "/home/vagrant/agentless-system-crawler/crawler/dockercontainer.py", line 85, in poll_docker_containers
host_namespace=host_namespace)
File "/home/vagrant/agentless-system-crawler/crawler/dockercontainer.py", line 177, in init
self._set_environment_specific_options()
File "/home/vagrant/agentless-system-crawler/crawler/dockercontainer.py", line 242, in _set_environment_specific_options
self.long_id, _options)
File "/home/vagrant/agentless-system-crawler/crawler/plugins/environments/kubernetes_environment.py", line 28, in get_container_namespace
container_meta = exec_dockerinspect(long_id)
File "/home/vagrant/agentless-system-crawler/crawler/utils/dockerutils.py", line 74, in exec_dockerinspect
inspect = client.inspect_container(long_id)
File "/usr/local/lib/python2.7/dist-packages/docker/utils/decorators.py", line 21, in wrapped
return f(self, resource_id, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/docker/api/container.py", line 197, in inspect_container
self._get(self._url("/containers/{0}/json", container)), True
File "/usr/local/lib/python2.7/dist-packages/docker/client.py", line 178, in _result
self._raise_for_status(response)
File "/usr/local/lib/python2.7/dist-packages/docker/client.py", line 173, in _raise_for_status
raise errors.NotFound(e, response, explanation=explanation)
docker.errors.NotFound: 404 Client Error: Not Found ("{"message":"No such container: aa8f6e35414553726334bea41e1b1d566549ec98abf32a62a2fd15d90275d01b"}")

How to Reproduce

Log Output

Debugging Commands Output

Output of docker version:

(paste your output here)

Output of docker info:

(paste your output here)

Output of python --version:

(paste your output here)

Output of pip freeze:

(paste your output here)