Deploy & Test Crawlers on Armada Env

[canturkisci]

Description

We should verify the deployment of crawler in the armada env.

How to Reproduce

• Use crawler deployment scripts to deploy reliably on an armada cluster.
• Show that crawler can emit frames correctly to a SAS/VA endpoint

[canturkisci]

@nadgowdas please update as you make progress on this. it would be good to get to some progress before our next checkpoint.

[nadgowdas]

@canturkisci
I am able to deploy crawler on IBM's public container service and send the frame to sas endpoint.

Deployment setup looks something like follows:

I am able to see the crawl frames being collected at the kafka topic in SAS.
I am also able to crawl existing as well as any new containers/pod that are created on container service.