Deploy and run WMAgent with Docker container

[amaltaro]

Impact of the new feature
WMAgent

Is your feature request related to a problem? Please describe.
Once we have all the WMAgent dependencies sorted out, we should start looking into running WMAgent from a Docker image, making the process of updating our baseline code easier by just stopping the container and pulling a new one.

Describe the solution you'd like
With WMAgent package in PyPi, and a WMAgent docker image uploaded to Gitlab registry, we should complete the cycle such that we are able to run WMAgent using a Docker image, completing the deprecation and dependency on RPM packages.

This is a meta-issue for commissioning a Docker-based WMAgent environment for central production. It depends on the following sub-tasks:

• Automated packaging and upload of WMCore services to Pypi #11276
• Create base docker image for WMAgent based on Red Hat flavor #11565
• Build a WMAgent docker image #8797
• Separate WMAgent docker files between base vs application image #11647
• Create a script to deploy WMAgent docker image #11564
• Refactor/reorganize agent initialization and manage actions #11627
• Missing /etc/ && /bin/ && /src/ toplevel areas in the pypi packages #11583
• Investigate integration HTCondor + WMAgent in a container setting #11570

These are now planned for Q1/2024 (updated on Feb/8/2024):

• Tier0 WMAgent deployment model for Alma9/RHEL9 #11890
• WMAgent: Implement all Oracle functionalities needed for the WMAgent initialization  #11720
• WMAgent: install/run MariaDB from Dockerhub #11313
• WMAgent: install/run CouchDB from Dockerhub #11312
• Condor is not able to access runtime scripts from WMAgent container #11946

And the following shall be finished in the beginning of Q2/2024:

• Evaluate performance of MariaDB in host vs container mode #11566
• Evaluate performance of CouchDB in host vs container mode #11567
• Estimate the need for --network=host in WMAgeent Docker builds #11635
• WMAgent: Implement adding of opportunistic resources with the new initialization model #11722
• WMAgent: Switch the new deployment model to less restrictive initialization mode #11990
• Pre-production integration and validation of WMAgent containerization solution #11945
• Setup multiple service accounts for WMAgent containers #11944
• Update Jenkins WMCore baseline and PR-test to run based on WMAgent/CouchDB/MariaDB containers #11568
• Commission schedd production VO boxes on Alma 9 #11973
• Investigate alternatives to increase performance on containerized MariaDB service  #11977
• [Meta] Create a WMAgent/WMCore-dev image to run unittests #11978
• Ensure WM docker images - and databases - properly handle signals #11979
• WMAgent container using wrong proxy path for job submission #11987
• Find instances of install/wmagent/COMPONENT_NAME/ComponentLog and config/wmagent/config.py in WMCore scripts #11993
• Different behavior between VOMS server failing WMAgent container run #11999
• WMAgent container crontab not functional #12000
• MariaDB container 10.6.5 issues (slowness and deadlock) #12007
• WMAgent docker image running with old VOMS servers #12030
• etc/rucio.cfg gets misconfigured upon WMAgent tag update #12034

while the following tickets will be addressed in a later stage, as they are not critical for the functionality of the containerized solution:

• WMAgent: Implement Schema validation functionalities #11925
• WMAgent: Retire obsolate Mysql prefixes for the agent deployment initialization and management scripts #11934
• WMAgent: Add SqlCl client for Oracle based agents  #11927
• WMAgent: Move the creation of the metadata table wma_init under WMCore  #11721

Describe alternatives you've considered
None

Additional context
Related to:

• Make CA certificates available in our WMAgent test container Make CA certificates available in our WMAgent test container #10225
• Unable to start agent in Docker container Unable to start agent in Docker container #9675

Dependent on:
#11312
and
#11313

[vkuznet]

@amaltaro we do already have wmagent Dockerfile and image but we do not have wmagent configuration. If you'll provide its configuration in services_config gitlab repo then I can test it in dev cluster.

[amaltaro]

Valentin, I am afraid we don't have budget to work on it this quarter, hence it has not been even considered for Q1/2023.
We will get back to this in one of the next quarters.

[amaltaro]

Meta-issue updated with other relevant tickets created in the last week or two. Please let me know if anything is still missing in here. @vkuznet @todor-ivanov @khurtado

[vkuznet]

@amaltaro , I think we are missing the part of deployment on a specific infrastructure. For example, if we'll deploy WMA using docker compose, then we'll need specific manifest file(s), etc. On top of that we should make a decision which orchestration to use. For details between different solution I suggest to google it, e.g. here is nice comparison. Therefore, I would suggest to create a dedicated issue related to that.

[amaltaro]

@vkuznet Valentin, that is a good point! We will need to have a final manifest/template integrating all the objects/containers required for running WMAgent in a containerized mode. If you feel like creating it, please go ahead. Otherwise I will get to it tomorrow.

[amaltaro]

@todor-ivanov thank you for updating this meta issue with the new tickets resulted from:
dmwm/CMSKubernetes#1410

[todor-ivanov]

As we discussed yesterday during the WMCore meeting, based on my previous experience with this containerization effort, I can point which are the most crucial and with highest priority issues to be resolved in order for WMCore Team to deliver this new functionality in the shortest possible timeline. I'll also add my high level view/feedback on the rest of the issues, but this will need to be confirmed during the dedicated effort on resolving them.

The 3 most important issues with absolutely no alternative for a workaround solution are:

• WMAgent: Implement all Oracle functionalities needed for the WMAgent initialization  #11720 - All of the database functionalities which are needed here are already well tested and working in the context of MariaDB, but we must add the needed Oracle alternatives to the manage script. One possible obstacle could be, the creation of an initial database schema dump and recording it during the deployment procedures for later validation and/or recovery. But this I believe is solvable.

• WMAgent: install/run MariaDB from Dockerhub #11313 - This is a must as well, but we already have some markers and basic construction on how should this docker image look like here: Add minimal MariaDB container for WMAgent CMSKubernetes#1412 . The default configurations from my.cnf here and the MariaDB server startup script are confirmed to be working on a direct deployment at the host. I also remember having the MariaDB container built and tested locally, but without the relevant manage and run.sh scripts for automating its startup.

• WMAgent: install/run CouchDB from Dockerhub #11312 - This is also unavoidable, but again we already have a 99% completed PR resolving the issue: Add minimal CouchDB container for WMAgent CMSKubernetes#1409 The container, built based on this PR, is well tested locally and working, but it is not uploaded to CERN registry.

Here follows the set of issues with feedback, which can be resolved with lower than maximum priority (as I said earlier, what ever I state here needs to be confirmed with a dedicated investigation/checks on the topic):

• WMAgent: Move the creation of the metadata table wma_init under WMCore  #11721 - This is good to have in the context of code organization, but things are working at the moment, even with creation of the init table during deployment.

• WMAgent: Implement adding of opportunistic resources with the new initialization model #11722 - This is again good to have in order to avoid manual work for adding those resources upon agent deployment, but if we decide, for the time being, we can live with some extra commands which need to be executed at the agent before we start it in production, we can resolve this issue with a medium priority as well.

• Evaluate performance of MariaDB in host vs container mode #11566 - So far, I have not observed any signs of the agent's performance being somehow affected if MariaDB is run inside a container vs. directly at the host. Of course I did not have the chance to do a high scale load test, but I am confident, we will have no scalability issues in that regard.

• WMAgent: install/run CouchDB from Dockerhub #11312 - Same as above.

• Estimate the need for --network=host in WMAgeent Docker builds #11635 - The short answer to this issue is: Yes ,for the time being, we need to use the --network=host option for the WMAgent container at runtime, and NO it is not needed at container build time. At least until we decide to completely move to a fully distributed solution and deploy WMAgents in kuberenetes clusters. But from what I see, there is a long way until we reach to that point.

• Update Jenkins WMCore baseline and PR-test to run based on WMAgent/CouchDB/MariaDB containers #11568 - This is indeed a must for us to have a stable CI/CD process, but this for sure will come last. And how difficult would that effort be, given the resolution of all other issues, I'd let @amaltaro or @vkuznet take a l look and share their opinion.

FYI @klannon @amaltaro

[amaltaro]

Thank you for sharing these insights, Todor.

I will soon start moving a few of these items under the Q1 quarter then, following your suggestion of: #11720, #11313, #11312. In addition, I inclined to say that the CI/Jenkins pipeline (issue #11568) should be updated accordingly and before we actually implement this new model in production, otherwise I fear it will be easier to insert issues into production.
We also need to pay closer attention to the Tier0 agent model and make sure that whenever central production is ready to migrate to the latest OS, that Tier0 agent can come along as well. For that, I created this new (meta-)issue: #11890

#11635 - The short answer to this issue is: Yes ,for the time being, we need to use the --network=host option for the WMAgent container at runtime, and NO it is not needed at container build time. At least until we decide to completely move to a fully distributed solution and deploy WMAgents in kuberenetes clusters. But from what I see, there is a long way until we reach to that point.

This is an important information and we better update the GH issue itself with this. Can you please update it?

[todor-ivanov]

hi @amaltaro

I inclined to say that the CI/Jenkins pipeline (issue #11568) should be updated accordingly and before we actually implement this new model in production, otherwise I fear it will be easier to insert issues into production.

Well then, it makes it 4 of them to get in as high priority. I am still positive about the outcome.