HPCC-30306 Allow arbitrary script based plane validation

[jakesmith]

This can be used catch problems with plane mounts

Type of change:

• This change is a bug fix (non-breaking change which fixes an issue).
• This change is a new feature (non-breaking change which adds functionality).
• This change improves the code (refactor or other change that does not change the functionality)
• This change fixes warnings (the fix does not alter the functionality or the generated code)
• This change is a breaking change (fix or feature that will cause existing behavior to change).
• This change alters the query API (existing queries will have to be recompiled)

Checklist:

• My code follows the code style of this project.
  • My code does not create any new warnings from compiler, build system, or lint.
• The commit message is properly formatted and free of typos.
  • The commit message title makes sense in a changelog, by itself.
  • The commit is signed.
• My change requires a change to the documentation.
  • I have updated the documentation accordingly, or...
  • I have created a JIRA ticket to update the documentation.
  • Any new interfaces or exported functions are appropriately commented.
• I have read the CONTRIBUTORS document.
• The change has been fully tested:
  • I have added tests to cover my changes.
  • All new and existing tests passed.
  • I have checked that this change does not introduce memory leaks.
  • I have used Valgrind or similar tools to check for potential issues.
• I have given due consideration to all of the following potential concerns:
  • Scalability
  • Performance
  • Security
  • Thread-safety
  • Cloud-compatibility
  • Premature optimization
  • Existing deployed queries will not be broken
  • This change fixes the problem, not just the symptom
  • The target branch of this pull request is appropriate for such a change.
• There are no similar instances of the same problem that should be addressed
  • I have addressed them here
  • I have raised JIRA issues to address them separately
• This is a user interface / front-end modification
  • I have tested my changes in multiple modern browsers
  • The component(s) render as expected

Smoketest:

• Send notifications about my Pull Request position in Smoketest queue.
• Test my draft Pull Request.

Testing:

[github-actions]

https://track.hpccsystems.com/browse/HPCC-30306
Jira updated

[ghalliday]

@jakesmith a few comments/questions.

[ghalliday]

Should include a description about how this is used. Also is it worth restricting the element type:
"items": { "type": "string" }

Does it need an associated documentation jira. (And similar for exception hooks, are they documeted?)

[jakesmith]

exceptionHandler are documented in helm/hpcc/docs/expert.md - probably enough?

validatePlaneScript isn't - and it feels like an expert setting too, perhaps it should go under a new expert: level under the plane to make it clear it is of that ilk - and then be documented in expert.md ..

[jakesmith]

Should include a description about how this is used. Also is it worth restricting the element type:
"items": { "type": "string" }

will change and add a description

[ghalliday]

Should there be a ; after each command?

[jakesmith]

no, it's because the block scalar passes it as a script (complete with line breaks) which bash interprets as individual commands (as it would in a script.
e.g. it's the equivalent of:

/bin/bash -c "
echo 'command 1'
echo 'command 2'
echo 'command 3'
"

The example in the JIRA is generated as:

          initContainers:
          - args:
            - |
              echo "Hello, World!"
              ls -l
              df -h
              end_time=$((SECONDS + 1200)); while [ $SECONDS -lt $end_time ]; do if mountpoint -q '/var/lib/HPCCSystems/hpcc-data'; then device=$(df /var/lib/HPCCSystems/hpcc-data | awk 'NR==2 {print $1}'); if [[ "$device" == "/dev/md0" ]]; then echo "correct mount device"; exit 0; else echo "wrong mount device \"$device\", will look again!"; fi; fi; echo waiting for mount point; sleep 5; done; echo failed; exit 1
            command:
            - /bin/bash
            - -c

.. and executes as expected.

[ghalliday]

Would it be worth combining this into the wait for mount container if possible? What is the extra overhead starting a new container?

[jakesmith]

well both are a kludge, but I wouldn't see both being used in tandem.
I was talking to the person behind the TF module that provisions the raided nvme on the nodes, and he thinks that it may require a new CSI behind the persistent volume to correctly wait for the mount to be fully ready.
But in the short term, in the next release, he is adding a marker file in the mount this mechanism could look for.

All less than ideal, once k8s pods see the pvc, that should be it - we shouldn't have to wait or check like this, but I think the provisioning of these raided nvme's is unusual and not fully supported by AKS out of the box.

[ghalliday]

minor: Slightly strange inconsistency between this and the wait for mount container - sh v bash and the way -c is passed.

[jakesmith]

agree, but there's about 50% use cases of /bin/sh vs /bin/bash in various places in _helpers.tpl
It is the only place that has "-c" as part of the command list though, I'll move it to 1st arg to be more consistent.

bash is more rich, I think it's okay to use here - perhaps should unify on it elsewhere?

[jakesmith]

@ghalliday - please see responses and new commit.

[ghalliday]

@jakesmith please squash