Integrate CLA Bot with FINOS Pure/Alloy instance of GitLab

[mcleo-d]

Description

FINOS needs the GitLab CLA Bot team to help integrate the system into a private instance of GitLab to help complete the following development story ...

• Enable CLA Bot to work in Gitlab akin to how it works in Github finos/open-developer-platform#15

Contact Details

Please leave a comment on finos/open-developer-platform#15 when you're ready to engage or contact the following FINOS team members direct ...

• https://github.com/maoo
• https://github.com/mcleo-d
• https://github.com/brooklynrob

[maoo]

Yay, first successful deployment! I'm adding below the list of operations (and GitLab/AWS configurations) I performed, would be great to add them to the project README.md.

I'm going to play a bit with the bot now, so I can find a way to configure it for the Alloy initiative.

Please find below the installation steps I followed and some simple suggestions to make deployment process smoother.

---

AWS Setup

1. Create a user on AWS via IAM dashboard (only enable programmatic access)
2. Assign AWSLambdaFullAccess, IAMFullAccess, AWSCloudFormationFullAccess, AmazonAPIGatewayInvokeFullAccess and AmazonS3FullAccess to the new user
3. Copy Access key ID and Secrect access key

Create GitLab User

1. Create a new user (ie "CLA Bot")
2. Create an Impersonation Token with scopes api, read_user and read_repository

Checkout code and deploy

git clone [email protected]:ScottLogic/gitlab-cla-bot.git
cd gitlab-cla-bot

npm install -g serverless

# Make sure to backup the ~/.aws folder, before running the next command
serverless config credentials --provider aws --key <Access key ID> --secret <Secrect access key>

# Set the GitLab Token in serverless.yml
sed -i.bak 's/YOUR_TOKEN_HERE/<GitLab Token>/g' serverless.yml

# Choose a unique S3 name
sed -i.bak 's/gitlab-cla-bot-logs/<New/unique S3 bucket name>/g' serverless.yml

# Choose your AWS region
sed -i.bak 's/us-east-2/<AWS region of choice>/g' serverless.yml

# Set the GitLab instance to use (if not gitlab.com)
sed -i.bak 's/gitlab.com/<GitLab instance domain of choice>/g' serverless.yml

# Deploy using AWS SAM and copy the POST URL reported under "endpoints:"
serverless deploy

Set GitLab WebHook

1. Access GitLab Project
2. Navigate Settings > Integrations menu
3. Paste the POST URL endpoint in the URL field
4. Paste the GitLab user Impersonation Token in the Secret Token field
5. Check Comment and Merge request events triggers
6. Click on Add webhook button

GitLab project setup

1. Create a GitLab project
2. Add the CLA Bot user (created above) as project Member, with Auditor role
3. Add a new file called contributors.json, with the list of allowed contributors for the project; copy the raw file URL
4. Add a new file called .clabot ; paste the content from default.json and add "contributors": "<contributors.json raw URL>", on line 2

Test the CLA bot

1. Submit a Merge Request against the project created above
2. Access AWS CloudWatch > Logs > Log Groups > /aws/lambda/gitlab-example-dev-handler
3. Check logs

Known issues

• If you get a 401 trying to access GitLab merge requests API, you may have a wrong Secret Token configured (as GITLAB_ACCESS_TOKEN) in you serverless.yml file.

Suggestions and improvements

• Rename serverless.yml to serverless.yml.sample and add serverless.yml to .gitignore
• Add .serverless to .gitignore
• Add detailed instructions in the README.md re. GitLab/AWS setup (see above)
• Debug IAM Permissions to find the smallest (and most fine-grained) amount of permissions that must be granted to the IAM user (see above, AWS Setup, step Check info retrievable from gitlab matches what we need #2)

[maoo]

Discovered a common issue with SAM; on every serverless deploy command, I get a:

  An error occurred: GitlabClaBotLoggingBucket - finos-gitlab-cla-bot-logs already exists.

Found a simple solution on https://www.npmjs.com/package/serverless-plugin-existing-s3 and submitted a PR on #36

[maoo]

I'm stuck on a 401 error logged in CloudWatch, when I hit Test while setting the GitLab Webhook (see step above)

    "errorType": "Error",
    "errorMessage": "Error: API request https://gitlab.com/api/v4/projects/14/merge_requests/1 failed with status 401",
    "stack": [
        "Error: Error: API request https://gitlab.com/api/v4/projects/14/merge_requests/1 failed with status 401",
        "    at _homogeneousError (/var/runtime/CallbackContext.js:13:12)",
        "    at postError (/var/runtime/CallbackContext.js:30:54)",
        "    at callback (/var/runtime/CallbackContext.js:42:7)",
        "    at /var/runtime/CallbackContext.js:105:16",
        "    at Runtime.handler (/var/task/src/index.js:71:5)",
        "    at processTicksAndRejections (internal/process/task_queues.js:94:5)"

I tried to:

1. Make the CLA Bot GitLab user an admin
2. Add sudo scope to the GitLab token created

But none of the solutions seem to work; any idea what could be the issue?

[maoo]

401 is fixed, thanks to @mcleo-d for spotting the issue! Here's the PR - #37 , which allows to configure the GitLab instance domain (in our case, it's not gitlab.com, updated configuration docs above)

[maoo]

@ColinEberhardt @sehaswell - I did some good progress with the deployment, see #35 (comment) . However, I still get an error: Error: A mechanism for verifying contributors has not been specified.

Maybe I'm using the wrong configuration in .clabot as pointer to the contributors list? I'm currently using "contributorListUrl": "<contributors.json raw URL>",, could you please confirm that is correct?

Thank you in advance!

[sehaswell]

... I still get an error: Error: A mechanism for verifying contributors has not been specified.

Maybe I'm using the wrong configuration in .clabot as pointer to the contributors list? I'm currently using "contributorListUrl": "<contributors.json raw URL>",, could you please confirm that is correct?

Thank you in advance!

Hi @maoo,
Firstly thanks for your PRs :)

The contributionVerifier module is next on my list of things to unit test, but I can see that it is looking for an object property "contributors" and then checking if the value is a URL so I suspect changing "contributorListUrl" to "contributors" might do the trick, or at least move you on to the next bug.

For info I have mostly been testing against an array of Gitlab usernames that looks like:

{
  "contributors": ["userOne"]
}

[maoo]

Thanks @sehaswell , that worked! And I got a cla-signed label, yay!

I had to use a gist file, because when I tried to use a file hosted by a GitLab repository, I got a 406:

Error: API request https://gitlab.alloy.finos.org/maoo/mao-rob-test/raw/master/contributors.json failed with status 406

Let me know if you experience this issue, meanwhile I'll contact GitLab support, as it may be something related with our installation.

Question, is it possible to add basic authentication to that call, ie adding a username and password environment variables that can be wired in the call? This way, I could host the contributors.json file in a private GitHub repository (as we currently do for the cla-bot).

Thanks!

[maoo]

Here's the gist I used as contributors list - https://gist.github.com/maoo/2b9c9deec284621940656cb22674ef37

[sehaswell]

Glad you've got there :)
The 406 sounds like something that could be fixable with the right headers - I don't know if Richard tested out this retrieval method much, so it's possible that twenty minutes and a bit of playing with Postman would give us a solution.

[maoo]

Thanks @sehaswell ! Please let me know if you manage to fetch the contributors list from a private GitLab repo; at FINOS we're currently using a public GitHub URL (see #38), which is not a viable solution.

[maoo]

@sehaswell @strangerich - any update on this? I'm currently using a private GitLab repo to host the contributions.json file, but I still get the 406 ; did you have any luck using this setup? TIA