Implement CI_JOB_TOKEN, so private includes can be accessed

[cs35-owncloud]

Hello,

sorry if I created this issue as a bug as it may be more of a feature request.
I've read the documentation, mostly the quirks section and this commit

Minimal .gitlab-ci.yml illustrating the issue

---
include:
  - project: 'ansible_hosting/ansible_config'
    file: '/ci/gitlab-ci-templates/.gitlab-ci.yml'
    ref: master

In the variables I specified a gitlab token:

~/.gitlab-ci-local/variables.yml

---
global:
  ANSIBLE_HOSTING_DEV_RO_USER: ro_token
  ANSIBLE_HOSTING_DEV_RO_TOKEN: xxxx

I'm using the above .gitlab-ci.yml on several projects to avoid rewriting the core ci in every projects.

Expected behavior
This should fetch the /ci/gitlab-ci-templates/.gitlab-ci.yml of the project ansible_hosting/ansible_config
It seems that gitlab-ci-local find the correct URL (probably because my current project come from the same gitlab URL) but it cannot use the token and prompts for a password:

[email protected]'s password: 

Is it possible to use this feature with gitlab's token ?

I also tried this version:

Minimal .gitlab-ci.yml illustrating the issue

---
include:
  - remote: 'https://gitlab.xxx.com/api/v4/projects/xxxx/repository/files/ci%2Fgitlab-ci-templates%2F.gitlab-ci%2Eyml/raw'

It doesn't work either and this time I get this error:

Remote include could not be fetched https://gitlab.xxx.com/api/v4/projects/xxx/repository/files/ci%2Fgitlab-ci-templates%2F.gitlab-ci%2Eyml/raw AxiosError: Request failed with status code 404

If I try to download the file with curl and the above token, it works.

curl  --location --header "PRIVATE-TOKEN: xxx" https://gitlab.xxx.com/api/v4/projects/xxx/repository/files/ci%2Fgitlab-ci-templates%2F.gitlab-ci%2Eyml/raw

If I create a public snippet on my gitlab, the include remote will work but it's not really a suitable solution for me.
Is there any other way so I can grab my ci file remotely with gitlab-ci-local ?
Thanks.

Host information
Ubuntu
gitlab-ci-local 4.38.1

Containerd binary
Docker version 24.0.5, build ced0996l.

Thanks !

[firecow]

I wonder how Gitlab CI knows that the remote include, which is basicly just a curl request needs a gitlab private-token.

What if the url was not gitlab.<company>.com, but pkg.example.com

They must be doing some sort of matching to find that the URL contains subdomain.domain matching the gitlab instance.

[cs35-owncloud]

Thanks for your fast answer, in my case I have a self hosted gitlab.
I guess it's checking the remote (git remote -v) and simply take the same domain to fetch.
That's a good question about the private-token 🤔
Maybe since it's in the same "context" and going through a runner that's already "logged in" it's bypassing the auth.
I will try to reach a ci file from an unauthorized project.

[cs35-owncloud]

OK that's weird.
So I created a project in a specific namespace with a private visibility.
In my other project on a different namespace, I defined the .gitlab-ci.yml to include the .gitlab-ci.yml from the first project.
And it works, the other project's ci got injected and started to run.

[firecow]

Well, if your user has access to both projects, I'm guessing the pipeline is generating a tmp token impersonating your access permissions, so you need to include a "ci-module" that your user don't have access to.

[cs35-owncloud]

OK I checked and it's because my user has admin access, I would have expected that the permissions were about the projects and not the users when you run the pipeline.

[bcouetil]

Indeed the access (or not) to remote include/trigger depends exclusively on the person launching the pipeline 😉

[firecow]

We need to implement a similar token feature that requires users to put a CI_JOB_TOKEN in their gcl variables.

This token will then be used by include fetching if git remote "matches" include url.

[cs35-owncloud]

It would be the best :)
Getting the base gitlab URL in this case and clone it would be the dream:

---
include:
  - project: 'ansible_hosting/ansible_config'
    file: '/ci/gitlab-ci-templates/.gitlab-ci.yml'
    ref: master

[alexey-vostrikov]

I have a workaround this problem manually

1. I define a variable in .gitlab-ci.yml

variables:
  JT: "JOB-TOKEN:"

2. I override variable for gitlab-ci-local in ~/.gitlab-ci-local/variables.yml

global:
  JT: "PRIVATE-TOKEN:"

3. I set a value of CI_JOB_TOKEN to my private token
   in ~/.gitlab-ci-local/variables.yml

group:
  gitlab.organization.local/:
    AUTHORIZATION_PASSWORD: "glpat-XxXx"
    CI_SERVER_PROTOCOL: "https"
    CI_SERVER_PORT: "443"
project:
  gitlab.organization.local/group/project:
    CI_PROJECT_ID: "actual project ID from gitlab copied manually"
global:
  CI_SERVER_URL: "${CI_SERVER_PROTOCOL}://${CI_SERVER_HOST}:${CI_SERVER_PORT}"
  CI_API_V4_URL: "${CI_SERVER_URL}/api/v4"
  CI_JOB_TOKEN: "${AUTHORIZATION_PASSWORD}"
  JT: "PRIVATE-TOKEN:"

4. I use following construction in scripts

job:
  script:
    - curl -H "${JT} ${CI_JOB_TOKEN}" --url "${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/..."

Also includes are working