arc 0.4.0 with service container or image fails on Failed to update oom_score_adj #2791

jandrunas · 2023-08-02T00:16:24Z

jandrunas
Aug 2, 2023

running a normal job works fine, but when trying to use a service container or specifying a container image in a work flow causes the job to fail with

Failed to update oom_score_adj for PID: 70. [WORKER 2023-08-01 23:48:59Z INFO ProcessInvokerWrapper] System.UnauthorizedAccessException: Access to the path '/proc/70/oom_score_adj' is denied. [WORKER 2023-08-01 23:48:59Z INFO ProcessInvokerWrapper] ---> System.IO.IOException: Permission denied [WORKER 2023-08-01 23:48:59Z INFO ProcessInvokerWrapper] --- End of inner exception stack trace ---

I even copied the exact example from the deep dive video in https://www.youtube.com/watch?v=_F5ocPrv6io

I can connect to the runner and the permissions on /proc look ok, but any attempt to do anything in /proc gives permissions denied even as root or with sudo

the jobs run just fine on hosted runners, and were running just fine using 0.27.4 with runner deployments

.github/workflows/stupid_test.yml

this fails

---
name: stupid_test
on:
  workflow_dispatch

jobs:
  build-and-test:
    runs-on: arc-runner-set
    container:
      image: ghcr.io/actions/actions-runner:latest
    steps:
      - name: Trials
        run: |
          echo "Hello World" >> $GITHUB_STEP_SUMMARY
          sleep 720
        env:
          REDIS_HOST: redis
          REDIS_PORT: 6379

this does not

---
name: stupid_test
on:
  workflow_dispatch

jobs:
  build-and-test:
    runs-on: arc-runner-set
    steps:
      - name: Trials
        run: |
          echo "Hello World" >> $GITHUB_STEP_SUMMARY
          sleep 720
        env:
          REDIS_HOST: redis
          REDIS_PORT: 6379

values.yaml

githubConfigUrl: "https://github.com/PollyEx"
githubConfigSecret: "controller-manager"
maxRunners: 20
minRunners: 1
containerMode:
  type: "kubernetes"
  kubernetesModeWorkVolumeClaim:
    accessModes: ["ReadWriteOnce"]
    storageClassName: "ebs"
    resources:
      requests:
        storage: 10Gi
        cpu: "1000m"
        memory: "2Gi"
template:
  spec:
    securityContext:
      fsGroup: 1001
    containers:
      - name: runner
        image: ghcr.io/actions/actions-runner:latest
        imagePullPolicy: Always
        command: ["/home/runner/run.sh"]
        env:
          - name: ACTIONS_RUNNER_CONTAINER_HOOKS
            value: /home/runner/k8s/index.js
          - name: ACTIONS_RUNNER_POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: ACTIONS_RUNNER_REQUIRE_JOB_CONTAINER
            value: "false"
        volumeMounts:
          - name: work
            mountPath: /home/runner/_work
        volumes:
          - name: work
            ephemeral:
              volumeClaimTemplate:
                spec:
                  accessModes: ["ReadWriteOnce"]
                  storageClassName: "ebs"
                  resources:
                    requests:
                      storage: 10Gi

ashishkurmi · 2023-09-27T02:09:34Z

ashishkurmi
Sep 27, 2023

I am experiencing this issue as well with the latest version of ARC when I run a GitHub Action step that uses a container:

[WORKER 2023-09-27 01:54:41Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6cb15225-bace-5236-3d64-8b6d88bf381f', success rate: 1/1.
[WORKER 2023-09-27 01:54:41Z INFO JobServerQueue] Try to append 1 batches web console lines for record '0bd5d730-eae6-5b54-0101-80d8a124934a', success rate: 1/1.
[WORKER 2023-09-27 01:54:41Z INFO JobServerQueue] Try to append 1 batches web console lines for record 'c64be972-a956-57ab-3a81-fb6cad8060ba', success rate: 1/1.
[WORKER 2023-09-27 01:54:41Z INFO ProcessInvokerWrapper] Starting process:
[WORKER 2023-09-27 01:54:41Z INFO ProcessInvokerWrapper]   File name: '/home/runner/externals/node16/bin/node'
[WORKER 2023-09-27 01:54:41Z INFO ProcessInvokerWrapper]   Arguments: '/home/runner/k8s/index.js'
[WORKER 2023-09-27 01:54:41Z INFO ProcessInvokerWrapper]   Working directory: '/home/runner/_work/link-arc/link-arc'
[WORKER 2023-09-27 01:54:41Z INFO ProcessInvokerWrapper]   Require exit code zero: 'False'
[WORKER 2023-09-27 01:54:41Z INFO ProcessInvokerWrapper]   Encoding web name:  ; code page: ''
[WORKER 2023-09-27 01:54:41Z INFO ProcessInvokerWrapper]   Force kill process on cancellation: 'False'
[WORKER 2023-09-27 01:54:41Z INFO ProcessInvokerWrapper]   Redirected STDIN: 'True'
[WORKER 2023-09-27 01:54:41Z INFO ProcessInvokerWrapper]   Persist current code page: 'False'
[WORKER 2023-09-27 01:54:41Z INFO ProcessInvokerWrapper]   Keep redirected STDIN open: 'False'
[WORKER 2023-09-27 01:54:41Z INFO ProcessInvokerWrapper]   High priority process: 'False'
[WORKER 2023-09-27 01:54:41Z INFO ProcessInvokerWrapper] Failed to update oom_score_adj for PID: 681.
[WORKER 2023-09-27 01:54:41Z INFO ProcessInvokerWrapper] System.UnauthorizedAccessException: Access to the path '/proc/681/oom_score_adj' is denied.
[WORKER 2023-09-27 01:54:41Z INFO ProcessInvokerWrapper]  ---> System.IO.IOException: Permission denied
[WORKER 2023-09-27 01:54:41Z INFO ProcessInvokerWrapper]    --- End of inner exception stack trace ---
[WORKER 2023-09-27 01:54:41Z INFO ProcessInvokerWrapper]    at System.IO.RandomAccess.WriteAtOffset(SafeFileHandle handle, ReadOnlySpan`1 buffer, Int64 fileOffset)
[WORKER 2023-09-27 01:54:41Z INFO ProcessInvokerWrapper]    at System.IO.Strategies.BufferedFileStreamStrategy.FlushWrite()
[WORKER 2023-09-27 01:54:41Z INFO ProcessInvokerWrapper]    at System.IO.Strategies.BufferedFileStreamStrategy.Dispose(Boolean disposing)
[WORKER 2023-09-27 01:54:41Z INFO ProcessInvokerWrapper]    at System.IO.StreamWriter.CloseStreamFromDispose(Boolean disposing)
[WORKER 2023-09-27 01:54:41Z INFO ProcessInvokerWrapper]    at System.IO.StreamWriter.Dispose(Boolean disposing)
[WORKER 2023-09-27 01:54:41Z INFO ProcessInvokerWrapper]    at System.IO.File.WriteAllText(String path, String contents)
[WORKER 2023-09-27 01:54:41Z INFO ProcessInvokerWrapper]    at GitHub.Runner.Sdk.ProcessInvoker.WriteProcessOomScoreAdj(Int32 processId, Int32 oomScoreAdj)
[WORKER 2023-09-27 01:54:41Z INFO ProcessInvokerWrapper] Process started with process id 681, waiting for process exit.
[WORKER 2023-09-27 01:54:41Z INFO ProcessInvokerWrapper] Close STDIN after the first redirect finished.
[WORKER 2023-09-27 01:54:41Z INFO ProcessInvokerWrapper] STDIN stream write finished.

1 reply

kmeratia Nov 25, 2023

Were you able to resolve this issue? any help is appreciated.

kmeratia · 2023-11-25T02:27:24Z

kmeratia
Nov 25, 2023

Were you able to resolve this issue? any help is appreciated.

0 replies

alexgnq · 2024-01-08T11:09:40Z

alexgnq
Jan 8, 2024

I'm also getting the same error using 0.8.1.

However Nikola seems to hint at it not being an issue:
#3132 (comment)

0 replies

asoltesz · 2024-02-02T17:58:19Z

asoltesz
Feb 2, 2024

I am also getting this error on 0.8.1 and the job fails with some kind of timeout.

Error on the GitHUB UI:

Error: connect ETIMEDOUT 172.30.0.1:443

It is possible that this is not the main reason.

What I can confirm that node with the index.js starts up and runs for a while.
Probably, index.js experiences some kind of problem.

It seems that there are no logs coming from index.js so we never learn the actual reason.

0 replies

sfiligoi · 2024-03-01T21:38:53Z

sfiligoi
Mar 1, 2024

Just noticed the same problem on our on-prem K8s cluster:

[2024-03-01 21:31:32Z INFO SelfUpdater] Validated Runner Hash matches /mnt/scratch/builder/_update/runner1.tar.gz : 56910d6628b41f99d9a1c5fe9df54981ad5d8c9e42fc14899dcc177e222e71c4
[2024-03-01 21:31:33Z INFO SelfUpdater] Which: 'tar'
[2024-03-01 21:31:33Z INFO SelfUpdater] Location: '/usr/bin/tar'
[2024-03-01 21:31:33Z INFO ProcessInvokerWrapper] Starting process:
[2024-03-01 21:31:33Z INFO ProcessInvokerWrapper]   File name: '/usr/bin/tar'
[2024-03-01 21:31:33Z INFO ProcessInvokerWrapper]   Arguments: '-xzf "/mnt/scratch/builder/_update/runner1.tar.gz"'
[2024-03-01 21:31:33Z INFO ProcessInvokerWrapper]   Working directory: '/mnt/scratch/builder/_update'
[2024-03-01 21:31:33Z INFO ProcessInvokerWrapper]   Require exit code zero: 'False'
[2024-03-01 21:31:34Z INFO ProcessInvokerWrapper]   Encoding web name:  ; code page: ''
[2024-03-01 21:31:34Z INFO ProcessInvokerWrapper]   Force kill process on cancellation: 'False'
[2024-03-01 21:31:34Z INFO ProcessInvokerWrapper]   Redirected STDIN: 'False'
[2024-03-01 21:31:34Z INFO ProcessInvokerWrapper]   Persist current code page: 'False'
[2024-03-01 21:31:34Z INFO ProcessInvokerWrapper]   Keep redirected STDIN open: 'False'
[2024-03-01 21:31:34Z INFO ProcessInvokerWrapper]   High priority process: 'False'
[2024-03-01 21:31:34Z INFO ProcessInvokerWrapper] Failed to update oom_score_adj for PID: 73.
[2024-03-01 21:31:34Z INFO ProcessInvokerWrapper] System.UnauthorizedAccessException: Access to the path '/proc/73/oom_score_adj' is denied.
 ---> System.IO.IOException: Permission denied
   --- End of inner exception stack trace ---
   at System.IO.RandomAccess.WriteAtOffset(SafeFileHandle handle, ReadOnlySpan`1 buffer, Int64 fileOffset)
   at System.IO.Strategies.BufferedFileStreamStrategy.FlushWrite()
   at System.IO.Strategies.BufferedFileStreamStrategy.Dispose(Boolean disposing)
   at System.IO.StreamWriter.CloseStreamFromDispose(Boolean disposing)
   at System.IO.StreamWriter.Dispose(Boolean disposing)
   at System.IO.File.WriteAllText(String path, String contents)
   at GitHub.Runner.Sdk.ProcessInvoker.WriteProcessOomScoreAdj(Int32 processId, Int32 oomScoreAdj)
[2024-03-01 21:31:36Z INFO ProcessInvokerWrapper] Process started with process id 73, waiting for process exit.

2 replies

aiell0 May 23, 2024

Anyone been able to fix this? Also seeing this issue.

trenslow Oct 15, 2024

seeing this same issue on runner version 0.8.2

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

arc 0.4.0 with service container or image fails on Failed to update oom_score_adj #2791

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 5 comments 3 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

arc 0.4.0 with service container or image fails on Failed to update oom_score_adj #2791

Replies: 5 comments · 3 replies

Replies: 5 comments 3 replies