IMAGE SCAN Job with pending executions

[leociociano]

Infrastructure:
Harbor Version v2.9.4-a6d707df
Running in K3S v1.25.3+k3s1

Error:
Since last night, there are IMAGE SCAN pending executions:

More Info:
I get the following logs from Database, Job Service, Trivy and Core services. There are several "unauthorized" errors but there was no perms or user modifications in the past days. Also, this is a Replicated Data from another Harbor (same version) and no errors found in the primary one. Projects in both Harbor deployments have the same perms.

In the Database I get ERROR: update or delete on table "execution" violates foreign key constraint "task_execution_id_fkey" on table "task", which is not related to authorization (as far as I know).

I've restarted every pod just in case. Still get the error.

There is no High CPU or Memory usage that can bring latency to the process.

What I expect to happen:
Image Scan executes succesfully without pending executions.

LOGS:

JobService

2024-10-21T16:37:50Z [INFO] [/jobservice/worker/cworker/c_worker.go:77]: Job incoming: {"name":"IMAGE_SCAN","id":"9509d0295a3072d263e24c11","t":1729429399,"args":null}
2024-10-21T16:37:50Z [INFO] [/jobservice/runner/redis.go:196]: Retrying job IMAGE_SCAN:9509d0295a3072d263e24c11, revision: 1729528670
2024-10-21T16:37:50Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: http://harbor-core:80/api/v2.0/internalconfig
2024-10-21T16:37:50Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: http://harbor-core:80/api/v2.0/internalconfig
2024-10-21T16:37:54Z [ERROR] [/jobservice/runner/redis.go:122]: Job 'IMAGE_SCAN:86e2d0241f49c75c79b6dfcd' exit with error: run error: check scan report with mime type application/vnd.security.vulnerability.report; version=1.1: running trivy wrapper: running trivy: exit status 1: 2024-10-21T16:37:50.458Z INFO Vulnerability scanning is enabled
2024-10-21T16:37:53.574Z FATAL image scan error: scan error: unable to initialize a scanner: unable to initialize an image scanner: 5 errors occurred:
* docker error: unable to inspect the image (harbor-core:80/project1/artifact2@sha256:7b543129f90cb8003b6c2a3bc07c10b3736f085faf784f59ccf89e601a90fa1e): Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
* containerd error: containerd socket not found: /run/containerd/containerd.sock
* podman error: unable to initialize Podman client: no podman socket found: stat podman/podman.sock: no such file or directory
* remote error: GET http://harbor-core:80/v2/project1/artifact2/manifests/sha256:7b543129f90cb8003b6c2a3bc07c10b3736f085faf784f59ccf89e601a90fa1e: UNAUTHORIZED: unauthorized to access repository: project1/artifact2, action: pull: unauthorized to access repository: project1/artifact2, action: pull
* remote error: GET http://harbor-core:80/v2/project1/artifact2/manifests/sha256:7b543129f90cb8003b6c2a3bc07c10b3736f085faf784f59ccf89e601a90fa1e: UNAUTHORIZED: unauthorized to access repository: project1/artifact2, action: pull: unauthorized to access repository: project1/artifact2, action: pull
: general response handler: unexpected status code: 500, expected: 200

---

Core

2024-10-21T16:38:47Z [INFO] [/pkg/task/dao/execution.go:471]: scanned out 1 executions with outdate status, refresh status to db
2024-10-21T16:38:47Z [INFO] [/pkg/task/dao/execution.go:512]: refresh outdate execution status done, 1 succeed, 0 failed
2024-10-21T16:38:48Z [ERROR] [/controller/scan/callback.go:97][error="robot account 1858391 not found" requestID="cbe871ab-88ca-429e-a7f8-b9d8cb616ea1" robot_id="1858391" status="Error" task_id="2325644"]: delete robot account failed
2024-10-21T16:38:49Z [WARNING] [/core/auth/ldap/ldap.go:73]: Not found an entry.
2024-10-21T16:38:49Z [WARNING] [/core/auth/authenticator.go:158]: Login failed, locking robot$project2+scanner-Trivy-af827656-8ee3-11ef-b4b6-d6838c791338, and sleep for 1.5s
2024-10-21T16:38:50Z [ERROR] [/server/middleware/security/basic_auth.go:72][client IP="10.42.0.199:42546" requestID="4ef1348a-4bde-493a-8b19-42706425890d" user agent="go-containerregistry/v0.19.0"]: failed to authenticate user:robot$project2+scanner-Trivy-af827656-8ee3-11ef-b4b6-d6838c791338, error:Failed to authenticate user, due to error 'Not found an entry'
2024-10-21T16:38:51Z [WARNING] [/core/auth/ldap/ldap.go:73]: Not found an entry.
2024-10-21T16:38:51Z [WARNING] [/core/auth/authenticator.go:158]: Login failed, locking robot$project2+scanner-Trivy-af827656-8ee3-11ef-b4b6-d6838c791338, and sleep for 1.5s
2024-10-21T16:38:52Z [ERROR] [/controller/scan/callback.go:97][error="robot account 1858392 not found" requestID="2c3d1edd-24e4-435c-b7a0-720e671cd045" robot_id="1858392" status="Success" task_id="2325645"]: delete robot account failed
2024-10-21T16:38:52Z [ERROR] [/server/middleware/security/basic_auth.go:72][client IP="10.42.0.199:42548" requestID="c7a69de8-445b-4402-8d87-97e37ba60761" user agent="go-containerregistry/v0.19.0"]: failed to authenticate user:robot$project2+scanner-Trivy-af827656-8ee3-11ef-b4b6-d6838c791338, error:Failed to authenticate user, due to error 'Not found an entry'
2024-10-21T16:38:54Z [ERROR] [/controller/scan/callback.go:97][error="robot account 1858393 not found" requestID="3f0a6212-ce35-4315-9516-6bb41882d8d5" robot_id="1858393" status="Error" task_id="2325646"]: delete robot account failed

---

Trivy

{"time":"2024-10-21T16:39:36.238943734Z","level":"ERROR","msg":"Running trivy failed","image_ref":"harbor-core:80/project1/artifact1@sha256:39c3289ff38d3243d275ba07adef965bfe44d124e23191e43a16c860d32395ba","exit_code":"1","std_out":"2024-10-21T16:39:33.092Z\t\u001b[34mINFO\u001b[0m\tVulnerability scanning is enabled\n2024-10-21T16:39:36.229Z\t\u001b[31mFATAL\u001b[0m\timage scan error: scan error: unable to initialize a scanner: unable to initialize an image scanner: 5 errors occurred:\n\t* docker error: unable to inspect the image (harbor-core:80/project1/artifact1@sha256:39c3289ff38d3243d275ba07adef965bfe44d124e23191e43a16c860d32395ba): Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?\n\t* containerd error: containerd socket not found: /run/containerd/containerd.sock\n\t* podman error: unable to initialize Podman client: no podman socket found: stat podman/podman.sock: no such file or directory\n\t* remote error: GET http://harbor-core:80/v2/project1/artifact1/manifests/sha256:39c3289ff38d3243d275ba07adef965bfe44d124e23191e43a16c860d32395ba: UNAUTHORIZED: unauthorized to access repository: project1/artifact1, action: pull: unauthorized to access repository: project1/artifact1, action: pull\n\t* remote error: GET http://harbor-core:80/v2/project1/artifact1/manifests/sha256:39c3289ff38d3243d275ba07adef965bfe44d124e23191e43a16c860d32395ba: UNAUTHORIZED: unauthorized to access repository: project1/artifact1, action: pull: unauthorized to access repository: project1/artifact1, action: pull\n\n\n"}
{"time":"2024-10-21T16:39:36.23925669Z","level":"ERROR","msg":"Scan failed","err":"running trivy wrapper: running trivy: exit status 1: 2024-10-21T16:39:33.092Z\t\u001b[34mINFO\u001b[0m\tVulnerability scanning is enabled\n2024-10-21T16:39:36.229Z\t\u001b[31mFATAL\u001b[0m\timage scan error: scan error: unable to initialize a scanner: unable to initialize an image scanner: 5 errors occurred:\n\t* docker error: unable to inspect the image (harbor-core:80/project1/artifact1@sha256:39c3289ff38d3243d275ba07adef965bfe44d124e23191e43a16c860d32395ba): Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?\n\t* containerd error: containerd socket not found: /run/containerd/containerd.sock\n\t* podman error: unable to initialize Podman client: no podman socket found: stat podman/podman.sock: no such file or directory\n\t* remote error: GET http://harbor-core:80/v2/project1/artifact1/manifests/sha256:39c3289ff38d3243d275ba07adef965bfe44d124e23191e43a16c860d32395ba: UNAUTHORIZED: unauthorized to access repository: project1/artifact1, action: pull: unauthorized to access repository: project1/artifact1, action: pull\n\t* remote error: GET http://harbor-core:80/v2/project1/artifact1/manifests/sha256:39c3289ff38d3243d275ba07adef965bfe44d124e23191e43a16c860d32395ba: UNAUTHORIZED: unauthorized to access repository: project1/artifact1, action: pull: unauthorized to access repository: project1/artifact1, action: pull\n\n\n"}

---

Database

2024-10-21 15:02:51.862 UTC [154] ERROR: update or delete on table "execution" violates foreign key constraint "task_execution_id_fkey" on table "task"
2024-10-21 15:02:51.862 UTC [154] DETAIL: Key (id)=(28921) is still referenced from table "task".
2024-10-21 15:02:51.862 UTC [154] STATEMENT: DELETE FROM execution WHERE id IN ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13,$14,$15,$16,$17,$18,$19,$20,$21,$22,$23,$24,$25,$26,$27,$28,$29,$30,$31,$32,$33,$34,$35,$36,$37,$38,$39)
2024-10-21 16:00:45.675 UTC [7398] ERROR: update or delete on table "execution" violates foreign key constraint "task_execution_id_fkey" on table "task"
2024-10-21 16:00:45.675 UTC [7398] DETAIL: Key (id)=(28921) is still referenced from table "task".
2024-10-21 16:00:45.675 UTC [7398] STATEMENT: DELETE FROM execution WHERE id IN ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13,$14,$15,$16,$17,$18,$19,$20,$21,$22,$23,$24,$25,$26,$27,$28,$29,$30,$31,$32,$33,$34,$35,$36,$37,$38,$39)
2024-10-21 16:30:27.800 UTC [10942] LOG: could not send data to client: Connection reset by peer
2024-10-21 16:30:27.800 UTC [10942] STATEMENT: SELECT T0."id", T0."cve_id", T0."registration_uuid", T0."package", T0."package_version", T0."package_type", T0."severity", T0."fixed_version", T0."urls", T0."cvss_score_v3", T0."cvss_score_v2", T0."cvss_vector_v3", T0."cvss_vector_v2", T0."description", T0."cwe_ids", T0."vendor_attributes" FROM "vulnerability_record" T0 WHERE T0."cve_id" IN ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17, $18, $19, $20, $21, $22, $23, $24, $25, $26, $27, $28, $29, $30, $31, $32, $33, $34, $35, $36, $37, $38, $39, $40, $41, $42, $43, $44, $45, $46, $47, $48, $49, $50, $51, $52, $53, $54, $55, $56, $57, $58, $59, $60, $61, $62, $63, $64, $65, $66, $67, $68, $69, $70, $71, $72, $73, $74, $75, $76, $77, $78, $79, $80, $81, $82, $83, $84, $85, $86, $87, $88, $89, $90, $91, $92, $93, $94, $95, $96, $97, $98, $99, $100, $101, $102, $103, $104, $105, $106, $107, $108, $109, $110, $111, $112, $113, $114, $115, $116, $117, $118, $119, $120, $121, $122, $123, $124, $125, $126, $127, $128, $129, $130, $131, $132, $133, $134, $135, $136, $137, $138, $139, $140, $141, $142, $143, $144, $145, $146, $147, $148, $149, $150, $151, $152, $153, $154, $155, $156, $157, $158, $159, $160, $161, $162, $163, $164, $165, $166, $167, $168, $169, $170, $171, $172, $173, $174, $175, $176, $177, $178, $179, $180, $181, $182, $183, $184, $185, $186, $187, $188, $189, $190, $191, $192, $193, $194, $195, $196, $197, $198, $199, $200, $201, $202, $203, $204, $205, $206, $207, $208, $209, $210, $211, $212, $213, $214, $215, $216, $217, $218, $219, $220, $221, $222, $223, $224, $225, $226, $227, $228, $229, $230, $231, $232, $233, $234, $235, $236, $237, $238, $239, $240, $241, $242, $243, $244, $245, $246, $247, $248, $249, $250, $251, $252, $253, $254, $255, $256, $257, $258, $259, $260, $261, $262, $263, $264, $265, $266, $267, $268, $269, $270, $271, $272, $273, $274, $275, $276, $277, $278, $279, $280, $281, $282, $283, $284, $285, $286, $287, $288, $289, $290, $291, $292, $293, $294, $295, $296, $297, $298, $299, $300, $301, $302, $303, $304, $305, $306, $307, $308, $309, $310, $311, $312, $313, $314, $315, $316, $317, $318, $319, $320, $321, $322, $323, $324, $325, $326, $327, $328, $329, $330, $331, $332, $333, $334, $335, $336, $337, $338, $339, $340, $341, $342, $343, $344, $345, $346, $347, $348, $349, $350, $351, $352, $353, $354, $355, $356, $357, $358, $359, $360, $361, $362, $363, $364, $365, $366, $367, $368, $369, $370, $371, $372, $373, $374, $375, $376, $377, $378, $379, $380, $381, $382, $383, $384, $385, $386, $387, $388, $389, $390, $391, $392, $393, $394, $395, $396, $397, $398, $399, $400, $401, $402, $403, $404, $405, $406, $407, $408, $409, $410, $411, $412, $413, $414, $415, $416, $417, $418, $419, $420, $421, $422, $423, $424, $425, $426, $427, $428, $429, $430, $431, $432, $433, $434, $435, $436, $437, $438, $439, $440, $441, $442, $443, $444, $445, $446, $447, $448, $449, $450, $451, $452, $453, $454, $455, $456, $457, $458, $459, $460, $461, $462, $463, $464, $465, $466, $467, $468, $469, $470, $471, $472, $473, $474, $475, $476, $477, $478, $479, $480, $481, $482, $483, $484, $485, $486, $487, $488, $489, $490, $491, $492, $493, $494, $495, $496, $497, $498, $499, $500, $501, $502, $503, $504, $505, $506, $507, $508, $509, $510, $511, $512, $513, $514, $515, $516, $517, $518, $519, $520) AND T0."registration_uuid" = $521
2024-10-21 16:30:27.800 UTC [10942] FATAL: connection to client lost