[BUG]: Finalize Job does not complete

[trichard-jabra]

What happened?

Our agents started hanging on the Finalize job step our pipelines since Wednesday September 26. Tried updating agents, re-installed agents. I tried with empty pipeline, runs for 5 seconds and tries to upload something until timeout (60 minutes).
Error is
Information, ArtifactHttpRetryMessageHandler.SendAsync: https://redacted.vsblob.visualstudio.com/redacted/_apis/blob/blobs/redacted attempt 1/6 failed with StatusCode InternalServerError, IsRetryableResponse True

Versions

Started on version 3.241.0 tested with 3.243.0, 3.243.1 and 3.244.1

Environment type (Please select at least one enviroment where you face this issue)

• Self-Hosted
• Microsoft Hosted
• VMSS Pool
• Container

Azure DevOps Server type

dev.azure.com (formerly visualstudio.com)

Azure DevOps Server Version (if applicable)

No response

Operation system

macOS 14.6.1 and 15.0

Version controll system

git

Relevant log output

[2024-09-26 08:38:54Z WARN VisualStudioServices] Attempt 2 of POST request to https://vsblobprodsu6weu.vsblob.visualstudio.com/A377fce0f-c74c-4ec5-b81c-fe89f488285a/_apis/blob/blobs/79665E6B7D4013C4960152C96F1D0B3BCA1B6399E35DD81636E596687B4C20BB00 failed (HTTP Status: InternalServerError, AFD Ref: Ref A: D09204BDF83149408F5E83745860BC00 Ref B: CPH30EDGE0617 Ref C: 2024-09-26T08:38:54Z). The operation will be retried in 13.1685756 seconds.
[2024-09-26 08:39:07Z WARN VisualStudioServices] Attempt 3 of POST request to https://vsblobprodsu6weu.vsblob.visualstudio.com/A377fce0f-c74c-4ec5-b81c-fe89f488285a/_apis/blob/blobs/79665E6B7D4013C4960152C96F1D0B3BCA1B6399E35DD81636E596687B4C20BB00 failed (HTTP Status: InternalServerError, AFD Ref: Ref A: 2377B567BA4249D98ED5280628A1FD7E Ref B: CPH30EDGE0617 Ref C: 2024-09-26T08:39:07Z). The operation will be retried in 18.2352374 seconds.
[2024-09-26 08:39:26Z WARN VisualStudioServices] Attempt 4 of POST request to https://vsblobprodsu6weu.vsblob.visualstudio.com/A377fce0f-c74c-4ec5-b81c-fe89f488285a/_apis/blob/blobs/79665E6B7D4013C4960152C96F1D0B3BCA1B6399E35DD81636E596687B4C20BB00 failed (HTTP Status: InternalServerError, AFD Ref: Ref A: 2424560C95204FA0A7F69763E7CD03EB Ref B: CPH30EDGE0617 Ref C: 2024-09-26T08:39:26Z). The operation will be retried in 26.5864818 seconds.
[2024-09-26 08:39:52Z WARN VisualStudioServices] Attempt 5 of POST request to https://vsblobprodsu6weu.vsblob.visualstudio.com/A377fce0f-c74c-4ec5-b81c-fe89f488285a/_apis/blob/blobs/79665E6B7D4013C4960152C96F1D0B3BCA1B6399E35DD81636E596687B4C20BB00 failed (HTTP Status: InternalServerError, AFD Ref: Ref A: 770F1390A06E4ABF8CA4763B45B4895E Ref B: CPH30EDGE0617 Ref C: 2024-09-26T08:39:52Z). The operation will be retried in 46.7367802 seconds.
[2024-09-26 08:40:39Z ERR  VisualStudioServices] Attempt 6 of POST request to https://vsblobprodsu6weu.vsblob.visualstudio.com/A377fce0f-c74c-4ec5-b81c-fe89f488285a/_apis/blob/blobs/79665E6B7D4013C4960152C96F1D0B3BCA1B6399E35DD81636E596687B4C20BB00 failed (HTTP Status: InternalServerError, AFD Ref: Ref A: BB69D243022F4596A663B722357D379D Ref B: CPH30EDGE0617 Ref C: 2024-09-26T08:40:39Z). The maximum number of attempts has been reached.
[2024-09-26 08:40:39Z INFO JobServer] Information, ArtifactHttpRetryMessageHandler.SendAsync: https://vsblobprodsu6weu.vsblob.visualstudio.com/A377fce0f-c74c-4ec5-b81c-fe89f488285a/_apis/blob/blobs/79665E6B7D4013C4960152C96F1D0B3BCA1B6399E35DD81636E596687B4C20BB00 attempt 1/6 failed with StatusCode InternalServerError, IsRetryableResponse True
[2024-09-26 09:41:18Z WARN VisualStudioServices] Attempt 1 of POST request to https://vsblobprodsu6weu.vsblob.visualstudio.com/A377fce0f-c74c-4ec5-b81c-fe89f488285a/_apis/blob/blobs/7EC386E5F9F46FE32C5E13F4D87C367D080266CBA33BE31F8AB7022668707CA000 failed (HTTP Status: InternalServerError, AFD Ref: Ref A: 96D87B0E830141268E04B9D306E0B7F6 Ref B: CPH30EDGE0912 Ref C: 2024-09-26T09:41:18Z). The operation will be retried in 10.8000993 seconds.
[2024-09-26 09:41:29Z WARN VisualStudioServices] Attempt 2 of POST request to https://vsblobprodsu6weu.vsblob.visualstudio.com/A377fce0f-c74c-4ec5-b81c-fe89f488285a/_apis/blob/blobs/7EC386E5F9F46FE32C5E13F4D87C367D080266CBA33BE31F8AB7022668707CA000 failed (HTTP Status: InternalServerError, AFD Ref: Ref A: A2348E86155842C09E9F8FE352D11675 Ref B: CPH30EDGE0912 Ref C: 2024-09-26T09:41:29Z). The operation will be retried in 13.5102755 seconds.
[2024-09-26 09:41:43Z WARN VisualStudioServices] Attempt 3 of POST request to https://vsblobprodsu6weu.vsblob.visualstudio.com/A377fce0f-c74c-4ec5-b81c-fe89f488285a/_apis/blob/blobs/7EC386E5F9F46FE32C5E13F4D87C367D080266CBA33BE31F8AB7022668707CA000 failed (HTTP Status: InternalServerError, AFD Ref: Ref A: 617167DCF7534E509006EDB15E5A3A8A Ref B: CPH30EDGE0912 Ref C: 2024-09-26T09:41:43Z). The operation will be retried in 15.7902204 seconds.
[2024-09-26 09:41:59Z WARN VisualStudioServices] Attempt 4 of POST request to https://vsblobprodsu6weu.vsblob.visualstudio.com/A377fce0f-c74c-4ec5-b81c-fe89f488285a/_apis/blob/blobs/7EC386E5F9F46FE32C5E13F4D87C367D080266CBA33BE31F8AB7022668707CA000 failed (HTTP Status: InternalServerError, AFD Ref: Ref A: EA7DE6A820E242CEABE294462DE2B2B0 Ref B: CPH30EDGE0912 Ref C: 2024-09-26T09:41:58Z). The operation will be retried in 22.7832669 seconds.
[2024-09-26 09:42:21Z WARN VisualStudioServices] Attempt 5 of POST request to https://vsblobprodsu6weu.vsblob.visualstudio.com/A377fce0f-c74c-4ec5-b81c-fe89f488285a/_apis/blob/blobs/7EC386E5F9F46FE32C5E13F4D87C367D080266CBA33BE31F8AB7022668707CA000 failed (HTTP Status: InternalServerError, AFD Ref: Ref A: 89F87323A6D44D72ACBD5488D9BFC49D Ref B: CPH30EDGE0912 Ref C: 2024-09-26T09:42:21Z). The operation will be retried in 39.0035194 seconds.
[2024-09-26 09:43:00Z ERR  VisualStudioServices] Attempt 6 of POST request to https://vsblobprodsu6weu.vsblob.visualstudio.com/A377fce0f-c74c-4ec5-b81c-fe89f488285a/_apis/blob/blobs/7EC386E5F9F46FE32C5E13F4D87C367D080266CBA33BE31F8AB7022668707CA000 failed (HTTP Status: InternalServerError, AFD Ref: Ref A: B528A501C10149328FDC5B719CE02E89 Ref B: CPH30EDGE0912 Ref C: 2024-09-26T09:43:00Z). The maximum number of attempts has been reached.
[2024-09-26 09:43:00Z INFO JobServer] Information, ArtifactHttpRetryMessageHandler.SendAsync: https://vsblobprodsu6weu.vsblob.visualstudio.com/A377fce0f-c74c-4ec5-b81c-fe89f488285a/_apis/blob/blobs/7EC386E5F9F46FE32C5E13F4D87C367D080266CBA33BE31F8AB7022668707CA000 attempt 5/6 failed with StatusCode InternalServerError, IsRetryableResponse True

[ivanduplenskikh]

@trichard-jabra, have you configured a proxy setting on your machine where the agents are hosted?
It appears the issue might be linked to Azure Pipelines API rather than the Azure Pipelines Agent itself. I recommend submitting a ticket at https://developercommunity.visualstudio.com/AzureDevOps to ensure the appropriate team can look into it.

[trichard-jabra]

@ivanduplenskikh No there is no proxy configured.
That is what I thought but wanted to make sure

https://developercommunity.visualstudio.com/t/Finalize-Job-does-not-complete/10754188?scope=follow&viewtype=all

[HenrikBacher]

I'm seeing the same issue, running on self-hosted mac as well
MacOS 14.7 - Xcode 15.4
Agent version 3.240.1, 3.244.1 & 3.243.1
Created a support case for it: 2409260050002843

[ivanduplenskikh]

@trichard-jabra, @HenrikBacher, I've forwarded your details regarding this issue to the specific team and will update you as soon as I have more information.

[Chw-Lho]

We are also seeing this issue with selfhosted agents running on Linux (Oracle Linux 8.7), also started from 05:00 AM 26/7-2024 GMT+1.

It doesn't happen with our Windows agents, only those on Linux.

We were running old v2 agents there, but have tried upgrading to v3.244.1 to no avail.

It actually runs the steps/tasks and you can see the logs, while it is running. But when the task is done, it can't seem to upload the logs to vsblob.visualstudio.com and therefor the logs are empty when looking in dev.azure.com after the step is done.

I have also written on the task created on developercommunity.visualstudio.com

[HenrikBacher]

I'm seeing the same thing that logs are visible during the run but not afterwards, furthermore I noticed a set of lines:
[2024-09-26 04:57:25Z INFO JobServerQueue] Web console line queue drained.
[2024-09-26 04:57:25Z INFO JobServerQueue] Uploading 18 files in one shot.
And next follows the the POST retries, the "Uploading 18 files..." matches with the number of tasks of the executed job

[Chw-Lho]

Seems like it actually works again now, maybe from around 8 PM last night.

[trichard-jabra]

Yes! Closing