Upload/remove failure with S3 storage

[thovu162020]

Technical Details:

• Vault setup using docker image 1.0.1194272500 (latest)

Problem
After letting an agent connecting to my vault for 30 days, the S3 storage suddenly stops working. All features of the vault work fine, except for the POST API endpoint /api/storage which the agent uses to send recordings to the vault.

Upon inspecting the vault's logs, I found 2 other issues from the logs: action RemoveMediaInBulk fails and the container frequently execute panic recovered:

time="2024-03-11T03:20:52Z" level=info msg="Cleanup: considering to remove 4 from S3"
time="2024-03-11T03:20:52Z" level=info msg="RemoveMediaInBulk: Starting removal..."
time="2024-03-11T03:20:52Z" level=error msg="RemoveMediaInBulk: Something went wrong while removing in bulk from S3: not yet implemented"
time="2024-03-11T03:20:52Z" level=info msg="RemoveMediaInBulk: Finished removal."
time="2024-03-11T03:20:52Z" level=info msg="Cleaning up media from providers"
[GIN] 2024/03/11 - 03:20:54 | 200 |     156.327µs |       127.0.0.1 | GET      "/"
[GIN] 2024/03/11 - 03:20:54 | 200 |     374.752µs |       127.0.0.1 | GET      "/"
[GIN] 2024/03/11 - 03:20:57 | 200 |     1.58368ms |      10.8.0.214 | POST     "/api/devices/heartbeat"
[GIN] 2024/03/11 - 03:20:57 | 200 |    1.612698ms |      10.8.0.214 | POST     "/api/devices/heartbeat"


2024/03/11 03:21:01 [Recovery] 2024/03/11 - 03:21:01 panic recovered:
runtime error: index out of range [0] with length 0
/usr/local/go/src/runtime/panic.go:113 (0x4c321e)
/go/pkg/mod/github.com/!eyevinn/[email protected]/avc/avcdecoderconfigurationrecord.go:64 (0x106675b)
/go/pkg/mod/github.com/!eyevinn/[email protected]/mp4/avcc.go:43 (0x1070f15)
/go/pkg/mod/github.com/!eyevinn/[email protected]/mp4/boxsr.go:166 (0x1076c92)
/go/pkg/mod/github.com/!eyevinn/[email protected]/mp4/visualsampleentry.go:126 (0x10be625)
/go/pkg/mod/github.com/!eyevinn/[email protected]/mp4/boxsr.go:166 (0x1076c92)
/go/pkg/mod/github.com/!eyevinn/[email protected]/mp4/container.go:131 (0x107a4f4)
/go/pkg/mod/github.com/!eyevinn/[email protected]/mp4/stsd.go:133 (0x10aaae7)
/go/pkg/mod/github.com/!eyevinn/[email protected]/mp4/boxsr.go:166 (0x1076c92)
/go/pkg/mod/github.com/!eyevinn/[email protected]/mp4/container.go:131 (0x107a4f4)
/go/pkg/mod/github.com/!eyevinn/[email protected]/mp4/stbl.go:98 (0x10a62e5)
/go/pkg/mod/github.com/!eyevinn/[email protected]/mp4/boxsr.go:166 (0x1076c92)
/go/pkg/mod/github.com/!eyevinn/[email protected]/mp4/container.go:131 (0x107a4f4)
/go/pkg/mod/github.com/!eyevinn/[email protected]/mp4/minf.go:58 (0x1094c08)
/go/pkg/mod/github.com/!eyevinn/[email protected]/mp4/boxsr.go:166 (0x1076c92)
/go/pkg/mod/github.com/!eyevinn/[email protected]/mp4/container.go:131 (0x107a4f4)
/go/pkg/mod/github.com/!eyevinn/[email protected]/mp4/mdia.go:56 (0x1090fa5)
/go/pkg/mod/github.com/!eyevinn/[email protected]/mp4/boxsr.go:166 (0x1076c92)
/go/pkg/mod/github.com/!eyevinn/[email protected]/mp4/container.go:131 (0x107a4f4)
/go/pkg/mod/github.com/!eyevinn/[email protected]/mp4/trak.go:58 (0x10b45f6)
/go/pkg/mod/github.com/!eyevinn/[email protected]/mp4/boxsr.go:166 (0x1076c92)
/go/pkg/mod/github.com/!eyevinn/[email protected]/mp4/container.go:131 (0x107a4f4)
/go/pkg/mod/github.com/!eyevinn/[email protected]/mp4/moov.go:70 (0x1096f64)
/go/pkg/mod/github.com/!eyevinn/[email protected]/mp4/box.go:339 (0x10745f2)
/go/pkg/mod/github.com/!eyevinn/[email protected]/mp4/file.go:176 (0x1085148)
/go/src/gitlab.com/kerberos-io/vault/api/mp4/fragmented.go:18 (0x10c6aa5)
/go/src/gitlab.com/kerberos-io/vault/api/storage/main.go:292 (0x13179c7)
/go/src/gitlab.com/kerberos-io/vault/api/main.go:244 (0x135e467)
/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:174 (0xaca3c1)
/go/pkg/mod/github.com/gin-gonic/[email protected]/recovery.go:102 (0xaca3ac)
/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:174 (0xac94c6)
/go/pkg/mod/github.com/gin-gonic/[email protected]/logger.go:240 (0xac94a9)
/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:174 (0xaca3c1)
/go/pkg/mod/github.com/gin-gonic/[email protected]/recovery.go:102 (0xaca3ac)
/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:174 (0xac94c6)
/go/pkg/mod/github.com/gin-gonic/[email protected]/logger.go:240 (0xac94a9)
/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:174 (0xac8550)
/go/pkg/mod/github.com/gin-gonic/[email protected]/gin.go:620 (0xac81b8)
/go/pkg/mod/github.com/gin-gonic/[email protected]/gin.go:576 (0xac7cfc)
/usr/local/go/src/net/http/server.go:2947 (0x77a0eb)
/usr/local/go/src/net/http/server.go:1991 (0x775306)
/usr/local/go/src/runtime/asm_amd64.s:1594 (0x4f9ac0)

[GIN] 2024/03/11 - 03:21:01 | 500 |    1.543388ms |      10.8.0.214 | POST     "/api/storage"
[GIN] 2024/03/11 - 03:21:01 | 500 |    1.561114ms |      10.8.0.214 | POST     "/api/storage"