Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Nest source encounters 409 and retry attempts hit 429 rate-limit #1397

Open
trvrnrth opened this issue Oct 12, 2024 · 1 comment
Open

Nest source encounters 409 and retry attempts hit 429 rate-limit #1397

trvrnrth opened this issue Oct 12, 2024 · 1 comment
Labels
bug Something isn't working

Comments

@trvrnrth
Copy link

I have a nest hello gen 1 wired doorbell source which, after some time experiences a 409 response:

error="streams: nest: wrong status: 409 Conflict, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door

It appears that there is no back-off on attempts to re-connect so these are then rate-limited:

error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door

The full error message grabbed from the Device Access Sample App at this time is:

{
  "error": {
    "code": 429,
    "message": "Rate limited for the GenerateWebRtcStream API for the user.",
    "status": "RESOURCE_EXHAUSTED"
  }
}

At this point a restart is not sufficient to re-connect. go2rtc must be stopped for some time to allow the rate limit to "clear" before being started again (in my case this is a case of manually stopping, waiting a few minutes then starting the HA Frigate add-on).

I am unclear what causes the initial 409 conflict response, but implementing a back-off on at least the 429 response would be good. I guess it might make sense to just back-off on any error which would lessen the chance of hitting the rate limit in the first place.

Here is a full extract from the logs at the time of last failure:

2024-10-12 15:16:29.799077795  16:16:29.798 WRN github.com/AlexxIT/go2rtc/internal/streams/producer.go:171 > error=EOF url=ffmpeg:front_door#audio=aac
2024-10-12 15:16:50.871397445  16:16:50.871 WRN [rtsp] error="streams: nest: wrong status: 409 Conflict, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:16:55.987067326  16:16:55.986 WRN [rtsp] error="streams: nest: wrong status: 409 Conflict, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:17:01.104080094  16:17:01.103 WRN [rtsp] error="streams: nest: wrong status: 409 Conflict, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:17:06.220382167  16:17:06.220 WRN [rtsp] error="streams: nest: wrong status: 409 Conflict, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:17:11.337320055  16:17:11.337 WRN [rtsp] error="streams: nest: wrong status: 409 Conflict, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:17:16.453962537  16:17:16.453 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:17:21.569329842  16:17:21.569 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:17:26.684904880  16:17:26.684 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:17:31.799685805  16:17:31.799 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:17:36.916763767  16:17:36.916 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:17:42.031527043  16:17:42.031 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:17:47.146558111  16:17:47.146 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:17:52.264210419  16:17:52.264 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:17:57.380472619  16:17:57.380 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:18:02.495376712  16:18:02.495 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:18:07.613469154  16:18:07.613 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:18:12.728694209  16:18:12.728 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:18:17.846215123  16:18:17.845 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:18:22.962979410  16:18:22.962 WRN [rtsp] error="streams: nest: wrong status: 409 Conflict, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:18:28.079459957  16:18:28.079 WRN [rtsp] error="streams: nest: wrong status: 409 Conflict, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:18:33.194993369  16:18:33.194 WRN [rtsp] error="streams: nest: wrong status: 409 Conflict, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:18:38.313387009  16:18:38.313 WRN [rtsp] error="streams: nest: wrong status: 409 Conflict, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:18:43.428449201  16:18:43.428 WRN [rtsp] error="streams: nest: wrong status: 409 Conflict, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:18:48.543098321  16:18:48.542 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:18:53.660881178  16:18:53.660 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:18:58.779714906  16:18:58.779 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:19:03.892740883  16:19:03.892 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:19:09.009398444  16:19:09.009 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:19:14.128929625  16:19:14.128 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:19:19.246458352  16:19:19.246 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:19:24.360992865  16:19:24.360 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:19:29.477766616  16:19:29.477 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:19:34.593178617  16:19:34.592 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:19:39.709458289  16:19:39.709 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:19:44.825299325  16:19:44.825 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:19:49.941154786  16:19:49.940 WRN [rtsp] error="streams: nest: wrong status: 409 Conflict, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:19:55.060783500  16:19:55.060 WRN [rtsp] error="streams: nest: wrong status: 409 Conflict, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:20:00.177158992  16:20:00.177 WRN [rtsp] error="streams: nest: wrong status: 409 Conflict, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:20:00.177373129  16:20:00.177 WRN github.com/AlexxIT/go2rtc/internal/rtsp/rtsp.go:225 > error="write tcp 127.0.0.1:8554->127.0.0.1:37294: write: broken pipe"
2024-10-12 15:20:05.294597204  16:20:05.293 WRN [rtsp] error="streams: nest: wrong status: 409 Conflict, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:20:05.294615471  16:20:05.294 WRN github.com/AlexxIT/go2rtc/internal/rtsp/rtsp.go:225 > error="write tcp 127.0.0.1:8554->127.0.0.1:37302: write: broken pipe"
2024-10-12 15:20:10.410614638  16:20:10.410 WRN [rtsp] error="streams: nest: wrong status: 409 Conflict, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:20:10.410963374  16:20:10.410 WRN github.com/AlexxIT/go2rtc/internal/rtsp/rtsp.go:225 > error="write tcp 127.0.0.1:8554->127.0.0.1:37314: write: broken pipe"
2024-10-12 15:20:15.525828154  16:20:15.525 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:20:20.643628744  16:20:20.643 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:20:25.758949395  16:20:25.758 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:20:30.874456050  16:20:30.874 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:20:30.874686424  16:20:30.874 WRN github.com/AlexxIT/go2rtc/internal/rtsp/rtsp.go:225 > error="write tcp 127.0.0.1:8554->127.0.0.1:45290: write: broken pipe"
2024-10-12 15:20:35.989727907  16:20:35.989 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:20:35.990225268  16:20:35.989 WRN github.com/AlexxIT/go2rtc/internal/rtsp/rtsp.go:225 > error="write tcp 127.0.0.1:8554->127.0.0.1:45292: write: broken pipe"
2024-10-12 15:20:41.106395530  16:20:41.106 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:20:41.106723038  16:20:41.106 WRN github.com/AlexxIT/go2rtc/internal/rtsp/rtsp.go:225 > error="write tcp 127.0.0.1:8554->127.0.0.1:45308: write: broken pipe"
2024-10-12 15:20:46.222196430  16:20:46.221 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:20:46.222541964  16:20:46.222 WRN github.com/AlexxIT/go2rtc/internal/rtsp/rtsp.go:225 > error="write tcp 127.0.0.1:8554->127.0.0.1:37156: write: broken pipe"
2024-10-12 15:20:51.336820870  16:20:51.336 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:20:51.336971870  16:20:51.336 WRN github.com/AlexxIT/go2rtc/internal/rtsp/rtsp.go:225 > error="write tcp 127.0.0.1:8554->127.0.0.1:37166: write: broken pipe"
2024-10-12 15:20:56.452101686  16:20:56.451 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:20:56.452461684  16:20:56.452 WRN github.com/AlexxIT/go2rtc/internal/rtsp/rtsp.go:225 > error="write tcp 127.0.0.1:8554->127.0.0.1:37168: write: broken pipe"
@AlexxIT AlexxIT added the bug Something isn't working label Oct 13, 2024
@trvrnrth
Copy link
Author

trvrnrth commented Oct 29, 2024

I had a bit of time to do some local debugging of this and realised I'd not provided my stream config in the issue above:

streams:
  front_door:
    - nest:?client_id=REDACTED&client_secret=REDACTED&device_id=REDACTED&project_id=REDACTED&refresh_token=REDACTED
    - "ffmpeg:front_door#audio=aac

The testing I've done suggests that the fixed 5 second retry I'm seeing stems from the ffmpeg audio transcode I have in place so I am currently testing with a 60 second timeout as follows to see if this helps:

      - "ffmpeg:front_door#audio=aac#input=-fflags nobuffer -flags low_delay -timeout 60000000 -user_agent go2rtc/ffmpeg -rtsp_flags prefer_tcp -i {input}"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants