t:connect Issues with WS2 API (Prior to tconnectsync v0.8)

[jwoglom]

Some portions of t:connect, specifically interactions with the therapy timeline on tconnectws2.tandemdiabetes.com, are not functioning properly and tconnectsync has been receiving errors from Tandem when trying to synchronize data over the past 24 hours.

This is showing up as the following in tconnectsync debug logs:

2022-07-15 14:43:54 DEBUG    Starting new HTTPS connection (1): tconnectws2.tandemdiabetes.com:443
Traceback (most recent call last):
  File "/home/tconnectsync/.local/share/virtualenvs/tconnectsync-wjwwe9KF/lib/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/home/tconnectsync/.local/share/virtualenvs/tconnectsync-wjwwe9KF/lib/python3.9/site-packages/urllib3/connectionpool.py", line 445, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/home/tconnectsync/.local/share/virtualenvs/tconnectsync-wjwwe9KF/lib/python3.9/site-packages/urllib3/connectionpool.py", line 440, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib/python3.9/http/client.py", line 1347, in getresponse
    response.begin()
  File "/usr/lib/python3.9/http/client.py", line 307, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.9/http/client.py", line 276, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

[kswick]

I'm getting the same error and I think it started around the same time. The last entry I got synced was 7/15 12:21am EDT (GMT-4). I'm running the pip install in a VM running via supervisor. Manual attempts to run failed with the same error even after stopping supervisor. However, I've found that if I stop supervisor and walk away for a but and then come back and try a manual run, it works. I did that and got caught up this morning. Then I restarted supervisor and got nothing for the rest of the day. I just repeated the process of stopping supervisor, waiting, and then doing a manual run and I am again caught up. Could Tandem have added a check based on frequency of requests? Or possibly related to recent updates for the limited launch of the remote bolus capability? I think I installed the t:connect mobile app update the morning on 7/15 which coincides with when the error started. I only just this evening installed the pump update so that's unlikely to be related.

[jwoglom]

I do wonder if there's any relation with the remote bolus update, which I installed on July 13th. Maybe it's additional load on tandem's servers as a result of the new version of the tconnect app which uploads more data? The sync issue started for me at 2022-07-14 11:13:00AM EST.

[jwoglom]

@kswick I think a root cause of this might be that Tandem began to tag some of tconnectsync's requests as automated and blocked them. I have released v0.7.2 with some workarounds which have gotten tconnectsync working again for me, at least as of this moment.

[kswick]

@jwoglom, that was my suspicion as well. I upgraded to 0.7.2 about 2 weeks ago but still am not getting data synced. I'm still getting errors about the remote connection being closed. Here's one instance of the stack trace. Then it repeats on the next tconnectsync cycle. I've stopped supervisor to give it a rest and planning to try a manual run later.

2022-08-06 21:55:19 INFO Enabled features: BASAL, BOLUS, IOB, PUMP_EVENTS
2022-08-06 21:55:20 INFO Logged in to AndroidApi successfully (expiration: 2022-08-07T05:55:20.123Z, in 7 hours, 59 minutes)
2022-08-06 21:55:20 INFO New reported t:connect data. (event index: 534784 last: None)
2022-08-06 21:55:20 INFO Downloading t:connect ControlIQ data
2022-08-06 21:55:20 INFO Logging in to ControlIQApi...
2022-08-06 21:55:23 INFO Logged in to ControlIQApi successfully (expiration: 2022-08-07T05:55:20.872Z, in 7 hours, 59 minutes)
2022-08-06 21:55:23 INFO Downloading t:connect CSV data
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 665, in urlopen
httplib_response = self._make_request(
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 421, in _make_request
six.raise_from(e, None)
File "", line 3, in raise_from
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 416, in _make_request
httplib_response = conn.getresponse()
File "/usr/lib/python3.8/http/client.py", line 1348, in getresponse
response.begin()
File "/usr/lib/python3.8/http/client.py", line 316, in begin
version, status, reason = self._read_status()
File "/usr/lib/python3.8/http/client.py", line 285, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/requests/adapters.py", line 439, in send
resp = conn.urlopen(
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 719, in urlopen
retries = retries.increment(
File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 400, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/usr/lib/python3/dist-packages/six.py", line 702, in reraise
raise value.with_traceback(tb)
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 665, in urlopen
httplib_response = self._make_request(
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 421, in _make_request
six.raise_from(e, None)
File "", line 3, in raise_from
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 416, in _make_request
httplib_response = conn.getresponse()
File "/usr/lib/python3.8/http/client.py", line 1348, in getresponse
response.begin()
File "/usr/lib/python3.8/http/client.py", line 316, in begin
version, status, reason = self._read_status()
File "/usr/lib/python3.8/http/client.py", line 285, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/kristen_swick/.local/bin/tconnectsync", line 8, in
sys.exit(main())
File "/home/kristen_swick/.local/lib/python3.8/site-packages/tconnectsync/init.py", line 87, in main
sys.exit(u.process(tconnect, nightscout, time_start, time_end, args.pretend, features=args.features))
File "/home/kristen_swick/.local/lib/python3.8/site-packages/tconnectsync/autoupdate.py", line 47, in process
added = process_time_range(tconnect, nightscout, time_start, time_end, pretend, features=features)
File "/home/kristen_swick/.local/lib/python3.8/site-packages/tconnectsync/process.py", line 55, in process_time_range
csvdata = tconnect.ws2.therapy_timeline_csv(time_start, time_end)
File "/home/kristen_swick/.local/lib/python3.8/site-packages/tconnectsync/api/ws2.py", line 80, in therapy_timeline_csv
req_text = self.get('therapytimeline2csv/%s/%s/%s?format=csv' % (self.userGuid, startDate, endDate))
File "/home/kristen_swick/.local/lib/python3.8/site-packages/tconnectsync/api/ws2.py", line 25, in get
r = self.session.get(self.BASE_URL + endpoint, headers=base_headers())
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 546, in get
return self.request('GET', url, **kwargs)
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python3/dist-packages/requests/adapters.py", line 498, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

[jwoglom]

@kswick I'm planning to do some refactorings so that tconnectsync doesn't use the ws2 API at all. Can hopefully finish that this week.

[sdneufer]

I am looking forward to trying the refactored version once available as we see the ws2 API having issues.

I started running tconnectsync on my macbook around July 24th. I am running tconnectsync through cron. I do get data, however out of the 36 times it ran today since midnight (CEST), it wrote 28 times to Nightscout. It got a ConnectionError and would have crashed however we modified the code to change it to retry same as an ApiException. It typically, but not always, hangs on the retries. Even when it does respond it can take many minutes.

If there is a different API that can be used this sounds like it would be worth trying.

[jwoglom]

@sdneufer @kswick please take a look at the latest release which removes the dependency on the WS2 API for bolus synchronization. #45 lists the remaining places which still use it. So long as you don't have any of them as enabled features tconnectsync won't query the ws2 api at all.

[sdneufer]

@jwoglom I upgraded to 0.8.1 and will let you know tomorrow night (CEST) how it does over the next 24 hours.

[rzachariassen]

I'm helping @sdneufer with looking at the code. Its not running at all, raising an exception at line 204 of parser/tconnect.py because it doesn't recognize the type of the therapy_event. Here's the value of data in the most recent failure, its getting a type of Basal but expecting either Bolus or CGM: {'duration': 0, 'percent': 0, 'value': 0.0}, 'baseBasalRate': 0.63, 'displayInHistory': 0, 'eventDateTime': '2022-08-10T00:00:00', 'note': {'id': 0, 'indexId': '205842', 'eventTypeId': 90, 'sourceRecordId': 0, 'eventId': 0, 'active': False}, 'noteDate': {}, 'requestDateTime': '0001-01-01T00:00:00', 'type': 'Basal', 'description': 'NDE', 'sourceRecId': 172151436, 'eventTypeId': 0, 'indexId': 0, 'uploadId': 0, 'interactive': 1, 'tempRateId': 0, 'tempRateCompleted': 0, 'tempRateActivated': 0}

We'll switch back to the previous version until we hear back.

[jwoglom]

@rzachariassen @sdneufer
Interesting... I have never seen Basal therapy events on my end before appearing on that endpoint. I only saw CGM or Bolus event types on my end, hence the assertion that's failing. Is Control IQ disabled? Or anything odd about their setup? Can you relate it to an event at that time which appears in the pump or in tconnect? Wondering what the "NDE" description means.

[sdneufer]

@jwoglom Control IQ is enabled. We are examining the data from the API's in both version 0.7.2 and 0.8.1 to get a view of my specific data..

[rzachariassen]

In 0.8.2 we see a number of records with type "Basal" and descriptions (note/EventTypeID and hypothesized meaning in parens) "NDE" (90, new day event) "EOD" (none, end of day event) "BRC" (3, basal rate change) and "PE" (11, pump error). I think NDE is like an initial value declaration and happens at midnight, the EOD happens milliseconds before NDE except for the first (oldest) NDE in the list. BRC is the normal basal rate change entry and PE happened at the same time as a BRC when the pump was suspended and control-iq turned off for a refill activity. Not sure why there's both an EOD and NDE. There is also a "BG" type that doesn't seem to be handled by the code. If it helps can send data via pm.

[jwoglom]

@rzachariassen if you can email me the API output of api.controliq.therapy_events and api.controliq.therapy_timeline that would be great. My email is on my Github profile. (I also added a section in the readme w instructions for the future: https://github.com/jwoglom/tconnectsync#tconnect-api-testing)

[jwoglom]

Also -- what version of the t:connect iOS or Android app are you using. Perhaps this is dependent on app version.

[jwoglom]

Closing this issue due to inactivity, and because workarounds were implemented in tconnectsync to avoid use of the ws2 api whenever possible. Please re-open or create a new issue if you're still experiencing issues.