Not properly accounting for market holydays (data shifts)

[pquan]

It looks like market holydays are not taken into account. For example, Thanksgiving 2024 was on Nov.28th, 2024. There was no market.
Downloading the daily NQ future from TV, I get a candle on 2024-11-28 and I'm missing the candle on 2024-11-29 which was the real open market day as follows:

                      symbol      open      high       low     close    volume
datetime
2024-11-04  CME_MINI:NQZ2024  20138.00  20248.50  20013.00  20086.00  520521.0
....
2024-11-26  CME_MINI:NQZ2024  20906.25  21024.75  20755.00  20993.50  505254.0
2024-11-27  CME_MINI:NQZ2024  20990.75  21019.00  20675.00  20813.00  520026.0
2024-11-28  CME_MINI:NQZ2024  20813.00  21017.50  20805.50  20993.50  339955.0

The data for 2024-11-28 should actually be the candle of 2024-11-29 (OHLCV values are correct for the 2024-11-29 date).

Further, looking at the TradingView Chart, i can confirm that the Candle for 2024-11-28 is missing, and the data is present only for 2024-11-27 and 2024-11-29.

My guess (to be verified) is that the same issue occurs on all market holidays and the data is shifted.

EDIT: Verifying the data that comes from TV, I see that the json value contains:

['i"', '134', '"v"', '', '1732748400.0', '20813.0', '21017.5', '20805.5', '20993.5', '339955.0', '}']

This means that, assuming the TV data is correct, some offset/timezone mangling is necessary to properly align the timestamp to a correct date.

A proposed fix:

Digging further, my guess is that:

• The timestamp returned by TV is localized in your particular timezone.
• The timestamp is for the beginning of the business day, and if the timestamp happens on a holiday, then it must be moved to the next business day.
  With these assumpions, he below proposed fix produces the correct timestamps:

from datetime import datetime, timedelta
import pytz

# Define the timestamp
timestamp = 1732748400  # 2024-11-28 00:00:00

tz = pytz.timezone("<your-local-timezone>")
dt = datetime.fromtimestamp(timestamp,tz).date()
print(f"Unadjusted date : {dt}")
if not is_business_day(dt):
    dt = next_business_day(dt)
print(f"UAdjusted date : {dt}")

the two functions is_business_day and next_business_day can be implemented using the pypi "holidays" package or the pandas_market_calendars package. your-local-timezone is the timezone of your TV account, or the default timezone if not using an account.

With this proposed fix, here's the corrected dates from the NQZ2024 contact:

                      symbol      open      high       low     close    volume adjusted_index
datetime
2024-05-27  CME_MINI:NQZ2024  19354.75  19450.75  19349.25  19420.75      74.0     2024-05-28
2024-06-19  CME_MINI:NQZ2024  20456.50  20601.00  20218.25  20286.00     728.0     2024-06-20
2024-07-04  CME_MINI:NQZ2024  20670.75  20893.75  20624.00  20875.75     817.0     2024-07-05
2024-09-02  CME_MINI:NQZ2024  19850.75  19919.25  19143.75  19234.50    4192.0     2024-09-03
2024-11-28  CME_MINI:NQZ2024  20813.00  21017.50  20805.50  20993.50  339955.0     2024-11-29

[pquan]

I'm still working on this issue, and unfortunately uncovered more timestamp troubles. Here's an example, using the same NQZ2024 contract. The timestamps returned by TV seem to not account for the specific day the day savings time start and end. Below is an extract from the returned data as a Timestamp and as a number. The data was downloaded from the Europe/Paris timezone (GMT+1 and GMT+2 when on day savings time).
The day savings end on Oct 27th, but the timestamp returned by TV seems to only change on Nov.1st. This results in a one-day jump and the weird timestamps on Oct 27 (sunday) at 23:00PM.

2024-10-23 00:00:00 1729641600000000000
2024-10-24 00:00:00 1729728000000000000
2024-10-25 00:00:00 1729814400000000000
2024-10-27 23:00:00 1730070000000000000
2024-10-28 23:00:00 1730156400000000000
2024-10-29 23:00:00 1730242800000000000
2024-10-30 23:00:00 1730329200000000000
2024-10-31 23:00:00 1730415600000000000
2024-11-04 00:00:00 1730678400000000000
2024-11-05 00:00:00 1730764800000000000

When changing the time zone from which you download data (like when using a VPN), the timestamps returned change as well, and they follow the local timezone from which you're downloading. However, the issue with the wrong day of DST end persists.

The problem seems to lie in the TV data itself and will require an ugly patch to fix, unless I am missing something, like perhaps a way to request timestamps explicitly in UTC.

I tried changing the wss message like this:

        self.__send_message("switch_timezone", [
                            self.chart_session, 'Europe/London'])

but the returned timestamps were identical. I'm out of ideas.