TWS datetime/pandas issue - Latest OSX and TWS

[HyperionSounds]

Dear Zipline Maintainers,

Before I tell you about my issue, let me describe my environment:

Environment

• Operating System: (Mac OS 10.13.6)

• Python Version: 3.6.3

• How did you install Zipline: (pip)

• Python packages: alembic==0.9.9 alpha-vantage==2.0.0 bcolz==0.12.1 Bottleneck==1.2.1 certifi==2018.4.16 chardet==3.0.4 click==6.7 contextlib2==0.5.5 cycler==0.10.0 cyordereddict==1.0.0 Cython==0.28.3 DateTime==4.2 decorator==4.3.0 empyrical==0.4.3 ib==0.8.0 IbPy2==0.8.0 idna==2.7 intervaltree==2.1.0 kiwisolver==1.0.1 Logbook==1.4.0 lru-dict==1.1.6 lxml==4.2.1 Mako==1.0.7 MarkupSafe==1.0 matplotlib==2.2.2 multipledispatch==0.5.0 networkx==1.11 numexpr==2.6.5 numpy==1.14.4 pandas==0.18.1 pandas-datareader==0.6.0 patsy==0.5.0 pyparsing==2.2.0 PyQt5==5.11.2 PyQt5-sip==4.19.12 python-dateutil==2.7.3 python-editor==1.0.3 pytz==2018.4 requests==2.19.0 requests-file==1.4.3 requests-ftp==0.3.1 scipy==1.1.0 simplejson==3.16.0 six==1.11.0 sortedcontainers==2.0.4 SQLAlchemy==1.2.8 statsmodels==0.9.0 tables==3.4.4 toolz==0.9.0 urllib3==1.23 wrapt==1.10.11 zipline-live==1.1.0.5 zope.interface==4.5.0

Now that you know a little about me, let me tell you about the issue I am
having:

Description of Issue

• I attempt running buyapple.py demo file, it will buy the contract, but then spits me an error regarding my time
• I have ran buyapple.py again and again, buying shares every time. I can now run it once per day, and then IB throws me an error in the way of datetime.

Here is how you can reproduce this issue on your machine:

Reproduction Steps

1. Updated Mac OS
2. Updated Interactive Brokers TWS
3. Attempt to run BuyApple.py
   ...

What steps have you taken to resolve this already?

I have attempted to reinstall zipline and its dependencies, uninstalled and reinstalled IB TWS, I have created a new account on TWS, which will work fine, until submitting and order, at which point the order will throw an error due to its datetime string, by my understanding.
...

Anything else?

Here is a description of the error:

(venv-zipline-live) HYPERiON:live Derek$ zipline run -f alpha-live.py --state-file alpha-live.state --realtime-bar-target realtime-bars/ --broker ib --broker-uri localhost:4001:1232 --bundle quantopian-quandl --data-frequency minute
[2018-07-31 16:49:09.683764] INFO: IB Broker: Connecting: localhost:4001:1232
Server Version: 76
TWS Time at connection:20180731 09:49:08 PST
[2018-07-31 16:49:09.804428] ERROR: IB Broker: <class 'str'> returned a result with an error set
ValueError: Error parsing datetime string "20180731 08:40:00" at position 9

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/Users/Derek/venv-zipline-live/lib/python3.6/site-packages/ib/ext/EReader.py", line 107, in run
while not self.isInterrupted() and self.processMsg(self.readInt()):
File "/Users/Derek/venv-zipline-live/lib/python3.6/site-packages/ib/ext/EReader.py", line 744, in processMsg
self.eWrapper().execDetails(reqId, contract, exec_)
File "/Users/Derek/venv-zipline-live/lib/python3.6/site-packages/zipline/gens/brokers/ib_broker.py", line 353, in execDetails
exec_time=pd.to_datetime(exec_detail.m_time),
File "/Users/Derek/venv-zipline-live/lib/python3.6/site-packages/pandas/util/decorators.py", line 91, in wrapper
return func(*args, **kwargs)
File "/Users/Derek/venv-zipline-live/lib/python3.6/site-packages/pandas/tseries/tools.py", line 291, in to_datetime
unit=unit, infer_datetime_format=infer_datetime_format)
File "/Users/Derek/venv-zipline-live/lib/python3.6/site-packages/pandas/tseries/tools.py", line 429, in _to_datetime
return _convert_listlike(np.array([arg]), box, format)[0]
File "/Users/Derek/venv-zipline-live/lib/python3.6/site-packages/pandas/tseries/tools.py", line 398, in _convert_listlike
require_iso8601=require_iso8601
File "pandas/tslib.pyx", line 2134, in pandas.tslib.array_to_datetime (pandas/tslib.c:41972)
File "pandas/tslib.pyx", line 2269, in pandas.tslib.array_to_datetime (pandas/tslib.c:40684)
File "pandas/tslib.pyx", line 2229, in pandas.tslib.array_to_datetime (pandas/tslib.c:39919)
File "pandas/src/datetime.pxd", line 141, in datetime._string_to_dts (pandas/tslib.c:85631)
SystemError: <class 'str'> returned a result with an error set
[2018-07-31 16:49:09.808153] ERROR: IB Broker: [None] None (<class 'str'> returned a result with an error set)
[2018-07-31 16:49:09.808440] ERROR: IB Broker: IB Connection closed
[2018-07-31 16:49:09.836936] ERROR: IB Broker: [504] Not connected (-1)

...

Sincerely,
$ HypeSound

[HyperionSounds]

After following the patch that fixes for ib data parsing here; it worked briefly, then breaks again with this:

Traceback (most recent call last):
File "/Users/Derek/venv-zipline-live/bin/zipline", line 11, in
sys.exit(main())
File "/Users/Derek/venv-zipline-live/lib/python3.6/site-packages/click/core.py", line 722, in call
return self.main(*args, **kwargs)
File "/Users/Derek/venv-zipline-live/lib/python3.6/site-packages/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "/Users/Derek/venv-zipline-live/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/Users/Derek/venv-zipline-live/lib/python3.6/site-packages/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/Users/Derek/venv-zipline-live/lib/python3.6/site-packages/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
File "/Users/Derek/venv-zipline-live/lib/python3.6/site-packages/zipline/main.py", line 102, in _
return f(*args, **kwargs)
File "/Users/Derek/venv-zipline-live/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func
return f(get_current_context(), *args, **kwargs)
File "/Users/Derek/venv-zipline-live/lib/python3.6/site-packages/zipline/main.py", line 316, in run
realtime_bar_target=realtime_bar_target
File "/Users/Derek/venv-zipline-live/lib/python3.6/site-packages/zipline/utils/run_algo.py", line 206, in _run
overwrite_sim_params=False,
File "/Users/Derek/venv-zipline-live/lib/python3.6/site-packages/zipline/algorithm_live.py", line 195, in run
daily_stats = super(self.class, self).run(*args, **kwargs)
File "/Users/Derek/venv-zipline-live/lib/python3.6/site-packages/zipline/algorithm.py", line 709, in run
for perf in self.get_generator():
File "/Users/Derek/venv-zipline-live/lib/python3.6/site-packages/zipline/algorithm.py", line 608, in get_generator
return self._create_generator(self.sim_params)
File "/Users/Derek/venv-zipline-live/lib/python3.6/site-packages/zipline/algorithm_live.py", line 126, in _create_generator
TradingAlgorithm._create_generator(self, sim_params)
File "/Users/Derek/venv-zipline-live/lib/python3.6/site-packages/zipline/algorithm.py", line 579, in _create_generator
self.initialize(*self.initialize_args, **self.initialize_kwargs)
File "/Users/Derek/venv-zipline-live/lib/python3.6/site-packages/zipline/algorithm_live.py", line 67, in initialize
checksum=self.algo_filename)
File "/Users/Derek/venv-zipline-live/lib/python3.6/site-packages/zipline/utils/serialization_utils.py", line 85, in load_context
loaded_state = pickle.load(f)
File "pandas/tslib.pyx", line 405, in pandas.tslib.Timestamp.new (pandas/tslib.c:9932)
File "pandas/tslib.pyx", line 1475, in pandas.tslib.convert_to_tsobject (pandas/tslib.c:26432)
TypeError: Cannot convert input to Timestamp

[stan4git]

From the 1st stack trace this is due to a timestamp formatting issue in ib_broker.py.
I suggest you could use a simple hack to unblock yourself first. I would submit a pull request to fix this in the long term.

Short term hacky solution:
In file "/Users/Derek/venv-zipline-live/lib/python3.6/site-packages/zipline/gens/brokers/ib_broker.py", line 353, in execDetails

Change:
exec_time=pd.to_datetime(exec_detail.m_time),
To:
exec_time=pd.to_datetime(exec_detail.m_time, format='%Y%m%d %H:%M:%S', errors='coerce'),

[HyperionSounds]

@stan4git thanks for the assistance stan, unfortunately the error persists..
I'm happy to submit a pull request once I can figure out how to fix this error. Has anyone else experienced the same issue?

ValueError: Error parsing datetime string "20180816 11:17:59" at position 9

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/Users/Derek/venv-zipline-live/bin/zipline", line 11, in
sys.exit(main())
File "/Users/Derek/venv-zipline-live/lib/python3.6/site-packages/click/core.py", line 722, in call
return self.main(*args, **kwargs)
File "/Users/Derek/venv-zipline-live/lib/python3.6/site-packages/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "/Users/Derek/venv-zipline-live/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/Users/Derek/venv-zipline-live/lib/python3.6/site-packages/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/Users/Derek/venv-zipline-live/lib/python3.6/site-packages/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
File "/Users/Derek/venv-zipline-live/lib/python3.6/site-packages/zipline/main.py", line 102, in _
return f(*args, **kwargs)
File "/Users/Derek/venv-zipline-live/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func
return f(get_current_context(), *args, **kwargs)
File "/Users/Derek/venv-zipline-live/lib/python3.6/site-packages/zipline/main.py", line 316, in run
realtime_bar_target=realtime_bar_target
File "/Users/Derek/venv-zipline-live/lib/python3.6/site-packages/zipline/utils/run_algo.py", line 206, in _run
overwrite_sim_params=False,
File "/Users/Derek/venv-zipline-live/lib/python3.6/site-packages/zipline/algorithm_live.py", line 195, in run
daily_stats = super(self.class, self).run(*args, **kwargs)
File "/Users/Derek/venv-zipline-live/lib/python3.6/site-packages/zipline/algorithm.py", line 709, in run
for perf in self.get_generator():
File "/Users/Derek/venv-zipline-live/lib/python3.6/site-packages/zipline/gens/tradesimulation.py", line 222, in transform
for capital_change_packet in every_bar(dt):
File "/Users/Derek/venv-zipline-live/lib/python3.6/site-packages/zipline/gens/tradesimulation.py", line 120, in every_bar
blotter.get_transactions(current_data)
File "/Users/Derek/venv-zipline-live/lib/python3.6/site-packages/zipline/finance/blotter_live.py", line 87, in get_transactions
all_transactions = list(self.broker.transactions.values())
File "/Users/Derek/venv-zipline-live/lib/python3.6/site-packages/zipline/gens/brokers/ib_broker.py", line 856, in transactions
self._update_transactions()
File "/Users/Derek/venv-zipline-live/lib/python3.6/site-packages/zipline/gens/brokers/ib_broker.py", line 895, in _update_transactions
dt=pd.to_datetime(exec_detail.m_time, utc=True),
File "/Users/Derek/venv-zipline-live/lib/python3.6/site-packages/pandas/util/decorators.py", line 91, in wrapper
return func(*args, **kwargs)
File "/Users/Derek/venv-zipline-live/lib/python3.6/site-packages/pandas/tseries/tools.py", line 291, in to_datetime
unit=unit, infer_datetime_format=infer_datetime_format)
File "/Users/Derek/venv-zipline-live/lib/python3.6/site-packages/pandas/tseries/tools.py", line 429, in _to_datetime
return _convert_listlike(np.array([arg]), box, format)[0]
File "/Users/Derek/venv-zipline-live/lib/python3.6/site-packages/pandas/tseries/tools.py", line 398, in _convert_listlike
require_iso8601=require_iso8601
File "pandas/tslib.pyx", line 2134, in pandas.tslib.array_to_datetime (pandas/tslib.c:41972)
File "pandas/tslib.pyx", line 2269, in pandas.tslib.array_to_datetime (pandas/tslib.c:40684)
File "pandas/tslib.pyx", line 2229, in pandas.tslib.array_to_datetime (pandas/tslib.c:39919)
File "pandas/src/datetime.pxd", line 141, in datetime._string_to_dts (pandas/tslib.c:85631)
SystemError: <class 'str'> returned a result with an error set

[stan4git]

As I mentioned in another thread, there is already one pull request to fix this. #112
I has been running fine with below fix:

exec_time=pd.to_datetime(exec_detail.m_time, infer_datetime_format=True)

[HyperionSounds]

Thanks @stan4git , but neither fix have worked for me. Unfortunately, I am still getting thrown an error with either fix you have suggested.