Project Meeting 2018.10.05

Multiprocessing

• Prototype configuration example on the Multi-Processing design page
• Rough prototype ActivitySim multiprocessing (asim mp) is running
  • Running on Windows as long as skims don’t exceed 2GB due to limitation of RawArray size in Python 2.7 on Windows due to sys.maxint
  • The runtime gains from single to mp are dependent on the machine RAM, processors available, number of HHs, chunk size, and amount of machine RAM to leave for other processes. We will test and document a number of configuration scenarios as we work on this.
  • Support for multi-processing the accessibility step has also been prototyped and will be added to asim
  • Running the entire example dataset (797,674 persons with 190 zones) on the SD modeling server took:

windows single process:
INFO - activitysim.core.tracing - Time to execute mp_simulation : 4383.497 sec (73.1 min)

windows with 3 processes:
INFO - activitysim.core.tracing - Time to execute mp_simulation : 2693.601 sec (44.9 min)

• Supporting both Python 2.7 and Python 3? #206
  • Tests show no limitation to size of RawArray in Python 3
  • Tests show that variable number of shared RawArraw objects can be passed to subprocess for implementation in 2.7.
  • Python 2 will not be maintained past 2020 - https://pythonclock.org
  • Started looking into porting to Python 3. It looks like we could have a code base that runs on both Python 2 and 3.
  • There are a few modules that don’t have Python 3 distributions
    • We should probably move orca into activitysim and strip out everything we don’t need
    • Someone needs to port omx
    • It appears zbox may already work with 3
  • Supporting 3 includes updating these things
• We decided to support both Python 2.7 and 3 and to roll the parts of orca we use into activitysim
  • We'll look into getting omx and zbox? updated to 3 outside of this project
  • This additional work will be done under the existing budget
  • We will gradually update asim to work under 2.7 and 3 by the end of this phase of work

Outreach

• The Florida Model Task Force has invited us to share the ActivitySim effort
• @Guy and @Ben to hopefully present