Project Meeting 2022.02.24

Agenda

1. Intro to sharrow, by CS
2. Admin Items
   • Upcoming meeting agendas

Notes

What is sharrow?

• The sharrow package is an extension of numba, and offers access to data formatting and a just-in-time compiler specifically for converting ActivitySim-style “specification” files into optimized, runnable functions that “can approach the speeds of C or FORTRAN”. The idea is to pay the cost of compiling these specification files only once, and then re-use the optimized results many times.
• Information about sharrow is available on github: https://activitysim.github.io/sharrow/.
• The User's Guide walk through the features and how they are used.

Features

• Recompiling
  • If you change the utility function, it will detect this change and recompile.
  • If you change the machine, code/utility function, or any data type, this will trigger an automatic recompile process.
  • If you change a value in the utility function, it does not need to recompile. If you change the format/data type of a value (for example, 3.5 to an integer), it will recompile.
  • If the user deletes the cached files, it will recompile. (Cached files are approximately 1-2GBs for everything, for the MTC Model. They are separate for every single line function.) If you don't delete these, they can be reused, automatically detected, and there is no need to recompile.
  • You can move the cache between machines, but they need to be identical. If they aren't, the change will be detected and sharrow will recompile. Numba keeps track of what works and what does not work for specific machines.
  • You will not be automatically alerted when sharrow is recompiling. You can see that it is happening by viewing the folder and seeing if files are being added with recent file modification dates.
• Compatibility and testing
  • Dependencies on numba
  • Also dependent on xarray and a few others, but these aren't very restrictive
  • Requires Python >= 3.9
  • Sharrow has been tested and works with the MTC code. While it is expected to work with other ActivitySim models, it has not been tested yet.