Hot Swapping
Loading bundles in OneBusAway core used to be a manual process that consists of manually creating a valid bundle file, moving the latest bundles to each server that uses the Transit Data Service, and finally loading each of the bundles by stopping and starting each Transit Data Server instance. To facilitate this process we've implemented some optional processes that help make these steps as automated as possible.
The Admin Server serves as the starting point for the bundle building process. An administrator uses the Transit Data Bundle Utility to generate a bundle package. During the bundle creation process a meta data file containing all the bundle paths, dates, and other statistical information is generated After completing the "deploy" step of the bundle build process, the bundle then becomes available for download by any of the existing TDS Servers. In order to make the newly staged bundle discoverable, the Admin server also provides an API that lists out all the currently available bundles. This API takes the previously generated bundle metadata and makes it accessible so that the Transit Data Server knows where to retrieve the bundle files.
After querying the Admin Server for the latest list of bundles, the TDS compares the list of bundles that it retrieved from the Admin Server to list of bundles that it currently has available. If the Admin Server is advertising a bundle that the Transit Data Server does not currently have, it attempts to download the files. During the download process the TDS verifies that the bundle files are all valid by checking the MD5 Checksum of all the bundle files for the expected values. If the MD5 Hash values of the files do not match it attempts to download the files again and marks the file bundle as corrupt in the event that the retry also fails. This is done to ensure that the TDS does not load potentially corrupted bundle files.
After downloading all newly available bundles, the Transit Data Service evaluates which bundle is the most appropriate to load
After deciding which bundle to load, the TDS then schedules the bundle swap to take place on the hour. This is done to make sure that all the TDS Servers load any new bundles at the same time to prevent synchronization issues.
When the scheduled bundle swapping task is run, its swaps in the latest bundle in real-time, preventing any potential downtime from having to stop the TDS to load in the new bundle.
The TDS continues to query the Admin Server for new bundles every 15 minutes
For added resiliency, the TDS fails over to use the bundles in its local bundle path if it cannot communicate with the Admin Server