-
Notifications
You must be signed in to change notification settings - Fork 10
History
The Smart Cities Data project began as the Smart Columbus Operating System, a component of the Smart Columbus initiative out of Columbus, Ohio, United States.
In 2016, the city of Columbus entered and won the United States Department of Transportation's first ever Smart Cities Challenge, a competitive grant process against 77 cities nationwide to develop the local economy, sustainable transportation and logistics technology, and improve the community. In order to coordinate the wealth of data a Smart and Connected city generates and solve real community problems by the application of targeted analysis of data, the Smart Columbus Operating System was envisioned as the hub of information generated and consumed by the rest of the Smart Columbus initiative and, eventually, the City as a whole.
As part of the grant, the USDOT strongly encouraged the data platform developed by Columbus to accommodate these requirements be built on the public cloud to reduce the cost requirements of maintaining the platform and that it be built from Free and Open Source Software. Additionally, this software should run equally on top of any public cloud provider's infrastructure. This combination of open source and cloud agnostic technology would ensure that adoption and maintenance costs for any other city would be minimized and the results of Columbus's efforts could benefit everyone.
The first iteration of what would become the Smart Cities Data platform was a customized local instance of the open data CKAN project with a customized interface built in the open source Joomla content management system. This provided the project with a jumping off point to gather lots of disparate data sets and make them available programmatically while maintaining reasonable access controls. However, this solution would not scale its operations to the desired level, was more difficult to maintain than desired, and would not support the need for realtime streaming data sources or adaptive transformations of data to which the project aspired.
For the next phase, dubbed the "operating system" of the Smart City, Hadoop was envisioned to provide the capabilities CKAN lacked and would natively support the scale of data and the streaming capabilities needed for a coordinator of connected vehicles and IoT devices. However, the ability of Hadoop at the time, particularly flavors not dependent on cloud vendor-specific implementations, to scale dynamically at reasonable costs was suboptimal. Additionally, significant development efforts would be necessary to add the dynamic data streaming and adaptive transformation capabilities via components native to the Hadoop ecosystem and such solutions would not be maintainable at the costs desired by the project.
After trial and error with all of these technologies, the team determined the best way to provide the desired functionality, at the lowest maintainable cost, was to develop it from scratch and let open source adoption drive future growth. While this may seem a stretch, developing a solution natively bypassing many of the constraints of the project allowed the team to target all the necessary feature requirements and rely on open source standards, standard cloud-provider services, and cutting edge technology to backfill the ongoing support and maintenance needs of the platform.