Geospatial Animations and Visualization
The purpose of this page is to capture some learnings from a recent few days of investigating technologies to generate and share geospatial visualizations of disease case data over time.
We have disease case data from different locations over time. Specifically right now we have measles case data from 954 towns in England and Wales covering 2 decades from 1944 to 1964. So that includes the reference data and simulation data. It's very helpful to visualize this data on a map as an animation over time. While these visualizations don't provide us with actionable quantitative outputs like a formal analysis does, they do inform our brains of patterns or non-patterns that serves to help create some intuition and gut-checks.
There have been a variety of technologies available for the years to help with these visualizations. We've also recently lost two of our experts who might have been able to take over these tasks. So we recently did a little exploration of current technologies, and sought to visualize our reference and simulation data. And then share them. Below is what we found.
We allowed ChatGPT to recommend technologies. It pointed us to Folium, which implements a Leaflet.js solution in the form a (large) HTML file which can be rendered in a browser. This is the solution we spent the most time with and ultimately got working. But there were definitely some issues and frustrations and learnings.
Note that all these visualizations were from after-the-fact data. None of these were being produced in real-time from a running simulation.
Sample screengrab
Code sample:
import folium
from folium.plugins import HeatMapWithTime
from engwaldata import data as engwal
# Create a map centered at a specific location
m = folium.Map(location=(engwal.places['London'].latitude,engwal.places['London'].longitude), zoom_start=12)
# Create a list to store the data for HeatMapWithTime
data = []
max_cases = 4103 # yes, by hand. obviously this can be automated.
all_cities = list(engwal.places.keys())
# Iterate over each place in engwal.places
for t in range(len(engwal.places["London"].cases)):
data.append([[engwal.places[place].latitude, engwal.places[place].longitude, engwal.places[place].cases[t]] for place in all_cities])
data[-1] = [ [ item[0], item[1], item[2]/max_cases ] for item in data[-1] ]
# Create HeatMapWithTime layer
heat_map = HeatMapWithTime(data,use_local_extrema=True)
# Add HeatMapWithTime layer to the map
heat_map.add_to(m)
# Save the map as an HTML file
m.save('disease_map_animation.html')`
Issues
It's hard to summarize here, but we found that folium does best when the case values are relatively uniform between 0 and 1. Starting off with all 0s -- as one might do in real life -- seems to mess it up. We ended up going to some lengths to normalize our data and also add a random tiny jitter to get around the issues we were encountering. This all sounds quite unsatisfactory and could be explored further. But if one's willing to work around those issues as they arise, Folium seems to be a very satisfactory solution relative to the alternatives.
Spent about 30 minutes on this. Really hard to get real map underneath.
Spent about 30-60 minutes on this. Seemed to get some working code but blew up browser twice with our data set, so discontinued.
Spent about 30 minutes on this. Didn't seem like it was for animation.
KML is great if you want to make people use Google Earth.
The html file produced by Folium can be tens of MB. To host this I created a Dockerfile, built a docker image with Jenkins, hosted in Artifactory, and deployed in my kubernetes cluster created in VMWare (Aria). Then produces an internally accessible IP address that folks can hit from their browser.
Here's a current example with the reference data visualization: http://10.24.49.67:8080/. (Note that this particular endpoint could easily go out of date.)
There is now a single script https://github.com/InstituteforDiseaseModeling/laser/blob/jbloedow/end_of_may_wip/jb/utils/folium_animate_from_sqlite.py which will take a simulation_output.csv file and create a sim_animation.html file. The script includes an instruction for how to serve this if one doesn't want to just load it directly into a browser.
We will soon be creating a pip installable package (laser? idmlaser? laser-core?) such that one could do:
python -m idmlaser.animate <output file>