[Question] How is the code handling huge amount of tiles ?

[idouiri9806]

Hi Jaffa!

Great project! I'm new to Flutter and would like to create something similar for my school project. I just want to understand the logic behind how the tiles are stored on the device and then retrieved. After the download is complete, how are the tiles for a specific region read and plotted at the correct location on the map? How are they stored — in XYZ folders, PMTiles, MBTiles, or vector formats?

I remember there was a document that explained all this. It was a great read, but I can't seem to find it anymore. If it's not too much trouble, could you explain it again? A short one would be ok. Thank you.

[JaffaKetchup]

Hi @idouiri9806,

Great to hear that you like the project! To be fair, a lot of the heavy lifting is done by flutter_map and ObjectBox, but all the connecting components and tile generation for bulk downloading is done by us.

---

In browse caching, flutter_map asks the tile provider for a tile at some specific coordinates - it is up to FMTCTileProvider to get an image with these coordinates, as well as the URL template and other metadata. The logic internally is extremely complex, and difficult to explain linearly. Most of the logic is within internal_get_bytes.dart (in v10), which also calls on readTile and writeTile from the thread worker. The actual comparison of what tile should be used to fill what tile coordinate is done based on the string equality of the 'storage-suitable UID' - the result of urlTransformer being run on the generated tile URL (see FM's TileProvider interface).

In the default implementation, a long-lived worker thread runs in the background that communicates with the database, to try to keep things fast. Whether this is actually necessary is a point for discussion - I can't find it right now, but there was a comment about this recently. The main thread is implemented in this file, whilst the worker thread is implemented in here.

In terms of how tiles are actually stored: they are just stored as blobs with some metadata in the ObjectBox database. In theory, the storage engine could actually be a seperate from FMTC - it should be able to store any binary blob, including vector tiles or raster tiles, or any other file type. It could make quite an advanced and interesting storage library 🤔.

---

In bulk downloading, the downloader gets a region, defined by its coordinates. There's a generator and counter algorithm for each region type - the counter is a generalisation/simplification/shortcut of the generator, that only needs to count tiles, it does not need to loop through them all (so, for example, it's easy to calculate the number of tiles in a rectangle region using multiplication, but the generator must use a nested loop to generate each tile's coordinates). At the start of the download, the counter is called, which allows the number to be reported in statistics, and a couple other bits and pieces.

The speed comes from heavy use of multi-threading (Isolates) and optimisation - there's probably loads more that could be done, but there's no point, since the bottleneck is pretty much always the network (FMTC can download at 1,500+ tiles/second on my laptop when the tile server is the testing tile server running locally).

The downloader is essentially split into 3 parts, each of which is running in an Isolate:

• Worker
  The responsibility of this thread is to continuously download tiles, add them to a queue to be written to the database or write them to a database, and ask the manager thread for a new tile, killing itself when there are no new tiles available. It should not be confused with the standard long-lived thread worker used throughout FMTC - in fact, during downloading, this is not used, and instead a special 'thread-safe' communicator is used, which can be passed through these threads, as the database connection itself cannot be passed to/from Isolates, and different methods are needed for efficiency and purpose whilst downloading. There may be more than one worker on each download.

• Tile generator
  The responsibility of this thread is to generate the next tile coordinate when asked by the manager, killing itself when there are no tiles left within the given region and range

• Manager
  The most complex. It connects multiple workers to the tile generator, and manages progress reporting, rate limiting, startup, teardown, recovery, etc. It starts the given number of threads as workers, then listens to them, each separately, not as a merged stream or similar - this is not necessarily a better way to do it, it's just how I designed it. When it receives a TileEvent (indication of a tile download) from a thread, it collates it with the other information it knows, and pushes it to the main thread (the resulting stream of startForeground). When it receives 0 (indicator of readiness for a new tile) from a thread, it asks the tile generator for a new tile coordinates, then waits for one, and finally passes it off to the thread. If there is no new thread, it sends null to the thread, prompting it to clean itself up (write any remaining queue, close http client, etc), then kill itself, which is detected as null message by the manager. When null is received, it closes the port used for communication with the thread, which is detected as the onDone event in the listener that was on it. This then fires a universal cancelSignal (which is also the same signal fired when the download is cancelled externally), which essentially blocks threads from getting new tile coordinates (because the generator is already shutdown by this point), causing each thread to kill itself when it tries to get new tile coordinates. Additionally, it also fires a local scope threadKilled signal (each thread has its own). The manager is waiting for every thread's killed signal to fire, then it starts the final cleanup.

Essentially, flow is controlled by a series of Completers and Streams via Isolate communication ports, and the fine detail can be quite overwhelming for beginners with Isolates. Indeed, I make no promises that what I have done is best practise :D.

---

To make things fast, it's just lots of Isolates, lots of lines of code to do simple tasks at first look (114 to write/update a single tile to multiple stores) - a lot of the complexity is added by keeping the statistics up to date, so that a statistic retrieval can be as fast as possible. For any more detail, I would advise looking at the implementation. Lookup is fast because of ObjectBox and the magic of indexes.

One thing you will notice is that mass/bulk operations can be quite slow - for example, deleting a large store may take some time. This is because we have to run through every tile that belongs to that store, and attempt to orphan it. If it is orphaned (it only belonged to that store), then we can delete it. If it still belongs to any other stores, we also have to update all of those store's statistics, as well as the tile itself. But we try to make some optimisations at the expense of memory-consumption to minimize the number of actual database writes made.

Again, a lot of the heavy lifting is done by ObjectBox - the storage engine - and flutter_map - which actually lays out the tiles. How those work is a different story, but I would recommend looking at the implementation of flutter_map's TileLayer to find out how the layout is done. The purpose of FMTC is to bridge those two libraries (and theoretically any other storage engine) in a way that is efficient and extremely flexible. The complexity added by v10 is quite large - as are the API changes coming 🫥.

I hope this was helpful!