diff --git a/docs/python/file_formats.rst b/docs/python/file_formats.rst index 5c8751e..4511b12 100644 --- a/docs/python/file_formats.rst +++ b/docs/python/file_formats.rst @@ -3,6 +3,8 @@ File formats We read and write a lot of CSV and JSON files. Their format should be consistent. +.. _format-json: + JSON ---- diff --git a/docs/python/performance.rst b/docs/python/performance.rst index f6da387..fa8b3f3 100644 --- a/docs/python/performance.rst +++ b/docs/python/performance.rst @@ -9,8 +9,27 @@ Performance :ref:`Django performance` -CPU profiling -------------- +Software performance depends on many choices: language (like :doc:`../rust/index` versus Python), framework (like `FastAPI versus Django `__), architecture (e.g. map-reduce), networking (e.g. batch requests), etc. Many choices are costly to change at a later date (e.g. full rewrite). + +Profiling +--------- + +Use profiling to: + +- Identify slow dependencies, in case faster alternatives can be easily swapped in +- Find major hotspots, like a loop that runs in exponential time instead of quadratic time +- Find minor hotspots, if changing language, etc. is too costly + +Once a hotspot is found, the solution might be to: + +- Call it once, via refactoring: for example, traversing JSON once for all CoVE calculations, instead of once for each calculation, in `lib-cove `__ +- Call it less, via batching: for example, :ref:`reducing the number of SQL queries in Django projects` +- Cache the results: for example, `caching mergers in Kingfisher Process `__ +- Process in parallel: for example, distributing work to multiple threads, like we do with :doc:`../services/rabbitmq` +- Replace it entirely: for example, using the :ref:`orjson` package instead of the ``json`` library + +CPU +~~~ For example: @@ -22,8 +41,8 @@ For example: To see where a running program is spending its time, use `py-spy top `__. -Memory profiling ----------------- +Memory +~~~~~~ For example: @@ -33,7 +52,6 @@ For example: time mprof run libcoveoc4ids data.json mprof plot - Reference ---------