COVID-19 Data by Johns Hopkins CSSE with addition of some data on Russia

Goals

1. Provide COVID19 dataset containing detailed information on Russia.
2. Maintain CSSE compatibility
3. Provide some higher level APIs for accessing the data.
4. Close the project after a more systematic approch is developed

• Upstream issue
• Generic local discussion (if any)

Disclamer: the author doesn't have relationships with any government or commercial organisations. The data provided here are collected from unreliable sources and may be not accurate. Use it at your own risk.

Отказ от ответственности: автор не имеет отношения к государственным или коммерческим организациям. Данные, приведенные здесь, собраны из ненадежных источников и могут быть неточными. Используйте их на свой страх и риск.

Contents

• csse_covid_19_data/csse_covid_19_daily_reports contains CSV files which were released by CSSE and later amended by us. Files released after March 25 were updates with additional information on Russian regions.
• csse_covid_19_data/csse_covid_19_time_series folder contains additional auto-generated timeseries on Russian regions.
• python3 folder contains Python development tools:
  • covid19ru.check module for checking certain invariants
  • covid19ru.fetch Yandex data fetcher
  • covid19ru.access Data accessor API
  • covid19ru.plot Matplotlib plotting

Data

Visualization

English version of the plot

Data source description

• https://github.com/CSSEGISandData/COVID-19
  • Upstream world data by CSSE.
• Rospotrebnadzor
  • The supposedly original official data source of COVID19 data in Russia. Data is published in Russian as a plain text. The source provides daily difference per region and current total for the whole state. Example: https://www.rospotrebnadzor.ru/about/info/news/news_details.php?ELEMENT_ID=14125
• Yandex COVID19 map
  • The Yandex company provides current per-region numbers.
• NovelCoronaVirusChannel at Telegram
  • Random COVID19 news in Russian.
• https://стопкоронавирус.рф//#

Update procedure

Originally, author filled the data on Moscow and Saint Petersburg manually, based on Rospotrebnadzor and NovelCoronaVirusChannel data. Starting from March, 25 we follow the below procedure:

1. Fetch hourly data from Yandex COVID map
   • Fetching is done by running monitor function of the fetcher script
   • The data is saved into pending folder, stamped with UTC time.
2. Fetch daily upstream updates by using regular git fetch manually.
3. If update is available,
   1. Rebase repository to upstream/master branch using git rebase
   2. For every csse_covid_19_data/csse_covid_19_daily_reports file which doesn't have russian details, do the following:
      1. Determine the update time of 'Russia' record found in the world data. The time is supposed to be UTC. The update time is often near 23:30 (supposedly UTC time).
      2. Find the russian details dump in pending folder which has the closest UTC timestamp.
      3. Update world information file by inserting russian details manually.
      4. Review the format compatibility (CSV fields order, date format, etc.).
      5. Update RU timeline by calling ru_timeline_dump() of access.py.
      6. Run the checker script.
      7. Update plots by running plot script.
      8. Commit the changes to this repository, forcebly push (due to rebase) here.

Related repos

• https://github.com/AlexxIT/YandexCOVID
• https://github.com/klevin92/covid19_moscow_cases
• https://github.com/wolfxyx/moscow-covid-19

Visualizations:

• https://github.com/AaronWard/covid-19-analysis

Roadmap

• Python code to check the correctness of CSV files
  • Python stub checking the validity of basic CSV structure (see ./python3/src/covid19ru/check.py )
  • Check less-trivial invariants
• Python API to access the CSV data. It should handle the CSV format change which happened around 23.03.2020
  • Pandas API (see ./python3/src/covid19ru/access.py)
  • Provide compatibility level for data before 23.03.2020
• Semi-automated data loader from Yandex. Ideally, we want to perform the following actions:
  • Collect Confirmed/Death/Recovered info for each Russian city (starting from 03-25-2020.csv)
  • Save this information in a temporary file to handle update gap
  • Set correct value of Longitude/Latitude for Russian regions
  • Figure out what does 'Active' field mean and how to get it.
    • Seems that it is just Confirmed-Deaths-Recovered. One have to update the data which miss this value.
• Make periodical dumps of rospotrebnadzor cite. Try to track possible source of data inconsistency.
• Auto-generate timeseries
• Daily update CSSE with Russian state information
• Find data on Russian regions for pre- 25.03.2020 period.

Log

14.04.2020

• Got an update on Komi republic.
• Decrease of confirmed cases in Amursk oblast is detected. The number falled from 17 to 6. Local news reported about 17 confirmed cases originally: https://vostok.today/33443-koronavirus-vs-prichiny-smertnosti-v-amurskoj-oblasti-bolshe-shansov-umeret-v-dtp-ili-ot-infarkta.html

13.04.2020

• Still no updates on Komi republic
• Added code to dump RU timeline. The up-to-date RU timeline file is available csse_covid_19_data/csse_covid_19_time_series/time_series_covid19_confirmed_RU.csv
• Also added deaths timeline csse_covid_19_data/csse_covid_19_time_series/time_series_covid19_deaths_RU.csv

12.04.2020

• No updates on Komi republic (3rd place among Russian regions) since 10.04.2020. Checked both Yandex and Rospotrebnadzor site.

08.04.2020

• More breaking changes from the upstream. The following daily data files have unmatching data foramt and extra symbols in the line ends:
  • 03-21-2020.csv
  • 03-29-2020.csv
  • 03-30-2020.csv
  • 04-06-2020.csv
• Updated issue CSSEGISandData#1523

01.04.2020

• More errors come from checker script, this time on Crimea:

  Error(file='COVID-19_plus_Russia/csse_covid_19_data/csse_covid_19_daily_reports/03-31-2020.csv',
  text='Confirmed decreased for Republic of Crimea from 20 to 16')
  

  That means that Yandex counters decrease their values. We can't name the reason, probably there were some corrections. One possible reason - splitting the Crimea into Crimea and Sevastopol.

30.03.2020

• Number of 'recovered' decreased in Sverdlovsk oblast
• Exact text of an error:

  Error(file='COVID-19_plus_Russia/csse_covid_19_data/csse_covid_19_daily_reports/03-29-2020.csv',
  text='Recovered decreased in Sverdlov oblast from 3 to 1 (oh no!)'),
  

25.03.2020

• Conflict resolved. 23-22-2020.csv file seemed to be damaged by the upstream admins.
• CSSEGISandData#1523
• Implemented Yandex data fetcher

23.03.2020

Upstream format change: now

• ,,Moscow,Russia,2020-03-24 00:00:00,55.75222,37.61556,262,1,9,,"Moscow, Russia"
• ,,"Saint Petersburg",Russia,2020-03-22 00:00:00,59.93863,30.31413,16,0,2,,"Saint Petersburg, Russia"

21.03.2020

We augmented CSV files from csse_covid_19_daily_reports folder by adding lines like:

• Moscow,Russia,2020-03-21T00:00:00,5,0,0,55.75222,37.61556
• "Saint Petersburg",Russia,2020-03-21T00:00:00,4,0,2,59.93863,30.31413

Original README.md starts here

2019 Novel Coronavirus COVID-19 (2019-nCoV) Data Repository by Johns Hopkins CSSE

This is the data repository for the 2019 Novel Coronavirus Visual Dashboard operated by the Johns Hopkins University Center for Systems Science and Engineering (JHU CSSE). Also, Supported by ESRI Living Atlas Team and the Johns Hopkins University Applied Physics Lab (JHU APL).

Visual Dashboard (desktop):
https://www.arcgis.com/apps/opsdashboard/index.html#/bda7594740fd40299423467b48e9ecf6

Visual Dashboard (mobile):
http://www.arcgis.com/apps/opsdashboard/index.html#/85320e2ea5424dfaaa75ae62e5c06e61

Lancet Article:
An interactive web-based dashboard to track COVID-19 in real time

Provided by Johns Hopkins University Center for Systems Science and Engineering (JHU CSSE):
https://systems.jhu.edu/

Data Sources:

• World Health Organization (WHO): https://www.who.int/
  
• DXY.cn. Pneumonia. 2020. http://3g.dxy.cn/newh5/view/pneumonia.
  
• BNO News: https://bnonews.com/index.php/2020/02/the-latest-coronavirus-cases/
  
• National Health Commission of the People’s Republic of China (NHC):
  http://www.nhc.gov.cn/xcs/yqtb/list_gzbd.shtml
  
• China CDC (CCDC): http://weekly.chinacdc.cn/news/TrackingtheEpidemic.htm
  
• Hong Kong Department of Health: https://www.chp.gov.hk/en/features/102465.html
  
• Macau Government: https://www.ssm.gov.mo/portal/
  
• Taiwan CDC: https://sites.google.com/cdc.gov.tw/2019ncov/taiwan?authuser=0
  
• US CDC: https://www.cdc.gov/coronavirus/2019-ncov/index.html
  
• Government of Canada: https://www.canada.ca/en/public-health/services/diseases/coronavirus.html
  
• Australia Government Department of Health: https://www.health.gov.au/news/coronavirus-update-at-a-glance
  
• European Centre for Disease Prevention and Control (ECDC): https://www.ecdc.europa.eu/en/geographical-distribution-2019-ncov-cases
• Ministry of Health Singapore (MOH): https://www.moh.gov.sg/covid-19
• Italy Ministry of Health: http://www.salute.gov.it/nuovocoronavirus
• 1Point3Arces: https://coronavirus.1point3acres.com/en
• WorldoMeters: https://www.worldometers.info/coronavirus/
• COVID Tracking Project: https://covidtracking.com/data. (US Testing and Hospitalization Data. We use the maximum reported value from "Currently" and "Cumulative" Hospitalized for our hospitalization number report ed for each state.)

Additional Information about the Visual Dashboard:
https://systems.jhu.edu/research/public-health/ncov/

Contact Us:

• Email: [email protected]
  
  

Terms of Use:

This GitHub repo and its contents herein, including all data, mapping, and analysis, copyright 2020 Johns Hopkins University, all rights reserved, is provided to the public strictly for educational and academic research purposes. The Website relies upon publicly available data from multiple sources, that do not always agree. The Johns Hopkins University hereby disclaims any and all representations and warranties with respect to the Website, including accuracy, fitness for use, and merchantability. Reliance on the Website for medical guidance or use of the Website in commerce is strictly prohibited.