download_osm_data.py got Error parsing message with type 'OSMPBF.Blob'

[loongmxbt]

Hi, I'm trying PyPSA-earth on a Mac M2 chip computer. All the previous steps works fine, but when get into the download_osm_data step, it has an error and I'm not sure if I've missed some previous steps or dependencies. Hope for some tips.

multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/Users/dragonszy/miniconda3/envs/pypsa-earth/lib/python3.10/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/Users/dragonszy/miniconda3/envs/pypsa-earth/lib/python3.10/multiprocessing/pool.py", line 51, in starmapstar
    return list(itertools.starmap(args[0], args[1]))
  File "/Users/dragonszy/miniconda3/envs/pypsa-earth/lib/python3.10/site-packages/earth_osm/extract.py", line 50, in filter_file_block
    entries.ParseFromString(read_blob(file, ofs, header))
  File "/Users/dragonszy/miniconda3/envs/pypsa-earth/lib/python3.10/site-packages/earth_osm/osmpbf/file.py", line 54, in read_blob
    blob.ParseFromString(file.read(header.datasize))
google.protobuf.message.DecodeError: Error parsing message with type 'OSMPBF.Blob'
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/dragonszy/Github/python/pypsa-earth/scripts/download_osm_data.py", line 108, in <module>
    eo.get_osm_data(
  File "/Users/dragonszy/miniconda3/envs/pypsa-earth/lib/python3.10/site-packages/earth_osm/eo.py", line 101, in get_osm_data
    df_feature = process_country(region, primary_name, feature_name, mp, update, data_dir)
  File "/Users/dragonszy/miniconda3/envs/pypsa-earth/lib/python3.10/site-packages/earth_osm/eo.py", line 38, in process_country
    primary_dict, feature_dict = get_filtered_data(region, primary_name, feature_name, mp, update, data_dir)
  File "/Users/dragonszy/miniconda3/envs/pypsa-earth/lib/python3.10/site-packages/earth_osm/filter.py", line 111, in get_filtered_data
    primary_dict = run_primary_filter(PBF_inputfile, primary_file, primary_name, mp)
  File "/Users/dragonszy/miniconda3/envs/pypsa-earth/lib/python3.10/site-packages/earth_osm/filter.py", line 70, in run_primary_filter
    primary_data = filter_pbf(PBF_inputfile, pre_filter, multiprocess)
  File "/Users/dragonszy/miniconda3/envs/pypsa-earth/lib/python3.10/site-packages/earth_osm/extract.py", line 93, in filter_pbf
    primary_entries = list(file_query(primary_entry_filter, pre_filter)) #list of named  tuples eg. Node(id,tags, lonlat)
  File "/Users/dragonszy/miniconda3/envs/pypsa-earth/lib/python3.10/site-packages/earth_osm/extract.py", line 66, in query_func
    entry_lists = pool.starmap(
  File "/Users/dragonszy/miniconda3/envs/pypsa-earth/lib/python3.10/multiprocessing/pool.py", line 375, in starmap
    return self._map_async(func, iterable, starmapstar, chunksize).get()
  File "/Users/dragonszy/miniconda3/envs/pypsa-earth/lib/python3.10/multiprocessing/pool.py", line 774, in get
    raise self._value
google.protobuf.message.DecodeError: Error parsing message with type 'OSMPBF.Blob'

Here is my data/osm folder:

├── osm
│   ├── pbf
│   │   ├── china-latest.osm.pbf
│   │   └── nigeria-latest.osm.pbf
│   └── power

[loongmxbt]

Seems the pbf file is broken, let me re-download and try again.

[loongmxbt]

Solve the previous problem by re-download the file. Stuck at the new problem. Hang on Compute population for hours. Already re-download the file https://data.worldpop.org/GIS/Population/Global_2000_2020/2020/CHN/chn_ppp_2020.tif

INFO:__main__:Stage 4/4 POP: Add population data to GADM GeoDataFrame
This is the repository path:  D:\Github\Python\pypsa-earth
Had to go 0 folder(s) up.
This is the repository path:  D:\Github\Python\pypsa-earth
Had to go 0 folder(s) up.
This is the repository path:  D:\Github\Python\pypsa-earth
Had to go 0 folder(s) up.
Download WorldPop : 100%|████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  7.25 countries/s]
Compute population :   0%|                                                                       | 0/3 [00:00<?, ?it/s]This is the repository path:  D:\Github\Python\pypsa-earth
Had to go 0 folder(s) up.
This is the repository path:  D:\Github\Python\pypsa-earth
Had to go 0 folder(s) up.
This is the repository path:  D:\Github\Python\pypsa-earth
Had to go 0 folder(s) up.

[pz-max]

Hey @loongmxbt, add population currently take for US & China up to a day and might also require a lot memory (64GB RAM should be fine). In case of Nigeria it's much faster (<<1 hour) and less memory intense. If you check the PR's we are currently working to reduce this to less than 1 hour.

[loongmxbt]

Hey @loongmxbt, add population currently take for US & China up to a day and might also require a lot memory (64GB RAM should be fine). In case of Nigeria it's much faster (<<1 hour) and less memory intense. If you check the PR's we are currently working to reduce this to less than 1 hour.

Thanks Max, I currently switch to a Windows computer (64GB) and set worldpop_method and gdp_method to false. Now I'm going to Verify lines overpassing nodes step and shows 3 hours total. I previously set the command the --cores 1 and leave it alone, does this step will increase speed by setting multiple cores?

[pz-max]

Sounds good. Yes, some parts are configured to benefit from multiprocessing. 👍🏽 Sent from Outlook for iOS<https://aka.ms/o0ukef>

…

________________________________ From: Saint Asky ***@***.***> Sent: Tuesday, April 4, 2023 1:06:22 PM To: pypsa-meets-earth/pypsa-earth ***@***.***> Cc: Maximilian Parzen ***@***.***>; Comment ***@***.***> Subject: Re: [pypsa-meets-earth/pypsa-earth] download_osm_data.py got Error parsing message with type 'OSMPBF.Blob' (Discussion #659) This email was sent to you by someone outside the University. You should only click on links or attachments if you are certain that the email is genuine and the content is safe. Hey @loongmxbt<https://github.com/loongmxbt>, add population currently take for US & China up to a day and might also require a lot memory (64GB RAM should be fine). In case of Nigeria it's much faster (<<1 hour) and less memory intense. If you check the PR's we are currently working to reduce this to less than 1 hour. Thanks Max, I currently switch to a Windows computer (64GB) and set worldpop_method and gdp_method to false. Now I'm going to Verify lines overpassing nodes step and shows 3 hours total. I previously set the command the --cores 1 and leave it alone, does this step will increase speed by setting multiple cores? — Reply to this email directly, view it on GitHub<#659 (reply in thread)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AOYZENP5VNUIVXRY6RBRJGLW7QFD5ANCNFSM6AAAAAAWPJN3VU>. You are receiving this because you commented.Message ID: ***@***.***> The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Is e buidheann carthannais a th’ ann an Oilthigh Dhùn Èideann, clàraichte an Alba, àireamh clàraidh SC005336.

[ekatef]

@loongmxbt thanks a lot for posting the error message. It has been very helpful to troubleshoot loading issues 🎉

Update: the issue is known in earth-osm and will be addressed in a systematic way.