Set OGR_GEOMETRY_ACCEPT_UNCLOSED_RING configuration option

[jkaucic]

Hello, I'm using osmextract to extract large portions of the osm.pbf file for Europe. I am using a standard config.ini and tweaked the vectortranslate_options argument to select only certain fields and tags. This works fine up to the point that after finishing the process tells me (very unspecificly): "Error in sf::gdal_utils(util = "vectortranslate", source = normalizePath(file_path), : gdal_utils vectortranslate: an error occured. The warnings all relate to "In CPL_gdalvectortranslate(source, destination, options, ... : GDAL Message 1: Non closed ring detected. To avoid accepting it, set the OGR_GEOMETRY_ACCEPT_UNCLOSED_RING configuration option to NO.

It seems that the vectortranslation finishes despite the error I receive, but does not extract all features (while when I do the same extraction for a smaller bounding box, no errors occur and many of valid features seem to have been excluded in the first extract as well).

Does the "Error in sf::gdal_utils" lead to osmextract stopping the process right away? Is there a possibility to change the "OGR_GEOMETRY_ACCEPT_UNCLOSED_RING configuration option to NO" and would this help in this situation? I am also not sure how to get more information on the "Error in sf::gdal_utils" that could give more hints to debug this situation. In another extract I also had the "OGR_GEOMETRY_ACCEPT_UNCLOSED_RING" warning but no "Error in sf::gdal_utils" and the extraction seemed to complete just fine.

Sorry for the long question and hoping that anyone has some clues on this issue.

[agila5]

Hi @jkaucic. Thank you very much for your question and for using this package. I will try to investigate the issue that you mentioned as soon as possible. Just a couple of questions:

1. Can you share a portion of your code to replicate the error?
2. How can you say that many of valid features seem to have been excluded in the first extract as well?

[jkaucic]

Thank you for your swift reply! I can provide you with the portion of the code, but it is a very complex query from the multipolygons layer, which took about 6 hours on a i5-8500 3,00 Gig processor with 16 GB RAM. It is about green spaces for the whole Europe.osm.pbf, combined from a selection of OSM tags (leisure, landuse and natural). Admittedly, these are huge data quantities, which is why I resorted to osmextract.

First, I define the vectortranslate_options:

poly_amenities_green_low_vectortranslate = c(
   #modify the output layer name
  "-nln", "OSM_20230203_Europe_green_low_polygon",
  # modify the CRS of the output.qpkg
  "-t_srs", "EPSG:3035",
  # SQL-like query where we select only the following fields
  "-select", "osm_id, name, leisure, landuse, natural, access, garden_type", 
  # SQL-like query where we filter only the features (keys/tags) we need
  "-where", "(leisure IN ('park', 'nature_reserve') AND access NOT IN ('private', 'no', 'customers', 'permit', 'license', 'restricted', 'agricultural', 'forestry')) OR (leisure = 'garden' AND access NOT IN ('private', 'no', 'customers', 'permit', 'license', 'restricted', 'agricultural', 'forestry') AND landuse NOT IN ('allotments', 'residential') AND garden_type IN ('community', 'botanical', 'public', 'flowerbed', 'municipal', 'street_side')) OR (landuse IN ('grass', 'forest', 'meadow', 'flowerbed', 'village_green') AND access NOT IN ('private', 'no', 'customers', 'permit', 'license', 'restricted', 'agricultural', 'forestry')) OR (natural IN ('beach', 'heath', 'wood', 'fell', 'grassland', 'scrub', 'tundra') AND access NOT IN ('private', 'no', 'customers', 'permit', 'license', 'restricted', 'agricultural', 'forestry'))"
  )

Then, I extract the gpkg layer from the pre-downloaded Europe.osm.pbf from Geofabrik (27.5 gig):

oe_get("Europe", layer = "multipolygons", provider = "geofabrik", force_download = FALSE, vectortranslate_options = poly_amenities_green_low_vectortranslate, extra_tags = c("leisure", "landuse", "access", "natural", "garden:type"), download_only = TRUE, skip_vectortranslate = FALSE, never_skip_vectortranslate = TRUE, quiet = FALSE)

This first extract for all of Europe yielded 802,277 features extracted (with 2,278 geometries empty) (strange that there are empty geometries there as well). To debug, I tried to extract only landuse=grass and this yielded 3,783,685 features (in about 7 hours processing time). Therefore, it is clear that in the complex extraction with the error message, a lot of features were not extracted.

Also visual inspection of an extract with all relevant tags only for Austria compared with the one for whole of Europe showed that the full extraction finished with a lot of features missing. So it seems that due to the error message the extraction finished, but was incomplete. However, I do not quite know what exactly the error was and how to define the query to have a full extraction of features.

Currently I'm trying to run a code with "-makevalid", "-nlt", "PROMOTE_TO_MULTI" and "-skipfailures" added to the vectortranslate_options. However, it seems that skipfailures slows down the process considerably.

[agila5]

Admittedly, these are huge data quantities, which is why I resorted to osmextract.

Wow, this is probably the largest example of using OSM data I've ever seen. It sounds like a super interesting use case (and, unfortunately, also a quite challenging problem to diagnose...). I'll do my best to run some tests in the next few days and report here in case I find any solution.

I can provide you with the portion of the code, but it is a very complex query from the multipolygons layer, which took about 6 hours on a i5-8500 3,00 Gig processor with 16 GB RAM.

What is the largest OSM extract where you observe this problem?

[jkaucic]

I am still running tests but I am wondering, if the operation just runs out of writable disk space (but this is not visible in the error message, or is it? Where could I find this information on the error). I am currently running an extract with only part of the query and task manager tells me that volume utilization is ranging from 30MB/sec to 160MB/s (i don't know how this actually translates into data being written). And if that process runs for 7 hours, then we get hundreds of gigs of data.

A question that comes to my mind is how the vectortranslate process actually works: Which data is actually written into the gpkg? Is it the final result of the query or are there any 'intermediate' writing steps (which are removed later). The operation mainly uses volume/disk and not really CPU and RAM.

Edit: When I right click on the GPKG during the process, it only shows me 64kb - so the file size is not updated during extraction. Could this be a hint to my question in the previous paragraph?

I have the feeling that the extract with errors leads to huge dumps of file sizes, even though only low amount of features extracted. The GPKG with the error message and 'only' 700k extracted features is huge with 2.86 gigabytes. I will compare this with the selective query and no gdal error. I assume that the extraction of 3+ Million features without error message is considerably smaller (extraction still running, i'll update soon)

[jkaucic]

Edit 2: I have the answer now:

40: In CPL_gdalvectortranslate(source, destination, options, ... :
GDAL Error 1: failed to execute insert : database or disk is full
41: In CPL_gdalvectortranslate(source, destination, options, ... :
GDAL Error 1: Unable to write feature 6258254 from layer multipolygons.
42: In CPL_gdalvectortranslate(source, destination, options, ... :
GDAL Error 1: Terminating translation prematurely after failed
translation of layer multipolygons (use -skipfailures to skip errors)
43: In CPL_gdalvectortranslate(source, destination, options, ... :
GDAL Error 1: sqlite3_exec(ROLLBACK) failed: cannot rollback - no transaction is active

It didn't show me those errors because I only saw the first 50 (which were unimportant ones compared to these; and, sorry, I'm quite new to R: I don't know how to see the full error protocol).

Some takeaways: The final gpkg after this failed extraction is 900MB containing a Simple feature collection with 271105 features and 7 fields (with 71106 geometries empty). I still have 71 GIG disk space left, so what does 'GDAL Error 1: failed to execute insert : database or disk is full' mean? Finally, -skipfailures in this case makes no sense because it just prolongs the extraction process, while for every new feature to be extracted, the disk still will be (seemingly) full.

[agila5]

Hi! Today I run some tests with the code that you attached (on a powerful VM with a lot of free space) considering the OSM extract for FR and GB and the function was executed without any problem. Tomorrow I will test it with the whole Europe.

I am still running tests but I am wondering, if the operation just runs out of writable disk space (but this is not visible in the error message, or is it? Where could I find this information on the error).

At the moment, I don't know... According to the tests I previously mentioned, the .gpkg file returned by the previous operations is smaller than the corresponding .pbf file (and I think that's reasonable since the .pbf file contains data for 5 different layers, although in a compressed format). For example, the .pbf file for France is approximately 4GB while your query returned 1.3GB of data (can you confirm those figures?)

A question that comes to my mind is how the vectortranslate process actually works: Which data is actually written into the gpkg? Is it the final result of the query or are there any 'intermediate' writing steps (which are removed later). The operation mainly uses volume/disk and not really CPU and RAM.

To be honest, I don't know all the details. I think that GDAL writes intermediate data steps to a so-called .gpkg.journal file but I think we should ask the GDAL experts. I will write an email in case I cannot figure it out.

Edit: When I right click on the GPKG during the process, it only shows me 64kb - so the file size is not updated during extraction. Could this be a hint to my question in the previous paragraph?

I don't think so since I see exactly the same behaviour but the process completes without any problem.

Some takeaways: The final gpkg after this failed extraction is 900MB containing a Simple feature collection with 271105 features and 7 fields (with 71106 geometries empty). I still have 71 GIG disk space left, so what does 'GDAL Error 1: failed to execute insert : database or disk is full' mean?

During the weekend I will run some tests using my laptop (which has just a few GB of free space) and I will try to replicate the error that you mentioned. I'll add some updates here ASAP.

[jkaucic]

Wow, thank you so much for your time and your help! This is very much appreciated!

[agila5]

I just completed the test on the whole Europe.pbf file and it runs without any problem in approximately 4h. The original file is approximately 28GB and the output .gpkg file is 16.6GB. However, I noticed that the ogr2ogr process creates some intermediate files (not immediately visible, at least to me) that occupy at least 30GB more. These intermediate files seem to be deleted at the end of the conversion process. Next week, I will run another test on a small laptop and add here an update. Anyway, I suspect that the problem is that, at some point, you are running out of writable disk space. Maybe you can explore some values for the argument -gt in ogr2ogr: https://gdal.org/programs/ogr2ogr.html#ogr2ogr

btw, check the following if you need to run some code al display all relevant warning messages:

Note that the length(last.warning) is maximally getOption("nwarnings") (at the time the warnings are generated) which is 50 by default. To increase, use something like
options(nwarnings = 10000)

[jkaucic]

Did you run the whole query including leisure, landuse and natural tags? 4h is impressive.. on what kind of system did you do that? I guess the main speed factor here would be disk writing time, so a ssd would do the trick, right?

[agila5]

Did you run the whole query including leisure, landuse and natural tags?

Yes!

on what kind of system did you do that?

TBH, I'm really a noob on how to use these VM... @Robinlovelace an easy way to get these info on ubuntu systems?

[Robinlovelace]

Not sure either sorry Andrea and @jkaucic. Keep us posted on progress, may well be useful for others!

[jkaucic]

Because my large vectortranslate operations always slow down considerably starting with 60% progress: Will setting -lco SPATIAL_INDEX=NO speed up the vectortranslate? As far as I understand, it is set to YES by ogr2ogr default. When I understand correctly the sf package https://r-spatial.org/r/2017/06/22/spatial-index.html , a spatial index is created from scratch for spatial operations on sf objects anyways - so a spatial index is not really needed (and added later to the gpkg anyways) and might slow the conversion from pbf to gpkg?

[agila5]

I don't know... It might be beneficial (see r-spatial/sf#1409) but I suspect that it might not solve the problem that you mentioned in the first message.

[jkaucic]

Just to answer my own question here: setting SPATIAL_INDEX to no doesn't increase performance. Also wouldn't recommend -skipfailures in this setting, as it slows up the process immensely (sets -gt to 1).