Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

viewParams stopped working after latest update of ows4R #183

Open
2 tasks
salvafern opened this issue Dec 11, 2024 · 7 comments
Open
2 tasks

viewParams stopped working after latest update of ows4R #183

salvafern opened this issue Dec 11, 2024 · 7 comments

Comments

@salvafern
Copy link
Collaborator

Adding these info here so I don't forget myself. This is an issue starting after latest patch of ows4R on 9 December 2024

Before eblondel/ows4R@ffab53a

# Example before https://github.com/eblondel/ows4R/commit/ffab53a31dd6eaccb7e50ee7b5615f5e20a5081a

library(emodnet.wfs)
library(stringr)
library(sf)

# emodnet.wfs is at HEAD
packageDescription("emodnet.wfs")$RemoteSha
#> [1] "70e5657774eab8488894efac32386ac356ae3a4c"

# ows4R is at dev version before December 9 2024
packageDescription("ows4R")$RemoteSha
#> [1] "1b317679e11e2acd106fc58b09a694ff19a6ae79"

# Connect to the WFS service
wfs_bio <- emodnet_init_wfs_client("biology_occurrence_data")
#> Loading ISO 19139 XML schemas...
#> Loading ISO 19115 codelists...
#> ✔ WFS client created successfully
#> ℹ Service: "https://geo.vliz.be/geoserver/Dataportal/wfs"
#> ℹ Version: "2.0.0"

# Paste here your URL created with the EurOBIS download toolbox
url <- "https://geo.vliz.be/geoserver/Dataportal/wfs?service=wfs&version=1.1.0&typeName=eurobis-obisenv&request=GetFeature&outputFormat=text%2Fcsv&viewParams=bounds%3Ageoid+%26%26+ARRAY%5B3324%5D%3Btaxaselection%3Amammals"

# Extract the query using a regular expression
query <- str_extract(url, "(?<=viewParams=)[^&]+")

# Pass the query to viewParams argument and query eurobis full data and parameters
df <- emodnet_get_layers(
  wfs_bio,
  layers = "eurobis-obisenv",
  viewParams = query,
  count = 5000, # You can limit the number of rows
  simplify = TRUE
)

# Inspect the data
message(paste0("There are ", nrow(df), " occurrences of marine mammals in the Alboran Sea"))
#> There are 4577 occurrences of marine mammals in the Alboran Sea

Created on 2024-12-11 with reprex v2.1.1

After eblondel/ows4R@ffab53a

library(emodnet.wfs)
library(stringr)
library(sf)

# emodnet.wfs is at HEAD
packageDescription("emodnet.wfs")$RemoteSha
#> [1] "70e5657774eab8488894efac32386ac356ae3a4c"

# ows4R is at dev version after 9 December 2024
packageDescription("ows4R")$RemoteSha
#> [1] "ffab53a31dd6eaccb7e50ee7b5615f5e20a5081a"

# Connect to the WFS service
wfs_bio <- emodnet_init_wfs_client("biology_occurrence_data")
#> Loading ISO 19139 XML schemas...
#> Loading ISO 19115 codelists...
#> ✔ WFS client created successfully
#> ℹ Service: "https://geo.vliz.be/geoserver/Dataportal/wfs"
#> ℹ Version: "2.0.0"

# Paste here your URL created with the EurOBIS download toolbox
url <- "https://geo.vliz.be/geoserver/Dataportal/wfs?service=wfs&version=1.1.0&typeName=eurobis-obisenv&request=GetFeature&outputFormat=text%2Fcsv&viewParams=bounds%3Ageoid+%26%26+ARRAY%5B3324%5D%3Btaxaselection%3Amammals"

# Extract the query using a regular expression
query <- str_extract(url, "(?<=viewParams=)[^&]+")

# Pass the query to viewParams argument and query eurobis full data and parameters
df <- emodnet_get_layers(
  wfs_bio,
  layers = "eurobis-obisenv",
  viewParams = query,
  count = 5000, # You can limit the number of rows
  simplify = TRUE
)
#> Warning: Download of layer "eurobis-obisenv" failed: Error in
#> curl::curl_fetch_memory(url, handle = handle): URL using bad/illegal format or
#> missing URL: URL rejected: Malformed input to a URL function

# Inspect the data
message(paste0("There are ", nrow(df), " occurrences of marine mammals in the Alboran Sea"))
#> There are  occurrences of marine mammals in the Alboran Sea

Created on 2024-12-11 with reprex v2.1.1

TODO:

  • Translate this into ows4R code and see if still fails, and if there is more information.
  • Open issue at ows4R if relevant

Environment (with ows4R@HEAD)

sessionInfo()
R version 4.4.2 (2024-10-31)
Platform: x86_64-pc-linux-gnu
Running under: Ubuntu 24.04.1 LTS

Matrix products: default
BLAS:   /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.12.0 
LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.12.0

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C               LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8     LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                  LC_ADDRESS=C               LC_TELEPHONE=C             LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

time zone: Europe/Brussels
tzcode source: system (glibc)

attached base packages:
[1] stats     graphics  grDevices datasets  utils     methods   base     

other attached packages:
[1] sf_1.0-19              stringr_1.5.1          emodnet.wfs_2.0.2.9000

loaded via a namespace (and not attached):
 [1] generics_0.1.3     utf8_1.2.4         bspm_0.5.7         class_7.3-22       KernSmooth_2.23-24 stringi_1.8.4      hms_1.1.3          magrittr_2.0.3     grid_4.4.2        
[10] fastmap_1.2.0      jsonlite_1.8.9     e1071_1.7-16       backports_1.5.0    DBI_1.2.3          whoami_1.3.0       httr_1.4.7         purrr_1.0.2        fansi_1.0.6       
[19] XML_3.99-0.17      codetools_0.2-20   geometa_0.8-0      cli_3.6.3          rlang_1.1.4        crayon_1.5.3       units_0.8-5        withr_3.0.2        cachem_1.1.0      
[28] tools_4.4.2        parallel_4.4.2     tzdb_0.4.0         memoise_2.0.1      checkmate_2.3.2    dplyr_1.1.4        curl_6.0.0         assertthat_0.2.1   vctrs_0.6.5       
[37] R6_2.5.1           proxy_0.4-27       lifecycle_1.0.4    classInt_0.4-10    pkgconfig_2.0.3    terra_1.8-0        pillar_1.9.0       glue_1.8.0         Rcpp_1.0.13-1     
[46] tidyselect_1.2.1   tibble_3.2.1       keyring_1.3.2      rstudioapi_0.17.1  ows4R_0.4-1        readr_2.1.5        compiler_4.4.2     askpass_1.2.1      openssl_2.2.2

Doing this as part of -> https://github.com/EMODnet/EMODnet-Biology-UN-Decade-webinar
@salvafern
Copy link
Collaborator Author

Or alternatively as a quick fix: force eblondel/ows4R@1b317679e11e2acd106fc58b09a694ff19a6ae79 as dependency of emodnet.wfs

@maelle
Copy link
Collaborator

maelle commented Dec 19, 2024

@salvafern how did you find the commit that created the problem? 🔍

@salvafern
Copy link
Collaborator Author

Just tried to run the example before and after the commit that caused the problem. I had been running it just a few days before and everything was fine so I was certain something had changed.

@maelle
Copy link
Collaborator

maelle commented Dec 19, 2024

Good detective work! I will run the code in a bit to see whether the traceback can help.

@maelle
Copy link
Collaborator

maelle commented Dec 19, 2024

I am getting a different error @salvafern (I had to set simplify to FALSE) 🫠

library("emodnet.wfs")


# Connect to the WFS service
wfs_bio <- emodnet_init_wfs_client("biology_occurrence_data")
#> Loading ISO 19139 XML schemas...
#> Loading ISO 19115 codelists...
#> ✔ WFS client created successfully
#> ℹ Service: "https://geo.vliz.be/geoserver/Dataportal/wfs"
#> ℹ Version: "2.0.0"

# Paste here your URL created with the EurOBIS download toolbox
url <- "https://geo.vliz.be/geoserver/Dataportal/wfs?service=wfs&version=1.1.0&typeName=eurobis-obisenv&request=GetFeature&outputFormat=text%2Fcsv&viewParams=bounds%3Ageoid+%26%26+ARRAY%5B3324%5D%3Btaxaselection%3Amammals"

# Extract the query using a regular expression
(query <- stringr::str_extract(url, "(?<=viewParams=)[^&]+"))
#> [1] "bounds%3Ageoid+%26%26+ARRAY%5B3324%5D%3Btaxaselection%3Amammals"

# Pass the query to viewParams argument and query eurobis full data and parameters
df <- emodnet_get_layers(
  wfs_bio,
  layers = "eurobis-obisenv",
  viewParams = query,
  count = 5000, # You can limit the number of rows
  simplify = FALSE
)
#> StartTag: invalid element name
#> Extra content at the end of the document
#> Warning: Download of layer "eurobis-obisenv" failed: Error: 1: StartTag: invalid element
#> name 2: Extra content at the end of the document

# Inspect the data
message(paste0("There are ", nrow(df), " occurrences of marine mammals in the Alboran Sea"))
#> There are  occurrences of marine mammals in the Alboran Sea

Created on 2024-12-19 with reprex v2.1.1

Session info 
sessioninfo::session_info()
#> ─ Session info ───────────────────────────────────────────────────────────────
#>  setting  value
#>  version  R version 4.4.1 (2024-06-14)
#>  os       Ubuntu 20.04.6 LTS
#>  system   x86_64, linux-gnu
#>  ui       X11
#>  language en_US.utf8
#>  collate  en_US.utf8
#>  ctype    en_US.utf8
#>  tz       Europe/Paris
#>  date     2024-12-19
#>  pandoc   3.1.11 @ /usr/lib/rstudio/resources/app/bin/quarto/bin/tools/x86_64/ (via rmarkdown)
#> 
#> ─ Packages ───────────────────────────────────────────────────────────────────
#>  package     * version    date (UTC) lib source
#>  askpass       1.2.1      2024-10-04 [1] CRAN (R 4.4.1)
#>  assertthat    0.2.1      2019-03-21 [1] CRAN (R 4.4.0)
#>  backports     1.5.0      2024-05-23 [1] CRAN (R 4.4.0)
#>  cachem        1.1.0      2024-05-16 [1] CRAN (R 4.4.0)
#>  checkmate     2.3.2      2024-07-29 [1] CRAN (R 4.4.0)
#>  class         7.3-22     2023-05-03 [4] CRAN (R 4.3.1)
#>  classInt      0.4-10     2023-09-05 [1] CRAN (R 4.4.0)
#>  cli           3.6.3      2024-06-21 [1] CRAN (R 4.4.0)
#>  codetools     0.2-20     2024-03-31 [1] CRAN (R 4.4.0)
#>  crayon        1.5.3      2024-06-20 [1] CRAN (R 4.4.0)
#>  curl          6.0.1      2024-11-14 [1] CRAN (R 4.4.1)
#>  DBI           1.2.3      2024-06-02 [1] CRAN (R 4.4.0)
#>  digest        0.6.37     2024-08-19 [1] CRAN (R 4.4.0)
#>  dplyr         1.1.4      2023-11-17 [1] CRAN (R 4.4.0)
#>  e1071         1.7-16     2024-09-16 [1] CRAN (R 4.4.1)
#>  emodnet.wfs * 2.0.2.9000 2024-11-26 [1] local
#>  evaluate      1.0.1      2024-10-10 [1] CRAN (R 4.4.1)
#>  fastmap       1.2.0      2024-05-15 [1] CRAN (R 4.4.0)
#>  fs            1.6.5      2024-10-30 [1] CRAN (R 4.4.1)
#>  generics      0.1.3      2022-07-05 [1] CRAN (R 4.4.0)
#>  geometa       0.8-0      2024-04-18 [1] CRAN (R 4.4.0)
#>  glue          1.8.0      2024-09-30 [1] CRAN (R 4.4.1)
#>  hms           1.1.3      2023-03-21 [1] CRAN (R 4.4.0)
#>  htmltools     0.5.8.1    2024-04-04 [1] CRAN (R 4.4.0)
#>  httr          1.4.7      2023-08-15 [1] CRAN (R 4.4.0)
#>  jsonlite      1.8.9      2024-09-20 [1] CRAN (R 4.4.1)
#>  KernSmooth    2.23-24    2024-05-17 [1] CRAN (R 4.4.0)
#>  keyring       1.3.2      2023-12-11 [1] CRAN (R 4.4.0)
#>  knitr         1.49       2024-11-08 [1] CRAN (R 4.4.1)
#>  lifecycle     1.0.4      2023-11-07 [1] CRAN (R 4.4.0)
#>  magrittr      2.0.3      2022-03-30 [1] CRAN (R 4.4.0)
#>  memoise       2.0.1      2021-11-26 [1] CRAN (R 4.4.0)
#>  openssl       2.3.0      2024-12-16 [1] CRAN (R 4.4.1)
#>  ows4R         0.4-1      2024-12-19 [1] Github (eblondel/ows4R@b3149cd)
#>  pillar        1.10.0     2024-12-17 [1] CRAN (R 4.4.1)
#>  pkgconfig     2.0.3      2019-09-22 [1] CRAN (R 4.4.0)
#>  proxy         0.4-27     2022-06-09 [1] CRAN (R 4.4.0)
#>  purrr         1.0.2      2023-08-10 [1] CRAN (R 4.4.0)
#>  R6            2.5.1      2021-08-19 [1] CRAN (R 4.4.0)
#>  Rcpp          1.0.13-1   2024-11-02 [1] CRAN (R 4.4.1)
#>  readr         2.1.5      2024-01-10 [1] CRAN (R 4.4.0)
#>  reprex        2.1.1      2024-07-06 [1] CRAN (R 4.4.1)
#>  rlang         1.1.4      2024-06-04 [1] CRAN (R 4.4.0)
#>  rmarkdown     2.29       2024-11-04 [1] CRAN (R 4.4.1)
#>  rstudioapi    0.17.1     2024-10-22 [1] CRAN (R 4.4.1)
#>  sessioninfo   1.2.2      2021-12-06 [1] CRAN (R 4.4.0)
#>  sf            1.0-19     2024-11-05 [1] CRAN (R 4.4.1)
#>  stringi       1.8.4      2024-05-06 [1] CRAN (R 4.4.0)
#>  stringr       1.5.1      2023-11-14 [1] CRAN (R 4.4.0)
#>  terra         1.8-5      2024-12-12 [1] CRAN (R 4.4.1)
#>  tibble        3.2.1      2023-03-20 [1] CRAN (R 4.4.0)
#>  tidyselect    1.2.1      2024-03-11 [1] CRAN (R 4.4.0)
#>  tzdb          0.4.0      2023-05-12 [1] CRAN (R 4.4.0)
#>  units         0.8-5      2023-11-28 [1] CRAN (R 4.4.0)
#>  vctrs         0.6.5      2023-12-01 [1] CRAN (R 4.4.0)
#>  whoami        1.3.0      2019-03-19 [1] CRAN (R 4.4.0)
#>  withr         3.0.2      2024-10-28 [1] CRAN (R 4.4.1)
#>  xfun          0.49       2024-10-31 [1] CRAN (R 4.4.1)
#>  XML           3.99-0.17  2024-06-25 [1] CRAN (R 4.4.0)
#>  yaml          2.3.10     2024-07-26 [1] CRAN (R 4.4.0)
#> 
#>  [1] /home/maelle/R/x86_64-pc-linux-gnu-library/4.4
#>  [2] /usr/local/lib/R/site-library
#>  [3] /usr/lib/R/site-library
#>  [4] /usr/lib/R/library
#> 
#> ──────────────────────────────────────────────────────────────────────────────

@maelle
Copy link
Collaborator

maelle commented Dec 19, 2024

@salvafern can you please confirm that if you run your example today, you get the same results?

@salvafern
Copy link
Collaborator Author

salvafern commented Dec 19, 2024 •
edited
Loading

Confirm, same results. Funny you get a different error. Only difference I can see is that you are on Ubuntu 20.04 and I am on 24.04

library(emodnet.wfs)
library(stringr)
library(sf)
#> Linking to GEOS 3.12.2, GDAL 3.8.4, PROJ 9.4.0; sf_use_s2() is TRUE
#> WARNING: different compile-time and runtime versions for GEOS found:
#> Linked against: 3.12.2-CAPI-1.18.2 compiled against: 3.12.1-CAPI-1.18.1
#> It is probably a good idea to reinstall sf (and maybe lwgeom too)

# emodnet.wfs is at HEAD
packageDescription("emodnet.wfs")$RemoteSha
#> [1] "70e5657774eab8488894efac32386ac356ae3a4c"

# ows4R is at dev version after 9 December 2024
packageDescription("ows4R")$RemoteSha
#> [1] "b3149cd141354d7b9c064d5a4da8bac51a26ace7"

# Connect to the WFS service
wfs_bio <- emodnet_init_wfs_client("biology_occurrence_data")
#> Loading ISO 19139 XML schemas...
#> Loading ISO 19115 codelists...
#> ✔ WFS client created successfully
#> ℹ Service: "https://geo.vliz.be/geoserver/Dataportal/wfs"
#> ℹ Version: "2.0.0"

# Paste here your URL created with the EurOBIS download toolbox
url <- "https://geo.vliz.be/geoserver/Dataportal/wfs?service=wfs&version=1.1.0&typeName=eurobis-obisenv&request=GetFeature&outputFormat=text%2Fcsv&viewParams=bounds%3Ageoid+%26%26+ARRAY%5B3324%5D%3Btaxaselection%3Amammals"

# Extract the query using a regular expression
query <- str_extract(url, "(?<=viewParams=)[^&]+")
query
#> [1] "bounds%3Ageoid+%26%26+ARRAY%5B3324%5D%3Btaxaselection%3Amammals"

# Pass the query to viewParams argument and query eurobis full data and parameters
df <- emodnet_get_layers(
  wfs_bio,
  layers = "eurobis-obisenv",
  viewParams = query,
  count = 5000, # You can limit the number of rows
  simplify = FALSE
)
#> Warning: Download of layer "eurobis-obisenv" failed: Error in
#> curl::curl_fetch_memory(url, handle = handle): URL using bad/illegal format or
#> missing URL: URL rejected: Malformed input to a URL function

Created on 2024-12-19 with reprex v2.1.1

Session info 
sessioninfo::session_info()
#> ─ Session info ───────────────────────────────────────────────────────────────
#>  setting  value
#>  version  R version 4.4.2 (2024-10-31)
#>  os       Ubuntu 24.04.1 LTS
#>  system   x86_64, linux-gnu
#>  ui       X11
#>  language (EN)
#>  collate  en_US.UTF-8
#>  ctype    en_US.UTF-8
#>  tz       Europe/Brussels
#>  date     2024-12-19
#>  pandoc   3.2 @ /usr/lib/rstudio/resources/app/bin/quarto/bin/tools/x86_64/ (via rmarkdown)
#> 
#> ─ Packages ───────────────────────────────────────────────────────────────────
#>  package     * version date (UTC) lib source
#>  cli           3.6.3   2024-06-21 [1] RSPM (R 4.4.0)
#>  digest        0.6.37  2024-08-19 [1] RSPM (R 4.4.0)
#>  evaluate      1.0.1   2024-10-10 [1] RSPM (R 4.4.0)
#>  fastmap       1.2.0   2024-05-15 [1] RSPM (R 4.4.0)
#>  fs            1.6.5   2024-10-30 [1] RSPM (R 4.4.0)
#>  glue          1.8.0   2024-09-30 [1] RSPM (R 4.4.0)
#>  htmltools     0.5.8.1 2024-04-04 [1] RSPM (R 4.4.0)
#>  knitr         1.49    2024-11-08 [1] RSPM (R 4.4.0)
#>  lifecycle     1.0.4   2023-11-07 [1] RSPM (R 4.4.0)
#>  reprex        2.1.1   2024-07-06 [1] RSPM (R 4.4.0)
#>  rlang         1.1.4   2024-06-04 [1] RSPM (R 4.4.0)
#>  rmarkdown     2.29    2024-11-04 [1] RSPM (R 4.4.0)
#>  rstudioapi    0.17.1  2024-10-22 [1] RSPM (R 4.4.0)
#>  sessioninfo   1.2.2   2021-12-06 [1] RSPM (R 4.4.0)
#>  withr         3.0.2   2024-10-28 [1] RSPM (R 4.4.0)
#>  xfun          0.49    2024-10-31 [1] RSPM (R 4.4.0)
#>  yaml          2.3.10  2024-07-26 [1] RSPM (R 4.4.0)
#> 
#>  [1] /usr/lib/R/site-library
#>  [2] /usr/lib/R/library
#>  [3] /usr/local/lib/R/site-library
#> 
#> ──────────────────────────────────────────────────────────────────────────────

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@maelle @salvafern