Standardise URL requests

[stitam]

In PR #379 @ethanbass added a new function try_url() which is essentially a wrapper around httr::RETRY() where some commonly used arguments are fixed. @Aariq suggested we could use this or a similar function for all httr::RETRY() querries in webchem. The issue I see with the current version is that quite often we use "POST" instead of "GET" and/or add some extra arguments. I think these limitations could be easily circumvented with the following structure:

try_url <- function(verb, qurl, ...){
  try(httr::RETRY(verb,
                  qurl,
                  httr::user_agent(webchem_url()),
                  terminate_on = 404,
                  quiet = TRUE,
                  ...), silent = TRUE)
}

Variations (not necessarily always but at least once):

• in bcpc we would add config = httr::config(accept_encoding = "identity")
• in chebi we would use "POST" and add httr::add_headers(headers) and body = body
• in chemspider we would add httr::add_headers(.headers = headers) and body = body and sometimes use "POST"
• in cir we would add httr::write_disk(path, overwrite = TRUE)
• in etox we would add handle = handle('') and body = body
• in pubchem we would add body = paste0("inchi=", query), body = paste0(from, "=", query)

What do you think of this approach?

[Aariq]

I like the idea mostly to (hopefully) make it easier for new developers to contribute code that matches our contribution guidelines. It's possible that it would make things more complicated and harder for new developers though. So I'd love to hear others thoughts about this. I just saw the try_url() function and it was reminiscent of webchem_sleep() and some of the other helpers we have.

[ethanbass]

I agree that this type of functionalization can be somewhat of a mixed bag; overall though I think it increases code readability and accessibility to newcomers by reducing clutter. One other thought: I wonder if it is still necessary to have a separate function for webchem_url? Would this function ever be called outside of try_url? If not, I think the functions could just be merged by adding one line with the webchem url at the top.