403 error

[balthasars]

library(tidysec)
ciks <- c(1588340, 1634222, 928633, 1368163, 1535602, 1291274, 1599469, 1599576, 1535631, 1793755, 1544599, 1068833, 1727588, 824468, 1321482, 903954, 861177, 1132716, 1610520, 1114446, 1615423, 1535660, 1535784, 1491719, 1528147, 1615423, 1615424, 1615305, 1649647, 1689918, 1658354, 1667654, 1649591, 1650162, 1649592, 1536550, 1641992)
get_13f(cik = ciks, year = 2021)
#> Searching SEC filings from 2021.
#> Error in get_check_parse_xml(url = link_to_quarter_dir): are_set_equal : The element '403' in httr::status_code(response) is not in 200.

^{Created on 2021-04-06 by the reprex package (v1.0.0)}

[balthasars]

Also for

library(tidysec)
# ciks <- c(1588340, 1634222, 928633, 1368163, 1535602, 1291274, 1599469, 1599576, 1535631, 1793755, 1544599, 1068833, 1727588, 824468, 1321482, 903954, 861177, 1132716, 1610520, 1114446, 1615423, 1535660, 1535784, 1491719, 1528147, 1615423, 1615424, 1615305, 1649647, 1689918, 1658354, 1667654, 1649591, 1650162, 1649592, 1536550, 1641992)
get_13f(cik = "1641992", year = 2020)
#> Searching SEC filings from 2020.
#> Error in get_check_parse_xml(url = link_to_quarter_dir): are_set_equal : The element '403' in httr::status_code(response) is not in 200.

^{Created on 2021-04-06 by the reprex package (v1.0.0)}

[balthasars]

and

library(tidysec)
# ciks <- c(1588340, 1634222, 928633, 1368163, 1535602, 1291274, 1599469, 1599576, 1535631, 1793755, 1544599, 1068833, 1727588, 824468, 1321482, 903954, 861177, 1132716, 1610520, 1114446, 1615423, 1535660, 1535784, 1491719, 1528147, 1615423, 1615424, 1615305, 1649647, 1689918, 1658354, 1667654, 1649591, 1650162, 1649592, 1536550, 1641992)
# get_13f(cik = "1641992", year = 2020)

tidysec::get_list_of_filings(cik = "cik", year = 2020, filing_type = "13F-HR")
#> Searching SEC filings from 2020.
#> tidysec caches the SEC's master index files for the time period you've downloaded up to now at/Users/balthasarsager/.rcache/tidysec/ .
#> Run `forget_tidysec_cache()` to delete the master indices if you no longer need them.
#> Error: Problem with `filter()` input `..1`.
#> x object 'form_type' not found
#> ℹ Input `..1` is `form_type %in% form_type_in & cik %in% cik_company`.

^{Created on 2021-04-06 by the reprex package (v1.0.0)}

[balthasars]

construct_link_to_filing_directory() is called on empty master_index

[balthasars]

Also, download_single_master_index_file()appears to get 403s `Error in open.connection(3L, "rb") : HTTP error 403.
i.a. upon Getting Q2 master index from https://www.sec.gov/Archives/edgar/full-index/2020/QTR2/master.idx

[balthasars]

response <- httr::GET("https://www.sec.gov/Archives/edgar/full-index/2021/QTR1/master.idx")
response
#> Response [https://www.sec.gov/Archives/edgar/full-index/2021/QTR1/master.idx]
#>   Date: 2021-04-06 16:40
#>   Status: 200
#>   Content-Type: application/octet-stream
#>   Size: 31.9 MB
#> <BINARY BODY>

^{Created on 2021-04-06 by the reprex package (v1.0.0)}

however

tidysec:::get_master_indices_for_single_year(2020)
#> Searching SEC filings from 2020.
#> Error in get_check_parse_xml(url = link_to_quarter_dir): are_set_equal : The element '403' in httr::status_code(response) is not in 200.

^{Created on 2021-04-06 by the reprex package (v1.0.0)}

[balthasars]

Has to do with rate limits:

Your request rate has exceeded the SEC’s threshold request per second rate. Your access to SEC.gov will be limited for 10 minutes.

[balthasars]

library(tidysec)
get_13f(cik = "1641992", year = 2020)
#> Searching SEC filings from 2020.
#> Request failed [403]. Retrying in 1 seconds...
#> Request failed [403]. Retrying in 3.2 seconds...
#> tidysec caches the SEC's master index files for the time period you've downloaded up to now at/Users/balthasarsager/.rcache/tidysec/ .
#> Run `forget_tidysec_cache()` to delete the master indices if you no longer need them.
#> Now parsing filings...
#> Searching SEC filings from 2020.
#> tidysec caches the SEC's master index files for the time period you've downloaded up to now at/Users/balthasarsager/.rcache/tidysec/ .
#> Run `forget_tidysec_cache()` to delete the master indices if you no longer need them.
#> Request failed [403]. Retrying in 1.3 seconds...
#> Request failed [403]. Retrying in 1.3 seconds...
#> Request failed [403]. Retrying in 1.3 seconds...
#> 👍Those filings are ready now, hehe.
#> # A tibble: 4 x 25
#>   cik    company_name  form_type date_filed quarter link_to_filing       filing 
#>   <chr>  <chr>         <chr>          <dbl>   <dbl> <chr>                <list> 
#> 1 16419… LGT CAPITAL … 13F-HR          2020       1 https://www.sec.gov… <tibbl…
#> 2 16419… LGT CAPITAL … 13F-HR          2020       2 https://www.sec.gov… <tibbl…
#> 3 16419… LGT CAPITAL … 13F-HR          2020       3 https://www.sec.gov… <tibbl…
#> 4 16419… LGT CAPITAL … 13F-HR          2020       4 https://www.sec.gov… <tibbl…
#> # … with 18 more variables: filing_number <chr>, submissionType <chr>,
#> #   filingManager <chr>, reportType <chr>, periodOfReport <chr>,
#> #   reportCalendarOrQuarter <chr>, form13FFileNumber <chr>,
#> #   stateOrCountry <chr>, signatureDate <chr>, tableEntryTotal <chr>,
#> #   tableValueTotal <chr>, otherIncludedManagersCount <chr>, coverPage <chr>,
#> #   link_to_primary_doc <chr>, other_included_managers_name <list>,
#> #   otherManagerForm13FFileNumber <list>,
#> #   other_managers_reporting_for_this_manager_file_number <list>,
#> #   other_managers_reporting_for_this_manager <list>

^{Created on 2021-04-06 by the reprex package (v1.0.0)}

(see commit)

[balthasars]

• examine initial example again

library(tidysec)
ciks <- c(1588340, 1634222, 928633, 1368163, 1535602, 1291274, 1599469, 1599576, 1535631, 1793755, 1544599, 1068833, 1727588, 824468, 1321482, 903954, 861177, 1132716, 1610520, 1114446, 1615423, 1535660, 1535784, 1491719, 1528147, 1615423, 1615424, 1615305, 1649647, 1689918, 1658354, 1667654, 1649591, 1650162, 1649592, 1536550, 1641992)
get_13f(cik = ciks, year = 2021)