Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Rewrite scraper #41

Open
wants to merge 17 commits into
base: master
Choose a base branch
from
Open

Rewrite scraper #41

wants to merge 17 commits into from

Commits on Mar 19, 2019

  1. Create function to create history coin lists as can be found on CMC, … 

    …e.g. https://coinmarketcap.com/historical/20150505/
    @sstoeckl
    sstoeckl committed Mar 19, 2019
    Configuration menu
    Copy the full SHA
    9c33fbf View commit details
    Browse the repository at this point in the history

  2. Add the possibility to scrape historical crypto_lists.

    @sstoeckl
    sstoeckl committed Mar 19, 2019
    Configuration menu
    Copy the full SHA
    771efe6 View commit details
    Browse the repository at this point in the history

  3. Add possibility to get lists of historical coins via the historical s… 

    …napshot facility of CMC. Needs srape_hist()
    @sstoeckl
    sstoeckl committed Mar 19, 2019
    Configuration menu
    Copy the full SHA
    92ca2a9 View commit details
    Browse the repository at this point in the history

  4. Prevent crypto from re-downloading the list if a list is provided.

    @sstoeckl
    sstoeckl committed Mar 19, 2019
    Configuration menu
    Copy the full SHA
    0a415e2 View commit details
    Browse the repository at this point in the history

Commits on Mar 20, 2019

  1. Adapt crypto_history() function to accept modified lists as input for… 

    … the coin retrieval e.g. gathered using a custom version of rypto_list().
    @sstoeckl
    sstoeckl committed Mar 20, 2019
    Configuration menu
    Copy the full SHA
    35645c5 View commit details
    Browse the repository at this point in the history

Commits on Mar 21, 2019

  1. Adapt crypto_history to new coinlist setting, create fake "ranknow", … 

    …adapt much more tidyverse-notation.
    @sstoeckl
    sstoeckl committed Mar 21, 2019
    Configuration menu
    Copy the full SHA
    b96315b View commit details
    Browse the repository at this point in the history

  2. Adapt crypto_history to new coinlist setting, create fake "ranknow", … 

    …adapt much more tidyverse-notation.
    @sstoeckl
    sstoeckl committed Mar 21, 2019
    Configuration menu
    Copy the full SHA
    4fe6b9b View commit details
    Browse the repository at this point in the history

Commits on Mar 22, 2019

  1. Take care of error 404 in scraper() when wrong slug was generated by … 

    …scraper_hist(). Output additional information on scraped data as attribute of crypto_history() (for backward compatibility).

Merge branch 'master' of https://github.com/sstoeckl/crypto

# Conflicts:
#	R/crypto_history.R
    @sstoeckl
    sstoeckl committed Mar 22, 2019
    Configuration menu
    Copy the full SHA
    6ba43c9 View commit details
    Browse the repository at this point in the history

  2. Take care of error 404 in scraper() when wrong slug was generated by … 

    …scraper_hist(). Output additional information on scraped data as attribute of crypto_history() (for backward compatibility).

Merge branch 'master' of https://github.com/sstoeckl/crypto

# Conflicts:
#	R/crypto_history.R
    @sstoeckl
    sstoeckl committed Mar 22, 2019
    Configuration menu
    Copy the full SHA
    7d8cddd View commit details
    Browse the repository at this point in the history

  3. Merge branch 'master' of https://github.com/sstoeckl/crypto

    @sstoeckl
    sstoeckl committed Mar 22, 2019
    Configuration menu
    Copy the full SHA
    cb4eeb3 View commit details
    Browse the repository at this point in the history

  4. rewrite history scraper. Does not need scraper() any more. Relies on … 

    …purrr's insistently() and safely() for scraping. Also, does first download all, the process data.
    @sstoeckl
    sstoeckl committed Mar 22, 2019
    Configuration menu
    Copy the full SHA
    0c7a5c0 View commit details
    Browse the repository at this point in the history

Commits on Mar 23, 2019

  1. Include possibly for download.

    @sstoeckl
    sstoeckl committed Mar 23, 2019
    Configuration menu
    Copy the full SHA
    3067111 View commit details
    Browse the repository at this point in the history

Commits on Mar 25, 2019

  1. Use "possibly" to take care of 404 problem imported through the histo… 

    …ric (dead) currencies. Code now runs smoothely.
    @sstoeckl
    sstoeckl committed Mar 25, 2019
    Configuration menu
    Copy the full SHA
    aded1a8 View commit details
    Browse the repository at this point in the history

Commits on Mar 26, 2019

  1. Adapt wait time to a little over one minute to avoid two waiting times.

    @sstoeckl
    sstoeckl committed Mar 26, 2019
    Configuration menu
    Copy the full SHA
    2022734 View commit details
    Browse the repository at this point in the history

Commits on Apr 18, 2019

  1. Finally managed to overcome the problem with the artificial creation … 

    …of slugs. I just take them from the url of the historical snapshots.
    @sstoeckl
    sstoeckl committed Apr 18, 2019
    Configuration menu
    Copy the full SHA
    9c6d19a View commit details
    Browse the repository at this point in the history

Commits on Jan 27, 2020

  1. Table positions on the website have changed. Also include another che… 

    …ck for empty tables in crypto_history.
    @sstoeckl
    sstoeckl committed Jan 27, 2020
    Configuration menu
    Copy the full SHA
    53b67f3 View commit details
    Browse the repository at this point in the history

  2. update RDs too

    @sstoeckl
    sstoeckl committed Jan 27, 2020
    Configuration menu
    Copy the full SHA
    330eb83 View commit details
    Browse the repository at this point in the history