just max-temp

[EtomicBomb]

The input script

seq $FROM $TO |
sed "s;^;$URL;" |
sed 's;$;/;' |
xargs -n1 -r curl --insecure |
grep gz |
tr -s ' \n' |
cut -d ' ' -f9 |
sed 's;^(.*)(20[0-9][0-9]).gz;\2/\1\2.gz;' |
sed "s;^;$URL;" |
xargs -n1 curl --insecure |
gunzip > "$input_dir/temperatures.txt"

doesn't just download the temperatures from 2015 it downloads from the whole range $FROM to $TO. It was taken from https://github.com/binpash/benchmarks/blob/6d33e8209fac4a93b93a603f8e619c9a7d9deeb0/max-temp/max-temp.sh

[vagos]

The input script

seq $FROM $TO | sed "s;^;$URL;" | sed 's;$;/;' | xargs -n1 -r curl --insecure | grep gz | tr -s ' \n' | cut -d ' ' -f9 | sed 's;^(.*)(20[0-9][0-9]).gz;\2/\1\2.gz;' | sed "s;^;$URL;" | xargs -n1 curl --insecure | gunzip > "$input_dir/temperatures.txt"

doesn't just download the temperatures from 2015 it downloads from the whole range $FROM to $TO. It was taken from https://github.com/binpash/benchmarks/blob/6d33e8209fac4a93b93a603f8e619c9a7d9deeb0/max-temp/max-temp.sh

Yep, that's correct.

[vagos]

How about we add a --small option to run.sh that runs the script for just two years (2014, 2015 ?) and then runs it for all years in any other case.

[EtomicBomb]

That makes sense but also the amount of data from just 2015 is insane. temperatures.txt is 34G and it takes longer to download than you would think because it fetches each segment from a different URL. Downloading one year, much less two years, really can't be in CI because it will literally take like an hour just to download. It would be feasible if we just download the data from the atlas server, which should be much faster than downloading all of the parts from noaa.

[vagos]

I see. How about we download a subset of a single year, then.
Try downloading just https://atlas.cs.brown.edu/data/noaa/1901/029070-99999-1901.gz

Just make sure that this is indeed a subset of the whole year.

[EtomicBomb]

That sounds reasonable!