Unable to download the files

[salawdeh]

Hello,

Thank you for sharing these data sets.
I was unsuccessfully trying to download these data sets using my AWS credentials. I always receive "An error occurred (403) when calling the HeadObject operation: Forbidden".
I am not sure if something wrong with my credentials or the bucket has some access restrictions.

Can you please help me please with that.

Thank you very much,

[tanumitra]

You can use AWS CLI or the easier s3cmd to download the data. First configure the tool to use your AWS account credentials and then use the appropriate command to download the s3 links listed below.

To download a data file (e.g., stream_tweets_byTimestamp.data) to the current folder, use the following command:

aws s3api get-object --request-payer requester --bucket credbank --key stream_tweets_byTimestamp.data stream_tweets_byTimestamp.data

OR

s3cmd get --requester-pays s3://credbank/stream_tweets_byTimestamp.data

Note that you will be charged a small data transfer fee by AWS, which can be offset by their free tier.

[salawdeh]

Thanks @TanuM,

It works now! :)

Regards,

[marcodegra]

Hi @salawdeh, have you finally downloaded the data?
If yes, could you please let me know how much is the dataset's size? So I can understand if I exceed my AWS free tier limits or not.

Best Regards

[salawdeh]

Hi @marcodegra, I think it was something around 21GB. Regards,

[HamdaSlimi]

I could not download the data. I created the AWS account and associated it with my credit card. When i introduce the command line given by tanumitra. After about 20 minutes i receive the error shown in the picture above. Please if anyone could help i need this DataSet for my thesis.

[tanumitra]

Hey @HamdaSlimi,
There seems to be a connectivity issue to AWS's us-west region from your location. As per aws/aws-cli#401 issue, you might want to retry a couple of times or call the aws command with --cli-read-timeout 0 option. Let us know if this solves the issue.

[HamdaSlimi]

Hey @tanumitra
I'm sorry i took so long to answer you. I attempted to resolve the issue using this parameter --cli-read-timeout 0. However, i encountered the following error [Errno 10054] An existing connection was forcibly closed by the remote host. When I researched it I found that passing the parameter --cli-read-timeout 0 results in this response from the server. The two sides of the connection disagreeing over whether the connection timed out or not during a keepalive. (Your code tries to reuse the connection just as the server is closing it because it has been idle for too long.)
As for the number of attempts, since my last email I attempted to retrieve the dataset nearly 10 times and each time it takes about 30 min before showing the error 10054 once more. What I would like to know is, is it possible that AWS may have restrictions concerning “region=us-east-2”.

[HamdaSlimi]

I was able to downloaded the DataSet FINALLY

[the-lost-explorer]

I was able to downloaded the DataSet FINALLY

How long does it take to download on a decent connection? I heard its 21GB on the above links.

[HamdaSlimi]

what do you mean by a decent connect? And you have to consider that the Dataset contains only the tweet ids and not the tweet text so crawling the tweets was the longest step in my case.

…

On Sat, Sep 22, 2018 at 7:13 AM Amey Parundekar ***@***.***> wrote: I was able to downloaded the DataSet FINALLY How long does it take to download on a decent connect? I heard its 21GB on the above links. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#5 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AhkueUSoiAWcfPGkcCD-o3P-RWIILj7Nks5uddUZgaJpZM4KLQzs> .

[sahar99]

@HamdaSlimi Please can you tell me how you solved the problem? or any sources can help to solve? because I had a similar problem, it takes several hours before stopped and showing that message "Read timeout on endpoint URL: "None" "

[HamdaSlimi]

Well it was a region related problem (i live in tunisia) when i asked a friend of mine who lives in england he was able to download it easily.

…

On Sat, Nov 3, 2018 at 12:21 PM sahar99 ***@***.***> wrote: @HamdaSlimi <https://github.com/HamdaSlimi> Please can you tell me how you solved the problem? or any sources can help to solve? because I had a similar problem, it takes several hours before stopped and showing that message "Read timeout on endpoint URL: "None" " — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#5 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AhkueWOBYN5fGPHx19AccHTEsUnhDPMeks5urXwegaJpZM4KLQzs> .

[jaeseok-huh]

I switched the region to us-east-2 and worked well.

[Marjan-Hosseini]

Is this dataset free? Do we need to pay for AWS credential?

[HamdaSlimi]

It is free; you only need to create an AWS account and download it. Le ven. 19 févr. 2021 à 21:27, Marjan Hosseini <[email protected]> a écrit :

…

Is this dataset free? Do we need to pay for AWS credential? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#5 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AIMS46NBO3255VMQNCJZTFLS73CU5ANCNFSM4CRNBTWA> .