Making sure archives are fetchable

[victorb]

To make sure all archives are still online and can be fetched, I'm currently running through them with refs -r.

• cdn.media.ccc.de | /ipfs/QmW84mqTYnCkRTy6VeRJebPWuuk8b27PJ4bWm2bL4nrEWb (@lgierth )
• arXiv | /ipfs/QmfXH9XtP7xmoTH8WAp4HNSduqWMwLTH8B8TvbTkdgzNAa
• IETF RFC Archive | /ipfs/QmNvTjdqEPjZVWCvRWsFJA1vK7TTw1g9JP6we1WBJTRADM
• Presidental Daily Briefs | /ipfs/Qme6epvZDj3vzHcFKdF1nZhbixjw8Bn4imGcKnbUyBJL89
• PGP Public Key Database | /ipfs/Qmbs2DxMBraF3U8F7vLAarGmZaSFry3vVY5zytuN3BxwaY
• XKCD | /ipfs/Qmb8wsGZNXt5VXZh1pEmYynjB6Euqpq3HYyeAdw2vScTkQ
• Online Encyclopedia of Integer Sequences | /ipfs/QmbXX6jkJSx1aH41nfBqPyZAgxe1m4CzMMVnhYz1xRyFDQ
• scholarpedia.org | /ipfs/Qmaskk1Egq5zmZsGTd7dwNiiK1cwfmx7k1StG1WJQjwGDm
• No-Intro Collection | /ipfs/QmTC8ADX28gEWH68SsuduWoropCad9g9bfK2b2WWwYggPv
• cdnjs | /ipfs/QmRrnfFUgx81KZR9ibEcxDXgevoj9e5DydB5v168yembnX
• alpine-linux 3.4 packages | /ipfs/QmRsvEpJggeu4HhoafzRFobV4sbwVVTXMrdb2p8XWv7bCS
• Project Apollo Archives | /ipfs/QmSnuWmxptJZdLJpKRarxBMS2Ju2oANVrgbr2xWbie9b2D
• textfiles.com | /ipfs/QmNoscE3kNc83dM5rZNUC5UDXChiTdDcgf16RVtFCRWYuU

[victorb]

These seems dead, can only get the listing of first level links but gets no further:

• Qmbs2DxMBraF3U8F7vLAarGmZaSFry3vVY5zytuN3BxwaY
• QmbXX6jkJSx1aH41nfBqPyZAgxe1m4CzMMVnhYz1xRyFDQ
• Qmaskk1Egq5zmZsGTd7dwNiiK1cwfmx7k1StG1WJQjwGDm

[ghost]

cdn.media.ccc.de | /ipfs/QmW84mqTYnCkRTy6VeRJebPWuuk8b27PJ4bWm2bL4nrEWb (@lgierth )

This one is also currently dead since it lives on a node whose badger datastore became unbootable. Working with the badger people to fix it.

[eminence]

Is this just to get a report? Or do you intend on pinning this stuff on protocol-labs hardware?

[ghost]

Mostly it's just to weed out things that went MIA, before DWeb Summit

[kevina]

Although it might not apply here, note that refs -r may not fetch the leaves if they are raw.

[ghost]

Oh what

[ghost]

But raw leaves can have references in them? Think about Git objects and the like

[kevina]

But raw leaves can have references in them?

No they can't. They are raw data with no structure and a leaf in the DAG tree. They by definition can not have links.

Think about Git objects and the like

I need examples.

[ghost]

Should add these existing archives to index.html and the datasets section of awesome-ipfs:

• Old Internet Files Old Internet Files #176
• No Intro collection No-Intro Collection as seen on archive.org #163 already there
• Web History Project Web History Project / 1997 Web History Day #159
• Pwned Passwords Pwned passwords #157
• MDSConnect MDSConnect raw data (Library of Congress) #152
• yarchive.net yarchive #76

[ghost]

They're all added/removed in #178

[h1z1]

Think this is the right place, heads up the example given to try is also MIA. Can't fetch it all from cloudflare either.

Arxiv.org CC-By Papers: https://ipfs.io/ipfs/QmfXH9XtP7xmoTH8WAp4HNSduqWMwLTH8B8TvbTkdgzNAa/