Failure when URL is too long aborts crawling

[kevinmarks]

If I patch this will it recrawl the whole site, or only files it doesn't have?

`Processing: http://the-toast.net/The%20dining%20room%20will%20have%20been%20dismantled%20by%20now,%20the%20tableware%20and%20furniture%20packed%20up.%20The%20final%20cuttlefish%20soba%20noodles%20and%20iced%20rose-petal%20broth%20have%20left%20the%20kitchen;%20the%20last%20wild%20duck%20has%20been%20barbecued%20and%20carved;%20no%20more%20sweet%20potatoes%20will%20appear%20at%20the%20pass,%20simmering%20in%20molten%20sugar.%20After%20two%20years%20of%20planning%20for%20a%20five-week%20run%20of%2032%20lunches%20and%2032%20dinners,%20with%203,584%20guests%20fed%20approximately%2057,350%20courses,%20the%20labour%20of%20love%20that%20was%20Noma%27s%20time%20in%20Tokyo%20came%20to%20an%20end%20yesterday.
Filename: The%20dining%20room%20will%20have%20been%20dismantled%20by%20now,%20the%20tableware%20and%20furniture%20packed%20up.%20The%20final%20cuttlefish%20soba%20noodles%20and%20iced%20rose-petal%20broth%20have%20left%20the%20kitchen;%20the%20last%20wild%20duck%20has%20been%20barbecued%20and%20carved;%20no%20more%20sweet%20potatoes%20will%20appear%20at%20the%20pass,%20simmering%20in%20molten%20sugar.%20After%20two%20years%20of%20planning%20for%20a%20five-week%20run%20of%2032%20lunches%20and%2032%20dinners,%20with%203,584%20guests%20fed%20approximately%2057,350%20courses,%20the%20labour%20of%20love%20that%20was%20Noma%27s%20time%20in%20Tokyo%20came%20to%20an%20end%20yesterday.
Directory: ./the-toast.net/
fs.js:558
return binding.open(pathModule._makeLong(path), stringToFlags(flags), mode);
^

Error: ENAMETOOLONG: name too long, open './the-toast.net/The%20dining%20room%20will%20have%20been%20dismantled%20by%20now,%20the%20tableware%20and%20furniture%20packed%20up.%20The%20final%20cuttlefish%20soba%20noodles%20and%20iced%20rose-petal%20broth%20have%20left%20the%20kitchen;%20the%20last%20wild%20duck%20has%20been%20barbecued%20and%20carved;%20no%20more%20sweet%20potatoes%20will%20appear%20at%20the%20pass,%20simmering%20in%20molten%20sugar.%20After%20two%20years%20of%20planning%20for%20a%20five-week%20run%20of%2032%20lunches%20and%2032%20dinners,%20with%203,584%20guests%20fed%20approximately%2057,350%20courses,%20the%20labour%20of%20love%20that%20was%20Noma%27s%20time%20in%20Tokyo%20came%20to%20an%20end%20yesterday.'
at Object.fs.openSync (fs.js:558:18)
at Object.fs.writeFileSync (fs.js:1223:33)
at Request._callback (/Users/kevinmarks/code/spiderpig/spider.js:145:12)
at Request.self.callback (/Users/kevinmarks/code/spiderpig/node_modules/request/request.js:188:22)
at emitTwo (events.js:106:13)
at Request.emit (events.js:191:7)
at Request. (/Users/kevinmarks/code/spiderpig/node_modules/request/request.js:1171:10)
at emitOne (events.js:96:13)
at Request.emit (events.js:188:7)
at IncomingMessage. (/Users/kevinmarks/code/spiderpig/node_modules/request/request.js:1091:12)
`

[aaronpk]

I don't think I ever made it do incremental crawls. That would be a good feature to add tho

[kevinmarks]

I'll have to read the code again - does it check for a local file before fetching the remote url? On 28 Jan 2018 14:20, "Aaron Parecki" <[email protected]> wrote: I don't think I ever made it do incremental crawls. That would be a good feature to add tho — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#8 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAGCwLV6Ix3Lw15cH8Y2xu3fuI12An9Yks5tPIIugaJpZM4RvsVc> .

[kevinmarks]

Looking at it, to make it incremental, I'd factor the url to path stuff into a function, call it at the beginning of process_link() and then see if a file exists already.
If it does, set visited to true.
Then, if the file ends in .html, read it and run the link parsing on it.

What I'm not sure about is how that would handle redirects.