My Web Crawler "FourBot"

A concurrent and performant web crawler using goroutines and channels.

Equipped with:

• Proper Network Error Handling
• Robots.txt Compliance
• Concurrency Control
• Rate Limting
• Enhanced Logging
• Graceful Shutdown

To run the web crawler, run the code below and input comma-separated URLs

go run cmd/goCrawl/main.go