WebScraper refactor into scrapeURL

[mogery]

Directives:

• reduce state
  • stateless, functional programming paradigms to reduce debugging complexity
    • state that is required (e.g. current logger) is passed in an immutable meta object
  • sovereign modules that do not see the whole state (see e.g. engines/fire-engine/scrape.ts, it only has logger, not the whole meta object)
• make the signal flow clear to ease debugging
  • intense verbosity in logging
  • modularity, make it clear where to add things in the future, make it easy to add things in the future without breaking stuff
    • define generic modules that can be implemented and appended to later (e.g. transformers, engines)
• better error handling
  • using a rust-like error model, exceptions are freely thrown instead of wrapping it into {success: false, error: ...} objects
  • errors that occur are always re-thrown, with the original metadata (e.g. stack) intact. if doing stuff like retries with a limit, previous errors are passed along using the cause property.
  • we may never swallow an error.
    • at points where errors are not directly re-thrown, rather put into an object/array (e.g. retry logic/EngineResultsTracker), non-expected errors should be explicitly logged and Sentry.captureException'd
  • errors are only transformed into a success object at the top level of scrapeURL, in order to avoid breaking other parts of the codebase. errors are passed in the error metadata
  • never determine what an error is by checking it's message -- if you need a specific error that is determinable by parts of the codebase, create a custom error class and use instanceof -- see error.ts for reference
• standalone
  • scrapeURL should never (even attempt to!) interface with the database. It should be its own standalone thing that could even be lifted out of firecrawl as a whole. To keep it fast, reliable, and maintainable, we need to keep its footprint minimal -- DB code can be handled by the surrounding bits that are tangled up in that anyways (e.g. queue-worker.ts)

[nickscamara]

• Add sb
• Integrate w/ v1
• Make crawl not crash if scrapeURL throws

[nickscamara]

• Reminder for Nick and Rafa to test scrape events + dashboard logging

[nickscamara]

• Redlock for emails, same issue in main right now. If someone is someone is doing a big crawl, it will send them multiple emails.