Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Prepare universal http interceptor for both static and browser crawlers tests #929

Open
Pijukatel opened this issue Jan 22, 2025 · 0 comments
Open

Prepare universal http interceptor for both static and browser crawlers tests #929

Pijukatel opened this issue Jan 22, 2025 · 0 comments
Labels
debt Code quality improvement or decrease of technical debt. t-tooling Issues with this label are in the ownership of the tooling team.

Comments

@Pijukatel
Copy link
Contributor

Pijukatel commented Jan 22, 2025 •
edited by vdusek
Loading

Currently in our tests sometimes respx is used to mock http traffic for static crawlers and for PlaywrightCrawler mostly real requests are done.
It would be convenient, faster and more robust to create a fixture that can mock both.
For Playwright related browser requests it can be done using custom browser BrowserPool, page.route for example:

class _StaticRedirectBrowserPool(BrowserPool):
    """BrowserPool for redirecting browser requests to static content."""

    async def new_page(
        self,
        *,
        page_id: str | None = None,
        browser_plugin: BaseBrowserPlugin | None = None,
        proxy_info: ProxyInfo | None = None,
    ) -> CrawleePage:
        crawlee_page = await super().new_page(page_id=page_id, browser_plugin=browser_plugin, proxy_info=proxy_info)
        await crawlee_page.page.route(
            '**/*',
            lambda route: route.fulfill(
                status=200, content_type='text/plain', body='<!DOCTYPE html><html><body>What a body!</body></html>'
            ),
        )
        return crawlee_page
@Pijukatel Pijukatel added debt Code quality improvement or decrease of technical debt. t-tooling Issues with this label are in the ownership of the tooling team. labels Jan 22, 2025
@Pijukatel Pijukatel changed the title Prepare universal http interceptor for both static and browser crawlers. Prepare universal http interceptor for both static and browser crawlers Jan 22, 2025
@Pijukatel Pijukatel changed the title Prepare universal http interceptor for both static and browser crawlers Prepare universal http interceptor for both static and browser crawlers tests Jan 22, 2025
@Pijukatel Pijukatel mentioned this issue Jan 22, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
debt Code quality improvement or decrease of technical debt. t-tooling Issues with this label are in the ownership of the tooling team.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@Pijukatel