From fe9777401f100ec9be7d56c9f63aeb0192013056 Mon Sep 17 00:00:00 2001 From: Jack Adamson Date: Mon, 25 Nov 2024 19:38:23 +0000 Subject: [PATCH] update README to document commandline arguments --- src/fetch/README.md | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/src/fetch/README.md b/src/fetch/README.md index 9ed3a5cd..ffdd01b0 100644 --- a/src/fetch/README.md +++ b/src/fetch/README.md @@ -97,6 +97,26 @@ Add to your Zed settings.json: ``` +### Customization - robots.txt + +By default, the server will obey a websites robots.txt file if the request came from the model (via a tool), but not if +the request was user initiated (via a prompt). This can be disabled by adding the argument `--ignore-robots-txt` to the +`args` list in the configuration. + +### Customization - User-agent + +By default, depending on if the request came from the model (via a tool), or was user initiated (via a prompt), the +server will use either the user-agent +``` +ModelContextProtocol/1.0 (Autonomous; +https://github.com/modelcontextprotocol/servers) +``` +or +``` +ModelContextProtocol/1.0 (User-Specified; +https://github.com/modelcontextprotocol/servers) +``` + +This can be customized by adding the argument `--user-agent=YourUserAgent` to the `args` list in the configuration. + ## Debugging You can use the MCP inspector to debug the server. For uvx installations: