diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index e5c18b8..a2b0da1 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -6,6 +6,7 @@ - [Programming](#programming) - [Submitting Your Work](#submitting-your-work) - [Expectations Regarding Timelines](#expectations-regarding-timelines) +- [Feature Coverage Documentation](#feature-coverage-documentation) - [Discuss \& Design First](#discuss--design-first) - [Getting the Source \& Installing](#getting-the-source--installing) - [Command Design](#command-design) @@ -27,16 +28,15 @@ No matter if you're a programmer or not there are many ways to contribute to the ### Non-Programming * [Set up synadm](README.md#installation) and report whether the installation, configuration and update processes work as described or could be improved. We don't have the resources to test on many operating systems, but you can [let us know](https://matrix.to/#/%23synadm%3Apeek-a-boo.at?via=jacksonchen666.com&via=maclemon.at&via=matrix.org&via=raim.ist) if it does work. -* Help keeping the [Implementation Status / Commands List](README.md#implementation-status--commands-list) chapter up to date. The Synapse project is steadily releasing new features for the _Synapse Admin API_. We can't keep up with updating this list but still think it is a handy overview of what's supported and what isn't. -* Help prioritizing what to code next: Pick a feature from the list you find in the [Implementation Status / Commands List](README.md#implementation-status--commands-list) chapter, open a GitHub issue and tell us what it is. If you don't find the feature in the list, please add it! +* Help keep the features list on [API to CLI Mapping](https://synadm.readthedocs.io/en/latest/features.html) up to date. * Improve the docs: The end-user documentation is entirely realized with the typical online help of Unix commands, an option named`--help/-h`. We believe that a top-priority in designing CLI tools is getting this information right. Spelling, wording and of course technical correctness are equally important. If it's not easily possible to stay brief and technically precise at the same time, we believe it is OK to prioritize precision over shortness. We also have documentation made with _Sphinx_ [available online](https://synadm.readthedocs.io/en/latest/index_cli_reference.html). ### Programming -* Pick an open issue on GitHub or an unticked feature from the [Implementation Status / Commands List](README.md#implementation-status--commands-list) chapter, start implementing and submit a pull-request when you are ready to receive our feedback. +* Pick an [open issue on GitHub](https://github.com/JOJ0/synadm/issues) or a not yet implemented [Synapse Admin API](https://synadm.readthedocs.io/en/latest/features.html) (see [API to CLI Mapping](https://synadm.readthedocs.io/en/latest/features.html)) and start implementing it, then submit a pull-request when you are ready. * If you'd like to receive "early feedback" even though you think your code is not ready yet, submit your PR already and set it to draft state. You can then mention us (`@` then followed by a GitHub username). * It's ok to open a Draft PR even if you don't want our feedback yet but it helps you in any way. We won't bother you until you ping us. -* Read the [Synapse Admin API docs](https://github.com/matrix-org/synapse/tree/master/docs/admin_api), pick a feature, implement it and send a pull-request. Don't forget to check if the feature is listed in the [Implementation Status / Commands List](README.md#implementation-status--commands-list) chapter already. If not, please add the command to the list (preferably in a separate commit). +* Don't forget to check if the feature you are submitting is listed on [API to CLI Mapping](https://synadm.readthedocs.io/en/latest/features.html) already. If not, please add it to the corresponding table (preferably in a separate commit). ## Submitting Your Work @@ -60,6 +60,31 @@ We are maintaining `synadm` in our spare time and currently are not sponsored by - Unfortunately we often see contributors submitting PR's, correcting a few of our suggested changes but then disappear, so if you realize you won't find the time at all to continue your work, **please just let us know**. - We believe that even the tiniest feature deserves a timely release. Sometimes we just can't make it soon enough because we don't find the time or we wait for something else because it makes sense to be released together. If a feature you submitted is merged and you'd need it urgently to be available on PyPi, **please let us know**. +## Feature Coverage Documentation + +We keep track of which Synapse Admin API's `synadm` supports in a set of tables on [API to CLI Mapping](https://synadm.readthedocs.io/en/latest/features.html). The structure of this page follows the layout of the official [Synapse Admin API documentation](https://element-hq.github.io/synapse/latest/usage/administration/admin_api/index.html). Each table represents one main chapter of the Synapse documentation. + +In our documentation source, the page is defined by [features.rst](https://github.com/JOJ0/synadm/blob/master/doc/source/features.rst), which contains multiple CSV files, each representing a table. To assist with maintaining this page, we offer a [web scraper tool](https://github.com/JOJ0/synadm/blob/master/scrape_docs.py) that pulls data from the Synapse Admin API docs and creates an initial version of such a CSV table. A basic usage example is + +``` +./scrape_docs.py -o csv https://element-hq.github.io/synapse/latest/admin_api/rooms.html +``` + +which prints a two-column CSV table containing restructuredText formatted hyperlinks in the left column. A corresponding `synadm` command is supposed to be added to the right hand side column manually. For example, you could add the following: + +``` +:option:`synadm user details USER_ID` +``` + +This would directly link to the `USER_ID` argument's documentation of that command. +Linking to an option is also possible: + +``` +:option:`synadm media list -r` +``` + +Due to a shortcoming of Sphinx it is currently not possible to link to a plain command (without any option or argument). Also see `scrape_docs.py --help` and the [existing CSV files](https://github.com/JOJ0/synadm/tree/master/doc/source/features). + ## Discuss & Design First diff --git a/doc/source/conf.py b/doc/source/conf.py index 2c2c09f..7541693 100644 --- a/doc/source/conf.py +++ b/doc/source/conf.py @@ -71,7 +71,7 @@ 'collapse_navigation': True, 'navigation_depth': 8, 'sticky_navigation': False, - 'prev_next_buttons_location': 'both' + 'prev_next_buttons_location': 'bottom' } diff --git a/doc/source/features.rst b/doc/source/features.rst new file mode 100644 index 0000000..2ce976a --- /dev/null +++ b/doc/source/features.rst @@ -0,0 +1,155 @@ +API to CLI Mapping +========================== + +The tables below show which Synapse Admin APIs are used by their corresponding ``synadm`` commands. +The left column's entries link to the official Synapse Admin API documentation, the right column's entries point to the ``synadm`` CLI reference. +To add missing entries to a table edit one of the CSV files in our `documentation source directory`_ and submit a `pull request`_ - see `feature coverage documentation`_ for details. + + +Account Validity +---------------- + +.. csv-table:: + :file: features/account_validity.csv + :header: "Synapse Admin API","synadm command(s)" + :widths: 1 1 + +Background Updates +------------------ + +.. csv-table:: + :file: features/background_updates.csv + :header: "Synapse Admin API","synadm command(s)" + :widths: 1 1 + +Delete Group +------------ + +.. csv-table:: + :file: features/delete_group.csv + :header: "Synapse Admin API","synadm command(s)" + :widths: 1 1 + +Event Reports +------------- + +.. csv-table:: + :file: features/event_reports.csv + :header: "Synapse Admin API","synadm command(s)" + :widths: 1 1 + +Experimental Features +--------------------- + +.. csv-table:: + :file: features/experimental_features.csv + :header: "Synapse Admin API","synadm command(s)" + :widths: 1 1 + +Media +----- + +.. csv-table:: + :file: features/media.csv + :header: "Synapse Admin API","synadm command(s)" + :widths: 1 1 + :keepspace: + +Purge History +------------- + +.. csv-table:: + :file: features/purge_history.csv + :header: "Synapse Admin API","synadm command(s)" + :widths: 1 1 + +Register Users +-------------- + +.. csv-table:: + :file: features/register_users.csv + :header: "Synapse Admin API","synadm command(s)" + :widths: 1 1 + +Registration Tokens +------------------- + +.. csv-table:: + :file: features/registration_tokens.csv + :header: "Synapse Admin API","synadm command(s)" + :widths: 1 1 + +Manipulate Room Membership +-------------------------- + +.. csv-table:: + :file: features/manipulate_room_membership.csv + :header: "Synapse Admin API","synadm command(s)" + :widths: 1 1 + +Rooms +----- + +.. csv-table:: + :file: features/rooms.csv + :header: "Synapse Admin API","synadm command(s)" + :widths: 1 1 + +Server Notices +-------------- + +.. csv-table:: + :file: features/server_notices.csv + :header: "Synapse Admin API","synadm command(s)" + :widths: 1 1 + +Statistics +---------- + +.. csv-table:: + :file: features/statistics.csv + :header: "Synapse Admin API","synadm command(s)" + :widths: 1 1 + +Users +----- + +.. csv-table:: + :file: features/users.csv + :header: "Synapse Admin API","synadm command(s)" + :widths: 1 1 + +Server Version +-------------- + +.. csv-table:: + :file: features/server_version.csv + :header: "Synapse Admin API","synadm command(s)" + :widths: 1 1 + +Federation +---------- + +.. csv-table:: + :file: features/federation.csv + :header: "Synapse Admin API","synadm command(s)" + :widths: 1 1 + +Miscellaneous +------------- + +.. csv-table:: + :file: features/miscellaneous.csv + :header: "Description","synadm command(s)" + :widths: 1 1 + + +.. _documentation source directory: + https://github.com/JOJ0/synadm/tree/master/doc/source/features/ +.. _feature request issue: + https://github.com/JOJ0/synadm/issues/new +.. _pull request: + https://github.com/JOJ0/synadm/blob/master/CONTRIBUTING.md#submitting-your-work +.. _feature coverage documentation: + https://github.com/JOJ0/synadm/blob/master/CONTRIBUTING.md#feature-coverage-documentation +.. |indent| unicode:: U+00A0 U+00A0 .. non-breaking two-space indentation diff --git a/doc/source/features/account_validity.csv b/doc/source/features/account_validity.csv new file mode 100644 index 0000000..c56288f --- /dev/null +++ b/doc/source/features/account_validity.csv @@ -0,0 +1 @@ +"`Renew account `_", diff --git a/doc/source/features/background_updates.csv b/doc/source/features/background_updates.csv new file mode 100644 index 0000000..076346b --- /dev/null +++ b/doc/source/features/background_updates.csv @@ -0,0 +1,3 @@ +"`Status `_", +"`Enabled `_", +"`Run `_", diff --git a/doc/source/features/delete_group.csv b/doc/source/features/delete_group.csv new file mode 100644 index 0000000..83415f7 --- /dev/null +++ b/doc/source/features/delete_group.csv @@ -0,0 +1 @@ +"`Delete a local group `_",:option:`synadm group delete GROUP_ID` diff --git a/doc/source/features/event_reports.csv b/doc/source/features/event_reports.csv new file mode 100644 index 0000000..427effc --- /dev/null +++ b/doc/source/features/event_reports.csv @@ -0,0 +1,3 @@ +"`Show reported events `_", +"`Show details of a specific event report `_", +"`Delete a specific event report `_", diff --git a/doc/source/features/experimental_features.csv b/doc/source/features/experimental_features.csv new file mode 100644 index 0000000..3c9f605 --- /dev/null +++ b/doc/source/features/experimental_features.csv @@ -0,0 +1,2 @@ +"`Enabling/Disabling Features `_", +"`Listing Enabled Features `_", diff --git a/doc/source/features/federation.csv b/doc/source/features/federation.csv new file mode 100644 index 0000000..d5bab9b --- /dev/null +++ b/doc/source/features/federation.csv @@ -0,0 +1,4 @@ +"`List of destinations `_", +"`Destination Details API `_", +"`Destination rooms `_", +"`Reset connection timeout `_", diff --git a/doc/source/features/manipulate_room_membership.csv b/doc/source/features/manipulate_room_membership.csv new file mode 100644 index 0000000..599f556 --- /dev/null +++ b/doc/source/features/manipulate_room_membership.csv @@ -0,0 +1 @@ +"`Edit Room Membership API `_",:option:`synadm room join ROOM_ID_OR_ALIAS` diff --git a/doc/source/features/media.csv b/doc/source/features/media.csv new file mode 100644 index 0000000..05fb5b2 --- /dev/null +++ b/doc/source/features/media.csv @@ -0,0 +1,15 @@ +"`Querying media `_","---" +"|indent| `List all media in a room `_",:option:`synadm media list -r` +"|indent| `List all media uploaded by a user `_",:option:`synadm media list -u` :option:`synadm user media -f` +"`Quarantine media `_","---" +"|indent| `Quarantining media by ID `_",:option:`synadm media quarantine -i` +"|indent| `Remove media from quarantine by ID `_",:option:`synadm media unquarantine -i` +"|indent| `Quarantining media in a room `_",:option:`synadm media quarantine -r` +"|indent| `Quarantining all media of a user `_",:option:`synadm media quarantine -u` +"|indent| `Protecting media from being quarantined `_",:option:`synadm media protect MEDIA_ID` +"|indent| `Unprotecting media from being quarantined `_", +"`Delete local media `_","---" +"|indent| `Delete a specific local media `_",:option:`synadm media delete -i` +"|indent| `Delete local media by date or size `_",:option:`synadm media delete -d` :option:`synadm media delete --size` +"|indent| `Delete media uploaded by a user `_", +"`Purge Remote Media API `_",:option:`synadm media purge -d` diff --git a/doc/source/features/miscellaneous.csv b/doc/source/features/miscellaneous.csv new file mode 100644 index 0000000..1304c28 --- /dev/null +++ b/doc/source/features/miscellaneous.csv @@ -0,0 +1,5 @@ +"The Configurator",:option:`synadm config -u` +"Any Synapse Admin API",:option:`synadm raw ENDPOINT` +"Native Matrix API Helpers","---" +"|indent| Matrix User Login",:option:`synadm matrix login USER_ID` +"|indent| Any Matrix API",:option:`synadm matrix raw ENDPOINT` \ No newline at end of file diff --git a/doc/source/features/purge_history.csv b/doc/source/features/purge_history.csv new file mode 100644 index 0000000..0ab0f26 --- /dev/null +++ b/doc/source/features/purge_history.csv @@ -0,0 +1,2 @@ +"`Purge history `_",:option:`synadm history purge -i` +"`Purge status query `_",:option:`synadm history purge-status PURGE_ID` diff --git a/doc/source/features/register_users.csv b/doc/source/features/register_users.csv new file mode 100644 index 0000000..42e9838 --- /dev/null +++ b/doc/source/features/register_users.csv @@ -0,0 +1 @@ +"`Shared-Secret Registration `_", diff --git a/doc/source/features/registration_tokens.csv b/doc/source/features/registration_tokens.csv new file mode 100644 index 0000000..b005069 --- /dev/null +++ b/doc/source/features/registration_tokens.csv @@ -0,0 +1,6 @@ +"`Registration token objects `_","---" +"`List all tokens `_",:option:`synadm regtok list -v` +"`Get one token `_",:option:`synadm regtok details TOKEN` +"`Create token `_",:option:`synadm regtok new -n` +"`Update token `_",:option:`synadm regtok update TOKEN` +"`Delete token `_",:option:`synadm regtok delete TOKEN` diff --git a/doc/source/features/rooms.csv b/doc/source/features/rooms.csv new file mode 100644 index 0000000..1099607 --- /dev/null +++ b/doc/source/features/rooms.csv @@ -0,0 +1,21 @@ +"`List Room API `_",:option:`synadm room list -f` +"`Room Details API `_",:option:`synadm room details ROOM_ID` +"`Room Members API `_",:option:`synadm room members ROOM_ID` +"`Room State API `_",:option:`synadm room state ROOM_ID` +"`Room Messages API `_", +"`Room Timestamp to Event API `_", +"`Block Room API `_","---" +"|indent| `Block or unblock a room `_",:option:`synadm room block -b` :option:`synadm room block -u` +"|indent| `Get block status `_",:option:`synadm room block-status ROOM_ID` +"`Delete Room API `_","---" +"|indent| `Version 1 (old version) `_",:option:`synadm room delete --v1` +"|indent| `Version 2 (new version) `_",:option:`synadm room delete ROOM_ID` +"|indent| `Status of deleting rooms `_","---" +"|indent| |indent| `Query by room_id `_",:option:`synadm room delete-status -r` +"|indent| |indent| `Query by delete_id `_",:option:`synadm room delete-status -d` +"|indent| `Undoing room deletions `_", +"`Make Room Admin API `_",:option:`synadm room make-admin -u` +"`Forward Extremities Admin API `_", +"|indent| `Check for forward extremities `_", +"|indent| `Deleting forward extremities `_", +"`Event Context API `_", diff --git a/doc/source/features/server_notices.csv b/doc/source/features/server_notices.csv new file mode 100644 index 0000000..415a631 --- /dev/null +++ b/doc/source/features/server_notices.csv @@ -0,0 +1 @@ +"`Server Notices `_",:option:`synadm notice send TO` diff --git a/doc/source/features/server_version.csv b/doc/source/features/server_version.csv new file mode 100644 index 0000000..ddc6daf --- /dev/null +++ b/doc/source/features/server_version.csv @@ -0,0 +1 @@ +"`Version API `_",:option:`synadm version` diff --git a/doc/source/features/statistics.csv b/doc/source/features/statistics.csv new file mode 100644 index 0000000..6ed4f3f --- /dev/null +++ b/doc/source/features/statistics.csv @@ -0,0 +1,2 @@ +"`Users' media usage statistics `_", +"`Get largest rooms by size in database `_", diff --git a/doc/source/features/users.csv b/doc/source/features/users.csv new file mode 100644 index 0000000..e73d3eb --- /dev/null +++ b/doc/source/features/users.csv @@ -0,0 +1,33 @@ +"`Query User Account `_",:option:`synadm user details USER_ID` +"`Create or modify account `_",:option:`synadm user modify USER_ID` +"`List Accounts `_","---" +"|indent| `List Accounts (V2) `_",:option:`synadm user list -f` +"|indent| `List Accounts (V3) `_", +"`Query current sessions for a user `_",:option:`synadm user whois USER_ID` +"`Deactivate Account `_",:option:`synadm user deactivate USER_ID` +"`Reset password `_",:option:`synadm user password USER_ID` +"`Get whether a user is a server administrator or not `_", +"`Change whether a user is a server administrator or not `_", +"`List room memberships of a user `_",:option:`synadm user membership USER_ID` +"`Account Data `_", +"`User media `_","---" +"|indent| `List media uploaded by a user `_",:option:`synadm user media -f` :option:`synadm media list -u` +"|indent| `Delete media uploaded by a user `_", +"`Login as a user `_",:option:`synadm user login USER_ID` +"`Allow replacing master cross-signing key without... `_", +"`User devices `_","---" +"|indent| `List all devices `_",:option:`synadm user prune-devices --list-only` +"|indent| `Create a device `_", +"|indent| `Delete multiple devices `_",:option:`synadm user prune-devices -d` :option:`synadm user prune-devices -s` +"|indent| `Show a device `_",:option:`synadm user prune-devices --list-only` (-i) +"|indent| `Update a device `_", +"|indent| `Delete a device `_",:option:`synadm user prune-devices -i` +"`List all pushers `_", +"`Controlling whether a user is shadow-banned `_",:option:`synadm user shadow-ban USER_ID` :option:`synadm user shadow-ban --unban` +"`Override ratelimiting for users `_","---" +"|indent| `Get status of ratelimit `_", +"|indent| `Set ratelimit `_", +"|indent| `Delete ratelimit `_", +"`Check username availability `_", +"`Find a user based on their ID in an auth provider `_",:option:`synadm user 3pid ADDRESS` +"`Find a user based on their Third Party ID `_",:option:`synadm user auth-provider EXTERNAL_USER_ID` diff --git a/doc/source/index.rst b/doc/source/index.rst index 0a39557..ecb2211 100644 --- a/doc/source/index.rst +++ b/doc/source/index.rst @@ -8,9 +8,9 @@ Welcome to synadm's documentation! .. toctree:: :maxdepth: 2 - :caption: Contents: index_cli_reference + features examples index_modules diff --git a/doc/source/index_cli_reference.rst b/doc/source/index_cli_reference.rst index d64ba24..29154c3 100644 --- a/doc/source/index_cli_reference.rst +++ b/doc/source/index_cli_reference.rst @@ -2,8 +2,7 @@ Command Line Reference ====================== .. toctree:: - :maxdepth: 1 - :caption: Contents: + :maxdepth: 3 synadm.cli.root synadm.cli.config diff --git a/doc/source/synadm.cli.config.rst b/doc/source/synadm.cli.config.rst index 1819a38..490320e 100644 --- a/doc/source/synadm.cli.config.rst +++ b/doc/source/synadm.cli.config.rst @@ -1,5 +1,5 @@ -Config -====== +Config Command +============== .. click:: synadm.cli:config_cmd :prog: synadm config diff --git a/doc/source/synadm.cli.group.rst b/doc/source/synadm.cli.group.rst index adac55f..64d0752 100644 --- a/doc/source/synadm.cli.group.rst +++ b/doc/source/synadm.cli.group.rst @@ -1,6 +1,6 @@ -Group -===== +Group Commands +============== .. click:: synadm.cli.group:group :prog: synadm group - :nested: full \ No newline at end of file + :nested: full diff --git a/doc/source/synadm.cli.history.rst b/doc/source/synadm.cli.history.rst index e4932bf..b7f61a7 100644 --- a/doc/source/synadm.cli.history.rst +++ b/doc/source/synadm.cli.history.rst @@ -1,6 +1,6 @@ -History -======= +History Commands +================ .. click:: synadm.cli.history:history :prog: synadm history - :nested: full \ No newline at end of file + :nested: full diff --git a/doc/source/synadm.cli.matrix.rst b/doc/source/synadm.cli.matrix.rst index 4a657de..c2a861f 100644 --- a/doc/source/synadm.cli.matrix.rst +++ b/doc/source/synadm.cli.matrix.rst @@ -1,5 +1,5 @@ -Matrix -====== +Matrix Commands +=============== .. click:: synadm.cli.matrix:matrix :prog: synadm matrix diff --git a/doc/source/synadm.cli.media.rst b/doc/source/synadm.cli.media.rst index 72f84f0..d882a91 100644 --- a/doc/source/synadm.cli.media.rst +++ b/doc/source/synadm.cli.media.rst @@ -1,6 +1,6 @@ -Media -===== +Media Commands +============== .. click:: synadm.cli.media:media :prog: synadm media - :nested: full \ No newline at end of file + :nested: full diff --git a/doc/source/synadm.cli.notice.rst b/doc/source/synadm.cli.notice.rst index 9ad7540..ceb26e6 100644 --- a/doc/source/synadm.cli.notice.rst +++ b/doc/source/synadm.cli.notice.rst @@ -1,5 +1,5 @@ -Notice -====== +Notice Commands +=============== .. click:: synadm.cli.notice:notice :prog: synadm notice diff --git a/doc/source/synadm.cli.raw.rst b/doc/source/synadm.cli.raw.rst index b386fdd..3aef114 100644 --- a/doc/source/synadm.cli.raw.rst +++ b/doc/source/synadm.cli.raw.rst @@ -1,5 +1,5 @@ -Raw -==== +Raw Command +=========== .. click:: synadm.cli.raw:raw_request_cmd :prog: synadm raw diff --git a/doc/source/synadm.cli.regtok.rst b/doc/source/synadm.cli.regtok.rst index d5d0f64..f329471 100644 --- a/doc/source/synadm.cli.regtok.rst +++ b/doc/source/synadm.cli.regtok.rst @@ -1,5 +1,5 @@ -Regtok -====== +Regtok Commands +=============== .. click:: synadm.cli.regtok:regtok :prog: synadm regtok diff --git a/doc/source/synadm.cli.room.rst b/doc/source/synadm.cli.room.rst index 8974f8b..d4a7718 100644 --- a/doc/source/synadm.cli.room.rst +++ b/doc/source/synadm.cli.room.rst @@ -1,6 +1,6 @@ -Room -==== +Room Commands +============= .. click:: synadm.cli.room:room :prog: synadm room - :nested: full \ No newline at end of file + :nested: full diff --git a/doc/source/synadm.cli.user.rst b/doc/source/synadm.cli.user.rst index b611a78..a79b235 100644 --- a/doc/source/synadm.cli.user.rst +++ b/doc/source/synadm.cli.user.rst @@ -1,6 +1,6 @@ -User -==== +User Commands +============= .. click:: synadm.cli.user:user :prog: synadm user - :nested: full \ No newline at end of file + :nested: full diff --git a/scrape_docs.py b/scrape_docs.py new file mode 100755 index 0000000..0320082 --- /dev/null +++ b/scrape_docs.py @@ -0,0 +1,74 @@ +#!/usr/bin/env python +# Prerequisites: pip install -e '.[scrape_docs]' +# Usage: scrape_docs.py --help + +import click +from bs4 import BeautifulSoup +import requests + + +@click.command() +@click.option( + '--output', '-o', default='csv', + type=click.Choice(['debug', 'rst', 'csv']), show_choices=True, + help='''Output format "debug" prints headline level, headline text and the + scraped link/anchor; "rst" gives a restructuredText formatted hyperlink + including indentation according to headline levels. Spaces at the beginning + of lines have a special meaning in rst documents, thus the hardcoded + replacement marker "|indent|" is used; "csv" is a two-column comma + separated value format that includes the results of "rst" as the left + column's contents.''') +@click.argument('URL') +def scrape(output, url): + '''Scrape one chapter of Admin API docs and spit out in various formats. + + URL is the address of the Synapse Admin API docs chapter. For example: + https://element-hq.github.io/synapse/latest/admin_api/rooms.html + + The default output format is "csv", which gives a two column CSV table + containing restructuredText formatted hyperlinks and a headline. + ''' + chapter = url + apidoc = requests.get(chapter).text + soup = BeautifulSoup(apidoc, 'html.parser') + + any_heading_tag = ['h1', 'h2', 'h3', 'h4', 'h5', 'h6'] + elements = soup.find_all([*any_heading_tag, 'a'],) + + if output == 'csv': + print('"Synapse Admin API","synadm command(s)"') + + for e in elements: + if e.name in any_heading_tag and output == 'debug': + print(f'{e.name}: {e.text}') + if e.name == 'a': + if e.parent.name in any_heading_tag: + link = e['href'] + if output == 'debug': + print(f'Element text:\t{e.text}\nLink/Anchor:\t{link}') + if output in ['rst', 'csv']: + parts = chapter.split('/admin_api/') + fulllink = f'{parts[0]}/admin_api/{parts[1]}{link}' + spacing = '' + for h in any_heading_tag: + if e.parent.name == h: + # h1 is no spacing (decrease by 1), + # h2 is 2 spaces, h3 is 4.... + # two literal spaces are replaced by '|indent| ' + spacing_count = int(e.parent.name[-1]) - 1 + for val in range(0, spacing_count * 2): + spacing += '|indent| ' + rst = f'{spacing}`{e.text} <{fulllink}>`_' + if output == 'csv': + left_col = f'"{rst}"' + print(f'{left_col},') + elif output == 'rst': + print(rst) + if output == 'debug': # Final spacing only with debug format + print() + +# print(soup.prettify()) + + +if __name__ == '__main__': + scrape() diff --git a/setup.py b/setup.py index 12be3a4..8c46775 100755 --- a/setup.py +++ b/setup.py @@ -56,6 +56,11 @@ "click-option-group>=0.5.2", "dnspython" ], + extras_require={ + "scrape_docs": [ + "beautifulsoup4" + ], + }, entry_points=""" [console_scripts] synadm=synadm.cli:root