Proposal to move wiki to a repo, possibly use GitHub pages

[mborgerson]

The wiki is outdated XboxDev/XboxDev#4, breaks frequently XboxDev/XboxDev#18 XboxDev/XboxDev#10 , isn't configured properly XboxDev/XboxDev#14 XboxDev/XboxDev#5, and is generally a pain to maintain. I don't care to spend any of my time fixing mediawiki issues or administering the service, and don't really want to provide root access to the box for someone else to do it either.

The XQEMU project operates entirely out of GitHub for free, with a custom domain even. Anyone is free to submit changes, review submitted changes, upload files, create/xref issues, clone all data, etc.

I propose to move away from mediawiki and move all of xboxdevwiki to a GitHub repository, possibly with a front-end like what is used for XQEMU, or https://docs.readthedocs.io or https://www.gitbook.com, or any static site generator. Even GitHub's UI for viewing Markdown files would be acceptable in the interim. I welcome feedback on this, including any potential alternatives we can also consider.

[dracc]

I vote in favor. 👍

[DiscoStarslayer]

I am for moving from wiki as well.

One advantage of a self hosted wiki is that we have control of the data, it's why we have the option to move away from it.

It's also easier to moderate and provide access to people using a repo instead of a wiki.

We should ensure whatever direction taken should have easy + regular backups so we do not risk losing research.

Not saying current suggestions introduce this risk, but it should be a hard requirement.

My +1 would be for https://docsify.js.org/ as you can release site updates without having to bug a maintainer to generate the static site files

[mborgerson]

While we discuss alternatives to mediawiki, I have created a backup of the wiki and plan to create a repository full of all of the changes in the coming days. I will use a fresh backup at the time we do the export to make sure I have the latest data.

In this backup I have e-mail addresses and names for many of the contributors and would like to include it in the Git commit history to give credit. However, to be conservative with their privacy, I will not include their full names and e-mail addresses in these commits unless they respond on this post with a 🚀 reaction, or contact me directly between now and when the potential change is made.

TL;DR: Please respond with 🚀 if you want your name and e-mail address to be published with any wiki commit history.

[JayFoxRox]

(In this post I raise some concerns that I have; unfortunately I'm not an expert on any of this, so I can't really propose alternatives. I can just list the issues that I perceive. Ideally someone can just respond with solutions to nullify them)

I'm a bit torn on this issue. I agree that Mediawiki is horrible (at least the way we use it) and I can also understand that it's frustrating to maintain as-is.
Also, tech wise, moving to git is probably a good decision on the backend - it makes it easier to work with dumps and decentralizing backups.

However, I worry that there are many drawbacks of moving that are not being considered yet:

Frontend wise, Mediawiki has traditionally been very good at attracting users. Its design screams "I'm a wiki, edit me.". I feel like mkdocs, read the docs, and other solutions don't do this very well (yet?).
"Edit on GitHub" buttons don't solve these issues in my opinion.

If we'd use git without a good frontend, I'm also personally sceptical if I'd be willing to fire up my local editor and manually push changes - especially if a good preview would be missing. I assume that setting up a local copy of the wiki (for preview) is a barrier for some contributors, too.

I'm aware that there are some web-UIs for git-wiki frontends, but generally my experience was even worse than the horrible Mediawiki editor we currently use.

Mediawiki also allows many formatting options through HTML or similar; many other platforms I have seen don't get this right. We need the following (probably more):

• Colored tables (already seen here for example: https://xboxdevwiki.net/Xbox_ADPCM)
• Copy/pasteable tables (see hidden commas in https://xboxdevwiki.net/Xbox_ADPCM#Step-Table)
• Filesystem views or tools to make some (https://xboxdevwiki.net/Hard_Drive_Files)
• Images which can be auto-thumbnailed, with caption, and anchored / aligned differently.
• Ideally timelines (we never got this in Mediawiki due to lack of administration)
• Code highlighting (we never got this in Mediawiki due to lack of administration)
• latex or something else for formuals (we never got this in Mediawiki due to lack of administration)
• Various headline levels
• Standard formatting options
• Table of contents

Mediawiki also has templates to implement features - I worry that some of those features will be lost when migrating to another platform (I feel like inlining templates is not an option).
I feel like most solutions lack some of these, and porting the existing content would be an issue.

GitHub flavored Markdown for example, struggles with most of this.
So using GitHub to preview changes would be tricky at best. Reviewing changes would be even more difficult.

On the topic of moving content: I don't think we should just "move all of xboxdevwiki to a GitHub repository". - I'm also not sure if we can properly add attribution to the Mediawiki changesets due to the reformatting that will be necessary (I assume proper attribution will make this a lot more complicated than it already is).

More importantly, I feel like since day 0, XboxDevWiki had a lack of organization and structure. We have many redundant articles (xbox-linux imports, but also our own articles), and some info is not where you'd expect it. We use different terms for the same thing and so on.

This feels like an opportunity where we could migrate content slowly to fix at least some of those issues. Particularly because neither Mediawiki or git are great at tracking movement of content (which is another issue which this move wouldn't solve).

For new content, I worry that if we handle changes through GitHub PRs, then the amount of work to review PRs would be immense, and probably much greater than fixing Mediawiki once per year. Traditionally we've been more strict on GitHub PRs (so I assume we'll do at least some unnecessary bikeshedding or nitpicking), and reviewing such wiki changes in GitHub is usually annoying (diffs for text typically aren't fun to review). On the existing wiki we just don't care, and then iterate. Getting this mindset on GitHub will probably be more difficult.

We also have similar issues on the XQEMU.com repository where reviews are not accepted or reviewed, and even getting a good preview can be troublesome.

[mborgerson]

@JayFoxRox thanks for providing your feedback! Please see my responses to your concerns inline below. I think most of the concerns have solutions that are generally compatible with the proposal. Let me know if you want to discuss any of them at greater length.

---

Frontend wise, Mediawiki has traditionally been very good at attracting users. Its design screams "I'm a wiki, edit me.". I feel like mkdocs, read the docs, and other solutions don't do this very well (yet?). "Edit on GitHub" buttons don't solve these issues in my opinion. If we'd use git without a good frontend, I'm also personally sceptical if I'd be willing to fire up my local editor and manually push changes - especially if a good preview would be missing. I assume that setting up a local copy of the wiki (for preview) is a barrier for some contributors, too. I'm aware that there are some web-UIs for git-wiki frontends, but generally my experience was even worse than the horrible Mediawiki editor we currently use.

I agree that people likely wouldn't be as motivated to contribute if the interface is not convenient. I personally think the GitHub interface to edit pages is very user friendly. Roughly the same amount of clicks as MediaWiki's and also has a preview mode. If it's not clear how to edit, we can likely provide a how-to page, or improve the templates to make it more obvious?

Mediawiki also allows many formatting options through HTML or similar; many other platforms I have seen don't get this right. We need the following (probably more).
Colored tables (already seen here for example: https://xboxdevwiki.net/Xbox_ADPCM)
Copy/pasteable tables (see hidden commas in https://xboxdevwiki.net/Xbox_ADPCM#Step-Table)
Filesystem views or tools to make some (https://xboxdevwiki.net/Hard_Drive_Files)
Images which can be auto-thumbnailed, with caption, and anchored / aligned differently.
Ideally timelines (we never got this in Mediawiki due to lack of administration)
Code highlighting (we never got this in Mediawiki due to lack of administration)
latex or something else for formuals (we never got this in Mediawiki due to lack of administration)
Various headline levels
Standard formatting options
Table of contents

Lots of these features are available with GitHub flavored markdown, and with the static site generators I'm pretty sure we can add additional formatting plugins to do all of these things.

Mediawiki also has templates to implement features - I worry that some of those features will be lost when migrating to another platform (I feel like inlining templates is not an option). I feel like most solutions lack some of these, and porting the existing content would be an issue.

I'm not sure we really make use of templates except on the home page and for some icons. Can you point me to some more cases where we are using this feature? Otherwise I'm not sure it's something we can't live without.

On the topic of moving content: I don't think we should just "move all of xboxdevwiki to a GitHub repository". - I'm also not sure if we can properly add attribution to the Mediawiki changesets due to the reformatting that will be necessary (I assume proper attribution will make this a lot more complicated than it already is).

The tools I've experimented with actually step through the entire wiki history and create commits which retain committer information, which can optionally be supplemented with e-mail addresses and real names to be more Git-ish.

More importantly, I feel like since day 0, XboxDevWiki had a lack of organization and structure. We have many redundant articles (xbox-linux imports, but also our own articles), and some info is not where you'd expect it. We use different terms for the same thing and so on. This feels like an opportunity where we could migrate content slowly to fix at least some of those issues. Particularly because neither Mediawiki or git are great at tracking movement of content (which is another issue which this move wouldn't solve).

I think trying to migrate content slowly will result in nothing happening--I don't personally want to copy-paste paragraphs from the wiki to a new file. I think a less tedious approach would be to migrate all of the content, then fixup articles in the new place. This also has the benefit of retaining authorship. I do agree that we have lots of stuff that can be cleaned up.

For new content, I worry that if we handle changes through GitHub PRs, then the amount of work to review PRs would be immense, and probably much greater than fixing Mediawiki once per year. Traditionally we've been more strict on GitHub PRs (so I assume we'll do at least some unnecessary bikeshedding or nitpicking), and reviewing such wiki changes in GitHub is usually annoying (diffs for text typically aren't fun to review). On the existing wiki we just don't care, and then iterate. Getting this mindset on GitHub will probably be more difficult.

I'm not sure this is a real technical limitation, but more of a policy decision. If we add more trusted reviewers to the repo, we can expedite the review process while gaining a better pulse on what is being changed + reject content that is not fit for the wiki.

We also have similar issues on the XQEMU.com repository where reviews are not accepted or reviewed, and even getting a good preview can be troublesome.

I think most changes that include actual content being added usually get reviewed and merged pretty fast on xqemu.com; though this is my gut feeling, haven't crunched numbers or anything. It's the PRs that try to change how things fundamentally work that sometimes stall out and require discussion/testing/etc.

[JayFoxRox]

I agree on most things or can at least tolerate the proposed solution. So I'll just respond to the things I disagree with / want to add more information:

I personally think the GitHub interface to edit pages is very user friendly. Roughly the same amount of clicks as MediaWiki's and also has a preview mode.

Please note that the Wikimedia editor we used was really bad. Wikipedia and other wikis had long switched to other ones. The GitHub editor is pretty bare bones (no formatting buttons or WYSIWYG?) and worse than what I assume a new Wikimedia installation with good (defacto standard) plugins would offer (see visual editing and preview of markup as you write it).

This is not a huge issue to me, but still something worth considering. Moving to elsewhere instead of Mediawiki will be a downgrade in UX.

Lots of these features are available with GitHub flavored markdown, and with the static site generators I'm pretty sure we can add additional formatting plugins to do all of these things.

I'm not sure about this. There are solutions to this, but I feel like this should be respected before converting all content or making a platform choice.

Also at this point the preview in the GitHub editor would also likely break.

I'm not sure we really make use of templates except on the home page and for some icons. Can you point me to some more cases where we are using this feature? Otherwise I'm not sure it's something we can't live without.

You can see the list of templates here http://xboxdevwiki.net/Special:AllPages?from=&to=&namespace=10

We do use it for various things. I'll try to categorize them a bit:

Should be preserved or converted to a new markup:

• FIXME and citation-needed are used almost everywhere to provide a tooltip note. This is used in many places and inlining would expand a lot and make it harder to find or change the style of these in the future.
• no, yes, unknown for colored table cells (also no-select although I'm unsure how to find out where or how it is used). This is used in many places and inlining would complicate tables.
• The XBDM article uses the xbdm command template to auto-format each entry. Inlining this would mean a lot of redundant code and possible style inconsistencies.
• overline to markup stuff like 6.666666..ms in the APU articles. Inlining this would produce ugly code and make it hard to use this feature in the future.
• hc is used for hidden commas to format tables copyable (probably only in the ADPCM article). Basically if you select the table in the wiki and copy it, you get a C array. Inlining this isn't too bad although I intend to use it in GPU articles in the future (for things like enums or other LUTs). This can be removed if the new platform allows CSV copy/paste of tables or something.
• Icons use fs- or input-. Inlining this would leave ugly filenames and make it hard to change the style of these in the future.

Can be inlined:

• We use the retrieved template to mark Xbox-Linux articles. This template has been broken since the beginning because we lack better template support which never landed. We should probably have preserved Xbox-Linux wiki separately anyway, and removed redundant content.
• We have at least one template for easier auto-categorization of content (such as game). This was never used much and just saved a bit of typing. However, the ability to tag articles by category is a useful feature that the new platform ideally also supports. We should use it more extensively.

Also please keep in mind that I have complained about our poor support for templates in our Mediawiki installation from very early on. I wanted to use templates much more than I eventually did, because we didn't have things like conditional execution (which are standard on Wikipedia, which also made it impossible to copy useful things like infoboxes).

So there are 2 independent problems:

• Preserving the templates that we do have to avoid inlining things.
• The loss of this potential when migrating away to another platform (which might not support templates out-of-the-box).

I'm fine if there won't be user-templates anymore, but we should still have the option of easily adding more features (so ideally a macro/template system exists in the new platform).
If we switch platforms we'd ideally keep the markup in the articles as-is and implement templates for the features we already use.

---

Another independent thing that I have noticed is links: We should preserve all links to xboxdevwiki.net.
There are links to xboxdevwiki.net from reddit, PRs, redump related source-code (in DIC), projects like ogx360 and elsewhere.

We should ideally add a warning splash screen for old links but also redirect to the new content location on the new platform.

If permalinks get outdated over time that's fine, but ideally we'd always link to an old revision.

This is also something which the new platform should ideally support. Mediawiki allows displaying old revisions; if we have a static site generator, that might only display the latest revision. We should ideally pick a solution where we can view individual revisions of the website.
Note that Mediawiki even has visual diffs in the rendered website. Ideally we'd also still have such features on the new platform.

Again: I consider moving from Mediawiki (instead of fixing our ancient installation) a risk to UX and loss of a lot of potential. If we invest a lot of time and effort to make a new platform work (converting content, adding templates, reviewing the conversion is fine, setting up new policies, fixing permalinks, ..), then it might still be easier to just invest that time into Mediawiki (and potentially moving servers or even just attracting an administrator).
I agree that our Mediawiki installation is horrible, but I don't think we should blame Mediawiki for that - we just don't use its potential. I consider the limiting factor with Mediawiki to be our lack of expertise with maintaining it and the server not being suitable (no e-mails / disk-full).
On a new platform we shift that administrative overhead to tech issues as we'll have to implement a lot on our own probably.

[mborgerson]

@JayFoxRox

I'm not sure about this. There are solutions to this, but I feel like this should be respected before converting all content or making a platform choice.

Can you make a suggestion here?

FIXME and citation-needed are used almost everywhere to provide a tooltip note. This is used in many places and inlining would expand a lot and make it harder to find or change the style of these in the future.

Why would it make this harder to find? You can grep the files for FIXME just as you could with source code.

no, yes, unknown for colored table cells (also no-select although I'm unsure how to find out where or how it is used). This is used in many places and inlining would complicate tables.

Do they need to be colored?

The XBDM article uses the xbdm command template to auto-format each entry. Inlining this would mean a lot of redundant code and possible style inconsistencies.

How about just a table?

overline to markup stuff like 6.666666..ms in the APU articles. Inlining this would produce ugly code and make it hard to use this feature in the future.

Is this necessary? Just use ellipses to indicate repeating values, or a fraction.

hc is used for hidden commas to format tables copyable (probably only in the ADPCM article). Basically if you select the table in the wiki and copy it, you get a C array. Inlining this isn't too bad although I intend to use it in GPU articles in the future (for things like enums or other LUTs). This can be removed if the new platform allows CSV copy/paste of tables or something.

Browsers let you select and copy table contents and paste this in to a compatible application such as LibreOffice calc. I don't think we need to do extra work here to make it convenient for C programmers.

Icons use fs- or input-. Inlining this would leave ugly filenames and make it hard to change the style of these in the future.

You could just use something like ![](icons/button-x.png). This doesn't see too bad to me?

We should preserve all links to xboxdevwiki.net. There are links to xboxdevwiki.net from reddit, PRs, redump related source-code (in DIC), projects like ogx360 and elsewhere.

I agree there is convenience of such redirects, but I do not think this is critical, and I do not plan to invest my time in setting them up. xboxdevwiki.net will resolve to the wiki, and I think motivated people will have no problem finding the correct article. If you do want to create forwarding rules, I'm not opposed to it.

We should ideally add a warning splash screen for old links but also redirect to the new content location on the new platform.

This is doable. We can put the wiki in read-only mode and put a banner at the top directing people to the new wiki over some temporary time span.

it might still be easier to just invest that time into Mediawiki (and potentially moving servers or even just attracting an administrator). I don't think we should blame Mediawiki for that - we just don't use its potential. I consider the limiting factor with Mediawiki to be our lack of expertise with maintaining it and the server not being suitable (no e-mails / disk-full).

This proposal is what I consider an easier-to-maintain, low-overhead alternative style of wiki that can be conveniently integrated with our existing workflows, among other benefits listed in the issue description. I agree that Mediawiki is not a bad platform (I'm a big fan of Wikipedia after all), but to be as clear as possible: I do not plan to put any of my time in administering this service in this way. Apparently neither do the 3 other individuals that have server admin access. If some motivated individual, with a demonstrated track record of community presence, chooses to step up and regularly maintain such a service, I will gladly accept. Ideally, if you wanted to do this @JayFoxRox as you are probably the one with most knowledge about mediawiki and how it should be used, that would be great.

[Teufelchen1]

I find it really hard to decide which option would be better longterm. Personally I think mediaWiki is the correct tool to use. How ever, not having to administrate a lot of stuff really is speaking for a GitHub solution as well as better backups.
One thing that comes to my mind is that one can also have kind of the same experience with media wiki, take a look at managed wiki installations.

[JayFoxRox]

@mborgerson

I'll only respond to points where I disagree with or have something to add.

Lots of these features are available with GitHub flavored markdown, and with the static site generators I'm pretty sure we can add additional formatting plugins to do all of these things
I'm not sure about this. There are solutions to this, but I feel like this should be respected before converting all content or making a platform choice.

Can you make a suggestion here?

No. But if we consider a platform, then we should pay attention to wether the system is scriptable and how awkward the preview will be in an editor (as part of GitHub or the Wiki web UI).

no, yes, unknown for colored table cells (also no-select although I'm unsure how to find out where or how it is used). This is used in many places and inlining would complicate tables.

Do they need to be colored?

Ideally yes, for readability and ability to skim over some larger lists. However, I can tolerate if this isn't done.

I'd still prefer to have templates for the yes/no table cells though; it keeps the wiki consistent and tables sortable (avoiding mixing of different "yes", "Yes", "tickmark").

The XBDM article uses the xbdm command template to auto-format each entry. Inlining this would mean a lot of redundant code and possible style inconsistencies.

How about just a table?

I feel like this would be a downgrade.

Each XBDM entry is already a table, but the template makes it easy to remain consistent (and to possibly also generate the table-of-contents, a command-by-version overview etc. - which we already had to maintain manually on MediaWiki due to lack of plugins).

overline to markup stuff like 6.666666..ms in the APU articles. Inlining this would produce ugly code and make it hard to use this feature in the future.

Is this necessary? Just use ellipses to indicate repeating values, or a fraction.

Not necessary, but ideally solved by something like LaTeX formula support (which was requested for our MediaWiki installation, but never added).

hc is used for hidden commas to format tables copyable (probably only in the ADPCM article). [...]

Browsers let you select and copy table contents and paste this in to a compatible application such as LibreOffice calc. I don't think we need to do extra work here to make it convenient for C programmers.

Fair enough.

Icons use fs- or input-. Inlining this would leave ugly filenames and make it hard to change the style of these in the future.

You could just use something like ![](icons/button-x.png). This doesn't see too bad to me?

It gets tricky when you have * ![](icons/file.png) Filename.bin (Size: 1020 bytes; Flags: Read-Only) <sup>[f00f]</sup> - it's much easier to just have a syntax checked {{file|Filename.bin|1020|READ_ONLY|hash=f00f}} or something.
It's kind of a first-world problem, but it really makes things easier.

For button combos this is also a difference between:

Press {{A}}{{B}}{{Dpad-Up}}{{Dpad-Down}} to do XYZ.

vs.

Press
![](icons/button-a.png)
![](icons/button-b.png)
![](icons/button-dpad-up.png)
![](icons/button-dpad-down.png)
to do XYZ.

The first option also allows us to automatically get tooltips or add modifiers in the future.

It's not critical, but still very very useful, and losing such capabilities will probably mean less contributions and less quality.

We should ideally add a warning splash screen for old links but also redirect to the new content location on the new platform.

This is doable. We can put the wiki in read-only mode and put a banner at the top directing people to the new wiki over some temporary time span.

Sounds good to me.

We should preserve all links to xboxdevwiki.net. There are links to xboxdevwiki.net from reddit, PRs, redump related source-code (in DIC), projects like ogx360 and elsewhere.

I agree there is convenience of such redirects, but I do not think this is critical, and I do not plan to invest my time in setting them up. xboxdevwiki.net will resolve to the wiki, and I think motivated people will have no problem finding the correct article. If you do want to create forwarding rules, I'm not opposed to it.

I consider keeping links alive absolutely critical. I'd probably be willing to create a splash screen and a list of forwards (at least for URLs that I can find in the wild).

If some motivated individual, with a demonstrated track record of community presence, chooses to step up and regularly maintain such a service, I will gladly accept. Ideally, if you wanted to do this @JayFoxRox as you are probably the one with most knowledge about mediawiki and how it should be used, that would be great.

I know MediaWiki as a user, but not as an administrator. I'm wouldn't know who to ask either.
However, likewise I'm also not sure about the pros and cons about various wiki frontends for git, coding languages used on the web or anything similar (the kind of expertise I expect we'll need when leaving MediaWiki behind).

@Teufelchen1

One thing that comes to my mind is that one can also have kind of the same experience with media wiki, take a look at managed wiki installations.

I don't think any of the managed MediaWiki solutions (that I'm aware of) are adequate. There are usually ads and lack of flexibility if necessary. Most wikis hosted on such platforms also have a severe lack of "professionalism".

---

The discussion has been a bit slow during February (also because of me, as I'm currently busy with other things; sorry).

If this move is still supposed to happen, then I'd like to see a concept / demo / prototype of some sort so we can see how this actually behaves in the real world (before shutting down or archiving the MediaWiki).

[espes]

I prefer keeping mediawiki. I don't like how hard it is to maintain with the snowflake instance.

I want to migrate the wiki to Docker+google cloud run. It'll still be hard to edit but at least then we can have all the config and code in a repo here and anyone can send a PR.

[JayFoxRox]

Thanks for your input @espes.

I prefer keeping mediawiki. [...] I want to migrate the wiki to Docker+google cloud run.

Can we enable e-mails then (see XboxDev/XboxDev#5 )? This is one of the major issues we've been having - we can't even recover broken accounts (as the "Forgot password" does not work). Or we'd at least need plugins / patches so it doesn't depend on functional e-mails.

[espes]

A docker for xboxdevwiki is now at https://github.com/xboxdev/xboxdevwiki. xboxdevwiki.net is now being served from google cloud run using that and a Cloud SQL db. I'll set up a CD workflow later this week so any changes to that repo is automatically deployed.

Right now file uploads don't work. GCR is stateless, so if we want file uploads we need to get an extension that enables uploads to google cloud storage. I wasn't in a hurry to do that since the old instance's disk was full anyway.

To fix email we can use a a mediawiki extension to use sendgrid or something. However, if all we need it for is resetting passwords I'm more inclined to just write a script that invokes changePassword.php.

I'll polish it off later this week.

[espes]

(Note xboxdevwiki.net is broken right this second because I'm waiting for the new certs to be rolled out)

[espes]

File uploads are enabled again. Let me know if there are any problems.

[JayFoxRox]

To fix email we can use a a mediawiki extension to use sendgrid or something. However, if all we need it for is resetting passwords I'm more inclined to just write a script that invokes changePassword.php.

I assume we only need password reset e-mails. This is definitely the one that we kept getting complaints about. If admins can reset peoples password, that'd also be fine.

I'm not sure if people would be using e-mail for something like notifications on their watched articles. Even then, it probably wouldn't be a critical feature to most people either.

I'm also not sure what other features depend on e-mails in MediaWiki.

We certainly wouldn't be sending announcements or anything like that.

[espes]

All changes pushed to master on the xboxdevwiki will now be automatically deployed. Let me know if there are any problems.