Importing Disqus comments when XML file is not available

[badolamgk]

Hello,

I was recently looking at Disqus's import and export page where I came across this statement:

Please note: exports may not be available for all sites, particularly those of a large size. If you've requested an export file more than twice and still have not received a download link from us, it's likely that an export for your site is currently unavailable.

I was wondering, in order to solve for cases like these, whether there should be a new Importer incorporated into Remark42's backend?

I have built a version of it in my local system, which incorporates Disqus API. The drawbacks are there for sure, notably having slower speed than our traditional method and the uncertainty of the completion of import process in one sitting (API usage quota is 1000 calls per hour).

Let me know your thoughts regarding this.

Many thanks.

[paskal]

I think it would be beneficial to create a tool that generates an XML file for such sites. Instead of just developing a solution for Remark42, this tool could address the broader issue for anyone who wants to export data from Disqus but is unable to.

On the other hand, it's likely someone has already encountered and solved this problem. I found a couple of projects, with the most recent being humanwhocodes/disqus-export written in NodeJS. This project outputs data in JSON, but it should be straightforward to add XML support. The code to download comments is less than 200 lines in JS, although it doesn't account for rate limiting.

Ideally, I would like to have a description in our documentation on how to handle this corner case with Disqus using that or any other third-party tool. Please let me know your thoughts and if you are interested in addressing this issue.

This is the first time I've heard of this problem, so I don't plan to work on it myself right now. However, I may reconsider if no one else picks it up and several people are affected.

[badolamgk]

So, you are suggesting that I should contribute to that project for XML support, and after that integration is sorted out, I should ping you for changes in documentation?

[paskal]

Correct, except that I do not insist on sticking to the tool I mentioned because it might be easier to ask ChatGPT to write the code in your favourite language from scratch, and it will work right away or after a few iterations.

Now thinking about it, that task could be solved by client-side JS so that the solution could be even embedded to the documentation page itself and export triggered from there in the user's browser. If it wouldn't be solved in some time, I'll try this variant myself as it sounds straightforward yet interesting.

[badolamgk]

Doing it in client-side JS may cause in 504 Gateway Timeout if there are a lot of comments.

In Disqus, comments are called Posts and the pages where they were posted are called Threads. The flow when involving Disqus API is as follows:

1. Query the posts > list endpoint. Now the API response, by default, only gives 25 comments (the limit can be increased to maximum of 100) and has a pagination system in place. So essentially, if you want to export the entirety of your comments, you would have to keep hitting this endpoint until the very last "page". [1]
2. Filter out comments marked as spam/deleted.
3. For each comment, find attributes of interest (for Remark42 Comment object, they are ID, author's name and username, actual message, parent ID of the comment, creation timestamp and page URL)
4. The concern in the above step is that you don't get the URL of the page but only the ID. So, you have to then query thread > details endpoint in order to get the URL. Since a page can have multiple comments, we can employ a in-memory cache to store page URL against page ID. [2]

So as you can see, it is not feasible to attached it in client-side JS.

...it might be easier to ask ChatGPT to write the code in your favourite language from scratch

As I said in my post, I have a local implementation built in place (in Go) which is directly integrated in the Import although as you had said that I should rather build a platform-agnostic solution that gives XML file. But for the scope of Remark42, could you help me understand the appeal of it, since I can directly call the Disqus APIs on my Dockerized application, create in memory variables and assign them to Comment object on the go? Thanks.

Image References:
[1]

[2]

[paskal]

I have a local implementation built in place

I think it would be great to release it with ability to export XML so that this class of problems would be solved for everyone in need, and output would be in the same format any import tool already expects.

Now, thinking of it, I might need to check if Remark42 has export in Disqus format as it would be very useful for the same reasons, presence of enormous amount of importers tailored to that format.

But for the scope of Remark42, could you help me understand the appeal of it, since I can directly call the Disqus APIs on my Dockerized application, create in memory variables and assign them to Comment object on the go?

Apologies, I do not understand that question, could you please clarify it?

[badolamgk]

Apologies, I do not understand that question, could you please clarify it?

Sorry for that. I just did a bit of digging and found out about the un-marshalling benefits (like accessing nested items like a>b>c or attributes using ,attr) in GoLang. Sorry for not noticing this earlier, as I have just started learning it and thought to test myself by contributing (like my auth PR here). XML is the way to go 🙌

I'll probably build that tool from scratch in Go. Thanks!

[badolamgk]

Just a heads up, I had started working on the tool here. I've been learning a lot with this and would just like to thank Remark42 to re-ignite my passion to build stuff. I've been juggling with office work and this, so the development timeline might be a bit slow.