Alternative to wow.tools for sourcing data

[seirl]

https://wow.tools will become read-only in 2023 due to a lack of maintainers, and has already stopped providing downloadable CSV exports.

We are currently using wow.tools to import data to SimpleArmory, because using the Blizzard API has proven very incomplete and unreliable, and in-game data is the only trustable source of info. Since wow.tools is getting discontinued, we will need to generate the CSV files ourselves from the game data by using DBC2CSV + a Listfile dump.

I personally don't have the bandwidth to work on this. My ideal workflow would be that $someone creates a periodical export of CSV files of WoW builds generated using DBC2CSV, potentially made available as a git repository or something like that, then I could change the import scripts so that they use these CSV dumps as an input, instead of fetching them from wow.tools.

Without this, we might have to go back to the old way of doing things: extremely broken scripts scraping wowhead to import data, data holes, manual JSON writing, unreliable data that we can't check, etc. If it has to come to that, I have to say that I will very likely lose interest in maintaining the entire data aspect of SimpleArmory, as I've spent many hours ensuring that we have very consistent and reliable data, and manual data entry will be a huge maintenance burden.

I tagged this issue as "help wanted". If you have any interest in helping generate these CSVs and the listfile, and dumping them somewhere, your help would be greatly appreciated.

[maxdekrieger]

I might have some spare time soon, but I'm not too familiar with the whole setup as is. Is it ok if I list my current understanding of the problem and you can correct me if I'm wrong?

So right now our data importer uses the wow.tools API to pull CSV files and reason over them. The wow.tools data will no longer be actively maintained from 2023 so we need a new source of CSV files. The most reliable source of data directly from wow builds, instead of manually crafted sources, so we need to find a new host of the build data to pull from, and then convert it to CSV using the DBC2CSV tool.

However, I don't see how the Listfile project would help since it seems to gather its data from wow.tools as well (judging from the commits: https://github.com/wowdev/wow-listfile/commits/master). In the November update from wow.tools they recommend other hosts such as wow.export or CASCExplorer.

Let me know if my understanding of the problem is correct and if you have any recommendations for a new host of the files.

[seirl]

That's close!

Right now it goes:

WoW client -> DBC2CSV -> wow.tools -> Simple Armory import scripts

Basically wow.tools has been doing the work for us to extract the data from the WoW client using DBC2CSV. But soon they will no longer do it, so we need to find a replacement (such as someone doing it for each build and uploading the resulting CSVs to a git repository or something).

[maxdekrieger]

Right, I can create a repo and push updates whenever a new wow retail build comes out, now I just need a way of getting notified about new builds...

Do you happen to know any service I can subscribe to using email or something to get notified?

Otherwise I can build something myself maybe. https://blizztrack.com/ looks promising for this data?

[seirl]

That's awesome to hear! I don't know much about that, but you can check out the wow.tools discord, there's a lot of people discussing potential replacements and ways to do that: https://discord.gg/Z5DeJH4NXS

[maxdekrieger]

I must be missing something, it's not that hard to export the data from a local installation using the wow.export and DBC2CSV tools. I'll check out the discord soon and see where my approach is naive 😅.

Files for the latest live build is here: https://github.com/maxdekrieger/wow-csv-from-db2s

[seirl]

Thanks, that's great! It's more annoying for me because I'm on linux and some of the tools I tried didn't work on wine. Were you also able to generate the listfile?

[maxdekrieger]

Err I didn't take a look at that yet, oops... I'm not quite sure on how to generate the listfile right now.

[maxdekrieger]

For now I'm not sure if listfiles can be generated at all, as Blizzard does not provide them I think. Judging from https://us.forums.blizzard.com/en/wow/t/community-tools-filenames/1179183, a lot of work goes into maintaining the listfile from wow.tools... I think you were right indeed and we have to resort to https://github.com/wowdev/wow-listfile haha. Took me some time but now I get it.

[maxdekrieger]

Update according to a conversation in the discord for the record:

We currently use the listfile for mapping icon ids to icon names but the manifestinterfacedata db2 file contains almost all of this data, with the icons belonging to the encrypted content being the exception.

[cassetto]

Jumping in just to say that I am willing to help as well, although my knowledge is extremely limited. I am willing to invest some time in training and educating myself on the topic, since I have been a fan of Simple Armory for a long time.

From my understanding it's a matter of data extraction from every new wow build and feeding it to the Simple Armory import script, without relying on wow.tools

As I train, please let me know if any help is needed on the more clerical side of things like data entry and such.

[maxdekrieger]

FTR: https://wago.tools/ is a decent alternative to wow.tools and it has responsive developers. It does not support hotfix files yet, which is why the the local export is a little better than wago.tools but according to the devs hotfix support is planned and in development already.

[kevinclement]

@seirl Looks like as I'm ramping back up I'm now running into this. Seems like we need to get this sorted, for both me to continue to work and for the project to stabalize how it does imports.

Is there an easy way for us to discuss this whole topic. maybe over discord. I can get a higher bandwidth brain dump that way and ask a few questions.

[maxdekrieger]

As a workaround, I manually did the exporting of the DB2 files, including the hotfix file to CSV files in this repo: https://github.com/maxdekrieger/wow-csv-from-db2s. This does require having an active subscription to access the DB2 and hotfix files from your local installation, and having a Windows PC to run some community tools. I hope the readme in my repo should is sufficient.

I'm not sure whether wago.tools supports hotfix data already, but you can reach the devs on their "WoW Datamining" Discord: https://discord.com/invite/qVZDRYfB4A. While @seirl is indeed the mastermind of the importer, I'm always up for a higher bandwith discussion on discord (id nexinmax97).

[seirl]

@kevinclement @maxdekrieger We can talk over discord yeah! My ID is also seirl.