fixer crashes scanning regions

[9hrack]

My when loading a specific chunk of the world,
the bukkit crashes and the bukkit says : I cannot load newer version of chunk

and scannning my world with this region fixer, it crashes :

Bug report:

**********
*** Exception while scanning:
*** r.-10.2.mca
**********
*** Printing the child's traceback:
*** Exception:<type 'exceptions.ValueError'>Unrecognised tag type
**********
*** File /home/centos/Minecraft-Region-Fixer-0.2.1/regionfixer_core/scan.py, line 695, in scan_region_file
***   entity_limit)**********
*** File /home/centos/Minecraft-Region-Fixer-0.2.1/regionfixer_core/scan.py, line 779, in scan_chunk
***   chunk = region_file.get_chunk(*coords)**********
*** File /home/centos/Minecraft-Region-Fixer-0.2.1/nbt/region.py, line 567, in get_chunk
***   return self.get_nbt(x, z)**********
*** File /home/centos/Minecraft-Region-Fixer-0.2.1/nbt/region.py, line 552, in get_nbt
***   return NBTFile(buffer=data)**********
*** File /home/centos/Minecraft-Region-Fixer-0.2.1/nbt/nbt.py, line 517, in __init__
***   self.parse_file()**********
*** File /home/centos/Minecraft-Region-Fixer-0.2.1/nbt/nbt.py, line 544, in parse_file
***   self._parse_buffer(self.file)**********
*** File /home/centos/Minecraft-Region-Fixer-0.2.1/nbt/nbt.py, line 391, in _parse_buffer
***   tag = TAGLIST[type.value](buffer=buffer)**********
*** File /home/centos/Minecraft-Region-Fixer-0.2.1/nbt/nbt.py, line 379, in __init__
***   self._parse_buffer(buffer)**********
*** File /home/centos/Minecraft-Region-Fixer-0.2.1/nbt/nbt.py, line 391, in _parse_buffer
***   tag = TAGLIST[type.value](buffer=buffer)**********
*** File /home/centos/Minecraft-Region-Fixer-0.2.1/nbt/nbt.py, line 304, in __init__
***   self._parse_buffer(buffer)**********
*** File /home/centos/Minecraft-Region-Fixer-0.2.1/nbt/nbt.py, line 314, in _parse_buffer
***   self.tags.append(TAGLIST[self.tagID](buffer=buffer))**********
*** File /home/centos/Minecraft-Region-Fixer-0.2.1/nbt/nbt.py, line 379, in __init__
***   self._parse_buffer(buffer)**********
*** File /home/centos/Minecraft-Region-Fixer-0.2.1/nbt/nbt.py, line 395, in _parse_buffer
***   raise ValueError("Unrecognised tag type")
**********

[Fenixin]

Thanks for the bug report. Could you please send me the region file that causes this?

[macfreek]

@deepnetworks Was this tested with the master branch code or the 1-3-update? I kind of expect that this was fixed by NBT 1.5, which was included in Minecraft-Region-Fixer in August in the 1-3-update branch, but only merged to master a week ago.

[ghost]

I have a world that also has many corrupted chunks. The scanner works fine with other worlds, but not the corrupted one. I am getting the same error as DeepNetworks, and I am using the most current and updated branch. Also Fenixin, this is a very amazing tool and thank you for all the time you put into it!

[Fenixin]

Hello!

@GeoffSon, could you please try to scan that same world with this branch? Is a WIP, but should detect a new kind of problem. Currently, this branch only works with python3.

[ghost]

@Fenixin Thank you for the quick response. I’ll be sure to try out this new branch!

[ghost]

I just tried to run the program with the new2to3 branch on Python3.7, and I got the following message.

Bug report:

---

*** Exception while scanning:
*** idcounts.dat

---

*** Printing the child's traceback:
*** Exception:<class 'TypeError'>a bytes-like object is required, not 'str'

---

*** File C:\Users\Geoffrey\Desktop\Region Fixer\Minecraft-Region-Fixer-new2to3\regionfixer_core\scan.py, line 615, in scan_data
*** _ = nbt.NBTFile(buffer=f)**********
*** File C:\Users\Geoffrey\Desktop\Region Fixer\Minecraft-Region-Fixer-new2to3\nbt\nbt.py, line 628, in init
*** self.parse_file()**********
*** File C:\Users\Geoffrey\Desktop\Region Fixer\Minecraft-Region-Fixer-new2to3\nbt\nbt.py, line 652, in parse_file
*** type = TAG_Byte(buffer=self.file)**********
*** File C:\Users\Geoffrey\Desktop\Region Fixer\Minecraft-Region-Fixer-new2to3\nbt\nbt.py, line 99, in init
*** self._parse_buffer(buffer)**********
*** File C:\Users\Geoffrey\Desktop\Region Fixer\Minecraft-Region-Fixer-new2to3\nbt\nbt.py, line 105, in _parse_buffer
*** self.value = self.fmt.unpack(buffer.read(self.fmt.size))[0]

The scan crashed after starting to scan structures and map data files. I believe the file causing the issue is idcounts.dat.

[ghost]

Actually, I just ran another scan, but I deleted the idcounts.dat file, and the scan worked! Thank you.

[macfreek]

@deepnetworks could you share your r.-10.2.mca file, either in public or private?
@GeoffSon Could you share your idcounts.dat file, either in public or private?

Both exceptions originate in the nbt library, which is used by Minecraft-Region-Fixer. I like to check if these exceptions are correct, and like to work with @Fenixin to see if we can make a patch that allows Minecraft-Region-Fixer to better report the error, and continue with other files.

[macfreek]

@Fenixin perhaps a better approach when a NBT file can't be read is first to:

• First ensure that it is a supported NBT version, and report if it is not
• If it is supported, and it can't be read, propose a solution to the user:
  • Recover the bad chunk or file from backup.
  • Delete the bad chunk or file

Of course, recovering or deleting a bad NBT file (.dat files) is something the user can do from the file browser, no need to support that from Minecraft-Region-Fixer, except a clear message which file is affected and what is wrong with it.

To delete or replace a chunk is possible with the NBT library. In 2013 I meticulously improved the region code to ensure even the most insanely corrupted regionfile is at least not further corrupted.

[ghost]

I don't mind sharing my idcounts.dat file.

idcounts.zip

[Fenixin]

@macfreek, that sounds great. I should look where the region file format version is stored and check if nbt cand open it before scanning. I haven't looked into it yet but, is there a nice why to ask nbt if a world is the correct format?

[Fenixin]

@GeoffSon, it's probably quite late for this but I have added a those errors to data scanning and regionfixer should point out that the file is broken instead of crashing.

So I have been looking how to check if the nbt version is supported or not and it seems that it's not that hard. From the Minecraft Wiki/level.dat there is TAG_Int that store the nbt version, currently 19133.

A funny thing that happens is that regionfixer is designed to be able to scan worlds and independent region files. Because the nbt version is only stored in leve.dat, if the user only provides a region file/regionset instead of a world directory there is no way to know what nbt version is being used, and consequently we don't know if the chunks/regions are really broken or nbt is failing because a new version of nbt format.

Another funny case that could happen when a new version of nbt rolls out and to look the nbt file version we need to open a nbt file. But I think it's unlikely that the nbt format changes that much.

I'm not really sure how to tackle this problems. Though I'm not sure is the best of solutions, right now I'm thinking in:

• If it's a world just check the version and print a big warning or stop the scan if a new nbt version is detected.
• If it's a few region files just print a warning that nbt version is unknown and corrupted chunks detected could be consecuence of a nbt format change. Also in the warning a "proceed at your own risk" text.

Any other ideas?

EDIT: Is there another way to check NBT version in a region file?