Debugging the Mass Effect Trilogy
Debugging issues in the Mass Effect trilogy can be difficult, especially if you don't know where to begin. As a developer, you should ensure you're taking consistent snapshots of your mod, as an issue that appears at one point may be triggered by changes further back then your last save. Many modders have learned the hard way when working on mods that a backup saves valuable time - once your file is broken, without a backup you have no way to get it back without recreating it.
Depending on which game you are modding, you will have different tools available. In order of how "easy" it is to debug a game, Mass Effect 3 is the easiest, followed by Mass Effect, and finally ending with the 'cursed' Mass Effect 2.
Each game has specific debugging tools for the game, in addition to the global tools we'll discuss below. ASI mods are native code mods that run along with the game and can provide additional features.
Mass Effect conveniently outputs program logs, which include crash error messages for some crashes. It is by far the most useful output, but it only typically only helps with crashes. The native SDK we have for ME1 is built differently than ME2/ME3 and is not easy to use, and as such, there are no game-hooking ASIs.
- The application log located in Documents/BioWare/Mass Effect/Logs
Mass Effect 2 removed the logging feature from the first game, as well as fully stripping out any useful debug methods. As such there is no way for us to tell why the game crashed (without resorting to a debugger & disasembler), or see internal debug methods (such as ClientMessage), which means developing mods for ME2 can be very frustrating when the game begins to crash.
- Kismet Logger ASI
- Function Logger ASI
- Streaming Levels HUD ASI
Mass Effect 3 does not have a logging feature, but they left an internal native method stub for 'appErrorF', which is the internal procedure call for logging error messages. By default this was a stub and did nothing, but with the ME3 Logger ASI installed, calls to this method are logged. It only catches handled error messages and crashes, unhandled errors that crash the application are unlikely to be logged. Common output from this log includes bad import/export/name index, I/O error operation on files, Serial Size mismatch, incorrect class declarations, and more.
- Kismet Logger ASI
- Function Logger ASI
- Streaming Levels HUD ASI
- ME3Logger ASI
P L A C E H O L D E R
Topics include:
- Available debugging tools for each game
- ProcMon intro
- ME3Logger + ME3Explorer debugging
- Mod Manager debugging
Process Monitor is a very useful utility that logs tons of things a process does, from file reading, registry key reading, subprocess start/stop, thread start/stop... While a lot of these features are not useful to us, ones such as ReadFile are. This section will cover the basics of using ProcMon.
You'll need to download ProcMon to get started, which you can download from Microsoft. Extract it, run it, and accept the EULA.
ProcMon logs everything by default on first run. You will need to make sure you immediately drop filtered events and stop the capture - ProcMon can crash your system if you leave it unattended, as it will use all available memory. Go to Filter > Drop Filtered Events.
The toolbar at the top of ProcMon is where useful actions are performed. The ones we will care about are Capture enable/disable, Autoscroll, Clear log, and Filter.
Capture enable/disable is the magnifying glass, with the items directly to the right of it being autoscroll, clear log, and filter, in order. At this time you should disable capture and clear the log. You can tell capture is disabled when the log is cleared by looking at the bottom left of the window.
When capturing logs for analysis, ensure you stop capturing once the event you are looking for has occurred. Otherwise, you will generate extra useless data that makes your work more difficult. Try to remember the timestamp that your event in-game occurs, which can narrow down your search.
A vital step in using procmon is filtering the events down to only what is relevant, otherwise you're going to dig through a lot of useless info. Sometimes it's useful to see everything, but often it will not be. Below is an example setting that I use to see which pcc files are being used by Mass Effect 2 (Origin version).
Common issues you will likely be looking for are things like the following:
- Is the right file actually being loaded? (E.g. is my DLC mounted high enough? Did I name a file wrong?)
- What was the last file that was read before the game died?
- Was the read offset correct (for things like what offset in a TFC is being accessed - rare, but may be useful)
- Was my package file actually fully read? Or was just the beginning of the file? This typically means there is something wrong in the package header itself, or the metadata about the package such as import or export tables.
ProcMon is a powerful tool that takes some playing around with to get a feel for the power of it. The filters allow you to greatly scope down what you're looking for. ProcMon can be used to discover application's hidden or lesser known features, such as it looking for flag files, or in areas that are unknown - like how Unreal Engine can load packages from the Documents folder first.
ME3Explorer has a few debugging tools available in it that can help find broken files. While it will not catch a lot of cases that can crash the game (since there are effectively infinite), many common ones stem from invalid properties and references.
Inside of Package Editor, there is a 'Debugging' menu. Once a package is loaded, you can use features in this menu to help identify issues in the package. It's not super robust - most of these are experiments that got promoted into usable features - but they can help identify issues, if you know the loaded file is the one causing issues.
You can find an entry that contains a specified offset in a file using this feature. You typically use this with output from ME3Logger to try and track down invalid data after finding instances of the invalid numerical value in an uncompressed package file.
You can use this to find an entry that has a specific Tag nameproperty. Not that useful for debugging game issues but can help track down actors that don't do what you want.
This feature will check for any entries that have a duplicate full-instanced index. Unreal Engine 3 will not hold duplicate objects in memory, and the full object path - the name of the object, and it's index - are used. This system can greatly complicate mod design, and easily increase memory usage if abused. It is a key part of Unreal Engine 3 and you should adhere to it as best as possible, even though it can be difficult at times.
Paths are constructed by concatenating each object in the heirarchy with a . - so, for example, in the below picture, the highlighted object's full path is Wwise_Samara_Loy_Cinedesign.Wwise_Samara_Loy_Cinedesign.
You may notice some objects end with a _0, _1, etc. These are 'indexes' of names, and can be used to help make memory-distinct objects. You can edit the index of the object name in the Metadata tab. Index values of 0 are not displayed, and values greater than zero are shown, but with the value decremented by 1 so they line up with Unreal's internal system.
Level object memory indexes (items under TheWorld object) are tied to the local level, so they can have same named objects as other levels. Objects outside of TheWorld however will be compared to existing items in memory.
Here's an example: You want to edit BioH_Vixen_00.pcc, which is ME2 Miranda. You change her hair, save the package, and then go to test it at in the starting level of BioP_ProCer. Miranda looks normal in the table scene - this is normal, as the Miranda shown here is not actually the BioH_Miranda, but rather another actor that has her mesh. You go to the end of the mission and upon meeting her... she looks like normal Miranda. What gives?
Inside of BioP_ProCer is a copy of Miranda - SFXPawn_Miranda, specifically, and this copy is kept in memory for the whole level. The Miranda at the end may be the same one from her BioH_Miranda.pcc file - but since most of her data was already in memory from BioP_ProCer by the time her BioH_Vixen file loaded, it saw it was already in memory and was never used.
To fix this, you must make the root object that is referenced, and sub objects too, have a different name. Renaming a single object is not enough as it may use a version already in memory that points to the original values.
For example, renaming Miranda's class itself will make the game use the new class, assuming it is the object that was referenced - but while her BioH's BIOG_HMF hair mesh has changed, it was not renamed, so it still used the version in memory, as it is still a duplicate by name.
Duplicate index checker in ME3Explorer only checks the local package, and ignores trashed items, as we purposely make these duplicate objects so they are not stored in memory more than once.
This item will enumerate all exports and look for references outside of the import and export tables. It will list anything that is not within range. It also checks binary references for classes that we support parsing, which is most of them. This feature does not check if the referenced object is of the expected type, however. We may change this in the future, it is fairly complicated to support.
ME3Tweaks Mod Manager, while mostly an end user tool, contains some developer features. The main one is the mod deployment feature, which runs a bunch of checks on a mod to identify common issues.
TODO: WRITE MOD MANAGER CHECKS HERE
ME3Logger is an asi mod for Mass Effect 3, and ages ago was directly included in the binkw32.dll bypass (not asi loader version). You can install this ASI from within ME3Explorer's ASI Manager, or ME3Tweaks Mod Manager's ASI Mod Manager (located in the tools menu).
When Mass Effect 3 is running, calls to appErrorF inside of the game's executable will be routed out to a file named 'me3log.txt' next to the game executable. If your game is crashing (or you've run out of ideas), looking at this log may help indicate what the problem is.
The following types of messages are reported by ME3Logger (but are not limited to) are fairly generic in nature and may not accurately describe the actual error that occurred. Some of the error messages include:
I/O error operating on file
- The file was attempted to be read in a way that produced an error. This could mean it was reading beyond the end of the file (or at a negative offset), or data may attempted to be loaded that is inconsistent with engine design. For example, reading the 7th or higher mip on a texture that is not externally stored, but the texture is missing the 'NeverStream' flag, will result in an I/O error reading on that file.
Import index out of bounds
- The listed import index will be a positive value. You need to add one, then multiple it by -1 to get the actual stored value. You can try searching a decompressed's package hex for this value (4 bytes, little endian) to try to figure out the offset of where this data is actually stored at.
Export index out of bounds
- The listed export index will be a positive value. You need to add 1 to get the get the actual stored value. You can try searching a decompressed's package hex for this value (4 bytes, little endian) to try to figure out the offset of where this data is actually stored at.
Name index out of bounds
- The listed name index will be a positive or negative value. Only 0 and positive values within the name table range are valid. You can try searching a decompressed's package hex for this value (4 bytes, little endian) to try to figure out the offset of where this data is actually stored at.
Once you've found the offset. You can use Package Editor's 'Find entry by offset' feature to locate what entry contains that offset of data.
