Missing notIndexed points in .ang files not re-filled by loadEBSD_ang in MTEX 5.11

[Tom-G-Uni-Vienna]

Hi all,

As I think you are aware, when MTEX imports EBSD data from .ang files, it does not import any points where the euler angles are all 4*pi, which is the .ang file's way of indicating completely not indexed points. Thus, an imported map ends up with "holes" where no data point is.

In the loadEBSD_ang interface there is a workaround whereby all resulting "holes" in the dataset are then re-filled with notIndexed EBSD pixels. In MTEX 5.10.2, this works fine, the code is (5.10.2 loadEBSD_ang.m from line 179):

  % reconstruct empty points previously removed by loadHelper
  % gridify might be easiest
  ebsd=ebsd.gridify;
  ind_no = isnan(ebsd.rotations);
  ebsd=EBSD(ebsd);
  ebsd(ind_no(:)).phase=notIndexedID;

In MTEX 5.11 (both versions), the line
ebsd=EBSD(ebsd);
has been removed from loadEBSD_ang, and the re-filling of not-indexed points does not occur. However, adding back in this line does not fix the problem, as it throws up an error with EBSD().

I think that maybe a change to EBSD() is to blame?

I tried to write a fix with the 'fill' option of gridify, but although
ind_no = isnan(ebsd.rotations)
then correctly finds the holes,
ebsd(ind_no(:)).phase=notIndexedID
does not work as I had hoped, and the resulting ebsd variable remains without any not indexed points.

To demonstrate the problem, first use the following code in MTEX 5.10.2 (here I create "artificial" holes in the test data just to demonstrate the issue, by removing all notIndexed points first):

mtexdata forsterite
ebsd_init=ebsd('indexed');
ebsd_final=ebsd_init.gridify;
ind_no = isnan(ebsd_final.rotations);
ebsd_final=EBSD(ebsd_final);
ebsd_final(ind_no(:)).phase=0;

in 5.10.2, length(ebsd_final) > length(ebsd_init), i.e. we create unindexed points to fill holes.

The code throws an error in 5.11. I think the error occurs with EBSD(), though carrying out the code line-by line (or deleting the EBSD() line) throws no errors, just results in ebsd_final and ebsd_init of the same length (so no added data points).

What would I like to happen?

I think there are two options: 1) fix the code in loadEBSD_ang or 2) change the behavour of ignoring all officially unindexed data points from .ang files, rendering the code in loadEBSD_ang unnecessary. I am not clear on why the step of ignoring all data points with 4*pi radian euler angles (.ang language for not indexed points) is being carried out in the first place...

best regards,
Tom Griffiths

PS: In case you are wondering why this matters: my scans are of crystals floating in glass with some gas bubbles - the gas bubbles give basically no EBSD signal and are thus not Indexed according to the software upon acquisition. This causes problems during grain reconstruction (I want the bubbles to be "filled" to stop any grains incorrectly expanding to fill these spaces).

[Tijmenvermeij]

Two possible solutions that might help solve your problem:

1. Mtex 5.11 has a new grain boundary calculation routine, using jc_voronoi, which is much faster and also allows to use a parameter 'alpha' that "specifies to which extend indexed regions are allowed to grow into not indexed regions". Maybe that can help you.
2. To get your nonIndexed points back, using gridify and identifying the nonIndexed points should do the trick (did a quick test myself):

ebsdNew= ebsd.gridify;
ebsdNew(isnan(ebsdNew.phaseId) = 'notIndexed';

Best,
Tijmen

[Tom-G-Uni-Vienna]

Hi Tijmen,

ebsdNew= ebsd.gridify;
ebsdNew(isnan(ebsdNew.phaseId)) = 'notIndexed';

Works great!

Forgive my ignorance, but will this syntax always give the correct numerical value for ebsdNew('notIndexed').phase?
This can be either 0 or -1, depending on the map...

The current code in loadEBSD_ang has a whole section that checks which should be used before (trying to) assign the correct phase number, so it seems that it is important to keep the right one?

If this new method would always pick the correct phase number, then these two lines of code could replace all of lines 171 to 183 in loadEBSD_ang !

If not, then this code would need to be expanded/altered before fixing the bug in loadEBSD_ang to make sure that the -1 / 0 distinction is maintained.

best regards,
Tom G

[Tijmenvermeij]

Hi,
Glad this works.
I'm sorry, I'm not very sure about all that, I'm not that informed into the workings of Mtex... I'm also not that sure if loadEBSD_ang really has a problem, I'll leave that to the experts.

Tijmen

[Tom-G-Uni-Vienna]

Just for any other readers, I can confirm that loadEBSD_ang has a bug, in so far as the code at lines 171 to 183 simply has no effect, and so an .ang file read in by the function does not have the missing notIndexed points added back in at present. In MTEX 5.10.2, the code still works and the notindexed points are re-added.

There are several possible solutions to this, including:
a) removing the code in loadhelper that explicitly drops the not indexed points in first place. As I said, I don't see why importing these points breaks anything (indeed, my "solution" to the problem for many years has been just altering this code in my MTEX to have the points be imported, with no bad effects I can find).
b) adapt the code in the loadEBSD_ang file
c) remove the relevant code in the loadEBSD_ang file seeing as it is not doing anything at present, as anyone wanting to add the notIndexed points later could do so with Tijmen's suggestion. However, they would have to notice that those points were missing (they are however only likely to be important for a subset of users).

Hopefully somebody from the core group of MTEX contributors can offer an opinion about which aligns best with their philosophy!

best regards,
Tom G

[ralfHielscher]

Sorry for my late reply. I did the following changes in loadEBSD_ang in develop:

1. pixels with 4*pi angles are not removed anymore - but the orientations are set to nan
2. nan-orientations get the phase notIndexed
3. gridify is not longer applied

Are those changes ok with you. From what I see the changed functions should also work in MTEX 5.11.

Ralf.

[Tom-G-Uni-Vienna]

Hi Ralf,

I think there would be one more tweak needed:

The removal of the pixels with 4*pi angle happens already in the loadHelper script, so if you do not change loadHelper and only edit loadEBSD_ang, there will still be holes in the dataset.

The code that is not working is code that attempts to re-fill the holes left by not importing certain pixels, loadEBSD_ang does not do any deleting of pixels itself that I can see.

If you would change things to make it so that 4*pi angle pixels are set with nan orientations and phase notIndexed already in loadHelper, then this would indeed fix the issue, because you would no longer need any special code in loadEBSD_ang to fill the holes.

best regards,
Tom G

[ralfHielscher]

Hi Tom,
this is what I did. The loadHelper already translated 4*pi values into NaN and be default removed those. However, when called with the option 'keepNaN' the loadHelper did not removed the NaN orientations. This is what I changed, I called loadHelper with this option are removed all the cleaning code in loadEBSD_ang.
Ralf.

[Tom-G-Uni-Vienna]

Hi Ralf,

In that case, that sounds great! Sorry for the misunderstanding, I wasn't aware that loadHelper works that way now!

best regards
TG