Some structures are laggy and produce duplicates when saving

[hiimjustin000]

Describe the bug
I've seen this occur many, many times. Some structures (Usually bigger ones, but a few small ones can trigger this bug as well), take tens of seconds to save, and oftentimes create a duplicate when saving is finished. I can tell it is a duplicate due to the ".conflict" suffix.

To Reproduce
Steps to reproduce the behavior:

1. Go to Data Type Manager
2. Find a large structure
3. Edit it
4. Save it (Note that this bug's appearance is not consistent)

Expected behavior
The structure would save properly, and not create a duplicate.

Screenshots

Attachments
If applicable, please attach any files that caused problems or log files generated by the software.

Environment (please complete the following information):

• OS: Windows 11 23H2 (10.0.22631.4602) / macOS Sequoia 15.1
• Java Version: 21.0.4 / 21.0.5
• Ghidra Version: 11.2.1
• Ghidra Origin: GitHub

Additional context
While writing this issue, I think I noticed what causes this bug. The smaller structures triggered the bug because they referenced the bigger structures (i.e. as a pointer), which would trigger the bug on their own. I noticed that while saving the small structures, duplicates of the larger ones being referenced would be created.

[DualTachyon]

In my experience, the "conflict" happens when your fields overlap. It can happen in various ways, one of them is making structure A a field of B and later making A larger. It depends also on whether you edit B afterwards and save it without resolving the conflict .

There are other ways it can happen, but I haven't figured it out.

I would suggest first to resolve conflicts you might have, by checking which fields are bigger than their allocated space.

Your steps to reproduce the issue will be difficult without a database containing this issue. I have edited large structures many many many times without getting a conflict. A conflict is almost always a resizing of a field done externally to the structure it's in.

[ghidra1]

There are many, many conditions which can cause performance and conflict related issues. So much depends on how the structures are being created and if any concurrent operations are occuring. Can you be very explicit with how to reproduce the situation. When building-up very complicated and inter-related structures avoiding conflicts can be quite challenging and frequently impossible when polymorphism applies to a data structure.

[ghidra1]

Are you able to share a gzf which contains the conflicts? It would be helpful to know how the datatypes were created or modified to trigger the conflicts.

[ghidra1]

Have you checked to see if the conflicts are actually different? There is a script which can help identify the root cause of a conflict. You can run it by selecting two conflicting structures within the tree then run the script FindDatatypeConflictCauseScript

[hiimjustin000]

Hello! I will share the script output of one of the conflict structures, with the gzf as well.
https://drive.google.com/file/d/1GdpTnCba8V-l9AX9oyuZBq15chZSCY1J/view?usp=sharing

FindDataTypeConflictCauseScript.java> Running...
FindDataTypeConflictCauseScript.java> Composite attributes differ: Length differs (14008/14000):
FindDataTypeConflictCauseScript.java>  /ClassDataTypes/GJBaseGameLayer/GJBaseGameLayer.conflict4 (StructureDB)
FindDataTypeConflictCauseScript.java>     used by >> /ClassDataTypes/PlayerObject/PlayerObject_data.conflict2
FindDataTypeConflictCauseScript.java>  /ClassDataTypes/GJBaseGameLayer/GJBaseGameLayer (StructureDB)
FindDataTypeConflictCauseScript.java>     used by >> /ClassDataTypes/PlayerObject/PlayerObject_data
FindDataTypeConflictCauseScript.java> Composite attributes differ: Length differs (14008/14000):
FindDataTypeConflictCauseScript.java>  /ClassDataTypes/GJBaseGameLayer/GJBaseGameLayer.conflict4 (StructureDB)
FindDataTypeConflictCauseScript.java>     used by >> /ClassDataTypes/ShaderLayer/ShaderLayer_data.conflict
FindDataTypeConflictCauseScript.java>  /ClassDataTypes/GJBaseGameLayer/GJBaseGameLayer.conflict1 (StructureDB)
FindDataTypeConflictCauseScript.java>     used by >> /ClassDataTypes/ShaderLayer/ShaderLayer_data
FindDataTypeConflictCauseScript.java> Composite attributes differ: Length differs (14008/14000):
FindDataTypeConflictCauseScript.java>  /ClassDataTypes/GJBaseGameLayer/GJBaseGameLayer.conflict4 (StructureDB)
FindDataTypeConflictCauseScript.java>     used by >> /ClassDataTypes/PlayerObject/PlayerObject_data.conflict2
FindDataTypeConflictCauseScript.java>  /ClassDataTypes/GJBaseGameLayer/GJBaseGameLayer.conflict7 (StructureDB)
FindDataTypeConflictCauseScript.java>     used by >> /ClassDataTypes/PlayerObject/PlayerObject_data.conflict4
FindDataTypeConflictCauseScript.java> Composite attributes differ: Length differs (14000/14008):
FindDataTypeConflictCauseScript.java>  /ClassDataTypes/GJBaseGameLayer/GJBaseGameLayer (StructureDB)
FindDataTypeConflictCauseScript.java>     used by >> /ClassDataTypes/PlayerObject/PlayerObject_data
FindDataTypeConflictCauseScript.java>  /ClassDataTypes/GJBaseGameLayer/GJBaseGameLayer.conflict (StructureDB)
FindDataTypeConflictCauseScript.java>     used by >> /ClassDataTypes/PlayerObject/PlayerObject_data.conflict
FindDataTypeConflictCauseScript.java> Composite attributes differ: Length differs (14000/14008):
FindDataTypeConflictCauseScript.java>  /ClassDataTypes/GJBaseGameLayer/GJBaseGameLayer.conflict1 (StructureDB)
FindDataTypeConflictCauseScript.java>     used by >> /ClassDataTypes/ShaderLayer/ShaderLayer_data
FindDataTypeConflictCauseScript.java>  /ClassDataTypes/GJBaseGameLayer/GJBaseGameLayer.conflict6 (StructureDB)
FindDataTypeConflictCauseScript.java>     used by >> /ClassDataTypes/ShaderLayer/ShaderLayer_data.conflict1
FindDataTypeConflictCauseScript.java> Finished!

[ghidra1]

They are clearly different which would justify a conflict. Which case do you have that you believe should not have been a conflict? How were the datatypes produced?

[hiimjustin000]

The issue is that I do all of these routine operations on smaller structures and they do not produce conflicts. Data types were produced by RecoverClassesFromRTTIScript.

[ghidra007]

@hiimjustin000 Thank you for the .gzf file. I used it to produce a fresh binary and was unable to reproduce your issue with just a run of the RTTI in 11.2.1 build. We would appreciate you listing the steps you took to create the conflicts. Perhaps starting with a freshly imported and analyzed program then a fresh run of the script on that newly analyzed program you can recreate the issue and let us know what caused the issue you are seeing. How did you end up with multiple <class_name>_data structures in your ClassDataTypes folders? The script only produces up to one per class.

[ghidra1]

@hiimjustin000 I just re-read your "Additional Context" which like you said may be the key and makes some sense. I will have to try that with a simpler case and see if I can reproduce. I think the issue may stem from the fact that the editor works with copies and any referenced composite that circle-back to the edited composite will appear as non-equivalent due to the changes. This may require a custom conflict handler to be used to force equivalence for those types that were copied into the editor based upon an ID-mapping.

[ghidra1]

@hiimjustin000 Unfortunately, I was unable to reproduce the conflict generation with a simple case with two structures with a circular reference via pointers:

struct A {
  struct B *  b;
 }

struct B {
  struct A * a;
}

Using the editor to add components to either A or B structure did not result in a conflict.

[ghidra1]

Using the gzf you supplied above, could you try doing an edit to cause a new conflict to occur than tell us what that edit was so that we can reproduce.