Slow compilation time of Reference Model in Parent model when reach "Updating sorted order display"

I have a model let's call it Model A and it's included as a model reference in a parent model. When I update diagram in Model A it takes ~30secs to compile. However when I update diagram in the parent model it is extremely slow and takes ~6-8mins just to compile model A only.

The compiling progresses well, till it gets to the process "Compiling: Updating sorted order display: Started" and this step alone takes up to 90% of the compiling time. My questions are what exactly occurs in this step? Is it necessary for compiling because display seems to imply that's it's a visual thing for the user. And is there a way that I can improve my compiling time in the parent model?