You shouldn`t be needing to map the shell widget yourself. Managing the top-level XmForm child of the XmDialog window will result, within Xm, to a call to XtPopup() for the shell. You`re basically mapping the window without having Xt aware of it, so any properties or other information on the window or its enclosing parent window is lost to the window manager.

Generally, beware of using the Xlib calls (X*, rather than Xt*) to do window manipulation. You`re probably outsmarting some code along the way that`s supposed to keep things working.