The only documentation available in a standard X distribution may be the lbxproxy(1) man page.
If you have access to the X source tree, then very interesting information on LBX is available there:
xc/doc/specs/Xext/lbx.mif
(Framemaker MIF)xc/doc/hardcopy/Xext/lbx.PS.Z
(Compressed Postscript)xc/doc/hardcopy/Xext/lbxTOC.html
(HTML)More detailed discussion of specific LBX algorithms is available here:
xc/doc/specs/Xext/lbxalg.mif
(Framemaker MIF)xc/doc/specs/Xext/lbxalg.PS.Z
(Compressed Postscript)If you don't have access to the X11 source, you can obtain these files from the X Consortium's FTP site.