First of all, I do realize that I can get discs that will mount properly in XBMC if I use ISO9660 + Joliet format; the thing is, I don't want to use that format, because it's old and I occasionally need more than 64 characters for things. I'm being a standards prick, I know, but I prefer to be able to burn all my DVDs in one format and know that they will work in any device I drop them in - set-top, Mac, PC, softmodded XBox, whatever.
Anyways, my results.
I created two ISO images of the same single file.
Image 1 was created with Nero 6.6.1.8, using UDF default settings and the "Force DVD-Video compatibility mode" box checked.
Image 2 was created with ImgBurn, using UDF as the filesystem and all default settings.
ISObuster does not seem to see any differences between the two ISO images. I burned both ISO images with ImgBurn to eliminate the possibility of Nero's ASPI layer having anything to do with it. The discs were consecutive media from the same spindle. The disc created with Nero mounts and plays just fine, the disc created with ImgBurn does not. XBMC (latest T3CH release) throws a "Path not found" error, which is what XBMC does when it can't mount a filesystem. So I think it's something in the way Nero is constructing the filesystem that makes it work.
I'm PMed Lightning UK with the download URLs for the images for his perusal.
Thanks again everyone for all your interest and assistance. If the answer ends up being "it works with ISO9660 + Joliet, suck it up", well, I'll be disappointed but that's fair. I would just really like to ditch Nero entirely in favour of ImgBurn and this is the last sticking point.