3D rmsd of substructure vs larger molecule

Ah, of course - the rotations mean you do need RMSD. That should be possible to calculate from the two molecules. I’m at the RDKit User Group for the next few days - I’ll have a think how to do it.

with a bit of google-ing I found that there is an RDkit function called
rdkit.Chem.rdMolAlign.GetAlignmentTransform
that also returns a transformation matrix, if I understood correctly.

You do understand correctly - that is how it currently calculates the RMSD (buried in the Java Snippet), but that is for the optimally aligned pair rather than the pair in their current positions.

Steve