Rotation matrix solution is not unique. You can use SVD(singular value
decomposition) to find an optimal one. Take correlation matrix R=A*transpost
(B), then do SVD on R: R=U*S*transpose(V) to get orthogonal matrices U and V
. The rotation matrix which minimizes the pairwise distance between A and B
is given by V*transpose(U). More details can be found in Golub's Matrix
Computation.
