The Fisher information matrix is simply the Hessian of KL divergence at the point where two distributions are equal.

Differential geometry is all about constructing things which are independent of the representation