Big speedup for gradient calculations

Also not clear if this is the best way to include it.  Probably need the std::variant approach here too.

Python compilation working again