As a professional automotive detailer and paint correction specialist I highly recommend water and vinegar to remove that stuff. You can also steam clean the leather. Don't use armor all on the interior instead use Chemical guys or Adams Polish on your interior. If you have any other questions feel free to ask me!

 

Ok white vinegar is taking it off. Amazing stuff.

 

UPDATE

not 100% satisfied with the vinegar so I bought a steam cleaner.

As for my driver door I think I got it enough and used Optimum to get the dark gloss back into the door.