I would leave it alone, since you don't need to remove it for your new gas cap. I took a look at mine as best I could. Use a flashlight if necessary. It appears that the rubber piece behind the hole is an arrowhead design. That's why you haven't been able to pull it out. On top of that, it appears that the tip of the piece of rubber touches the metal cap on the far side, so it would be difficult to push it out.
If you just have to get it out, I think you're going to have to come up with a way to break the rubber piece apart, with the risk of damaging the paint. You could try to drill it out, use a ***** punch to break it up, slide a screwdriver with a small head and a long shaft in from the side and beat on it or think of some other way to physically destroy it.
As I said, I would leave it alone.
2007 Mustang GT Coupe Premium
High Performance White/Dark Charcoal Leather
Rear Spoiler Delete
Interior Sport Appearance Package
MGW EJECT Power Point Plug
The more you know, the less you don't know.