you have to use a polish and a random orbital to remove swirl marks. It will be next to impossible to polish out swirl marks by hand.
You can hide them by using wax filled with fillers, but they will re-appear and this isn't a fix.
and yes brushes at those places will scratch your car
__________________
|