Nope you do not. All you need to do is get a breaker bar/large torque wrench down to the belt tensioner to pivot it counter clock-wise. There should be plenty of room for this both above and below the engine. I know for sure its easy to access from below. If you do need to end up taking it off its easy as hell so it shouldn't be much of a hassle. Why do you need to remove the belt? Is it worn?
If you are having a hard time visualizing this let me know I can post pictures. I'm pretty sure our pulley setups should be the same but they could be different. I have an 03.