As a potential best man, it's not your job to judge the groom, his choices, his marriage, his wedding, his shoes or anything else. It's your job to assess the talent of the bridesmaids and start thinking about trying to fuck them.
I think the only reason to ever say no to being best man is if you have some aversion to fun, strippers, feeling special, feeling strippers, fun feelings, stripper fun, or free cake.
Seriously, being the best man is the greatest thing ever. You get to make a speech, and you're kind of expected to be a little drunk and say inappropriate shit. If the maid of honor is single, no matter what you look like or act like, she will fuck you. And fucking someone wearing purple taffeta on a washing machine is pretty badass, even were you not in attendance at a wedding reception.
I guess if you feel you have to bow out, because you've decided that being someone's best man is somehow an endorsement of their life choices, you could try saying something like, "I'm not a very good friend." That'll get the whole thing across in a hurry.
Short Answer: There's no graceful way to tell someone who thinks you're their best friend that you're not. Maybe it would suit you to try something passive like, "I'm going to be sick that day."