indeed, there's nothing inherently racist about a white man pretending to be a black man, but the reason a black guy impersonating a white man isn't generally going to be seen as racist is that there's no cultural baggage attached to it.

dressing up in blackface and insisting it's not racist because nobody is being persecuted is akin, as simonj said, to tossing around the word "nigger" and insisting you're not meaning to insult anybody so you should be allowed to use it whenever you want. there's no equivalent for black-on-white racism, because there's no cultural baggage around black-on-white racism.