First of all, Whedon is undoubtedly incredibly gifted. I remember when I was new to Buffy fandom, I read how many of the writers he worked with would have the experience of having written an episode, and Whedon having written ONE SCENE in that episode, and fans inevitably and unwittingly pointing to the scene Whedon wrote as their favorite.
I tried it out for myself. What was an episode that had one scene I particularly liked? I remembered the Buffy/Spike scene in Hell's Bells.
Guys, guess who wrote that scene?
Of course talent does not excuse bad behavior - and talent and bad behavior can coexist in the same person.
I will not be leaving Buffy fandom or any other Whedon fandoms. I might think twice before picking up a new Whedon fandom, but I'm not throwing out what I already have. As many others have pointed out, while Whedon was gifted, he hardly worked alone. Throwing out Whedon fandoms would mean throwing out not only his work but the work of those who are criticizing him, and many, many others.
However even if it was Whedon alone, I wouldn't be walking away from those fandoms. I still enjoy Alfred Hitchcock movies, after all. There's a man who, when quoted as saying "actors are cattle", publicly clarified it by saying what he really meant was "actors should be treated like cattle".
(I used to find that quote hilarious, but with all the discussion that's come up around Whedon's actions and mistreatment of actors in general, I'm less amused)
It's easier to swallow Hitchcock because he died such a long time ago. But frankly, it's going on twenty years since Buffy and Angel were on the air. It's over a decade since I watched them for the first time, and I watched them late. I can't take back that decade of fannishness - and I don't think I would if I could. I can be realistic about the kind of person Whedon turned out to be without letting go of that fannishness.