Oh God, is this what having friends is like? My head hurts already. If you don't like J then don't talk to him. I don't see why you have to cut off ties with J's friends then, but if J twists their opinion of you around then I guess you'll just have to burn bridges. If J and his friends are so against everything that you stand for, I wouldn't think it's worth the time.
It's very hard to just simply "not talk to [insert person here]", that's for sure. Even when they are against what you stand for, it sure as heck isn't easy to do it without destroying ties with others. And doing that is difficult enough in its own right.
To be honest, doing whichever thing you want to do to deal with J will get talked about behind your back. Ditching J will lead to it, as will sticking with J - which you probably know. Otherwise...it's an awkward scenario :/
Despite what I said in response to Hahex...you mention that you're "10%" of a group that's stopped talking to J? You could - in theory - follow the 90%. And those people who are Js friends that believe him over you...are a waste of time, honestly.
(sorry if I'm not much help)