You're "evacuating yourself" all over a stance that allows you your rights then, Rabbi.
At current, your Hypothetical Neo-Nazi Neighbor's beliefs are looked at by our society with contempt: while he has a right to his beliefs, and the right to express his opinion, most in society would look on those beliefs with contempt. On the other hand, your "anti-Nazi" beliefs are considered acceptable, even commendable.
But what happens if/when society shifts, and the acceptability of those beliefs switch?
Right now, you're (basically) advocating the violation of one's First Amendment rights because the viewpoint espoused is unpopular. But, if we do that to THIS set of beliefs, are you going to be so supporting of such violations when the shoe's on the other foot? It IS within the realm of possibility that antisemitism could become "socially acceptable", at which point the roles reverse.
For that matter, what about "pro-gun" views? Conservative views in general? There's a fairly large portion of society right now that sees such as "dangerous", and "not worth protecting". Where, exactly, do you draw the line there?
Going back to your Hypothetical Neo-Nazi, I think I can illustrate the point (and how I look at it). If all he's doing is saying "The Nazis had the best form of government.", or even just "The Nazis made the most scientific advancements.", then we pretty much have to let him go ahead and believe/say what he wants (of course, we are well within our rights to point and laugh). If, however, he started with things like "The Nazis should have been allowed to finish the job.", or "The Jews are subhuman, and should be locked away from society.", then something needs to be done to kinda keep him quiet.
