For me this was the main reason which kept me from going vegan I have to admit. That I’d always have to justify myself for something you seriously shouldn’t have to justify for at all.
And yes, at first I got reactions like “this is so exaggerated”, “but what are you allowed to eat now?”, “isn’t that unhealthy?”, “Vegan? Oh come on..” etc. But I think if you really stand up for something you believe in it’s not a big deal. If people won’t leave you alone just say something like “Please let’s not talk about this anymore. I respect you and your choice, please also respect mine.”. It’s also helpful when you know what you’re talking about, for example the facts why veganism is a good thing and where you get all your nutrients from.
And don’t worry about being impolite when refusing food someone offers you. I know it can be difficult when someone for example made a cake and you don’t want to hurt that person’s feelings. But you can just let them know that it’s nothing personal and that it looks good and that you’re sure it also tastes great. Hopefully they will understand.
People also will get used to you being vegan after a while so it will get easier. My father who at first was so refusing now even tries out some special vegan food like vegan meat products and stuff like that and when I visit my parents they cook vegan. They might not fully understand it, they still think most animal products they buy come from “happy animals”, but they respect my choice. And you’d be surprised how many people even agree with you and want to learn more about veganism. I guess they’re just like me before going vegan and maybe you: Knowing veganism is a good thing but not being vegan yourself (yet).
You should also consider maybe to break with people that don’t respect what you’re doing at all and who constantly want to bring you down. They will just make you unhappy. And I’m not talking about little jokes. I guess jokes about vegans and also meateaters are just a normal thing which you shouldn’t be bothered about.