Pissing people off by stimming

I do a little noise stim, mostly without even realizing it (I puff my cheek up a little and push the air out and make a raspberry sound) and for some reason it makes my friend and brother go absolutely ballistic for no reason, they start crying and moaning and literally screaming at me about how its so god awfully annoying and its oh so terrible (but I dont say shit when they do annoying stuff?)

Am I the asshole for not really caring or should I try to stop