Relationship problems?

what to do when you have history with a guy that you have two kids with and decided to give him another chance..... but family members keep on whispering and telling u that hes no good, and they just dont approve of him..... and although you are saying stop talking to me like that..... they just dont stop. basically trying to change your mind..... and its driving me crazy..... y cant they just lemme alone! its my life im 24 geez and i have two kids w him ............... ugh!