It all depends on why they are anti-Mormon. Some of those people either were once part of the Mormon faith, but later fell from their faith or were excommunicated from the Church. Unfortunately, they feel shunned from family and friends after leaving the LDS Church. Another group of people are Evangelical Christians (often are Biblical literalist) feel that Mormonism is a cult that they must get rid of by any means. 
 
However, from my experiences in Utah, Mormons are very accepting of other faiths in the state (feel they have just as much right to be there as they do) and the general US population holds little grudges to people of a different faith. Be nice to the people and they will be nice back. Many missionaries are understanding as well if you tell them that you are comfortable with your current beliefs and are not interested in investigation the Mormon faith. 
 
Don't feel that you need to hang around with anti-Mormons just because they share your religious beliefs (no one wants to hang around a negative person). Tell them thanks for the information, but you are not interested in hearing about Mormon beliefs.