Why do Ghanaians stigmatize being well built (Macho man)

I am a corporate guy, a Christian and happily married with kids. I workout to keep in shape and to build muscle mass for health and physical vital. I do this not because I have the time but I consider it a vital aspect of health. But I often get attacked in the office by people who associate me with bad stuff like womanizing, drinking alcohol and not focused on work until they see your credentials and output. At church I get similar comments associating me with being carnally minded. People even ask who I am trying to impress for working out. This is getting too much, can it affect my career progress? should I consider people’s perception or just ignore them?