Should I sell my automatic for a manual?

I currently have a AT 2014 scion fr-s which was my first car. I have only added a cat-back exhaust, replaced the front and rear lights, and a few other things. I have headers on the way and have many plans for the car; however I feel like I will regret doing all this work on an automatic. Would love your opinions or advice. Thanks