Thoughts on changing GOGC

We run an api that peaks at about 500 rps, so not a super big system but not small. Anyway, we started noticing that the p95 and p99 response times were going up a bit as we were brining on more traffic.

And also noticed that GC pause times were also going up and so was frequency (as to be expected). So someone suggested to raise GOGC as well since we have a relatively so memory footprint and our API does have a lot of allocs. This did reduce p95 and p99 by a bit so we kept raising GOGC. GOGC is now at 800.

To me, this "just raise GOGC" is kinda a red flag; personally I like to know exactly what is happening and there is no such thing as free lunch. I knew we did some things pretty inefficiently, so I went and refactored quite a bit and reduced allocs by around 60%. It definitely helped things a bit but when comparing the old code with GOGC=800 vs new code with GOGC=100, the old code still performs better.

So all that is just context to ask: do you change GOGC often? When do you do it? How do you decide when to do it? Are there better ways to address this?