How is Secular/Scientific Buddhism a Problem?

Just to preface, All I want is to be rid of the suffering of anxiety and the perception of dogma is distressing to me and sort of pushes me away from the practice. I know Secular/Scientific Buddhism gets a lot of criticism here, but as a Westerner, I do have trouble accepting seemingly unverifiable metaphysical claims such as literal “life-to-life” rebirth or other literal realms of existence, in which other-worldly beings dwell, for which there is insufficient evidence. My response to these claims is to remain agnostic until I have sufficient empirical evidence, not anecdotal claims. Is there sufficient evidence for rebirth or the heavenly or hellish realms to warrant belief? If it requires accepting what the Buddha said on faith, I don’t accept it.

I do, however, accept the scientifically verified physical and mental health benefits of meditation and mindfulness practice. I’ve seen claims on this subreddit that Secular/Scientific Buddhism is “racist” and I don’t see how. How is looking at the Buddhist teachings in their historical context and either accepting them, suspending judgement, or rejecting them due to lack of scientific evidence “racist”?