I want to improve my Chinese but...

I have been living in Taipei for quite some times now, one of my main objective was to be able to speak Chinese at a conversational level, however it seems I have a felt a wall in my progress, I started off taking classes and trying to find community of Chinese speakers in my community but it seems that it's quite hard for me to even fit in, I told them that I can understand and engage in conversation albeit in a mix english-chinese but it seems that I couldn't keep up since they would either talk with their own peers or talk in a way as if I'm fluent, and when they finally realize that I'm a foreigner with a limited vocab they just start speaking English even if I asked to just speak in Chinese and it hampers how I can progress, is there anyone who felt like this, or maybe if you are Chinese speaker, I wished to be able to naturally talk and immersed myself in Chinese dialogue, we could do stuff or talk about something, I hope people understand my struggles and could give inputs