Running offline LLM (ChatBot) on uConsole
I don't use any "AI" chatbots, because I have concerns with privacy, IP's, biases, when it comes to services as ChatGPT which are hosted on distant server and every query goes and is stored (and probably used) there. Just recently I start looking for a way to use the advantages of LLM's in terms acceptable to me, and found all that offline engines and the impressive models openly available. After experimenting for several days on my PC, I decided that this would be a perfect use case for my uConsole, which honestly gets little use since I got it. Yes, it runs and performs quite well actually, I finally see my uConsole as a really useful "Pocket Assistant" with impressive amount of information locally on the machine.
I was simply following the instructions from here:
https://www.maketecheasier.com/run-llm-on-raspberry-pi/
Even the model I took is the recommended one, and despite being only 3.5B it's actually quite capable. It's not uncensored per se, but I was able to get the answers I wanted just by asking the right questions.