AI: Simple offline nsfw AI chatbot guide (Patreon)

Google is doing a crackdown on chatbots, particularily nsfw ones, running on its platform Colab. That was how most people were playing with nsfw AI chatbots for performance reasons. The future for nsfw chatbots seems to be offline, so here's a quick & simple guide for people that are new to it and want to try it.

The go to UI for text AI is KoboldAI and I talked about that before but for performance & simplicity I recommend OobaBooga.

Using this approach I got to run the Pygmalion-6B model offline on a pretty old gpu and it gives responses in 5-10s which is fantastic for such a big model. I imagine the response time is almost instant on great GPUs.

1. Install
Follow the install section https://github.com/oobabooga/text-generation-webui

2. Download AI model
Once you follow the install steps there, you can run download-model.bat , hit L None of the above, and paste mayaeary/pygmalion-6b_dev-4bit-128g  and hit enter.

This will download the Pygmalion-6B model which is very smart. Until recently you had to have a fantastic GPU to be able to run it offline. But this version https://huggingface.co/mayaeary/pygmalion-6b_dev-4bit-128g (the 4bit version) is a minimized/optimized version of the model, allowing it to run on less performant GPUs with little impact on text quality. You still need a nVidia card though, with AMD cards I think it's gonna be extremely slow.

You can find out more about Pygmalion on the subreddit https://www.reddit.com/r/PygmalionAI/

3. Add parameters
Right-click and Edit the start-webui.bat file and add to the call python line these parameters:
call python server.py  --wbits 4 --groupsize 128

This will allow OobaBoga to run 4bit models like that model. Kobold AI is prettier, but doesn't work with these kind of models yet and the responses would be considerably slower.

Myself I removed --auto-devices from that same line , it seemed to be much slower with it and caused problems sometimes but might be just for my GPU. That parameter makes it break the AI model on both CPU & GPU automatically (which is bad for performance but necessary sometimes to run huge models).

4. Run
If the install went ok, when you run start-webui.bat it loads for a few seconds and at the end it will show: Running on local URL:  http://127.0.0.1:7860

Then if you go in a browser to http://127.0.0.1:7860 you will see the OogaBooga UI.


5. Chatting
The simplest way to use it is to go to Interface mode tab and select chat as the Mode, hit Apply. Now when you go back to the Text Generation tab, it will look more like this:

There's also a Character tabs where you can customize the character, change the backstory and how it will behave. You can also save presets in \text-generation-webui\characters, similar to the Example files there.

Depending on how good your GPU is, after a while the AI will stop answering to you. It's because the conversation got so big that the GPU is running out of memory, you'll see errors in the cmd window too. At that point you have to either clear the conversation, or remove some of the replies to make space for new ones.