Gradio Chatbot App for Open OnDemand (OOD)
Wake Forest University (WFU) HPC Team
Primary Contact: Sean Anderson (anderss@wfu.edu)
chatbot.py script created by Cody Stevens (stevca9@wfu.edu).
This is a quick way to get a simple LLM chatbot app in OOD. This is known to
work with the Meta Llama-3.1-8B-Instruct model and using Gradio.app as a
frontend interface for interacting with it.
Obtaining an LLM
Make a directory for your LLM:
mkdir -p ${HOME}/llm
For this example, we'll use a pre-processed model from HuggingFace that can be
downloaded directly in the GGUF format:
wget https://huggingface.co/professorf/Meta-Llama-3-1-8B-Instruct-f16-gguf/resolve/main/llama-3-1-8b-instruct-f16.gguf -o ${HOME}/llm/llama-3-1-8b-instruct-f16.gguf
This model is around 16 GB in size.
Installing the Python environment
Now create a fresh Python environment using any version of Python 3 that you
want. Activate the environment after creation:
python3 -m venv ${HOME}/env-chatbot # create environment
. ${HOME}/env-chatbot/bin/activate # activate environment
Lastly, install the required modules into the environment:
python3 -m pip install llama_cpp_python gradio openai matplotlib