README.md

ExUI

This is a simple, lightweight browser-based UI for running local inference using ExLlamaV2.

Overview of features

• Friendly, responsive and minimalistic UI
• Persistent sessions
• Multiple instruct formats
• Speculative decoding
• Supports EXL2, GPTQ and FP16 models

Screenshots

Running locally

First, clone this repository and install requirements:

git clone https://github.com/turboderp/exui
cd exui
pip install -r requirements.txt

Then run the web server with the included server.py:

python server.py

Running locally with pre-made scripts

On windows:

run.cmd

On Linux:

start.sh

Running on Google Colab

Colab note for Exui is avalible [here]https://colab.research.google.com/github/turboderp/exui/blob/master/colab.ipynb

Your browser should automatically open on the default IP/port. Config and sessions are stored in ~/exui by default.

Prebuilt wheels for ExLlamaV2 are available here. Installing the latest version of Flash Attention is recommended.

More to come

Stay tuned.