Skip to content

Commit 14e6bae

Browse files
oobaboogaoobabooga
authored
Merge pull request #6838 from oobabooga/dev
Merge dev branch
2 parents bb1905e + 9025848 commit 14e6bae

24 files changed

+514
-205
lines changed

‎README.md

Lines changed: 9 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@ Its goal is to become the [AUTOMATIC1111/stable-diffusion-webui](https://github.
1212

1313
## Features
1414

15-
- Supports multiple text generation backends in one UI/API, including [Transformers](https://github.com/huggingface/transformers), [llama.cpp](https://github.com/ggerganov/llama.cpp), and [ExLlamaV2](https://github.com/turboderp-org/exllamav2). [TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM) is supported via its own [Dockerfile](https://github.com/oobabooga/text-generation-webui/blob/main/docker/TensorRT-LLM/Dockerfile), and the Transformers loader is compatible with libraries like [AutoGPTQ](https://github.com/PanQiWei/AutoGPTQ), [AutoAWQ](https://github.com/casper-hansen/AutoAWQ), [HQQ](https://github.com/mobiusml/hqq), and [AQLM](https://github.com/Vahe1994/AQLM), but they must be installed manually.
15+
- Supports multiple text generation backends in one UI/API, including [Transformers](https://github.com/huggingface/transformers), [llama.cpp](https://github.com/ggerganov/llama.cpp), [ExLlamaV3](https://github.com/turboderp-org/exllamav3), and [ExLlamaV2](https://github.com/turboderp-org/exllamav2). [TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM) is supported via its own [Dockerfile](https://github.com/oobabooga/text-generation-webui/blob/main/docker/TensorRT-LLM/Dockerfile), and the Transformers loader is compatible with libraries like [AutoGPTQ](https://github.com/PanQiWei/AutoGPTQ), [AutoAWQ](https://github.com/casper-hansen/AutoAWQ), [HQQ](https://github.com/mobiusml/hqq), and [AQLM](https://github.com/Vahe1994/AQLM), but they must be installed manually.
1616
- OpenAI-compatible API with Chat and Completions endpoints – see [examples](https://github.com/oobabooga/text-generation-webui/wiki/12-%E2%80%90-OpenAI-API#examples).
1717
- Automatic prompt formatting using Jinja2 templates.
1818
- Three chat modes: `instruct`, `chat-instruct`, and `chat`, with automatic prompt templates in `chat-instruct`.
@@ -78,25 +78,19 @@ conda activate textgen
7878

7979
| System | GPU | Command |
8080
|--------|---------|---------|
81-
| Linux/WSL | NVIDIA | `pip3 install torch==2.4.1 torchvision==0.19.1 torchaudio==2.4.1 --index-url https://download.pytorch.org/whl/cu121` |
82-
| Linux/WSL | CPU only | `pip3 install torch==2.4.1 torchvision==0.19.1 torchaudio==2.4.1 --index-url https://download.pytorch.org/whl/cpu` |
83-
| Linux | AMD | `pip3 install torch==2.4.1 torchvision==0.19.1 torchaudio==2.4.1 --index-url https://download.pytorch.org/whl/rocm6.1` |
84-
| MacOS + MPS | Any | `pip3 install torch==2.4.1 torchvision==0.19.1 torchaudio==2.4.1` |
85-
| Windows | NVIDIA | `pip3 install torch==2.4.1 torchvision==0.19.1 torchaudio==2.4.1 --index-url https://download.pytorch.org/whl/cu121` |
86-
| Windows | CPU only | `pip3 install torch==2.4.1 torchvision==0.19.1 torchaudio==2.4.1` |
81+
| Linux/WSL | NVIDIA | `pip3 install torch==2.6.0 torchvision==0.21.0 torchaudio==2.6.0 --index-url https://download.pytorch.org/whl/cu124` |
82+
| Linux/WSL | CPU only | `pip3 install torch==2.6.0 torchvision==0.21.0 torchaudio==2.6.0 --index-url https://download.pytorch.org/whl/cpu` |
83+
| Linux | AMD | `pip3 install torch==2.6.0 torchvision==0.21.0 torchaudio==2.6.0 --index-url https://download.pytorch.org/whl/rocm6.1` |
84+
| MacOS + MPS | Any | `pip3 install torch==2.6.0 torchvision==0.21.0 torchaudio==2.6.0` |
85+
| Windows | NVIDIA | `pip3 install torch==2.6.0 torchvision==0.21.0 torchaudio==2.6.0 --index-url https://download.pytorch.org/whl/cu124` |
86+
| Windows | CPU only | `pip3 install torch==2.6.0 torchvision==0.21.0 torchaudio==2.6.0` |
8787

8888
The up-to-date commands can be found here: https://pytorch.org/get-started/locally/.
8989

90-
For NVIDIA, you also need to install the CUDA runtime libraries:
90+
If you need `nvcc` to compile some library manually, you will additionally need to install this:
9191

9292
```
93-
conda install -y -c "nvidia/label/cuda-12.1.1" cuda-runtime
94-
```
95-
96-
If you need `nvcc` to compile some library manually, replace the command above with
97-
98-
```
99-
conda install -y -c "nvidia/label/cuda-12.1.1" cuda
93+
conda install -y -c "nvidia/label/cuda-12.4.1" cuda
10094
```
10195

10296
#### 3. Install the web UI
@@ -143,19 +137,6 @@ Then browse to
143137
3) Manually install AutoGPTQ: [Installation](https://github.com/PanQiWei/AutoGPTQ#install-from-source).
144138
* Perform the from-source installation - there are no prebuilt ROCm packages for Windows.
145139

146-
##### Older NVIDIA GPUs
147-
148-
1) For Kepler GPUs and older, you will need to install CUDA 11.8 instead of 12:
149-
150-
```
151-
pip3 install torch==2.4.1 torchvision==0.19.1 torchaudio==2.4.1 --index-url https://download.pytorch.org/whl/cu118
152-
conda install -y -c "nvidia/label/cuda-11.8.0" cuda-runtime
153-
```
154-
155-
2) bitsandbytes >= 0.39 may not work. In that case, to use `--load-in-8bit`, you may have to downgrade like this:
156-
* Linux: `pip install bitsandbytes==0.38.1`
157-
* Windows: `pip install https://github.com/jllllll/bitsandbytes-windows-webui/raw/main/bitsandbytes-0.38.1-py3-none-any.whl`
158-
159140
##### Manual install
160141

161142
The `requirements*.txt` above contain various wheels precompiled through GitHub Actions. If you wish to compile things manually, or if you need to because no suitable wheels are available for your hardware, you can use `requirements_nowheels.txt` and then install your desired loaders manually.

‎css/chat_style-Dark.css

Lines changed: 128 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,128 @@
1+
.message {
2+
display: grid;
3+
grid-template-columns: 60px minmax(0, 1fr);
4+
padding-bottom: 28px;
5+
font-size: 18px;
6+
font-family: Roboto, Arial, sans-serif; /* Modern font */
7+
line-height: 1.5;
8+
}
9+
10+
.circle-you,
11+
.circle-bot {
12+
background-color: #2b2b2b; /* Darker background for circles */
13+
border-radius: 50%; /* Perfect circle */
14+
border: 1px solid #4a90e2; /* Soft blue border */
15+
box-shadow: 0 4px 8px rgb(0 0 0 / 50%); /* Soft shadow for depth */
16+
}
17+
18+
.circle-bot img,
19+
.circle-you img {
20+
border-radius: 50%; /* Make images circular */
21+
width: 100%;
22+
height: 100%;
23+
object-fit: cover;
24+
}
25+
26+
.circle-you, .circle-bot {
27+
width: 64px; /* Smaller size for modern look */
28+
height: 64px;
29+
}
30+
31+
.text {
32+
padding-left: 12px; /* Reduced padding for a cleaner layout */
33+
color: #f0f0f0; /* Light text color for readability */
34+
}
35+
36+
.text p {
37+
margin-top: 2px;
38+
}
39+
40+
.username {
41+
padding-left: 10px;
42+
font-size: 20px;
43+
font-weight: bold;
44+
color: #e0e0e0; /* Light gray text */
45+
transition: color 0.3s ease; /* Smooth color transition */
46+
}
47+
48+
.username:hover {
49+
color: #4a90e2; /* Blue color on hover */
50+
}
51+
52+
.message-body {
53+
position: relative;
54+
border: 1px solid rgb(255 255 255 / 10%); /* Soft white border */
55+
border-radius: 8px; /* Slightly rounded corners */
56+
padding: 15px;
57+
background: #1e1e1e; /* Dark background */
58+
box-shadow: 0 4px 10px rgb(0 0 0 / 30%); /* Subtle shadow for depth */
59+
transition: background 0.3s ease; /* Smooth transition for background */
60+
}
61+
62+
.message-body:hover {
63+
background: #252525; /* Slightly lighter on hover */
64+
}
65+
66+
/* Adds 2 extra lines at the top and bottom of the message */
67+
.message-body::before,
68+
.message-body::after {
69+
content: "";
70+
position: absolute;
71+
left: 10px;
72+
right: 10px;
73+
height: 1px;
74+
background-color: rgb(255 255 255 / 5%); /* Faded lines for subtle separation */
75+
}
76+
77+
.message-body::before {
78+
top: 4px;
79+
}
80+
81+
.message-body::after {
82+
bottom: 4px;
83+
}
84+
85+
.message-body img {
86+
max-width: 300px;
87+
max-height: 300px;
88+
border-radius: 10px; /* Rounded corners for images */
89+
}
90+
91+
.message-body p {
92+
margin-bottom: 0 !important;
93+
font-size: 16px !important;
94+
line-height: 1.5 !important;
95+
color: #e0e0e0 !important; /* Light color for text */
96+
}
97+
98+
.message-body p em {
99+
color: #a6a6a6 !important; /* Softer gray for emphasized text */
100+
}
101+
102+
@media screen and (width <= 688px) {
103+
.message {
104+
display: grid;
105+
grid-template-columns: 60px minmax(0, 1fr);
106+
padding-bottom: 25px;
107+
font-size: 15px;
108+
font-family: Roboto, Arial, sans-serif; /* Modern font */
109+
line-height: 1.5;
110+
}
111+
112+
.circle-you, .circle-bot {
113+
width: 40px; /* Smaller size for mobile */
114+
height: 40px;
115+
}
116+
117+
.text {
118+
padding-left: 10px; /* Reduced padding for mobile */
119+
}
120+
121+
.message-body p {
122+
font-size: 14px !important; /* Smaller text for mobile */
123+
}
124+
125+
.username {
126+
font-size: 18px; /* Smaller username for mobile */
127+
}
128+
}

‎docker/amd/Dockerfile

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@ RUN --mount=type=cache,target=/var/cache/apt,sharing=locked,rw \
1313
WORKDIR /home/app/
1414
RUN git clone https://github.com/oobabooga/text-generation-webui.git
1515
WORKDIR /home/app/text-generation-webui
16-
RUN GPU_CHOICE=C LAUNCH_AFTER_INSTALL=FALSE INSTALL_EXTENSIONS=TRUE ./start_linux.sh --verbose
16+
RUN GPU_CHOICE=B LAUNCH_AFTER_INSTALL=FALSE INSTALL_EXTENSIONS=TRUE ./start_linux.sh --verbose
1717
COPY CMD_FLAGS.txt /home/app/text-generation-webui/
1818
EXPOSE ${CONTAINER_PORT:-7860} ${CONTAINER_API_PORT:-5000} ${CONTAINER_API_STREAM_PORT:-5005}
1919
WORKDIR /home/app/text-generation-webui

‎docker/intel/Dockerfile

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@ RUN --mount=type=cache,target=/var/cache/apt,sharing=locked,rw \
1313
WORKDIR /home/app/
1414
RUN git clone https://github.com/oobabooga/text-generation-webui.git
1515
WORKDIR /home/app/text-generation-webui
16-
RUN GPU_CHOICE=E LAUNCH_AFTER_INSTALL=FALSE INSTALL_EXTENSIONS=TRUE ./start_linux.sh --verbose
16+
RUN GPU_CHOICE=D LAUNCH_AFTER_INSTALL=FALSE INSTALL_EXTENSIONS=TRUE ./start_linux.sh --verbose
1717
COPY CMD_FLAGS.txt /home/app/text-generation-webui/
1818
EXPOSE ${CONTAINER_PORT:-7860} ${CONTAINER_API_PORT:-5000} ${CONTAINER_API_STREAM_PORT:-5005}
1919
# set umask to ensure group read / write at runtime

‎extensions/ngrok/README.md

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -9,9 +9,9 @@ the `settings.json` file, see the Examples below. Retrieve your authtoken on the
99

1010
# Documentation
1111

12-
For a list of all available options, see [the configuration documentation](https://ngrok.com/docs/ngrok-agent/config/) or [the connect example](https://github.com/ngrok/ngrok-py/blob/main/examples/ngrok-connect-full.py).
12+
For a list of all available options, see [the configuration documentation](https://ngrok.com/docs/ngrok-agent/config/) or [the forward example](https://github.com/ngrok/ngrok-python/blob/main/examples/ngrok-forward-full.py).
1313

14-
The ngrok Python SDK is [on github here](https://github.com/ngrok/ngrok-py). A quickstart guide and a full API reference are included in the [ngrok-py Python API documentation](https://ngrok.github.io/ngrok-py/).
14+
The ngrok Python SDK is [on github here](https://github.com/ngrok/ngrok-py). A quickstart guide and a full API reference are included in the [ngrok-py Python API documentation](https://ngrok.github.io/ngrok-python/).
1515

1616
# Running
1717

@@ -66,4 +66,4 @@ To add an authtoken instead of using the NGROK_AUTHTOKEN environment variable:
6666
"authtoken_from_env":false
6767
}
6868
}
69-
```
69+
```

0 commit comments

Comments
 (0)