[REQUEST] It is very difficult to service exlamav2 using RestfullAPI.

[nalgae]

Problem

Hello.

I'm using the open source you revealed well.

Restfull API is essential when using exllamav2 for actual services.

I would appreciate it if you could provide an example source code that can be processed simultaneously using Restfull API (FastAPI).

It is very difficult to service exlamav2 using API.

Below is the code I'm using right now, I don't know if I'm using it correctly.

Thank you.

[christopherthompson81]

You're looking for tabbyAPI

[nalgae]

I want a level where I can do actual commercial service using exlamav2.