[Feature request] Add simple HTTP API server like in llama.cpp with api like OpenAI

[pythops]

For more infos here
https://github.com/ggerganov/llama.cpp/blob/master/examples/server/README.md

[austinvhuang]

Great suggestion, if there's others who interested please +emoji above and we'll prioritize this :)

[pythops]

Just for the update: llama.cpp added support for gemma models
ggml-org/llama.cpp#5631

[loretoparisi]

Just for the update: llama.cpp added support for gemma models

ggml-org/llama.cpp#5631

Also with 💎Gemma in 🦙Llama.CPP you get CUDA, Neon and AMD GPUs support!
And - in theory - running into the browser if you can compile to WASM.

[omkar806]

adding a api like support would be great these models can be used on cpu for smaller tasks.
+1 for this.

[zeerd]

I have a question: why using http but not websocket?

As I known, the answer token is generated one word by one word.
And, seems, http has no function to do multi-responses for one call.
Which means , http need to gather the whole answer before trans it back.

[ufownl]

I have a question: why using http but not websocket?

As I known, the answer token is generated one word by one word. And, seems, http has no function to do multi-responses for one call. Which means , http need to gather the whole answer before trans it back.

WebSocket is more suitable for instant messenger style UI but may not be ideal for other UI types. And I think it is better to integrate gemma.cpp as a module into the web backend framework than to implement the HTTP/WebSocket API directly.

Here is my WebSocket online demo solution, and you can try it here or via this Kaggle notebook. In this solution gemma.cpp is a module of OpenResty which makes it easy to implement WebSocket or HTTP API.

[Gopi-Uppari]

Could you please confirm if this issue is resolved for you with the above comment ? Please feel free to close the issue if it is resolved ?

Thank you.

[leszko7]

ok app

[Zeenat30]

Yes it is

…

On Wed, 16 Oct 2024, 19:14 Leszek, ***@***.***> wrote: ok app — Reply to this email directly, view it on GitHub <#1 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFCLH3EJX5QCVDE3KMMDK53Z32NFVAVCNFSM6AAAAABQATJCOOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMJXGQZTSMRZG4> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>