Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

LLM integration proof-of-concept #2549

Draft
wants to merge 1 commit into
base: master
Choose a base branch
from
Draft

LLM integration proof-of-concept #2549

wants to merge 1 commit into from

Conversation

PalmerAL
Copy link
Collaborator

This PR sets up the basic infrastructure to run an LLM inside Min using node-llama-cpp inside a utility process. Any llama.cpp-formatted model file should work; the model can be configured by updating modelPath inside llmService.mjs. My testing so far has been with either this model or this one.

My original intent with this was to see if it was possible to generate high-quality page summaries to display in the searchbar. Unfortunately, with llama-3.2-1b, the quality of the summaries seems quite poor. llama-3.2-3b does much better, but keeping the model loaded requires around 5GB of memory. I think this means that any use case that requires the model to continually be loaded in the background is infeasible, but it might work in a situation where the user explicitly requests to use it, which would allow us to load the model for a brief period of time and then immediately unload it. I'm planning to experiment with language translation (replacing the current cloud-based version) and with an explicit "summarize page" command, but if anyone has additional ideas for where this could be useful, I'd be happy to test them.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant