Skip to content

[FEATURE] Support passing image as input to LLM #496

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
badmonster0 opened this issue May 16, 2025 · 0 comments
Open

[FEATURE] Support passing image as input to LLM #496

badmonster0 opened this issue May 16, 2025 · 0 comments
Labels
enhancement New feature or request help wanted Extra attention is needed

Comments

@badmonster0
Copy link
Member

Currently our LLM interface and ExtractByLlm builtin function only supports text prompt. We want it to support image inputs too.

  • Extend LlmGenerateRequest to accept image bytes as input:
    pub struct LlmGenerateRequest<'a> {
  • Implement existing LLM clients to pass this to different APIs (usually as base64)
  • Extend the ExtractByLlm function to accept an optional image as input
  • Can be tested by the existing image_search_example example. It's using customized code to send image to LLM now. We should be able to replace it with the ExtractByLlm function.

❤️ Contributors, please refer to 📙Contributing Guide.
Unless the PR can be sent immediately (e.g. just a few lines of code), we recommend you to leave a comment on the issue like I'm working on it or Can I work on this issue? to avoid duplicating work. Our Discord server is always open and friendly.

@badmonster0 badmonster0 added enhancement New feature or request help wanted Extra attention is needed labels May 16, 2025
@badmonster0 badmonster0 moved this to 🌈 Triaging in 🐳 CocoIndex Roadmap May 16, 2025
@badmonster0 badmonster0 moved this from 🌈 Triaging to 🐬 Prioritized in 🐳 CocoIndex Roadmap May 16, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request help wanted Extra attention is needed
Projects
Status: 🐬 Prioritized
Development

No branches or pull requests

1 participant