MultimodalQnA audio features completion #1698

mhbuehler · 2025-03-19T22:16:47Z

Description

This PR completes the third and final phase of the RFC for MultimodalQnA image and audio support. The changes in GenAIExamples are listed below. The accompanying PR in GenAIComps is opea-project/GenAIComps#1433.

New Features:

Added playable audio/TTS query responses from megaservice API
Added ability to upload, record, and send audio captions to dataprep API
Enabled retrieved audio files to be displayed in multimodal query results box
Combined text, image, and audio query types into a unified multimodal text box
Added ability to list and delete files in the vector store
Parameterized UI timeout

Bug Fixes:

Fixed PDF ingestion status
Fixed PDF clearing behavior

Issues

Type of change

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds new functionality)
Breaking change (fix or feature that would break existing design and interface)
Others (enhancement, documentation, validation, etc.)

Dependencies

Version upgrades:

gradio: 5.11.0 -> 5.22.0 or higher
gradio_pdf: 0.0.19 -> 0.0.20

Tests

Updated:

MultimodalQnA/tests/test_compose_on_gaudi.sh
MultimodalQnA/tests/test_compose_on_xeon.sh
MultimodalQnA/tests/test_compose_on_rocm.sh

Authors

Co-authored-by: Harsha Ramayanam harsha.ramayanam@intel.com
Co-authored-by: Melanie Buehler melanie.h.buehler@intel.com
Co-authored-by: Dina Suehiro Jones dina.s.jones@intel.com
Co-authored-by: Omar Khleif omar.khleif@intel.com

* Added tests + updated docs for asr mp3 change Signed-off-by: okhleif-IL <omar.khleif@intel.com> * addressed review comments Signed-off-by: okhleif-IL <omar.khleif@intel.com> --------- Signed-off-by: okhleif-IL <omar.khleif@intel.com>

* Added logic for showing/deleting files from vector store Signed-off-by: Harsha Ramayanam <harsha.ramayanam@intel.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Added message to show when vector store is empty Signed-off-by: Harsha Ramayanam <harsha.ramayanam@intel.com> * Update MultimodalQnA/ui/gradio/multimodalqna_ui_gradio.py Co-authored-by: Dina Suehiro Jones <dina.s.jones@intel.com> --------- Signed-off-by: Harsha Ramayanam <harsha.ramayanam@intel.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Dina Suehiro Jones <dina.s.jones@intel.com>

* Parameterize UI timeout and increase default Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com> * Add new variable to compose.yaml Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com> * Update READMEs Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com> --------- Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>

…phase3

Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>

…phase3

Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>

…ts (#58) * MultimodalQnA README and diagram updates for phase 3 enhancements Signed-off-by: dmsuehir <dina.s.jones@intel.com> * Wording Signed-off-by: dmsuehir <dina.s.jones@intel.com> * Update to remove your_* vars Signed-off-by: dmsuehir <dina.s.jones@intel.com> * Updates based on review comments Signed-off-by: dmsuehir <dina.s.jones@intel.com> --------- Signed-off-by: dmsuehir <dina.s.jones@intel.com>

* added TTS linkage to backend Signed-off-by: okhleif-IL <omar.khleif@intel.com> * added modalities as a toggle Signed-off-by: okhleif-IL <omar.khleif@intel.com> * doc updates and code refactor Signed-off-by: okhleif-IL <omar.khleif@intel.com> * added tts test to megaservice tests Signed-off-by: okhleif-IL <omar.khleif@intel.com> * addressed recent review comments Signed-off-by: okhleif-IL <omar.khleif@intel.com> --------- Signed-off-by: okhleif-IL <omar.khleif@intel.com>

* Add test for image and audio data ingestion Signed-off-by: dmsuehir <dina.s.jones@intel.com> * README updates Signed-off-by: dmsuehir <dina.s.jones@intel.com> * Add Gaudi tests Signed-off-by: dmsuehir <dina.s.jones@intel.com> * Add note about matching base names in test Signed-off-by: dmsuehir <dina.s.jones@intel.com> --------- Signed-off-by: dmsuehir <dina.s.jones@intel.com>

* fixed test and added tts validation Signed-off-by: okhleif-IL <omar.khleif@intel.com> * added gaudi test, reverted -speech change Signed-off-by: okhleif-IL <omar.khleif@intel.com> --------- Signed-off-by: okhleif-IL <omar.khleif@intel.com>

…phase3

* Enable audio caption upload in the UI Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com> * Improve handling of unsupported audio formats Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com> * Improve label and exception Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com> * Replace exception with error message so audio component still works Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com> --------- Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>

…phase3

Signed-off-by: dmsuehir <dina.s.jones@intel.com>

…se (#64) Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>

* Add missing env vars for MMQnA UI data prep endpoints Signed-off-by: dmsuehir <dina.s.jones@intel.com> * Remove dockerfile branch Signed-off-by: dmsuehir <dina.s.jones@intel.com> --------- Signed-off-by: dmsuehir <dina.s.jones@intel.com>

Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>

* first commit for tts addition Signed-off-by: okhleif-IL <omar.khleif@intel.com> * added TTS linkage to backend Signed-off-by: okhleif-IL <omar.khleif@intel.com> * removed unused import Signed-off-by: okhleif-IL <omar.khleif@intel.com> * added necessary env vars Signed-off-by: okhleif-IL <omar.khleif@intel.com> * reworked temp tts toggle logic Signed-off-by: okhleif-IL <omar.khleif@intel.com> * added modalities as a toggle Signed-off-by: okhleif-IL <omar.khleif@intel.com> * removed print statement Signed-off-by: okhleif-IL <omar.khleif@intel.com> * removed gaudi from tts Signed-off-by: okhleif-IL <omar.khleif@intel.com> * doc updates and code refactor Signed-off-by: okhleif-IL <omar.khleif@intel.com> * added tts test to megaservice tests Signed-off-by: okhleif-IL <omar.khleif@intel.com> * remove log diles Signed-off-by: okhleif-IL <omar.khleif@intel.com> * addressed recent review comments Signed-off-by: okhleif-IL <omar.khleif@intel.com> * Added Logic for audio responses & refactored code to align with new gradio version Signed-off-by: Harsha Ramayanam <harsha.ramayanam@intel.com> * Minr bug fixes and UI changes Signed-off-by: Harsha Ramayanam <harsha.ramayanam@intel.com> * UI layout update & handling empty text with spaces Signed-off-by: Harsha Ramayanam <harsha.ramayanam@intel.com> * Updates on review comments Signed-off-by: Harsha Ramayanam <harsha.ramayanam@intel.com> * Update on review comments Signed-off-by: Harsha Ramayanam <harsha.ramayanam@intel.com> * Update MultimodalQnA/ui/gradio/multimodalqna_ui_gradio.py Co-authored-by: Dina Suehiro Jones <dina.s.jones@intel.com> * Some updates to review comments. More to come after testing Signed-off-by: Harsha Ramayanam <harsha.ramayanam@intel.com> * Restrict file media types to known/working formats Signed-off-by: Harsha Ramayanam <harsha.ramayanam@intel.com> * Remove extra whitespace Signed-off-by: Harsha Ramayanam <harsha.ramayanam@intel.com> * Fix test_compose_on_gaudi.sh script's diff not syncing with phase3 Signed-off-by: Harsha Ramayanam <harsha.ramayanam@intel.com> * Changes per review comments Signed-off-by: Harsha Ramayanam <harsha.ramayanam@intel.com> * Added single space to the pload Signed-off-by: Harsha Ramayanam <harsha.ramayanam@intel.com> * Added logic to flush chatbot assistant's voice reponse .wav Signed-off-by: Harsha Ramayanam <harsha.ramayanam@intel.com> * Fixed issue where assistant's image is not sent Signed-off-by: Harsha Ramayanam <harsha.ramayanam@intel.com> * Revert build yaml Signed-off-by: Harsha Ramayanam <harsha.ramayanam@intel.com> * Clear diff Signed-off-by: Harsha Ramayanam <harsha.ramayanam@intel.com> * changes per review Signed-off-by: Harsha Ramayanam <harsha.ramayanam@intel.com> * small change Signed-off-by: Harsha Ramayanam <harsha.ramayanam@intel.com> * Update Dockerfile Revert Dockerfile --------- Signed-off-by: okhleif-IL <omar.khleif@intel.com> Signed-off-by: Harsha Ramayanam <harsha.ramayanam@intel.com> Co-authored-by: okhleif-IL <omar.khleif@intel.com> Co-authored-by: Dina Suehiro Jones <dina.s.jones@intel.com>

github-actions · 2025-03-19T22:17:01Z

Dependency Review

✅ No vulnerabilities or license issues found.

Scanned Files

MultimodalQnA/ui/gradio/requirements.txt

for more information, see https://pre-commit.ci

Signed-off-by: Harsha Ramayanam <harsha.ramayanam@intel.com>

MultimodalQnA/Dockerfile

* Point git clones to corresponding fork of comps Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com> * Use --depth 1 Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com> --------- Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>

* Initial commit to add logic Signed-off-by: Harsha Ramayanam <harsha.ramayanam@intel.com> * changes per review comments from closed PR #66 Signed-off-by: Harsha Ramayanam <harsha.ramayanam@intel.com> * Modified screenshot for audio query Signed-off-by: Harsha Ramayanam <harsha.ramayanam@intel.com> * rename tmp_dir2 to audio_tmp_dir Signed-off-by: Harsha Ramayanam <harsha.ramayanam@intel.com> --------- Signed-off-by: Harsha Ramayanam <harsha.ramayanam@intel.com>

for more information, see https://pre-commit.ci

…phase3

MultimodalQnA/Dockerfile

MultimodalQnA/tests/test_compose_on_gaudi.sh

…phase3

* Specify MMQnA UI to use pytantic==2.10.6 Signed-off-by: Dina Suehiro Jones <dina.s.jones@intel.com> * Formatting Signed-off-by: Dina Suehiro Jones <dina.s.jones@intel.com> --------- Signed-off-by: Dina Suehiro Jones <dina.s.jones@intel.com>

MultimodalQnA/ui/gradio/requirements.txt

…phase3

* Revert git clone changes after tests pass Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com> * Updated Dockerfile with opea/comps-base Signed-off-by: Dina Suehiro Jones <dina.s.jones@intel.com> --------- Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com> Signed-off-by: Dina Suehiro Jones <dina.s.jones@intel.com> Co-authored-by: Dina Suehiro Jones <dina.s.jones@intel.com>

MultimodalQnA/README.md

MultimodalQnA/docker_compose/intel/hpu/gaudi/README.md

MultimodalQnA/tests/test_compose_on_gaudi.sh

MultimodalQnA/ui/gradio/requirements.txt

Signed-off-by: Harsha Ramayanam <harsha.ramayanam@intel.com>

Signed-off-by: Dina Suehiro Jones <dina.s.jones@intel.com>

ashahba

LGTM!
For the one file we are downloading from an archived repo, we can always come back to it and decide.

Signed-off-by: okhleif-IL <omar.khleif@intel.com> Signed-off-by: Harsha Ramayanam <harsha.ramayanam@intel.com> Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com> Signed-off-by: dmsuehir <dina.s.jones@intel.com> Signed-off-by: Dina Suehiro Jones <dina.s.jones@intel.com> Co-authored-by: Omar Khleif <omar.khleif@intel.com> Co-authored-by: Harsha Ramayanam <harsha.ramayanam@intel.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Dina Suehiro Jones <dina.s.jones@intel.com> Co-authored-by: Liang Lv <liang1.lv@intel.com> Co-authored-by: Abolfazl Shahbazi <12436063+ashahba@users.noreply.github.com> Signed-off-by: Chingis Yundunov <c.yundunov@datamonsters.com>

okhleif-IL and others added 22 commits February 5, 2025 10:24

Merge branch 'opea-project:main' into mmqna-phase3

4f99bd1

Merge branch 'main' of github.com:mhbuehler/GenAIExamples into mmqna-…

cd0a364

…phase3

Update retrieval endpoints in READMEs (#55)

e145e93

Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>

Merge branch 'main' of github.com:mhbuehler/GenAIExamples into mmqna-…

311e6b6

…phase3

Modifies a megaservice test to verify correct apple color (#57)

fc4f46c

Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>

Merge branch 'main_upstream' into mmqna-phase3

51c6ec5

Added TTS validation test (#59)

a0b8177

* fixed test and added tts validation Signed-off-by: okhleif-IL <omar.khleif@intel.com> * added gaudi test, reverted -speech change Signed-off-by: okhleif-IL <omar.khleif@intel.com> --------- Signed-off-by: okhleif-IL <omar.khleif@intel.com>

Merge branch 'main' of github.com:mhbuehler/GenAIExamples into mmqna-…

6f39ff1

…phase3

Merge branch 'main' of github.com:mhbuehler/GenAIExamples into mmqna-…

4954f77

…phase3

Merge branch 'main' of github.com:mhbuehler/GenAIExamples into mmqna-…

dc5065d

…phase3

Fix spoken audio responses on Gaudi and add tests (#63)

401570f

Signed-off-by: dmsuehir <dina.s.jones@intel.com>

Clearing PDF clears textbox and version upgrade fixes interactive=Fal…

4aa013c

…se (#64) Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>

Fix first query test (#65)

36ee073

Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>

[pre-commit.ci] auto fixes from pre-commit.com hooks

82e76f0

for more information, see https://pre-commit.ci

mhbuehler mentioned this pull request Mar 19, 2025

MultimodalQnA audio features completion opea-project/GenAIComps#1433

Merged

3 tasks

mhbuehler and others added 2 commits March 19, 2025 15:21

Merge branch 'main' into mmqna-phase3

d2d77a0

Modified Dockerfile for megaservice tests to pass until merge

8c963eb

Signed-off-by: Harsha Ramayanam <harsha.ramayanam@intel.com>

HarshaRamayanam reviewed Mar 19, 2025

View reviewed changes

MultimodalQnA/Dockerfile Outdated Show resolved Hide resolved

ashahba added the v1.3 label Mar 19, 2025

HarshaRamayanam and others added 2 commits March 21, 2025 13:58

[pre-commit.ci] auto fixes from pre-commit.com hooks

60c8754

for more information, see https://pre-commit.ci

joshuayao requested review from lvliang-intel, letonghan and XinyaoWa March 24, 2025 02:47

Merge branch 'main' of github.com:mhbuehler/GenAIExamples into mmqna-…

0012f34

…phase3

dmsuehir reviewed Mar 24, 2025

View reviewed changes

MultimodalQnA/Dockerfile Outdated Show resolved Hide resolved

dmsuehir reviewed Mar 24, 2025

View reviewed changes

MultimodalQnA/tests/test_compose_on_gaudi.sh Outdated Show resolved Hide resolved

dmsuehir added 5 commits March 25, 2025 08:16

Merge branch 'main' of github.com:mhbuehler/GenAIExamples into mmqna-…

131941d

…phase3

Merge branch 'main' of github.com:mhbuehler/GenAIExamples into mmqna-…

9ae5cdd

…phase3

Merge branch 'main' of github.com:mhbuehler/GenAIExamples into mmqna-…

f1089a6

…phase3

Merge branch 'main' of github.com:mhbuehler/GenAIExamples into mmqna-…

4a1d430

…phase3

dmsuehir reviewed Mar 27, 2025

View reviewed changes

MultimodalQnA/ui/gradio/requirements.txt Show resolved Hide resolved

dmsuehir and others added 6 commits March 31, 2025 08:23

Merge branch 'main' of github.com:mhbuehler/GenAIExamples into mmqna-…

949a349

…phase3

Merge branch 'main' of github.com:mhbuehler/GenAIExamples into mmqna-…

4f66887

…phase3

Merge branch 'main' of github.com:mhbuehler/GenAIExamples into mmqna-…

e7e90d5

…phase3

Merge branch 'main' into mmqna-phase3

7dbe6cd

Merge branch 'main' of github.com:mhbuehler/GenAIExamples into mmqna-…

11f4292

…phase3

ashahba requested changes Apr 2, 2025

View reviewed changes

MultimodalQnA/ui/gradio/requirements.txt Outdated Show resolved Hide resolved

HarshaRamayanam and others added 2 commits April 2, 2025 14:59

Bump gradio version

3ce053f

Signed-off-by: Harsha Ramayanam <harsha.ramayanam@intel.com>

Update UI_TIMEOUT to 240 and minor doc update (#74)

307675f

Signed-off-by: Dina Suehiro Jones <dina.s.jones@intel.com>

ashahba approved these changes Apr 2, 2025

View reviewed changes

lvliang-intel approved these changes Apr 3, 2025

View reviewed changes

Merge branch 'main' into mmqna-phase3

8df3088

ashahba merged commit bbd5344 into opea-project:main Apr 3, 2025
19 checks passed

ashahba deleted the mmqna-phase3 branch April 3, 2025 04:45

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

MultimodalQnA audio features completion #1698

MultimodalQnA audio features completion #1698

Uh oh!

mhbuehler commented Mar 19, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Mar 19, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ashahba left a comment

Uh oh!

Uh oh!

Uh oh!

MultimodalQnA audio features completion #1698

MultimodalQnA audio features completion #1698

Uh oh!

Conversation

mhbuehler commented Mar 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Issues

Type of change

Dependencies

Tests

Authors

Uh oh!

github-actions bot commented Mar 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Dependency Review

Scanned Files

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ashahba left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

mhbuehler commented Mar 19, 2025 •

edited

Loading

github-actions bot commented Mar 19, 2025 •

edited

Loading