the embedding dimension(1536) is not equal to (1024) in database

[VladimirARodionov]

[2025-02-13 14:17:04,833] ERROR embedding.py:176 - services.embedding - (builtins.AssertionError) the embedding dimension(1536) is not equal to (1024) in database.
[SQL: SELECT rag_vec.distance AS score, rag_doc.id, rag_doc.src_id, rag_doc.content, rag_doc.type, rag_doc.target_id, rag_doc.parent_id, rag_doc.seq, rag_doc.metadata
FROM rag_vec LEFT OUTER JOIN rag_doc ON rag_vec.doc_id = rag_doc.id LEFT OUTER JOIN rag_src ON rag_doc.src_id = rag_src.id
WHERE rag_vec.embedding MATCH ? AND k=5]
Traceback (most recent call last):
File "/home/vrodionov/viewstore/openai/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1815, in _execute_context
context = constructor(
File "/home/vrodionov/viewstore/openai/venv/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 1474, in _init_compiled
l_param: List[Any] = [
File "/home/vrodionov/viewstore/openai/venv/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 1476, in
flattened_processorskey
File "/home/vrodionov/viewstore/openai/venv/lib/python3.10/site-packages/sqlalchemy/sql/type_api.py", line 2052, in process
fixed_process_param(value, dialect)
File "/home/vrodionov/viewstore/openai/venv/lib/python3.10/site-packages/sqlalchemy_vectorstores/databases/sa_types.py", line 23, in process_bind_param
assert len(value) == self._dim, f"the embedding dimension({len(value)}) is not equal to ({self._dim}) in database."
AssertionError: the embedding dimension(1536) is not equal to (1024) in database.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/vrodionov/viewstore/openai/services/embedding.py", line 145, in ask
results = self.vs.search_by_vector(query, top_k=5)
File "/home/vrodionov/viewstore/openai/venv/lib/python3.10/site-packages/sqlalchemy_vectorstores/vectorstores/sqlite.py", line 30, in search_by_vector
docs = [x._asdict() for x in con.execute(stmt)]
File "/home/vrodionov/viewstore/openai/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1418, in execute
return meth(
File "/home/vrodionov/viewstore/openai/venv/lib/python3.10/site-packages/sqlalchemy/sql/elements.py", line 515, in _execute_on_connection
return connection._execute_clauseelement(
File "/home/vrodionov/viewstore/openai/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1640, in _execute_clauseelement
ret = self._execute_context(
File "/home/vrodionov/viewstore/openai/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1821, in _execute_context
self._handle_dbapi_exception(
File "/home/vrodionov/viewstore/openai/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 2355, in _handle_dbapi_exception
raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
File "/home/vrodionov/viewstore/openai/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1815, in _execute_context
context = constructor(
File "/home/vrodionov/viewstore/openai/venv/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 1474, in _init_compiled
l_param: List[Any] = [
File "/home/vrodionov/viewstore/openai/venv/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 1476, in
flattened_processorskey
File "/home/vrodionov/viewstore/openai/venv/lib/python3.10/site-packages/sqlalchemy/sql/type_api.py", line 2052, in process
fixed_process_param(value, dialect)
File "/home/vrodionov/viewstore/openai/venv/lib/python3.10/site-packages/sqlalchemy_vectorstores/databases/sa_types.py", line 23, in process_bind_param
assert len(value) == self._dim, f"the embedding dimension({len(value)}) is not equal to ({self._dim}) in database."
sqlalchemy.exc.StatementError: (builtins.AssertionError) the embedding dimension(1536) is not equal to (1024) in database.
[SQL: SELECT rag_vec.distance AS score, rag_doc.id, rag_doc.src_id, rag_doc.content, rag_doc.type, rag_doc.target_id, rag_doc.parent_id, rag_doc.seq, rag_doc.metadata
FROM rag_vec LEFT OUTER JOIN rag_doc ON rag_vec.doc_id = rag_doc.id LEFT OUTER JOIN rag_src ON rag_doc.src_id = rag_src.id
WHERE rag_vec.embedding MATCH ? AND k=5]

[VladimirARodionov]

after manually change in code in class SqliteVector dim: int = 1024 to dim: int = 1536 error disappear

[liunux4odoo]

after manually change in code in class SqliteVector dim: int = 1024 to dim: int = 1536 error disappear

You should specify the dim argument mannually when initialize the vector store.

[VladimirARodionov]

I do set it vs = SqliteVectorStore(sqlite_db, dim=1536, embedding_func=embed_func, fts_tokenize="jieba"), but it does not work for class SqliteVector

[liunux4odoo]

I will check it. Thanks for feedback.