feat: replace verifier with retriever #26
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
p0deje
force-pushed
the
2-pass-verifier
branch
2 times, most recently
from
94c033b to
d04308a
Compare
botocore.errorfactory.ModelErrorException: An error occurred (ModelErrorException) when calling the Converse operation: The system encountered an unexpected error during processing. Try your request again.
It's more consistent to other agents
p0deje
force-pushed
the
2-pass-verifier
branch
from
d04308a to
31b0053
Compare
This reverts commit 174aca9.
p0deje
force-pushed
the
2-pass-verifier
branch
from
964dbdb to
d29ed05
Compare
p0deje
changed the title
sh3pik
reviewed
Feb 4, 2025
alumnium/alumni.py
Outdated
| """ | ||
| Checks a given statement using the verifier. | ||
|
|
||
| Args: | ||
| statement: The statement to be checked. | ||
| vision: A flag indicating whether to use a vision-based verification via a screenshot. Defaults to False. | ||
| retries: The number of retries to check the statement. Defaults to the value set in the LoadingDetectorAgent. |
There was a problem hiding this comment.
@p0deje hm, there is no LoadingDetectorAgent anymore, right?
sh3pik
reviewed
Feb 4, 2025
alumnium/__init__.py
Outdated
| logger = logging.getLogger(__name__) | ||
|
|
||
| level = getenv("ALUMNIUM_LOG_LEVEL", None) | ||
| if getenv("ALUMNIUM_DEBUG", "0") == "1": |
There was a problem hiding this comment.
Is it kept for backward compatibility?
There was a problem hiding this comment.
Not really, I can remove it if you think it's unuseful.
There was a problem hiding this comment.
I think it might be more confusing than helpful.
I imagine the situation when ALUMNIUM_DEBUG=1 and ALUMNIUM_LOG_LEVEL=info. If I look at them in the .env file, I can forget which one gets a priority
`ALUMNIUM_LOG_LEVEL=debug` should be used instead
It looks like it's not so useful as long as we use CoT in the retriever agent structured output (explanation string BEFORE value string)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What started initially as an attempt to implement 2-pass verification to increase its stability ended up being a complete replacement of
verifier/confirmation_checkerwithretriever/extractoragents. For now it seems to work good enough, though I would like to further improve the implementations by utilizing chain-of-thought approach for structured outputs. This can be done in follow-up PRs.Overall, the new design for verification is to:
true/falseoutput of the retriever result.Smaller additions in this PR:
ALUMNIUM_LOG_LEVELwhich compliments existingALUMNIUM_DEBUG.<septo%SEP%because Haiku treats the former as opening HTML tag and likes to append the closing tag (e.g.<sep>FOO</sep>).