Skip to content

fix: Casting to float lose precision in some cases #10423

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 4 commits into
base: develop
Choose a base branch
from

Conversation

EnascutaRazvan
Copy link

What does this PR do?
I reproduced the case and it seems like when casting to float the precision is lost
Related to -> #10391

Motivation
Because i use this database everyday

Related issues
#10391

Additional Notes
I don't know exactly why that cast to float was used, but it seems generates some problems

@tglman
Copy link
Member

tglman commented Mar 19, 2025

Hi,

Thank you, just triggered the ci build on this, if all pass will merge it and port it to 3.2.x

Bye

@tglman
Copy link
Member

tglman commented Mar 19, 2025

Hi,

Tests are failing, so we do have cases where we rely on this conversion, looking in the code is possible to avoid this conversion specifying the double type at the end of the number, so something like: 35.927675D would this work for your use case ?

Regards

@tglman
Copy link
Member

tglman commented Mar 19, 2025

Hi,

Maybe another possible approach is replace the check: if (Math.abs(returnValue) < Float.MAX_VALUE) with a reconversion to the double value, to see if the value still matches, like:

float floatValue = (float) returnValue;
double check = (double) flatValue;
if(check == returnValue) {
 finalValue= flatValue;
} else {
finalValue = returnValue;
}

Can you verify if a check like this works in your case and update the PR?

@tglman
Copy link
Member

tglman commented Mar 19, 2025

One last thing (hopefully ...)

in case you update this PR can you add a test case for the case conversion error in OInsertStatementExecutionTest or in OUpdateStatementExecutionTest so we avoid to get regression in future.

@EnascutaRazvan
Copy link
Author

I ll try it tomorrow, ty

@tglman
Copy link
Member

tglman commented Mar 20, 2025

Hi,

Tried to run the tests, they fail for formatting, you can run mvn spottless::apply to fix all the formatting issues.

(waiting for the test as well )

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

2 participants