JSON report aggregate tool #91
Conversation
|
Thank you for working on this. I probably won't be able to take a detailed look for a few days. Based on the description I'd say you're good to merge. Leaving it up to you if you want to wait for a full review or if you want to merge early. |
|
Before merging should we rename this script from |
|
Maybe |
reccmp/tools/denoise.py
Outdated
| parser.add_argument( | ||
| "--diff", type=Path, metavar="<files>", nargs="+", help="Report files to diff." | ||
| ) | ||
| parser.add_argument( | ||
| "--output", | ||
| "-o", | ||
| type=Path, | ||
| metavar="<file>", | ||
| help="Where to save the aggregate file.", | ||
| ) | ||
| parser.add_argument( | ||
| "--samples", | ||
| type=Path, | ||
| metavar="<files>", | ||
| nargs="+", | ||
| help="Report files to aggregate.", | ||
| ) |
There was a problem hiding this comment.
If neither of --diff and --samples is passed, then the program will exit silently.
I think this is an error condition.
There was a problem hiding this comment.
Thanks! See what you think now.
There was a problem hiding this comment.
Looks good! I'd just have checked the args namespace.
The main
reccmp-reccmpscript will save the report of the comparison run to a JSON file if you use the--jsoncommand line option. The report contains data for each orig. address with the then-current entity name, accuracy score, and flags to mark a function stub and effective match. The HTML display has the same data (also JSON format), but with unified diff output for each address included.This update adds the
reccmp-denoisescript to combine multiple JSON reports into a single "noise reduced" report. (Resolves #88.) Provide the list of report files you want to aggregate in the--samplesoption, and the output file with--output.Our strategy to create the aggregate is this: for each orig. address in any of the sample reports, use the highest accuracy score from any report. The hierarchy is:
To clarify, what we mean by "noise" is: variations in the generated assembly in in functions not directly impacted by the code change under review.
This update also adds some rigidity for importing and exporting JSON data. This has been working fine for a while, but without any set schema. The only file format in the field is version 1, and this is marked in the JSON file to protect against possible future versions.
We now have the internal
ReccmpStatusReportclass and somepydanticschemas to serialize and deserialize the JSON report files. We could now restructure the data into a new schema version and this framework should make the conversion stable.reccmp-denoisealso allows you to diff two different report files. This is not possible withreccmp-reccmpbecause you can only diff the saved report against the current comparison run. If you specify--diffand--samplesin the same run, we will diff the first file in the--difflist against the newly generated aggregate report.