Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Barretenberg C++ binary overhaul #11459

Open
wants to merge 152 commits into
base: master
Choose a base branch
from
Open

feat: Barretenberg C++ binary overhaul #11459

wants to merge 152 commits into from

Conversation

codygunton
Copy link
Contributor

@codygunton codygunton commented Jan 23, 2025

Overhaul the Barretenberg binary and its API.

  • Breaks up bb main into different files organized by proving system / IVC scheme.
  • Make UltraHonk conform to the new API interface introduced earlier for Client IVC.
  • Refines the API a bit.
  • Introduces CLI11 to: provide help / documentation; validate opts (options can be required, exlusive of each other, validated against predicates like "path exists" or "string is in list"); also allows for easy environment variable aliasing.

This could definitely use some more a help.

  • Lots of documentation needed
  • Defaults are set in a weird and inconsistent way and that information isn't included in the documentation.
  • The help menus are perhaps too verbose. Subcommands can't inherit options or flags so we end up repeating.
  • Empty string cannot be passed and parsed to a "nothing argument" which can lead to frustrating debugging...
  • Little option validation is actually implemented.
  • Deprecated options aren't noted but they could be.

It was requested that the default change from UltraPlonk to UltraHonk, but we get rid of a default set of commands altogether. As a workaround, we can have users set BB_SCHEME=ultra_honk.

Newly created issues: AztecProtocol/barretenberg#1252, AztecProtocol/barretenberg#1253, AztecProtocol/barretenberg#1254, AztecProtocol/barretenberg#1255, AztecProtocol/barretenberg#1256, AztecProtocol/barretenberg#1257, AztecProtocol/barretenberg#1258, AztecProtocol/barretenberg#1259

Resolves AztecProtocol/barretenberg#1260

NB the line count is large because 1) CLI11 is a single 11k-line header; 2) I moved a lot of functions and some git mvs didn't show up as such. Main new code is api_ultra_honk.hpp.

@ludamad
Copy link
Collaborator

ludamad commented Jan 24, 2025

Merging master as boxes to get CI passing here

size_t required_crs_size = prover.proving_key->proving_key.circuit_size;
if constexpr (Flavor::HasZK) {
// Ensure there are enough points to commit to the libra polynomials required for zero-knowledge sumcheck
if (required_crs_size < curve::BN254::SUBGROUP_SIZE * 2) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: I prefer required_crs_size = std::max(required_crs_size, curve::BN254::SUBGROUP_SIZE * 2);

Copy link
Contributor Author

@codygunton codygunton Feb 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah this is nice, changed

const std::filesystem::path& vk_path) override
{
const bool ipa_accumulation = flags.ipa_accumulation;
if (ipa_accumulation) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not sure there is already a warning if multiple of these are set

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right these should be made exclusive of each other

@@ -51,6 +52,7 @@ inline std::vector<uint8_t> read_file(const std::string& filename, size_t bytes

inline void write_file(const std::string& filename, std::vector<uint8_t> const& data)
{
info("writing file to ", filename);
Copy link
Collaborator

@ludamad ludamad Feb 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we want this to stay in? maybe verbose only? I'm all for explicitness, but usually better to be like 'Writing proof to file ...' e.g. case by case

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no removed it

}
functions_string = format(functions_string, "\n]}");

const char* jsonData = functions_string.c_str();
Copy link
Collaborator

@ludamad ludamad Feb 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

mind clearing up this nonsense (I know not yours) and just doing std::cout << functions_string;

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sure

}

/**
* @brief Writes a sting to stdout
*
* @param str The raw string to write to stdout
*/
inline void writeStringToStdout(const std::string& str)
inline void write_string_to_stdout(const std::string& str)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we just delete this and replace every use with std::cout << str;

Copy link
Collaborator

@ludamad ludamad Feb 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(answer can be no / in fast follow)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

Copy link
Collaborator

@ludamad ludamad left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have marked a few things as blocking, but if you just want this in as-is I'm also happy to accept those in a fast-follow PR. Otherwise the non-blocking changes you may want to do in a fast follow PR regardless.

@ludamad
Copy link
Collaborator

ludamad commented Feb 15, 2025

Great push overall, it's nice to see things with meaningful flags and not just esoteric combinations

@@ -40,7 +40,7 @@ for artifact in $artifacts_to_process; do

echo "Generating verification key for function $fn_name"
# BB outputs the verification key to stdout as raw bytes, however, we need to base64 encode it before storing it in the artifact
verification_key=$($BB write_vk_for_ivc -b ${fn_artifact_path} -o - | base64)
verification_key=$($BB write_vk --scheme client_ivc -b ${fn_artifact_path} -o - | base64)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess this is immediately going to become confusing when we add a flow to write a CIVC VK. I wonder if this should just become the write_vk for mega honk..

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I thought about this but then it's the only mega honk command which seems weird. It think it makes sense to put it behind the --input_type flag in CIVC?

@codygunton codygunton added e2e-all CI: Enables this CI job. e2e Enables e2e jobs in BB change PRs. labels Feb 18, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
e2e Enables e2e jobs in BB change PRs. e2e-all CI: Enables this CI job.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Only construct circuit once when writing two vks.
4 participants