An upgraded protocol definition for flexible and scalable decentralized governance

[brickpop]

A data-encoding protocol, intended to allow developing decentralized and fast-changing components within a flexible environment, where iterations can be made in a fault-tolerant fashion.

This proposal is in line with the ideas presented by @arnaucube, and also with some of the best practices of the Clean Architecture model.

This proposal is conceived to host the current and future developments, topologies and architectures.

Reasoning

The current flow in which models are defined (Vochain > rest of components), has brought friction and development issues, since other components learn about the schema's when the whole features are iin master, and reconsidering changes is a no-go. If changes are made to a JSON model or Vochain enum's, errors and mismatches don't appear until weeks or months later (sometimes before critical deadlines).

The Vochain is acting as a source of truth, that all remote components are coupled against, even if data fields and internal edge cases are not relevant to them. This increases the chances on developers missing out.

Main ideas

In order to adopt a stronger development model for the future, the proposed path is to gradually replace the current data serialization and low-level JSON API's by a global set of protobuf models. Not all at once.

Defined in a project of its own in the first place, and implemented across the stack only after all the affected parties validate it. This should not imply changing the implementation of the Vochain. Just the transport adapters.

Proposed global models: https://github.com/vocdoni/dvote-protobuf/blob/draft/protocol-v2/src/protocol/

The data models involve two different domains:

• The transport layer for connecting to remote services or even local resources
• The bytes of all payloads being wrapped (metadata, params, ballots, proofs, etc), which can be unambiguously signed and verified at anytime

For this proposal to be successful, there should be a Protocol coordinator, as suggested by @mvdan. This is, someone with the role of assuring the consistency and soundness of the whole spec and accommodating the feedback of everyone before any changes are made.

In addition to this:

• Everything has its own data model
  • Transactions, receipts, requests, responses, signed content, ballot variants, census proof variants, etc.
• Data models act as a global interface understood by everyone
  • Components can keep their own internal data models
    • Nobody else is expected to know about them
• Different use cases have their own submodel
  • Only the relevant fields are present
  • Meaningful names only
  • No ambiguous names to be used in many use cases
• Naming consistency
  • Camel case
  • No synonyms, one official name only.
• Communication data models are agnostic to the transport layer used (HTTP, WebSocket, GRPC, IPFS pub/sub, Swarm, local files, etc)

Key benefits:

• The model itself is testable and can be validated before starting to write any adapter code (see here)
  • Allowing to make sure that a change or a fix has been pulled from every component
    • Compilation won't work otherwise
  • Automated type generation for many languages
• Adding new features (submodels) without breaking the existing ones
• Breaking large models into smaller meaningful bits
• Allowing to simplify the structs and handlers on the Go side
  • No need to declare every potential GW response struct fields
    • No more limitations on their types or attributes
    • No more omitempty in every potential GW response field
• Upgradeability
  • Immutable data can be kept without breaking
  • Legacy services can coexist with new ones, using the same models

Data transport examples (fragment)

Every interaction is a message body, which may or may not be signed (similar to what the Vochain does now, but fitted to act as the only communication wrapper for all):

message Message {
	// The bytes of the {Body} model being signed
	bytes body = 1;

	// Optionally signed. Not expected when:
	// - Submitting a vote
	// - Performing a read-only operation
	// 
	optional bytes signature = 2;

	SignatureType signatureType = 3;
}

// The body contains the serialized bytes from a `Body`, which can host three types of interactions.
message Body {
	bytes id = 1;
	// UNIX timestamp
	int32 timestamp = 2;

	oneof body {
		// Vochain, VocOne, ...
		Transaction transaction = 11;
		TransactionReceipt receipt = 12;

		// To remote services
		Request request = 13;
		Response response = 14;
	}
}

A message body can have one of: Transaction, TransactionReceipt, Request and Response. These oneof definitions act as an implicit enum. Only relevant Transaction fields will appear, only relevant Request fields will exist, etc. In additioin, all these payloads are self-contained, 100% signable and verifiable, irregardless of JSON marshaling, field ordering, string encoding, etc.

All the potential operations are immediately visible to the developer.

// Census Service, File gateways, and similar nodes expect `Request` models, and return `Response` models, all wrapped within a `Message > Body`.
message Request {
	oneof body {
		// ELECTION
		GetElection getElection = 1;
		GetElectionList getElectionList = 2;
		GetOrganization getOrganization = 3;
		GetBallot getBallot = 4;
		GetElectionBallots getElectionBallots = 5;
		GetElectionKeys getElectionKeys = 6;
		GetElectionCircuitInfo getElectionCircuitInfo = 7;
		GetElectionResults getElectionResults = 8;
		GetElectionResultsWeight getElectionResultsWeight = 9;

		// CENSUS
		NewCensus newCensus = 31;
		AddCensusKeys addCensusKeys = 32;
		GetCensusRoot getCensusRoot = 33;
		GetCensusSize getCensusSize = 34;
		PublishCensus publishCensus = 35;
		GetCensusProof getCensusProof = 36;
		DumpCensus dumpCensus = 37;

		// FILE STORAGE
		PinFile pinFile = 51;
		FetchFile fetchFile = 52;

		// NETWORK
		GetBlockStatus getBlockStatus = 61;
		GetBlockCount getBlockCount = 62;
		EstimateElectionPrice estimateElectionPrice = 63;

		// TRANSACTIONS
		GetTransaction getTransaction = 81;
		GetRawTransactionMessage getRawTransactionMessage = 82;
		WaitTransaction waitTransaction = 83;
	}
}

For each one of them, we can easily find the definition of their Request and their corresponding Response. One is always next to each other. Naming leaves no doubt about the hierarchy.

// Request
message GetElectionCircuitInfo {
	bytes electionId = 1;
}
// Response
message GetElectionCircuitInfoResponse {
    // Circuit index
    int32 index = 1;
    // The prefix of the URI to fetch the artifacts from
    string baseUri = 2;
    // Relative path where the circuit is hosted
    string circuitPath = 3;
    // Maximum census size supported by the circuit
    int32 maxSize = 4;

    bytes witnessHash = 5;
    bytes zKeyHash = 6;
    bytes vKeyHash = 7;
}

In general, responses can be a success or an error, each with their own fields and an optional body (similar for transactions)

// Responses can either be successful or fail. The `Response` body depends on the type of request made originally, and the caller is responsible for deserializing the bytes into the according model, if any.
message Response {
	oneof body {
		ResponseSuccess success = 1;
		ResponseError error = 2;
	}
}

message ResponseSuccess {
	// Serialized response, related to the request model
	bytes body = 1;
}
message ResponseError {
	string message = 1;

	// Serialized response, related to the request model
	bytes body = 2;
}

Wrapped data examples (fragment)

An election can be defined as follows. It can specify many censuses, for which voters need to provide a proof in their ballot.

// The following model defines the declaration of an election. These are just the settings, not the human readable information.
message Election {
	// See census.proto
	Census mainCensus = 1;
	Census secondaryCensus = 2;
	Census tertiaryCensus = 3;

	// Used to select the zk circuit, determine the cost, etc
	int32 censusSize = 11;

	// Settings of the questions that people can vote
	repeated Proposal proposals = 12;

	// How data and voters should be protected
	Privacy privacy = 13;
	// When to start and end
	Lifecycle lifecycle = 14;

	string metadataUri = 15;
}

Each election has N proposals (questions). Each question can have a different type of voting mechanic, have its own status (ready, paused, ended, etc)

message Proposal {
	oneof proposal {
        	// Binary
		ApprovalProposal approval = 1;
		// choose one of [0, ... N-1]
        	SingleChoiceProposal singleChoice = 2;
		// allocate quadratic points below a max sum
        	QuadraticProposal quadratic = 3;
		// 1w1v: Voting the ordered indexes
        	RankedProposal ranked = 4;
		// assign % to each option
        	SpreadProposal spread = 5;
	}
}

// Submodels
message ApprovalProposal {}
message SingleChoiceProposal {
    int32 optionCount = 1;
}
message QuadraticProposal {
    int32 optionCount = 1;
    float costExponent = 2;
    // Assigning points from 0 to maxValue
    int32 maxValue = 3;
    // The exponentiated sum of values must not exceed maxSum
    int32 maxSum = 4;
}
message RankedProposal {
    int32 optionCount = 1;
    // Up to how many indexes can be ranked
    int32 maxItems = 2;
}
message SpreadProposal {
    int32 optionCount = 1;
}

Certain flags like the Privacy ones can be defined here:

// The models below affect the whole election definition.
message Privacy {
	bool realTimeResults = 1;

	enum CensusProofs {
		// Signed vote. Standard proof(s) are expected.
        	PLAIN = 0;
		// Anonymous votes, submitted right away.
        	// The standard proof is part of the ZK circuit inputs. The ZK proof is expected.
        	ZK_SNARKS = 1;
		// Anonymous votes, with voter registration prior to the election.
        	// The standard proof is part of the ZK circuit inputs. The ZK proof is expected.
        	ZK_SNARKS_PREREGISTER = 2;
	}
	CensusProofs censusProof = 2;
}

When the censusProof is ZK_SNARKS or ZK_SNARKS_PREREGISTER, all voters are expected to submit a ProofZkSnark for each census entry. Otherwise, a plain proofs like ProofArbo, ProofCSP, StorageProofErc20, etc are expected. See below.

---

When voting, a ballot can have two use cases: Signed (non-anonymous census proofs) or Anonymous (using ZK proofs, nothing is signed):

message Ballot {
    oneof body {
        // When Election.privacy.censusProof == PLAIN
        SignedBallot signedBallot = 1;

        // When Election.privacy.censusProof != PLAIN
        BallotBody anonymousBallot = 2;
    }
}

message SignedBallot {
    bytes ballot = 1; // Serialized {BallotBody} bytes
    bytes signature = 2; // sign(bytes(BallotBody))

    SignatureType signatureType = 3;
}

message BallotBody {
    bytes electionId = 1;
    bytes nullifier = 2;
    
    // The proof(s), according to the Election census types defined
    // Or [ProofZkSnark] if anonymous
    repeated Proof proofs = 3;
    
    message VoteList {
        // Submitting only only one vote?
        bool partial = 1;
        
        // [v1, v2, v3...] when partial == false
        // [v3] when partial == true
        repeated Vote votes = 2;

        // Index of the proposal being submitted.
        // Set to 0 when `partial = false` (ignored)
        int32 submittedIndex = 3;
    }
    VoteList votes = 4;
}

Votes can be sent all at once or be submitted proposal by proposal (step by step).

The kind of Proof models will be 1:1 mapped to the respective Census'es used to define the election.

message Proof {
	oneof body {
        	// SIGNED PROOFS
		ProofNone none = 1;

        	// Always weighted
		ProofArbo arbo = 11;
		ProofCSP csp = 12;

		StorageProofErc20 erc20 = 21;
		StorageProofErc721 erc721 = 22;
		StorageProofErc1155 erc1155 = 23;
		StorageProofErc777 erc777 = 24;
		StorageProofMiniMe miniMe = 30;

        	// ANONYMOUS PROOFS
		ProofZkSnark zkSnark = 100;
	}
}

If the primaryCensus defined a CensusArbo model like:

message CensusArbo {
	bytes censusRoot = 1;
	string censusUri = 2;
}

Then, voters will need to provide a first proof like:

message ProofArbo {
	repeated bytes siblings = 1;
}

If the secondaryCensus defined a CensusErc20 (PoH or similar) like:

message CensusErc20 {
	bytes tokenAddress = 1;
	int32 balanceMapSlot = 2;
	optional StorageProofErc20 proof = 3;
	// Ethereum block at which the state root is taken
	int32 sourceEthereumBlock = 4;
}

Then voters are expected to encode a ProofErc20 like:

message StorageProofErc20 {
	bytes key = 1;
	bytes value = 2;
	repeated bytes proof = 3;
}

Important note: The ballot will be signed by one wallet only. This means that all proofs need to refer to that wallet's address.

The beauty of this is that new types of censuses, are not coupled with existing models and schemes. Censuses can also be defined as CensusNone, in which case, nothing is required.

---

Simmilarly, depending on the type of aggregation requested when defining a Proposal above (ApprovalProposal, SingleChoiceProposal, QuadraticProposal, RankedProposal and SpreadProposal), their Results will have an according data representation.

message PendingResults {}
message ApprovalResult {
    // How many (weighted) rejections
    string rejected = 1;
    // How many (weighted) approvals
    string approved = 2;
}
message SingleChoiceResult {
    // How many (weighted) votes each index got
    repeated string votes = 1;
}
message QuadraticResult {
    // How many (weighted) points each index got
    repeated string points = 1;
}
message RankedResult {
    message RankedChoiceResult {
        message RankedChoicePositionResult {
            // The rank position of the choice, for which the points below have been achieved
            int32 position = 1;
            // How many (weighted) points the choice got at the current rank position
            string points = 2;
        }

        // The results of the choice at every valid rank position
        repeated RankedChoicePositionResult entries = 1;
    }

    // The results of each choice
    repeated RankedChoiceResult choices = 1;
}
message SpreadResult {
    // How many (weighted) points each index got
    repeated string points = 1;
}

This way, consumers get an already friendly representation for most of the cases. The only case where aggregation is a bit more involved is when representing RankedResult's. There, we need to express the votes that each option received for each potential weight, but the rest are unaffected.

[NateWilliams2]

I’m a big fan in general. one suggestion I have is to ensure that fields/settings are only accessible if they are relevant. For example, in the election model we have a censusSize field, which a comment notes is only relevant to non-anonymous elections: https://github.com/vocdoni/dvote-protobuf/blob/1ac419517d30e0fd43002b66cb52d134daa3c758/src/protocol/election.proto#L24

I think situations like this can be confusing- I probably won’t be looking at comments in the protobuf models when implementing a component, so I might assume that censusSize is relevant and important to any process, and then encounter unknown bugs if I’m relying on what’s stored in this field. I definitely encountered bugs like this with our current architecture when implementing the app + explorer.

Just a general design thought. Maybe in this specific situation we could allow the CensusProofs models to contain their own configs, where we could store CensusSize or any similar proof-specific setting in the future.

[NateWilliams2]

Another comment: with the way protobuf serializes/deserializes, would we be able to add new Proposal models in the future and still use these models to deserialize the existing/legacy elections we have stored?