Mass deployment with Hypercore

[urbien]

The purpose of this issue is to coordinate what is a "must have" for mass deployment V1, leaving the rest for V1.1. Please read the README and FAQ.

Meta goal - Maximum Decentralization

For example, we can use Hypercore ecosystem not just for decentralized data structures, but also as:

• DHT instead of DNS, to avoid central DNS operators (and domain registrars).
• Network encryption + authentication with Noise instead of TLS, to avoid Certificate Authorities

This pushes the Internet further away from centralized operators and makes it simpler, and more reliable.

Towards the same decentralization goal, we look at algorithms that do not require central actors to coordinate shared state across distributed peers, like CRDT and HLC. This is in contrast to database and filesystems systems that require a Zookeeper, etcd or any other kind of central manager.

Somewhat controversially, we believe adding a cloud peer will answer a dozen issues that went unresolved throughout Hypercore's history. Why is it controversial? Because cloud is often associated with centralization and loss of personal control. So our take on the cloud is novel, something that does not exist today at a 'consumer-level' - a Personal Cloud (pC) node, not a data aggregation point like in all the SaaS apps. We also believe the pC needs to be free. Or rather freemium, like dropbox, but unlike dropbox, that is not open source, has access to everyone's data, and was built just to store of data.

pC should provide maximum isolation in the cloud that banks today are getting, it is open source and should be built, even in V1, to run apps, created by the community. Deployment of pCs will serve the same goal of increasing decentralization:

• pC will provide availability that p2p apps today thoroughly lack, serve as a personal signaling and relay point, when direct connectivity is not possible, which is a huge reliability issue in p2p apps today, and guarantee the data durability
• any Data Center should be able to choose to offer pC, from big to small, and all the way to Edge machines on 5G towers. Same goes for running pC at home or company premises. A person should have full mobility of their pC between Data Centers, machines, and countries.

Problem

For Hypercore to go mainstream and for the mainstream apps to use Hypercore we need to solve a number of problems. This issue just links to issues that cover those needs and maybe adds some comments for why we think they are essential.

Direction

Iterative design, learning the powers of Hypercore bringing to the surface its hidden gems.

The main thrust here is to create reliability with the help of Personal Cloud (pC).

pC creates stable nodes that are always on the network and create the cloud-grade durability of Hypercore data structures. This then will be the foundation for AI-enabled Cloud apps.

pC nodes will maintain DHT stability, help with signaling when Hyperswarm is not connectable, and provide relay services for Hypercores from peers can't connect directly (due to network restrictions). At the same time, we need to take care for pC to not violate P2P principles, unlike super-nodes. pC logically is just an extension of a personal device, and it does not do any type of traffic aggregation or aggregate data storing, that personal devices, for which Hypercore was built, would not do. The DHT services for the community of pCs may be only exception.

Issues

1. Reliability of data. We need to backup Hypercores to S3 S3 for backup / restore and database streaming #1 and create multiple pCs replicas.

2. Reliability of the network. We need fallback modes Hyperswarm network reliability and fallback #3 and Hypercore network reliability and fallback #4

3. User-friendliness. This requires key rotation key rotation and revocation #5, full mobile and web browser support, and multi-device support multi-writer for single user on multi-devices and clouds #7

4. One store. Users do not care about managing multiple feeds. Corestore fulfills that (need to review its security model). But Corestore does not merge changes from all my feeds between devices and my pC replicas multi-writer for single user on multi-devices and clouds #7

Planned experiments

A repo with all tests will be published within couple of days.

1. Streaming Hyperbee from a static state in S3 (works)
2. Backing up Hypercore to S3, in shards (segments). Why shards? See S3 for backup / restore and database streaming #1.
3. Union of Hyperbees from a shard on disk and in s3. Note that union support only 2 Hyperbees (works)
4. Union of Hypertries. Different from mountable Hypertries, as it preserves the keys, while mounting prefixes them (works)
5. Union of Hyperdrives. (working on it)
6. Key rotation with verification (using multi-key). Needed for mobility as new machine has its own keypair. Research how to sign keypair transition, in view of not having an old keypair.
7. Restore from S3 with key rotation, as new machine does not have old keys.
8. Multi-writer. multi-writer for single user on multi-devices and clouds #7 and transacting in-time  #9