Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix "cannot insert into generated column snapshot.id" #208

Open
samuelstroschein opened this issue Dec 12, 2024 — with Linear · 1 comment
Open

fix "cannot insert into generated column snapshot.id" #208

samuelstroschein opened this issue Dec 12, 2024 — with Linear · 1 comment

Comments

Copy link
Member

samuelstroschein commented Dec 12, 2024

Context

The snapshot id column being forcibly generated leads to:

  • frustration because a simple insert doesn't work
  • needs to be explained in the docs
  • potentially large performance cost

CleanShot 2024-12-12 at 18.09.52@2x.png

Proposal

Make the sha256 hash the default for snapshot.id but lift the restriction on snapshot ids being the hash of the content.

  • We are not using snapshot ids for comparing changes because it would be flaky.
  • The main purpose is data de-duplication.
  • As such, it's enough to have a default du-duplicated id.
Copy link
Member Author

samuelstroschein commented Dec 12, 2024

The performance cost of forcibly generated snapshot ids is severe.

Copying snapshots from one lix to another or even sending it over the network when syncing could entail hundreds of thousands of snapshots. Even though the snapshot.id has already been computed, the receiver must re-compute the snapshot id.

    const lix1 = await openLix()	

    // potentially 100.000's snapshots
    const snapshots1 = await lix.db.selectFrom("snapshot").selectAll().execute();

	const lix2 = await openLix({});

	await lix2.db
		.insertInto("snapshot")
       // need to re-compute the id for all snapshots
		.values(snapshots1.map((c) => ({ content: c.content })))
		.execute();

	await applyChanges({ lix: lix2, changes: changes1 });

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant