Add support for auto generating sqlite schema

[habdelra]

This PR adds support for auto generating the sqlite schema from the postgres DB. The command pnpm schema automatically create a sqlite schema from the postgres db. To do this we leverage a SQL AST (actually a CST).

TODO:

• add developer documentation for now to create a new migration
• add developer documentation for running a migration manually
• add developer documentation for generating the sqlite schema

[github-actions]

Test Results

584 tests  ±0   580 ✔️ ±0   8m 21s ⏱️ -6s
    1 suites ±0       4 💤 ±0 
    1 files   ±0       0 ❌ ±0 

Results for commit 4d4c3df. ± Comparison against base commit 563cc3a.

♻️ This comment has been updated with latest results.

[habdelra]

I opted to adjust this migration instead of making a new one since we don't actually have any data yet to migrate

[backspace]

very cool!

[lukemelia]

Suggested change

The boxel system also uses SQLite in order to run the DB in the browser as part of running browser tests (and eventually we may run the realm server in the browser to provide a local index). After you have completed your migration you then need to generate a new schema SQL file for SQLite. To generate a new SQLite schema, from `packages/realm-server`, execute:

Boxel also uses SQLite in order to run the DB in the browser as part of running browser tests (and eventually we may run the realm server in the browser to provide a local index). We treat the Postgres database schema as the source of truth and derive the SQLite schema from it. Therefore, once you author and apply a migration, you should generate a new schema SQL file for SQLite. To generate a new SQLite schema, from `packages/realm-server`, execute:

[lukemelia]

Also, would it be possible and desirable to automatically run pnpm make-schema after pnpm migrate up or pnpm migrate down?

[habdelra]

it's possible. not sure if it's desirable. one thing to keep in mind is that the application itself can run migrations (like a django app). so it's not necessarily the case that migrations will always happen as a result of running pnpm migrate up or pnpm migrate down. in order to make sure things keep in sync, though, I did add some logic when the host app starts to make sure the sqlite schema is up-to-date. My thinking was that I wanted for us to work with it for bit and based on dev feedback we can adjust as necessarily.

[habdelra]

I think another reason why we might want to keep the pg migration and sqlite schema generation decoupled for now is that the sqlite schema conversion is still pretty fresh. I could see scenarios where we start using a new constraint type that we haven't yet added support for in the sqlite schema generation script. in which case it might be easier for debugging if the migration and schema generation were separate.