feat(NODE-6507): generate encryption configuration on mongoose connect

[baileympearson]

Summary

This PR contains the meat and potatoes of the automatic CSFLE/QE integration into mongoose. With the changes included, users can now create encrypted schemas, instantiate models with them, insert and query using their model and have the resultant documents be encrypted and decrypted automatically.

This PR contains a lot of test changes. I tried to be as exhaustive to ensure as much coverage as possible, and because the schema format is unlikely to change (changing the format of encryptedFieldsMap or schemaMap would break a lot of drivers and applications). So ideally these tests are not a large maintenance burden.

Examples

See the integration tests in encryption.test.ts for an example of how the API works.

[vkarpov15]

I'd also like to see tests covering:

1. Subdocuments
2. Document arrays
3. Maps

If you can do CSFLE with nested paths, you should be able to also do CSFLE with subdocuments, which are slightly different.

[baileympearson]

I'd also like to see tests covering:

• Subdocuments
• Document arrays
• Maps
  If you can do CSFLE with nested paths, you should be able to also do CSFLE with subdocuments, which are slightly different.

I believe I now have tests for all of these. Let me know if I'm still missing anything

[vkarpov15]

Potential edge case here: discriminator base schema defines nested path, discriminator child schema defines subdocument with same path but different options.

const schema = new Schema({
  name: {
    first: { type: String, encrypt: { keyId: [keyId], algorithm } }
  }
});

const discriminatorSchema = new Schema({
  name: new Schema({ first: Number }) // Different type, no encryption, stored as same field in MDB
});

schema.eachpath() doesn't account for subdocuments because subdocuments have a distinct schema (it does account for nested paths though because nested paths do not have their own schema)

[baileympearson]

Good catch - I wrote a test for this scenario and you were right.

I've been trying out some different approaches and I can't find a better solution than something like this:

  function* allPaths(schema, prefix) {
    for (const path of Object.keys(schema.paths)) {
      const fullPath = prefix != null ? `${prefix}.${path}` : path;
      if (schema.path(path).instance === 'Embedded') {
        yield* allPaths(schema.path(path).schema, fullPath);
      } else {
        yield fullPath;
      }
    }
  }

  const paths = new Set(allPaths(schema));

  for (const path of allPaths(model.schema)) {
    if (paths.has(path) && (model.schema._hasEncryptedField(path) || schema._hasEncryptedField(path))) {
      throw new Error(`cannot declare an encrypted field on child schema overriding base schema. key=${path}`);
    }
  }

This generates all possible paths in the schema (the above doesn't handle arrays, but encryption on fields in arrays isn't supported so that's out of scope here). Not my first choice, because this feels brittle. I'll keep looking at it but

1. Can you think of a better solution here than recursively iterating over all paths + nested schemas?
2. If not, does a utility that I could use like allPaths exist somewhere?

[vkarpov15]

I don't think recursively checking all paths is necessary here because you just want to find conflicts in the top-level paths. So if discriminator schema has a path pathname with an encrypted field, and root schema has a nested path with rootSchema.nested[pathname.split('.')[0]], you can already call that a conflict and throw an error. Similarly, if discriminator schema has a nested path pathname but in root schema you have rootSchema.paths[pathname] then you can also throw an error.

[baileympearson]

Not necessarily, right? That would throw an error in scenarios where the child schema provides a subdocument with the same root path but doesn't modify the encrypted path. ex:

const schema = new Schema({
  name: {
    first: { type: String, encrypt: { keyId: [keyId], algorithm } }
  }
});

const discriminatorSchema = new Schema({
  name: new Schema({ age: Number }) // Different path, no encryption
});

I'd expect this to be fine, because there isn't a conflicting path for name.first.