openapi: Add masks field to /v1/compile results.

[philipaconrad]

What Changed?

This PR adds an object property to each /v1/compile return type, allowing us to include masking rules in the output for later consumption by downstream tooling.

What I have in mind for masking rules looks roughly like:

{
  "masks": {
    "tickets.assignee": { "replace": ["***"] },
    "tickets.customer.id": { "md5hash": [] },
    "tickets.customer.phone": { "lastN": [4] }
  }
}

Where the key is the column name, and the value is an object of the form {"mask_function": [args...]}

Open Questions

• Do we want to allow providing masks as an HTTP request argument? (Ex: add a new key like options.masks)
• Do we want to allow shorthand syntaxes? (Ex: "tickets.assignee": "***" would indicate a replacement with the constant string ***)
• This design makes a number of assumptions. Are there any obvious edge cases that this design would be poor for handling?

[srenatus]

Do we want to allow shorthand syntaxes? (Ex: "tickets.assignee": "***" would indicate a replacement with the constant string ***)

I had been thinking about something related reading your PR message -- I'm on the fence about encoding masking as function+args: it suggests an openness to the system that we perhaps don't need. Unless of course, want to support exactly that, but I think that's a product decision; we could make both work. I think the simpler, more static approach would be to declare certain well-known mask options, and encode them more directly:

{
  "masks": {
    "tickets.assignee": { "replace": "***" },
    "tickets.customer.id": { "md5hash": {} },
    "tickets.customer.phone": { "lastN": 4 }
  }
}

(💭 Unrelated: Do we support three-part refs for masks? What column of which table does tickets.customer.id refer to?)

[philipaconrad]

Do we support three-part refs for masks? What column of which table does tickets.customer.id refer to?

I hadn't considered this problem because up until just recently, we were not making hard assumptions about what a.b refs were implying. 😅

In the LINQ/EF Core use case, 3-part refs usually imply a JOIN being used to look up a field, like jumping from a Ticket, to the associated Customer object. In an ORM context, this makes sense, elsewhere, less so.

For the masking use case, I'd assume that the LINQ+UCAST library would have to do one of the following:

• Just ignore fields that require JOINs. (Easy, but leaves pitfalls for users)
• Warn on JOIN fields? (Still doesn't save users from pitfalls)
• Actually try to hide the JOIN'd field result, by either decorator, or some kind of overlay "MaskedX" class? (I'm struggling with this actively in my prototypes right now; not sure if it's entirely worth the hassle. It's trivial to mangle fields in-place when it's a plain old datatype struct, much more tricky when it's a "smarter" object.)

What makes this tricky/hard is that in the tickets.customer.id case, the field-to-be-masked actually does not live in the Tickets table at all-- the mask would need to apply for the Customer object that is fetched to display the ID.

[philipaconrad]

I had been thinking about something related reading your PR message -- I'm on the fence about encoding masking as function+args: it suggests an openness to the system that we perhaps don't need.

I agree with this opinion-- I'd been mulling over the initial sketch, and realized pretty quickly that unconstrained lists of args are going to be a pain in strongly-typed languages, like C# and Go. Having a more fixed structure as you suggested, with known key-value pairs, should be much easier to scale across languages.

(EDIT: I've gone and updated the spec to follow a stronger, more locked-down format, as discussed.)

[srenatus]

I hadn't considered this problem because up until just recently, we were not making hard assumptions about what a.b refs were implying. 😅

In the LINQ/EF Core use case, 3-part refs usually imply a JOIN being used to look up a field, like jumping from a Ticket, to the associated Customer object. In an ORM context, this makes sense, elsewhere, less so.

I suppose I hadn't thought about anything not "a.b" until kurt wanted "a" 😅 So "a.b.c" hadn't crossed my mind, and I don't know if, as a consequence, the compile machinery will let that pass. I think an action item here would be to put some LINQ-specific test cases into the EOPA sources. Perhaps we even need to make pkg/compile/check.go adjust its checks (for unknown ref length, specifically) based on the supported features. I.e. if you only ask for ucast.linq, that'll be OK, if you ask for sql.postgresql and ucast.linq, it won't be -- none of the other targets support "a.b.c" refs.