Make flattened synthetic source concatenate object keys on scalar/object mismatch

[parkertimmins]

There is an issue where for Flattened fields with synthetic source, if there is a key with a scalar value, and a duplicate key with an object value, one of the values will be left out of the produced synthetic source.

This fixes the issue by replacing the problematic object with paths to each of its keys. These paths consist of the concatenation of all keys going down to a given scalar, joined by .. For example, they are of the form foo.bar.baz. This applies recursively, so that every value within the object, no matter how nested, will be accessible through a full specified path.

For example if the following flattened field values is indexed:

{
   "a": {
       "b": 5
       "b": {
          "c": 10,
          "d": { "e": 15 }
       }
   }
}

The following synthetic source will be produced:

{
   "a": {
      "b": 5
      "b.c": 10,
      "b.d.e": 15
   }
}

Fixes #122936

[kkrik-es]

What if the conflict doesn't happen on the first part of the prefix, e.g.

field {
  path {
    to: 10
    to {
      foo: bar
    }
  }
}

Would this be caught here?

[parkertimmins]

Yep, so this will become:

field {
  path {
    to: 10
    to.foo: bar
 }
}

When it get to the first key/value field.path.to|10 it will take the else block and traverse down into the object, adding field and path to the openObject context. When it reaches the key value field.path.to.foo|bar that object will still be open, and seeing that lastScalarSingleLeaf has a value of to, and that to is the first token in the startPrexix (to.foo), it will make a concatenated path.

(Updated a test to this situation to verify it)

[parkertimmins]

Interestingly, there was a test which had a scalar/object mismatch. But it produced duplicate keys. When the xcontent was converted to jsont these duplicate keys originally threw an error, but now just drop the duplicates. (Something must have changed in xcontent stuff since the issue was opened to cause this change from an error to deduplication)

[elasticsearchmachine]

Pinging @elastic/es-storage-engine (Team:StorageEngine)

[github-actions]

🔍 Preview links for changed docs:

• docs/reference/elasticsearch/mapping-reference/flattened.md

🔔 The preview site may take up to 3 minutes to finish building. These links will become live once it completes.

[kkrik-es]

Maybe add also: `b.field("key1", "baz");

[parkertimmins]

Wasn't able to add a key1 field because there is a key1 object. I went ahead and added a few more keys with scalar values at the root level and in nested objects.

[kkrik-es]

Let's also add something for a and a.b.

[parkertimmins]

I think it's worth leaving this one as is so that the concatenated keys occur in a nested object. But testSingleObject has keys a, a.b, and a.b.c, so it covers the scenario where keys in the root object are concatenated.

[kkrik-es]

Nice, some minor comments to address but otherwise it looks good.

[elasticsearchmachine]

💔 Backport failed

Status	Branch	Result
❌	8.19	Commit could not be cherrypicked due to conflicts
✅	9.0
❌	8.18	Commit could not be cherrypicked due to conflicts

You can use sqren/backport to manually backport by running backport --upstream elastic/elasticsearch --pr 129600

[parkertimmins]

💚 All backports created successfully

Status	Branch	Result
✅	8.19
✅	8.18

Questions ?

Please refer to the Backport tool documentation