Explain ignore_above better
#129284
Explain ignore_above better
#129284
Conversation
This concept is complicated. Closes elastic#128991
nik9000
added
>docs
elasticsearchmachine
added
Team:Docs
|
Pinging @elastic/es-docs (Team:Docs) |
|
Pinging @elastic/es-search-foundations (Team:Search Foundations) |
| If you need to never reject documents, this should have some value `<=8191`. All documents with | ||
| more characters will just skip building the index for this field. | ||
|
|
||
| The defaults are complicated. It's `2147483647` (effectively unbounded) in standard indices and |
There was a problem hiding this comment.
Consider using bullets for defaults/dynamic mapping info for readability
Co-authored-by: Liam Thompson <32779855+leemthompo@users.noreply.github.com>
| : Do not index any field containing a string with more characters than this value. This is important because {{es}} | ||
| will reject entire documents if they contain keyword fields that exceed `32766` bytes when UTF-8 encoded. | ||
|
|
||
| To avoid any risk of document rejection, set this value to `8191` or less. Fields with strings exceeding this | ||
| length will be excluded from indexing. |
There was a problem hiding this comment.
Does this work on text fields? Or only keyword fields?
Also further down you say:
`logsdb` indices: `8191`. `keyword` fields longer than `8191` characters won't be indexed, but the documents are
accepted and the values unindexed values are available from `_source.
Does the previous statement only apply to logsdb indices? Or to standard indices as well? If both, that feels important.
What about this:
Skip indexing of a keyword value whose UTF-8–encoded size is larger than
ignore_above. The value is still kept in_source, but the field won’t be searchable or aggregatable.If you do not set
ignore_above, {es} will reject entire documents if they contain one or morekeywordfields exceeding a UTF-8–encoded size of32766.To avoid any risk of document rejection, set this value to
8191or less.
| The defaults are complicated: | ||
| * Standard indices: `2147483647` (effectively unbounded). Documents containing `keyword` fields longer than `32766` | ||
| bytes will be rejected. | ||
| * `logsdb` indices: `8191`. `keyword` fields longer than `8191` characters won't be indexed, but the documents are | ||
| accepted and the values unindexed values are available from `_source. |
There was a problem hiding this comment.
What about a table for this information?
The defaults are complicated:
Index type Default Effect Standard indices 2147483647(effectively unbounded)Documents will be rejected if any keyword exceeds 32766bytes.logsdbindices8191Documents are never rejected. Keywords exceding this limit are still kept in _source, but won’t be searchable or aggregatable.
There was a problem hiding this comment.
Ahh I see this is in definition list already, so maybe a table won't work. But if you like my wording you can update accordingly.
There was a problem hiding this comment.
Me like that wording :)
| * The [dynamic mapping](docs-content://manage-data/data-store/mapping/dynamic-mapping.md) for string fields | ||
| defaults to a `text` field with a [sub](/reference/elasticsearch/mapping-reference/multi-fields.md)-`keyword` | ||
| field with an `ignore_above` of `256`. String fields longer than 256 characters are available for full text | ||
| search but won't have a value in their `.keyword` sub-field they can not do exact matching over _search. |
There was a problem hiding this comment.
This part I struggle to understand. But it feels separate from the defaults above? Maybe this can be in a new paragraph. I think you're saying that...
When ES finds a new string field without an explicit mapping, it automatically:
- Maps the field to a text field so the entire value is searchable with full-text search.
- Adds a sub keyword field with
ignore_aboveset to256bytes. This means that values less than 256 bytes are available for exact matching over_search. Values longer than that are still searchable via thetextfield, but are not indexed as keywords.
This concept is complicated.
Closes #128991