From 3f41c24f518ed5d25d007eeae555d3a48a7d88ad Mon Sep 17 00:00:00 2001 From: Florent Le Borgne Date: Mon, 3 Feb 2025 11:51:51 +0100 Subject: [PATCH 1/7] add base 'write to scratch' content to Explore and Analyze --- explore-analyze/ai-assistant.md | 36 ++++++++++- explore-analyze/alerts.md | 25 +++++++- explore-analyze/index.md | 59 +++++++++++++++++++ explore-analyze/query-filter/filtering.md | 44 +++++++++++++- explore-analyze/query-filter/tools.md | 10 +++- .../query-filter/tools/playground.md | 2 +- 6 files changed, 169 insertions(+), 7 deletions(-) diff --git a/explore-analyze/ai-assistant.md b/explore-analyze/ai-assistant.md index 402639cd7..67749b817 100644 --- a/explore-analyze/ai-assistant.md +++ b/explore-analyze/ai-assistant.md @@ -23,4 +23,38 @@ mapped_urls: % - [ ] ./raw-migrated-files/docs-content/serverless/security-ai-assistant.md % - [ ] ./raw-migrated-files/docs-content/serverless/ai-assistant-knowledge-base.md -$$$token-limits$$$ \ No newline at end of file +$$$token-limits$$$ + +**AI Assistant** is a chat-based interactive tool that uses generative AI and ELSER, Elastic’s proprietary semantic search model, to help you with a variety of tasks related to Elasticsearch and Kibana, including: + +- **Constructing queries**: Assists you in building queries to search and analyze your data, including converting queries from other languages to [ES|QL](query-filter/languages/esql-rest.md). +- **Indexing data**: Guides you on how to index data into Elasticsearch. +- **Using APIs**: Calls Elasticsearch APIs on your behalf if you need specific operations performed. +- **Generating sample data**: Helps you create sample data for testing and development purposes. +- **Visualizing and analyzing data**: Assists you in creating visualizations and analyzing your data using Kibana. +- **Troubleshooting**: Explains errors, messages, and suggests remediation. + +AI Assistant requires specific privileges and a generative AI connector. + +% Check [Configure AI Assistant](../deploy-manage/) for more details on how to enable and configure it. + +The capabilities and ways to interact with AI Assistant can differ for each solution. Find more information in the respective solution docs: + +% - [AI Assistant for Search](../solutions/search/) +- [AI Assistant for Observability](../solutions/observability/observability-ai-assistant.md) +- [AI Assistant for Security](../solutions/security/ai/ai-assistant.md) + +## Prompt best practices [rag-for-esql] +Elastic AI Assistant allows you to take full advantage of the Elastic platform to improve your operations. It can help you write an ES|QL query for a particular use case, or answer general questions about how to use the platform. Its ability to assist you depends on the specificity and detail of your questions. The more context and detail you provide, the more tailored and useful its responses will be. + +To maximize its usefulness, consider using more detailed prompts or asking for additional information. For instance, after asking for an ES|QL query example, you could ask a follow-up question like, “Could you give me some other examples?” You can also ask for clarification or further exposition, for example "Please provide comments explaining the query you just gave." + +In addition to practical advice, AI Assistant can offer conceptual advice, tips, and best practices for enhancing your security measures. You can ask it, for example: + +- “How do I set up a machine learning job in Elastic Security to detect anomalies in network traffic volume over time?” +- “I need to monitor for unusual file creation patterns that could indicate ransomware activity. How would I construct this query using EQL?” + +## Your data and AI Assistant [ai-assistant-data-information] +Elastic does not use customer data for model training. This includes anything you send the model, such as alert or event data, detection rule configurations, queries, and prompts. However, any data you provide to AI Assistant will be processed by the third-party provider you chose when setting up the generative AI connector as part of the assistant setup. + +Elastic does not control third-party tools, and assumes no responsibility or liability for their content, operation, or use, nor for any loss or damage that may arise from your using such tools. Please exercise caution when using AI tools with personal, sensitive, or confidential information. Any data you submit may be used by the provider for AI training or other purposes. There is no guarantee that the provider will keep any information you provide secure or confidential. You should familiarize yourself with the privacy practices and terms of use of any generative AI tools prior to use. diff --git a/explore-analyze/alerts.md b/explore-analyze/alerts.md index 1ea124ca8..2cb23e539 100644 --- a/explore-analyze/alerts.md +++ b/explore-analyze/alerts.md @@ -15,4 +15,27 @@ mapped_urls: $$$alerting-concepts-actions$$$ -$$$alerting-concepts-conditions$$$ \ No newline at end of file +$$$alerting-concepts-conditions$$$ + +Alerting tools in Elasticsearch and Kibana provide functionality to monitor data and notify you about significant changes or events in real time. This page provides an overview of how the key components work. + +## Alerts +Alerts are notifications generated when specific conditions are met. These notifications are sent to you through channels that you previously set such as email, Slack, webhooks, PagerDuty, and so on. Alerts are created based on rules, which define the criteria for triggering them. Rules monitor the data indexed in Elasticsearch and evaluate conditions on a defined schedule to identify matches. For example, a threshold rule can generate an alert when a value crosses a specific threshold, while a machine learning rule activates an alert when an anomaly detection job identifies an anomaly. + +## Cases +Cases are a collaboration and tracking tool, which is particularly useful for incidents or issues that arise from alerts. You can group related alerts into a case for easier management, add notes and comments to provide context, track investigation progress, and assign cases to team members or link them to external systems. Cases ensure that teams have a central place to track and resolve alerts efficiently. + +## Maintenance windows +If you have a planned outage, maintenance windows prevent rules from generating notifications in that period. Alerts still occur but their notifications are suppressed. + +### Workflow Example + +1. **Rule Creation**: You set up a rule to monitor server logs for failed login attempts exceeding 5 within a 10-minute window. +1. **Alert Generation**: When the rule's condition is met, an alert is created. +1. **Notification**: The alert runs an action, such as sending a Slack message or an email, unless a maintenance window is active. +1. **Case Management**: If the alert is part of an ongoing investigation, it's added to a case for further analysis and resolution. + +By combining these tools, Elasticsearch and Kibana enable incident response workflows, helping teams to detect, investigate, and resolve issues efficiently. + +## Watcher +You can use Watcher for alerting and monitoring specific conditions in your data. It enables you to define rules and take automated actions when certain criteria are met. Watcher is a powerful alerting tool for custom use cases and more complex alerting logic. It allows advanced scripting using Painless to define complex conditions and transformations. diff --git a/explore-analyze/index.md b/explore-analyze/index.md index d8428fa71..0a2495eb9 100644 --- a/explore-analyze/index.md +++ b/explore-analyze/index.md @@ -16,3 +16,62 @@ mapped_urls: % Internal links rely on the following IDs being on this page (e.g. as a heading ID, paragraph ID, etc): $$$elasticsearch-explore-your-data-visualizations-save-to-the-visualize-library$$$ + +The Elasticsearch platform and its UI, also known as Kibana, provide a comprehensive suite of tools to help you search, interact with, explore, and analyze your data effectively. These features empower you to gain deep insights, uncover trends, and take actionable steps based on your findings. This page is an overview of the key capabilities. + +## Querying and filtering +Elasticsearch’s robust query capabilities enable you to retrieve specific data from your datasets. Using the Query DSL (Domain Specific Language), you can build powerful, flexible queries that support: + +- Full-text search +- Boolean logic +- Fuzzy matching +- Proximity searches +- Semantic search +- …and more. + +These tools simplify refining searches and pinpointing relevant information in real-time. + +## Scripting +Scripting makes custom data manipulation and transformation possible during search and aggregation processes. Using scripting languages like Painless, you can calculate custom metrics, perform conditional logic, or adjust data dynamically in search time. This flexibility ensures tailored insights specific to your needs. + +## Aggregations +Aggregations provide advanced data analysis, enabling you to extract actionable insights. With aggregations, you can calculate statistical metrics (for example, sums, averages, medians), group data into buckets (histograms, terms, and so on), or perform nested and multi-level analyses. Aggregations transform raw data into structured insights with ease. + +## Geospatial Analysis +The geospatial capabilities enable analysis of location-based data, including distance calculations, polygon and bounding box queries, and geohash grid aggregations. This functionality is necessary for logistics, real estate, and IoT industries, where location matters. + +## Machine Learning +Elasticsearch integrates machine learning for proactive analytics, helping you to: +- Detect anomalies in time-series data +- Forecast future trends +- Analyze seasonal patterns +- Perform powerful NLP operations such as semantic search +- Machine learning models simplify complex predictive tasks, unlocking new opportunities for optimization. + +## Discover +Discover lets you interact directly with raw data. Use Discover to: +- Browse documents in your indices +- Apply filters and search queries +- Visualize results in real-time + +It’s the starting point for exploratory analysis. + +## Dashboards +Dashboards serve as centralized hubs for visualizing and monitoring data insights. With Dashboards, you can: +- Combine multiple visualizations into a single, unified view +- Display data from multiple indices or datasets for comprehensive analysis +- Customize layouts to suit specific workflows and preferences + +Dashboards provide an interactive and cohesive environment to explore trends and metrics at a glance. + +## Panels and visualizations +Panels and visualizations are the core elements that populate your dashboards, enabling dynamic data representation. They support diverse chart types, Interactive filtering, and drill-down capabilities to explore data further. These building blocks transform raw data into clear, actionable visuals, allowing users to analyze and interpret results effectively. + +## Reporting and sharing +You can share your work and findings with colleagues and stakeholders or generate reports. Report generation can be scheduled or on-demand. You can choose from multiple formats (for example, PDF, CSV). These tools ensure that actionable insights reach the right people at the right time. +Alerting +You can set up alerts to monitor your data continuously. Alerts notify you when specific conditions are met. This ensures timely action on critical issues. + +## Bringing it all together +Elasticsearch's features integrate seamlessly, offering an end-to-end solution for exploring, analyzing, and acting on data. If you want to explore any of the listed features in greater depth, refer to their respective documentation pages and check the provided hands-on examples and tutorials. + diff --git a/explore-analyze/query-filter/filtering.md b/explore-analyze/query-filter/filtering.md index 1b8aa6b59..60cab1292 100644 --- a/explore-analyze/query-filter/filtering.md +++ b/explore-analyze/query-filter/filtering.md @@ -4,7 +4,7 @@ mapped_urls: - https://www.elastic.co/guide/en/kibana/current/set-time-filter.html --- -# Filtering +# Filtering in Kibana % What needs to be done: Write from scratch @@ -15,4 +15,44 @@ mapped_urls: % Internal links rely on the following IDs being on this page (e.g. as a heading ID, paragraph ID, etc): -$$$_finding_your_apps_and_objects$$$ \ No newline at end of file +$$$_finding_your_apps_and_objects$$$ + +This page describes the common ways Kibana offers in most apps for filtering data and refining your initial search queries. + +Some apps provide more options, such as [Dashboards](../dashboards.md). + +## Time filter [set-time-filter] + +Display data within a specified time range when your index contains time-based events, and a time-field is configured for the selected [{{data-source}}](../find-and-organize/data-views.md). The default time range is 15 minutes, but you can customize it in [Advanced Settings](https://www.elastic.co/guide/en/kibana/current/advanced-options.html). + +1. Click ![calendar icon](../../../images/kibana-time-filter-icon.png ""). +2. Choose one of the following: + + * **Quick select**. Set a time based on the last or next number of seconds, minutes, hours, or other time unit. + * **Commonly used**. Select a time range from options such as **Last 15 minutes**, **Today**, and **Week to date**. + * **Recently used date ranges**. Use a previously selected data range. + * **Refresh every**. Specify an automatic refresh rate. + + :::{image} ../../../images/kibana-time-filter.png + :alt: Time filter menu + :width: 300px + ::: + +3. To set start and end times, click the bar next to the time filter. In the popup, select **Absolute**, **Relative** or **Now**, then specify the required options. + + :::{image} ../../../images/kibana-time-relative.png + :alt: Time filter showing relative time + :class: screenshot + ::: + +The global time filter limits the time range of data displayed. In most cases, the time filter applies to the time field in the data view, but some apps allow you to use a different time field. + +Using the time filter, you can configure a refresh rate to periodically resubmit your searches. + +To manually resubmit a search, click the **Refresh** button. This is useful when you use Kibana to view the underlying data. + +## Additional filters [autocomplete-suggestions] + +Structured filters are a more interactive way to create {{es}} queries, and are commonly used when building dashboards that are shared by multiple analysts. Each filter can be disabled, inverted, or pinned across all apps. Each of the structured filters is combined with AND logic on the rest of the query. + +![Add filter popup](../../../images/kibana-add-filter-popup.png "") \ No newline at end of file diff --git a/explore-analyze/query-filter/tools.md b/explore-analyze/query-filter/tools.md index e766a5da1..b98329c5e 100644 --- a/explore-analyze/query-filter/tools.md +++ b/explore-analyze/query-filter/tools.md @@ -5,12 +5,18 @@ mapped_pages: # Query tools [devtools-kibana] -**Dev Tools** contains tools that you can use to interact with your data. +Elasticsearch offers tools that you can use to query your data, manage those queries, and optimize them to be as efficient as possible. | | | | --- | --- | -| [Console](tools/console.md) | Interact with the REST APIs of {{es}} and {{kib}}, including sending requestsand viewing API documentation. | +| [Saved queries](tools/saved-queries.md) | Save your searches and queries to reuse them later. | +| [Console](tools/console.md) | Interact with the REST APIs of {{es}} and {{kib}}, including sending requests and viewing API documentation. | | [{{searchprofiler}}](tools/search-profiler.md) | Inspect and analyze your search queries. | | [Grok Debugger   ](tools/grok-debugger.md) | Build and debug grok patterns before you use them in your data processing pipelines. | | [Painless Lab](../scripting/painless-lab.md) | [beta] Test and debug Painless scripts in real-time. | +| [Playground](tools/playground.md) | Combine your Elasticsearch data with the power of large language models (LLMs) for retrieval augmented generation (RAG), using a chat interface. | + + + + diff --git a/explore-analyze/query-filter/tools/playground.md b/explore-analyze/query-filter/tools/playground.md index 6019c9e31..2a7785f7d 100644 --- a/explore-analyze/query-filter/tools/playground.md +++ b/explore-analyze/query-filter/tools/playground.md @@ -10,6 +10,6 @@ Use the Search Playground to test and edit {{es}} queries visually in the UI. Th Find Playground in the {{es-serverless}} UI under **{{es}} > Build > Playground**. ::::{note} -ℹ️ The Playground documentation currently lives in the [{{kib}} docs](../../../solutions/search/rag/playground.md). +ℹ️ For more details, check the full [Playground documentation](../../../solutions/search/rag/playground.md). :::: From a4e3c664c3e7ee0c6e6fa523a63a3547942ea9cb Mon Sep 17 00:00:00 2001 From: Florent Le Borgne Date: Mon, 3 Feb 2025 12:26:31 +0100 Subject: [PATCH 2/7] img paths --- explore-analyze/query-filter/filtering.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/explore-analyze/query-filter/filtering.md b/explore-analyze/query-filter/filtering.md index 60cab1292..d39dfff4a 100644 --- a/explore-analyze/query-filter/filtering.md +++ b/explore-analyze/query-filter/filtering.md @@ -25,7 +25,7 @@ Some apps provide more options, such as [Dashboards](../dashboards.md). Display data within a specified time range when your index contains time-based events, and a time-field is configured for the selected [{{data-source}}](../find-and-organize/data-views.md). The default time range is 15 minutes, but you can customize it in [Advanced Settings](https://www.elastic.co/guide/en/kibana/current/advanced-options.html). -1. Click ![calendar icon](../../../images/kibana-time-filter-icon.png ""). +1. Click ![calendar icon](../../images/kibana-time-filter-icon.png). 2. Choose one of the following: * **Quick select**. Set a time based on the last or next number of seconds, minutes, hours, or other time unit. @@ -55,4 +55,4 @@ To manually resubmit a search, click the **Refresh** button. This is useful when Structured filters are a more interactive way to create {{es}} queries, and are commonly used when building dashboards that are shared by multiple analysts. Each filter can be disabled, inverted, or pinned across all apps. Each of the structured filters is combined with AND logic on the rest of the query. -![Add filter popup](../../../images/kibana-add-filter-popup.png "") \ No newline at end of file +![Add filter popup](../../images/kibana-add-filter-popup.png "") \ No newline at end of file From d5ac8e3ff39a9b4f783bcc01a644e6ae8159d7ab Mon Sep 17 00:00:00 2001 From: Florent Le Borgne Date: Tue, 4 Feb 2025 13:13:00 +0100 Subject: [PATCH 3/7] query languages intro, query dsl, and esql --- .../alerts/kibana/rule-type-es-query.md | 2 +- explore-analyze/discover/try-esql.md | 2 +- explore-analyze/geospatial-analysis.md | 2 +- explore-analyze/query-filter.md | 86 +-------- explore-analyze/query-filter/filtering.md | 6 +- explore-analyze/query-filter/languages.md | 82 +-------- .../query-filter/languages/esorql.md | 22 --- .../query-filter/languages}/esql-examples.md | 0 .../languages}/esql-getting-started.md | 14 +- .../query-filter/languages/esql-kibana.md | 64 ++++--- .../query-filter/languages/esql-rest.md | 10 +- .../query-filter/languages/esql.md | 68 ++++++++ .../query-filter/languages/querydsl.md | 165 +++++++++++++++++- explore-analyze/toc.yml | 4 +- .../serverless/security-about-rules.md | 2 +- .../serverless/security-rules-create.md | 2 +- .../serverless/security-timelines-ui.md | 2 +- .../elasticsearch-reference/esql.md | 32 ---- .../elasticsearch-reference/query-dsl.md | 39 ----- .../query-filter-context.md | 93 ---------- .../elasticsearch-reference/search-analyze.md | 4 +- raw-migrated-files/kibana/kibana/esql.md | 2 +- .../kibana/kibana/managing-data-views.md | 2 +- .../kibana/kibana/search-ai-assistant.md | 2 +- .../security-docs/security/about-rules.md | 2 +- .../security-docs/security/rules-ui-create.md | 2 +- .../security-docs/security/timelines-ui.md | 2 +- .../create-an-elasticsearch-query-rule.md | 2 +- 28 files changed, 313 insertions(+), 402 deletions(-) delete mode 100644 explore-analyze/query-filter/languages/esorql.md rename {raw-migrated-files/elasticsearch/elasticsearch-reference => explore-analyze/query-filter/languages}/esql-examples.md (100%) rename {raw-migrated-files/elasticsearch/elasticsearch-reference => explore-analyze/query-filter/languages}/esql-getting-started.md (96%) create mode 100644 explore-analyze/query-filter/languages/esql.md delete mode 100644 raw-migrated-files/elasticsearch/elasticsearch-reference/esql.md delete mode 100644 raw-migrated-files/elasticsearch/elasticsearch-reference/query-dsl.md delete mode 100644 raw-migrated-files/elasticsearch/elasticsearch-reference/query-filter-context.md diff --git a/explore-analyze/alerts/kibana/rule-type-es-query.md b/explore-analyze/alerts/kibana/rule-type-es-query.md index ebf5fdb75..727b65871 100644 --- a/explore-analyze/alerts/kibana/rule-type-es-query.md +++ b/explore-analyze/alerts/kibana/rule-type-es-query.md @@ -37,7 +37,7 @@ When you create an {{es}} query rule, your choice of query type affects the info If you use [KQL](../../query-filter/languages/kql.md) or [Lucene](../../query-filter/languages/lucene-query-syntax.md), you must specify a data view then define a text-based query. For example, `http.request.referrer: "https://example.com"`. - If you use [ES|QL](../../query-filter/languages/esorql.md), you must provide a source command followed by an optional series of processing commands, separated by pipe characters (|). [8.16.0] For example: + If you use [ES|QL](../../query-filter/languages/esql.md), you must provide a source command followed by an optional series of processing commands, separated by pipe characters (|). [8.16.0] For example: ```sh FROM kibana_sample_data_logs diff --git a/explore-analyze/discover/try-esql.md b/explore-analyze/discover/try-esql.md index 5fa8c554d..91b7a75a0 100644 --- a/explore-analyze/discover/try-esql.md +++ b/explore-analyze/discover/try-esql.md @@ -10,7 +10,7 @@ The Elasticsearch Query Language, {{esql}}, makes it easier to explore your data In this tutorial we’ll use the {{kib}} sample web logs in Discover and Lens to explore the data and create visualizations. ::::{tip} -For the complete {{esql}} documentation, including tutorials, examples and the full syntax reference, refer to the [{{es}} documentation](../query-filter/languages/esorql.md). For a more detailed overview of {{esql}} in {{kib}}, refer to [Use {{esql}} in Kibana](../query-filter/languages/esql-kibana.md). +For the complete {{esql}} documentation, including tutorials, examples and the full syntax reference, refer to the [{{es}} documentation](../query-filter/languages/esql.md). For a more detailed overview of {{esql}} in {{kib}}, refer to [Use {{esql}} in Kibana](../query-filter/languages/esql-kibana.md). :::: diff --git a/explore-analyze/geospatial-analysis.md b/explore-analyze/geospatial-analysis.md index ddee42ece..7100bc159 100644 --- a/explore-analyze/geospatial-analysis.md +++ b/explore-analyze/geospatial-analysis.md @@ -34,7 +34,7 @@ Data is often messy and incomplete. [Ingest pipelines](../manage-data/ingest/tra ## ES|QL [esql-query] -[ES|QL](query-filter/languages/esorql.md) has support for [Geospatial Search](https://www.elastic.co/guide/en/elasticsearch/reference/current/esql-functions-operators.html#esql-spatial-functions) functions, enabling efficient index searching for documents that intersect with, are within, are contained by, or are disjoint from a query geometry. In addition, the `ST_DISTANCE` function calculates the distance between two points. +[ES|QL](query-filter/languages/esql.md) has support for [Geospatial Search](https://www.elastic.co/guide/en/elasticsearch/reference/current/esql-functions-operators.html#esql-spatial-functions) functions, enabling efficient index searching for documents that intersect with, are within, are contained by, or are disjoint from a query geometry. In addition, the `ST_DISTANCE` function calculates the distance between two points. * [`ST_INTERSECTS`](https://www.elastic.co/guide/en/elasticsearch/reference/current/esql-functions-operators.html#esql-st_intersects) * [`ST_DISJOINT`](https://www.elastic.co/guide/en/elasticsearch/reference/current/esql-functions-operators.html#esql-st_disjoint) diff --git a/explore-analyze/query-filter.md b/explore-analyze/query-filter.md index eda1430d7..fd1db0818 100644 --- a/explore-analyze/query-filter.md +++ b/explore-analyze/query-filter.md @@ -7,88 +7,20 @@ mapped_pages: You can use {{es}} as a basic document store to retrieve documents and their metadata. However, the real power of {{es}} comes from its advanced search and analytics capabilities. -You’ll use a combination of an API endpoint and a query language to interact with your data. - - -## REST API [search-analyze-rest-api] - -Use REST APIs to manage your {{es}} cluster, and to index and search your data. For testing purposes, you can submit requests directly from the command line or through the Dev Tools [Console](query-filter/tools/console.md) in {{kib}}. From your applications, you can use a [client](https://www.elastic.co/guide/en/elasticsearch/client/index.md) in your programming language of choice. - -Refer to [first steps with Elasticsearch](../solutions/search/get-started.md) for a hands-on example of using the `_search` endpoint, adding data to {{es}}, and running basic searches in Query DSL syntax. - - -## Query languages [search-analyze-query-languages] - -{{es}} provides a number of query languages for interacting with your data. - -**Query DSL** is the primary query language for {{es}} today. - -**{{esql}}** is a new piped query language and compute engine which was first added in version **8.11**. - -{{esql}} does not yet support all the features of Query DSL. Look forward to new {{esql}} features and functionalities in each release. - -Refer to [Query languages](#search-analyze-query-languages) for a full overview of the query languages available in {{es}}. - - -### Query DSL [search-analyze-query-dsl] - -[Query DSL](query-filter/languages/querydsl.md) is a full-featured JSON-style query language that enables complex searching, filtering, and aggregations. It is the original and most powerful query language for {{es}} today. - -The [`_search` endpoint](../solutions/search/querying-for-search-searching-with-the-search-api.md) accepts queries written in Query DSL syntax. - +## Querying -#### Search and filter with Query DSL [search-analyze-query-dsl-search-filter] - -Query DSL support a wide range of search techniques, including the following: - -* [**Full-text search**](../solutions/search/full-text.md): Search text that has been analyzed and indexed to support phrase or proximity queries, fuzzy matches, and more. -* [**Keyword search**](https://www.elastic.co/guide/en/elasticsearch/reference/current/keyword.html): Search for exact matches using `keyword` fields. -* [**Semantic search**](../solutions/search/semantic-search/semantic-search-semantic-text.md): Search `semantic_text` fields using dense or sparse vector search on embeddings generated in your {{es}} cluster. -* [**Vector search**](../solutions/search/vector/knn.md): Search for similar dense vectors using the kNN algorithm for embeddings generated outside of {{es}}. -* [**Geospatial search**](https://www.elastic.co/guide/en/elasticsearch/reference/current/geo-queries.html): Search for locations and calculate spatial relationships using geospatial queries. - -Learn about the full range of queries supported by [Query DSL](query-filter/languages/querydsl.md). - -You can also filter data using Query DSL. Filters enable you to include or exclude documents by retrieving documents that match specific field-level criteria. A query that uses the `filter` parameter indicates [filter context](query-filter/languages/querydsl.md#filter-context). - - -#### Analyze with Query DSL [search-analyze-data-query-dsl] - -[Aggregations](aggregations.md) are the primary tool for analyzing {{es}} data using Query DSL. Aggregrations enable you to build complex summaries of your data and gain insight into key metrics, patterns, and trends. - -Because aggregations leverage the same data structures used for search, they are also very fast. This enables you to analyze and visualize your data in real time. You can search documents, filter results, and perform analytics at the same time, on the same data, in a single request. That means aggregations are calculated in the context of the search query. - -The folowing aggregation types are available: - -* [Metric](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-metrics.html): Calculate metrics, such as a sum or average, from field values. -* [Bucket](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket.html): Group documents into buckets based on field values, ranges, or other criteria. -* [Pipeline](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-pipeline.html): Run aggregations on the results of other aggregations. - -Run aggregations by specifying the [search API](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html)'s `aggs` parameter. Learn more in [Run an aggregation](aggregations.md#run-an-agg). - - -### {{esql}} [search-analyze-data-esql] - -[Elasticsearch Query Language ({{esql}})](query-filter/languages/esorql.md) is a piped query language for filtering, transforming, and analyzing data. {{esql}} is built on top of a new compute engine, where search, aggregation, and transformation functions are directly executed within {{es}} itself. {{esql}} syntax can also be used within various {{kib}} tools. - -The [`_query` endpoint](query-filter/languages/esql-rest.md) accepts queries written in {{esql}} syntax. - -Today, it supports a subset of the features available in Query DSL, but it is rapidly evolving. +You’ll use a combination of an API endpoint and a query language to interact with your data. -It comes with a comprehensive set of [functions and operators](https://www.elastic.co/guide/en/elasticsearch/reference/current/esql-functions-operators.html) for working with data and has robust integration with {{kib}}'s Discover, dashboards and visualizations. +- Elasticsearch provides a number of [query languages](/explore-analyze/query-filter/languages.md). From Query DSL to the newest ES|QL, find the one that's most appropriate for you. -Learn more in [Getting started with {{esql}}](../solutions/search/get-started.md), or try [our training course](https://www.elastic.co/training/introduction-to-esql). +- You can call Elasticsearch's REST APIs by submitting requests directly from the command line or through the Dev Tools [Console](/explore-analyze/query-filter/tools/console.md) in {{kib}}. From your applications, you can use a [client](https://www.elastic.co/guide/en/elasticsearch/client/index.md) in your programming language of choice. +- A number of [tools](/explore-analyze/query-filter/tools.md) are available for you to save, debug, and optimize your queries. -## List of available query languages [search-analyze-data-query-languages-table] +% todo: update link to the best target +% > If you're just getting started with Elasticsearch, check [first steps with Elasticsearch](/solutions/search/get-started.md) for a hands-on example of using the `_search` endpoint, adding data to {{es}}, and running basic searches in Query DSL syntax. -The following table summarizes all available {{es}} query languages, to help you choose the right one for your use case. +## Filtering -| Name | Description | Use cases | API endpoint | -| --- | --- | --- | --- | -| [Query DSL](query-filter/languages/querydsl.md) | The primary query language for {{es}}. A powerful and flexible JSON-style language that enables complex queries. | Full-text search, semantic search, keyword search, filtering, aggregations, and more. | [`_search`](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html) | -| [{{esql}}](query-filter/languages/esorql.md) | Introduced in **8.11**, the Elasticsearch Query Language ({{esql}}) is a piped query language language for filtering, transforming, and analyzing data. | Initially tailored towards working with time series data like logs and metrics.Robust integration with {{kib}} for querying, visualizing, and analyzing data.Does not yet support full-text search. | [`_query`](query-filter/languages/esql-rest.md) | -| [EQL](query-filter/languages/eql.md) | Event Query Language (EQL) is a query language for event-based time series data. Data must contain the `@timestamp` field to use EQL. | Designed for the threat hunting security use case. | [`_eql`](https://www.elastic.co/guide/en/elasticsearch/reference/current/eql-apis.html) | -| [Elasticsearch SQL](query-filter/languages/sql.md) | Allows native, real-time SQL-like querying against {{es}} data. JDBC and ODBC drivers are available for integration with business intelligence (BI) tools. | Enables users familiar with SQL to query {{es}} data using familiar syntax for BI and reporting. | [`_sql`](https://www.elastic.co/guide/en/elasticsearch/reference/current/sql-apis.html) | -| [Kibana Query Language (KQL)](query-filter/languages/kql.md) | {{kib}} Query Language (KQL) is a text-based query language for filtering data when you access it through the {{kib}} UI. | Use KQL to filter documents where a value for a field exists, matches a given value, or is within a given range. | N/A | +When querying your data in Kibana, additional options let you filter the results to just the subset you need. Some of these options are common to most Elastic apps. Check [Filtering in Kibana](/explore-analyze/query-filter/filtering.md) for more details on how to recognize and use them in the UI. diff --git a/explore-analyze/query-filter/filtering.md b/explore-analyze/query-filter/filtering.md index d39dfff4a..5fa940bc9 100644 --- a/explore-analyze/query-filter/filtering.md +++ b/explore-analyze/query-filter/filtering.md @@ -32,15 +32,13 @@ Display data within a specified time range when your index contains time-based e * **Commonly used**. Select a time range from options such as **Last 15 minutes**, **Today**, and **Week to date**. * **Recently used date ranges**. Use a previously selected data range. * **Refresh every**. Specify an automatic refresh rate. - - :::{image} ../../../images/kibana-time-filter.png + :::{image} ../../images/kibana-time-filter.png :alt: Time filter menu :width: 300px ::: 3. To set start and end times, click the bar next to the time filter. In the popup, select **Absolute**, **Relative** or **Now**, then specify the required options. - - :::{image} ../../../images/kibana-time-relative.png + :::{image} ../../images/kibana-time-relative.png :alt: Time filter showing relative time :class: screenshot ::: diff --git a/explore-analyze/query-filter/languages.md b/explore-analyze/query-filter/languages.md index e8e210a66..832fe2ebb 100644 --- a/explore-analyze/query-filter/languages.md +++ b/explore-analyze/query-filter/languages.md @@ -3,92 +3,18 @@ mapped_pages: - https://www.elastic.co/guide/en/elasticsearch/reference/current/search-analyze.html --- -# Query languages [search-analyze] - -You can use {{es}} as a basic document store to retrieve documents and their metadata. However, the real power of {{es}} comes from its advanced search and analytics capabilities. - -You’ll use a combination of an API endpoint and a query language to interact with your data. - - -## REST API [search-analyze-rest-api] - -Use REST APIs to manage your {{es}} cluster, and to index and search your data. For testing purposes, you can submit requests directly from the command line or through the Dev Tools [Console](tools/console.md) in {{kib}}. From your applications, you can use a [client](https://www.elastic.co/guide/en/elasticsearch/client/index.html) in your programming language of choice. - -Refer to [first steps with Elasticsearch](../../solutions/search/get-started.md) for a hands-on example of using the `_search` endpoint, adding data to {{es}}, and running basic searches in Query DSL syntax. - - -## Query languages [search-analyze-query-languages] +# Query languages [search-analyze-query-languages] {{es}} provides a number of query languages for interacting with your data. -**Query DSL** is the primary query language for {{es}} today. - -**{{esql}}** is a new piped query language and compute engine which was first added in version **8.11**. - -{{esql}} does not yet support all the features of Query DSL. Look forward to new {{esql}} features and functionalities in each release. - -Refer to [Query languages](../query-filter.md#search-analyze-query-languages) for a full overview of the query languages available in {{es}}. - - -### Query DSL [search-analyze-query-dsl] - -[Query DSL](languages/querydsl.md) is a full-featured JSON-style query language that enables complex searching, filtering, and aggregations. It is the original and most powerful query language for {{es}} today. - -The [`_search` endpoint](../../solutions/search/querying-for-search-searching-with-the-search-api.md) accepts queries written in Query DSL syntax. - - -#### Search and filter with Query DSL [search-analyze-query-dsl-search-filter] - -Query DSL support a wide range of search techniques, including the following: - -* [**Full-text search**](../../solutions/search/full-text.md): Search text that has been analyzed and indexed to support phrase or proximity queries, fuzzy matches, and more. -* [**Keyword search**](https://www.elastic.co/guide/en/elasticsearch/reference/current/keyword.html): Search for exact matches using `keyword` fields. -* [**Semantic search**](../../solutions/search/semantic-search/semantic-search-semantic-text.md): Search `semantic_text` fields using dense or sparse vector search on embeddings generated in your {{es}} cluster. -* [**Vector search**](../../solutions/search/vector/knn.md): Search for similar dense vectors using the kNN algorithm for embeddings generated outside of {{es}}. -* [**Geospatial search**](https://www.elastic.co/guide/en/elasticsearch/reference/current/geo-queries.html): Search for locations and calculate spatial relationships using geospatial queries. - -Learn about the full range of queries supported by [Query DSL](languages/querydsl.md). - -You can also filter data using Query DSL. Filters enable you to include or exclude documents by retrieving documents that match specific field-level criteria. A query that uses the `filter` parameter indicates [filter context](languages/querydsl.md#filter-context). - - -#### Analyze with Query DSL [search-analyze-data-query-dsl] - -[Aggregations](../aggregations.md) are the primary tool for analyzing {{es}} data using Query DSL. Aggregrations enable you to build complex summaries of your data and gain insight into key metrics, patterns, and trends. - -Because aggregations leverage the same data structures used for search, they are also very fast. This enables you to analyze and visualize your data in real time. You can search documents, filter results, and perform analytics at the same time, on the same data, in a single request. That means aggregations are calculated in the context of the search query. - -The folowing aggregation types are available: - -* [Metric](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-metrics.html): Calculate metrics, such as a sum or average, from field values. -* [Bucket](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket.html): Group documents into buckets based on field values, ranges, or other criteria. -* [Pipeline](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-pipeline.html): Run aggregations on the results of other aggregations. - -Run aggregations by specifying the [search API](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html)'s `aggs` parameter. Learn more in [Run an aggregation](../aggregations.md#run-an-agg). - - -### {{esql}} [search-analyze-data-esql] - -[Elasticsearch Query Language ({{esql}})](languages/esorql.md) is a piped query language for filtering, transforming, and analyzing data. {{esql}} is built on top of a new compute engine, where search, aggregation, and transformation functions are directly executed within {{es}} itself. {{esql}} syntax can also be used within various {{kib}} tools. - -The [`_query` endpoint](languages/esql-rest.md) accepts queries written in {{esql}} syntax. - -Today, it supports a subset of the features available in Query DSL, but it is rapidly evolving. - -It comes with a comprehensive set of [functions and operators](https://www.elastic.co/guide/en/elasticsearch/reference/current/esql-functions-operators.html) for working with data and has robust integration with {{kib}}'s Discover, dashboards and visualizations. - -Learn more in [Getting started with {{esql}}](../../solutions/search/get-started.md), or try [our training course](https://www.elastic.co/training/introduction-to-esql). - - -## List of available query languages [search-analyze-data-query-languages-table] - -The following table summarizes all available {{es}} query languages, to help you choose the right one for your use case. | Name | Description | Use cases | API endpoint | | --- | --- | --- | --- | | [Query DSL](languages/querydsl.md) | The primary query language for {{es}}. A powerful and flexible JSON-style language that enables complex queries. | Full-text search, semantic search, keyword search, filtering, aggregations, and more. | [`_search`](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html) | -| [{{esql}}](languages/esorql.md) | Introduced in **8.11**, the Elasticsearch Query Language ({{esql}}) is a piped query language language for filtering, transforming, and analyzing data. | Initially tailored towards working with time series data like logs and metrics.Robust integration with {{kib}} for querying, visualizing, and analyzing data.Does not yet support full-text search. | [`_query`](languages/esql-rest.md) | +| [{{esql}}](languages/esql.md) | Introduced in **8.11**, the Elasticsearch Query Language ({{esql}}) is a piped query language language for filtering, transforming, and analyzing data. | Initially tailored towards working with time series data like logs and metrics.Robust integration with {{kib}} for querying, visualizing, and analyzing data.Does not yet support full-text search. | [`_query`](languages/esql-rest.md) | | [EQL](languages/eql.md) | Event Query Language (EQL) is a query language for event-based time series data. Data must contain the `@timestamp` field to use EQL. | Designed for the threat hunting security use case. | [`_eql`](https://www.elastic.co/guide/en/elasticsearch/reference/current/eql-apis.html) | | [Elasticsearch SQL](languages/sql.md) | Allows native, real-time SQL-like querying against {{es}} data. JDBC and ODBC drivers are available for integration with business intelligence (BI) tools. | Enables users familiar with SQL to query {{es}} data using familiar syntax for BI and reporting. | [`_sql`](https://www.elastic.co/guide/en/elasticsearch/reference/current/sql-apis.html) | | [Kibana Query Language (KQL)](languages/kql.md) | {{kib}} Query Language (KQL) is a text-based query language for filtering data when you access it through the {{kib}} UI. | Use KQL to filter documents where a value for a field exists, matches a given value, or is within a given range. | N/A | +> {{esql}} does not yet support all the features of Query DSL. Look forward to new {{esql}} features and functionalities in each release. + diff --git a/explore-analyze/query-filter/languages/esorql.md b/explore-analyze/query-filter/languages/esorql.md deleted file mode 100644 index dfd353ce7..000000000 --- a/explore-analyze/query-filter/languages/esorql.md +++ /dev/null @@ -1,22 +0,0 @@ ---- -mapped_urls: - - https://www.elastic.co/guide/en/elasticsearch/reference/current/esql.html - - https://www.elastic.co/guide/en/elasticsearch/reference/current/esql-getting-started.html - - https://www.elastic.co/guide/en/elasticsearch/reference/current/esql-using.html - - https://www.elastic.co/guide/en/elasticsearch/reference/current/esql-examples.html - - https://www.elastic.co/guide/en/kibana/current/esql.html ---- - -# ES|QL - -% What needs to be done: Refine - -% Scope notes: everything but language reference. Merge the pages about Kibana. Add links to reference's new location - -% Use migrated content from existing pages that map to this page: - -% - [ ] ./raw-migrated-files/elasticsearch/elasticsearch-reference/esql.md -% - [ ] ./raw-migrated-files/elasticsearch/elasticsearch-reference/esql-getting-started.md -% - [ ] ./raw-migrated-files/elasticsearch/elasticsearch-reference/esql-using.md -% - [ ] ./raw-migrated-files/elasticsearch/elasticsearch-reference/esql-examples.md -% - [ ] ./raw-migrated-files/kibana/kibana/esql.md \ No newline at end of file diff --git a/raw-migrated-files/elasticsearch/elasticsearch-reference/esql-examples.md b/explore-analyze/query-filter/languages/esql-examples.md similarity index 100% rename from raw-migrated-files/elasticsearch/elasticsearch-reference/esql-examples.md rename to explore-analyze/query-filter/languages/esql-examples.md diff --git a/raw-migrated-files/elasticsearch/elasticsearch-reference/esql-getting-started.md b/explore-analyze/query-filter/languages/esql-getting-started.md similarity index 96% rename from raw-migrated-files/elasticsearch/elasticsearch-reference/esql-getting-started.md rename to explore-analyze/query-filter/languages/esql-getting-started.md index de5f1450c..0d5e16a64 100644 --- a/raw-migrated-files/elasticsearch/elasticsearch-reference/esql-getting-started.md +++ b/explore-analyze/query-filter/languages/esql-getting-started.md @@ -103,7 +103,7 @@ After switching to {{esql}} mode, the query bar shows a sample query. You can re To make it easier to write queries, auto-complete offers suggestions with possible commands and functions: -:::{image} ../../../images/elasticsearch-reference-esql-kibana-auto-complete.png +:::{image} /images/elasticsearch-reference-esql-kibana-auto-complete.png :alt: esql kibana auto complete ::: @@ -116,7 +116,7 @@ You can adjust the editor’s height by dragging its bottom border to your likin Each {{esql}} query starts with a [source command](https://www.elastic.co/guide/en/elasticsearch/reference/current/esql-commands.html#esql-source-commands). A source command produces a table, typically with data from {{es}}. -:::{image} ../../../images/elasticsearch-reference-source-command.svg +:::{image} /images/elasticsearch-reference-source-command.svg :alt: A source command producing a table from {es} ::: @@ -143,7 +143,7 @@ from sample_data A source command can be followed by one or more [processing commands](https://www.elastic.co/guide/en/elasticsearch/reference/current/esql-commands.html#esql-processing-commands), separated by a pipe character: `|`. Processing commands change an input table by adding, removing, or changing rows and columns. Processing commands can perform filtering, projection, aggregation, and more. -:::{image} ../../../images/elasticsearch-reference-esql-limit.png +:::{image} /images/elasticsearch-reference-esql-limit.png :alt: A processing command changing an input table ::: @@ -167,7 +167,7 @@ FROM sample_data | LIMIT 3 ### Sort a table [esql-getting-started-sort] -:::{image} ../../../images/elasticsearch-reference-esql-sort.png +:::{image} /images/elasticsearch-reference-esql-sort.png :alt: A processing command sorting an input table ::: @@ -205,7 +205,7 @@ There are many other processing commands, like [`KEEP`](https://www.elastic.co/g You can chain processing commands, separated by a pipe character: `|`. Each processing command works on the output table of the previous command. The result of a query is the table produced by the final processing command. -:::{image} ../../../images/elasticsearch-reference-esql-sort-limit.png +:::{image} /images/elasticsearch-reference-esql-sort-limit.png :alt: Processing commands can be chained ::: @@ -308,7 +308,7 @@ FROM sample_data {{esql}} enables you to [enrich](https://www.elastic.co/guide/en/elasticsearch/reference/current/esql-enrich-data.html) a table with data from indices in {{es}}, using the [`ENRICH`](https://www.elastic.co/guide/en/elasticsearch/reference/current/esql-commands.html#esql-enrich) command. -:::{image} ../../../images/elasticsearch-reference-esql-enrich.png +:::{image} /images/elasticsearch-reference-esql-enrich.png :alt: esql enrich ::: @@ -421,5 +421,5 @@ For more about data processing with {{esql}}, refer to [Data processing with DIS ## Learn more [esql-getting-learn-more] -To learn more about {{esql}}, refer to [{{esql}} reference](https://www.elastic.co/guide/en/elasticsearch/reference/current/esql-language.html) and [*Using {{esql}}*](../../../explore-analyze/query-filter/languages/esorql.md). +To learn more about {{esql}}, refer to [{{esql}} reference](https://www.elastic.co/guide/en/elasticsearch/reference/current/esql-language.html). diff --git a/explore-analyze/query-filter/languages/esql-kibana.md b/explore-analyze/query-filter/languages/esql-kibana.md index 30253d483..38944cddc 100644 --- a/explore-analyze/query-filter/languages/esql-kibana.md +++ b/explore-analyze/query-filter/languages/esql-kibana.md @@ -4,13 +4,15 @@ mapped_pages: - https://www.elastic.co/guide/en/elasticsearch/reference/current/esql-kibana.html --- - - # Using ES|QL in Kibana [esql-kibana] - You can use {{esql}} in {{kib}} to query and aggregate your data, create visualizations, and set up alerts. +More specifically, {{esql}} is a powerful tool in Kibana that can help you with specific solution use cases. For example: + +- {{observability}}: {{esql}} makes it much easier to analyze metrics, logs and traces from a single query. Find performance issues fast by defining fields on the fly, enriching data with lookups, and using simultaneous query processing. Combining {{esql}} with {{ml}} and AiOps can improve detection accuracy and use aggregated value thresholds. +- Security: Use {{esql}} to retrieve important information for investigation by using lookups. Enrich data and create new fields on the go to gain valuable insight for faster decision-making and actions. For example, perform a lookup on an IP address to identify its geographical location, its association with known malicious entities, or whether it belongs to a known cloud service provider all from one search bar. {{esql}} ensures more accurate alerts by incorporating aggregated values in detection rules. + This guide shows you how to use {{esql}} in Kibana. To follow along with the queries, load the "Sample web logs" sample data set by selecting **Sample Data** from the **Integrations** page in {{kib}}, selecting **Other sample data sets**, and clicking **Add data** on the **Sample web logs** card. @@ -21,12 +23,12 @@ This guide shows you how to use {{esql}} in Kibana. To follow along with the que This will hide the {{esql}} user interface from various applications. However, users will be able to access existing {{esql}} artifacts like saved searches and visualizations. -## Get started with {{esql}} [esql-kibana-get-started] +## The {{esql}} editor [esql-kibana-get-started] -To get started with {{esql}} in Discover, open the main menu and select **Discover**. Next, select **Try ES|QL** from the application menu bar. +To get started with {{esql}}, go to **Discover**. Next, select **Try ES|QL** from the application menu bar. -## The query bar [esql-kibana-query-bar] +### The query bar [esql-kibana-query-bar] After switching to {{esql}} mode, the query bar shows a sample query. For example: @@ -45,7 +47,7 @@ Click the **ES|QL help** button to open the in-product reference documentation f To make it easier to write queries, auto-complete offers suggestions with possible commands and functions: -:::{image} ../../../images/elasticsearch-reference-esql-kibana-auto-complete.png +:::{image} /images/elasticsearch-reference-esql-kibana-auto-complete.png :alt: esql kibana auto complete ::: @@ -89,10 +91,16 @@ You can reuse your recent {{esql}} queries in the query bar. In the query bar, c You can then scroll through your recent queries: -:::{image} ../../../images/elasticsearch-reference-esql-discover-query-history.png +:::{image} /images/elasticsearch-reference-esql-discover-query-history.png :alt: esql discover query history ::: +### Query help + +{{esql}} features in-app help and suggestions, so you can get started faster and don’t have to leave the application to check syntax. + +![The ES|QL syntax reference and the autocomplete menu](/images/kibana-esql-in-app-help.png "") + ### Starred queries [esql-kibana-starred-queries] @@ -104,12 +112,12 @@ From the **Recent** tab, you can star any queries you want. In the **Starred** tab, find all the queries you have previously starred. -:::{image} ../../../images/elasticsearch-reference-esql-discover-query-starred.png +:::{image} /images/elasticsearch-reference-esql-discover-query-starred.png :alt: esql discover query starred ::: -## The results table [esql-kibana-results-table] +### Organizing the query results [esql-kibana-results-table] For the example query, the results table shows 10 rows. Omitting the `LIMIT` command, the results table defaults to up to 1000 rows. Using `LIMIT`, you can increase the limit to up to 10,000 rows. @@ -149,17 +157,17 @@ FROM kibana_sample_data_logs ``` -## Time filtering [esql-kibana-time-filter] +### Time filtering [esql-kibana-time-filter] To display data within a specified time range, you can use the standard time filter, custom time parameters, or a WHERE command. -### Standard time filter [_standard_time_filter] +#### Standard time filter [_standard_time_filter] The standard [time filter](../filtering.md) is enabled when the indices you’re querying have a field named `@timestamp`. -### Custom time parameters [_custom_time_parameters] +#### Custom time parameters [_custom_time_parameters] If your indices do not have a field named `@timestamp`, you can use the `?_tstart` and `?_tend` parameters to specify a time range. These parameters work with any timestamp field and automatically sync with the [time filter](../filtering.md). @@ -178,7 +186,7 @@ FROM kibana_sample_data_logs This example uses `50` buckets, which is the maximum number of buckets. -### WHERE command [_where_command] +#### WHERE command [_where_command] You can also limit the time range using the [`WHERE`](https://www.elastic.co/guide/en/elasticsearch/reference/current/esql-commands.html#esql-where) command and the [`NOW`](https://www.elastic.co/guide/en/elasticsearch/reference/current/esql-functions-operators.html#esql-now) function. For example, if the timestamp field is called `timestamp`, to query the last 15 minutes of data: @@ -203,19 +211,19 @@ FROM kibana_sample_data_logs The resulting visualization is a bar chart showing the top 3 countries: -:::{image} ../../../images/elasticsearch-reference-esql-kibana-bar-chart.png +:::{image} /images/elasticsearch-reference-esql-kibana-bar-chart.png :alt: esql kibana bar chart ::: -To make changes to the visualization, like changing the visualization type, axes and colors, click the pencil button (![esql icon edit visualization](../../../images/elasticsearch-reference-esql-icon-edit-visualization.svg "")). This opens an in-line editor: +To make changes to the visualization, like changing the visualization type, axes and colors, click the pencil button (![esql icon edit visualization](/images/elasticsearch-reference-esql-icon-edit-visualization.svg "")). This opens an in-line editor: -:::{image} ../../../images/elasticsearch-reference-esql-kibana-in-line-editor.png +:::{image} /images/elasticsearch-reference-esql-kibana-in-line-editor.png :alt: esql kibana in line editor ::: -You can save the visualization to a new or existing dashboard by clicking the save button (![esql icon save visualization](../../../images/elasticsearch-reference-esql-icon-save-visualization.svg "")). Once saved to a dashboard, you’ll be taken to the Dashboards page. You can continue to make changes to the visualization. Click the options button in the top-right (![esql icon options](../../../images/elasticsearch-reference-esql-icon-options.svg "")) and select **Edit ES|QL visualization** to open the in-line editor: +You can save the visualization to a new or existing dashboard by clicking the save button (![esql icon save visualization](/images/elasticsearch-reference-esql-icon-save-visualization.svg "")). Once saved to a dashboard, you’ll be taken to the Dashboards page. You can continue to make changes to the visualization. Click the options button in the top-right (![esql icon options](/images/elasticsearch-reference-esql-icon-options.svg "")) and select **Edit ES|QL visualization** to open the in-line editor: -:::{image} ../../../images/elasticsearch-reference-esql-kibana-edit-on-dashboard.png +:::{image} /images/elasticsearch-reference-esql-kibana-edit-on-dashboard.png :alt: esql kibana edit on dashboard ::: @@ -224,19 +232,19 @@ You can save the visualization to a new or existing dashboard by clicking the sa You can use {{esql}} queries to create panels on your dashboards. To add a panel to a dashboard, under **Dashboards**, click the **Add panel** button and select {{esql}}. -:::{image} ../../../images/elasticsearch-reference-esql-dashboard-panel.png +:::{image} /images/elasticsearch-reference-esql-dashboard-panel.png :alt: esql dashboard panel ::: -Check the {{esql}} query by clicking the Panel filters button (![Panel filters button on panel header](../../../images/elasticsearch-reference-dashboard_panel_filter_button.png "")): +Check the {{esql}} query by clicking the Panel filters button (![Panel filters button on panel header](/images/elasticsearch-reference-dashboard_panel_filter_button.png "")): -:::{image} ../../../images/elasticsearch-reference-esql-dashboard-panel-query.png +:::{image} /images/elasticsearch-reference-esql-dashboard-panel-query.png :alt: esql dashboard panel query ::: -You can also edit the {{esql}} visualization from here. Click the options button in the top-right (![esql icon options](../../../images/elasticsearch-reference-esql-icon-options.svg "")) and select **Edit ESQL visualization** to open the in-line editor. +You can also edit the {{esql}} visualization from here. Click the options button in the top-right (![esql icon options](/images/elasticsearch-reference-esql-icon-options.svg "")) and select **Edit ESQL visualization** to open the in-line editor. -:::{image} ../../../images/elasticsearch-reference-esql-dashboard-panel-edit-visualization.png +:::{image} /images/elasticsearch-reference-esql-dashboard-panel-edit-visualization.png :alt: esql dashboard panel edit visualization ::: @@ -245,19 +253,19 @@ You can also edit the {{esql}} visualization from here. Click the options button The {{esql}} [`ENRICH`](https://www.elastic.co/guide/en/elasticsearch/reference/current/esql-commands.html#esql-enrich) command enables you to [enrich](https://www.elastic.co/guide/en/elasticsearch/reference/current/esql-enrich-data.html) your query dataset with fields from another dataset. Before you can use `ENRICH`, you need to [create and execute an enrich policy](https://www.elastic.co/guide/en/elasticsearch/reference/current/esql-enrich-data.html#esql-set-up-enrich-policy). If a policy exists, it will be suggested by auto-complete. If not, click **Click to create** to create one. -:::{image} ../../../images/elasticsearch-reference-esql-kibana-enrich-autocomplete.png +:::{image} /images/elasticsearch-reference-esql-kibana-enrich-autocomplete.png :alt: esql kibana enrich autocomplete ::: Next, you can enter a policy name, the policy type, source indices, and optionally a query: -:::{image} ../../../images/elasticsearch-reference-esql-kibana-enrich-step-1.png +:::{image} /images/elasticsearch-reference-esql-kibana-enrich-step-1.png :alt: esql kibana enrich step 1 ::: Click **Next** to select the match field and enrich fields: -:::{image} ../../../images/elasticsearch-reference-esql-kibana-enrich-step-2.png +:::{image} /images/elasticsearch-reference-esql-kibana-enrich-step-2.png :alt: esql kibana enrich step 2 ::: @@ -278,7 +286,7 @@ FROM kibana_sample_data_logs You can use {{esql}} queries to create alerts. From Discover, click **Alerts** and select **Create search threshold rule**. This opens a panel that enables you to create a rule using an {{esql}} query. Next, you can test the query, add a connector, and save the rule. -:::{image} ../../../images/elasticsearch-reference-esql-kibana-create-rule.png +:::{image} /images/elasticsearch-reference-esql-kibana-create-rule.png :alt: esql kibana create rule ::: diff --git a/explore-analyze/query-filter/languages/esql-rest.md b/explore-analyze/query-filter/languages/esql-rest.md index be3700976..9b95063ec 100644 --- a/explore-analyze/query-filter/languages/esql-rest.md +++ b/explore-analyze/query-filter/languages/esql-rest.md @@ -1,12 +1,12 @@ --- -navigation_title: "REST API" +navigation_title: "{{esql}} query API" mapped_pages: - https://www.elastic.co/guide/en/elasticsearch/reference/current/esql-rest.html --- -# REST API [esql-rest] +# {{esql}} query API [esql-rest] @@ -36,7 +36,7 @@ James S.A. Corey |Leviathan Wakes |561 |2011-06-02T00:00:00.000Z ### Kibana Console [esql-kibana-console] -If you are using [Kibana Console](../tools/console.md) (which is highly recommended), take advantage of the triple quotes `"""` when creating the query. This not only automatically escapes double quotes (`"`) inside the query string but also supports multi-line requests: +If you are using [Kibana Console](/explore-analyze/query-filter/tools/console.md) (which is highly recommended), take advantage of the triple quotes `"""` when creating the query. This not only automatically escapes double quotes (`"`) inside the query string but also supports multi-line requests: ```console POST /_query?format=txt @@ -72,7 +72,7 @@ The URL parameter takes precedence over the HTTP headers. If neither is specifie | Binary | | `cbor` | `application/cbor` | [Concise Binary Object Representation](https://cbor.io/) | | `smile` | `application/smile` | [Smile](https://en.wikipedia.org/wiki/Smile_(data_interchange_format)) binary data format similarto CBOR | -| `arrow` | `application/vnd.apache.arrow.stream` | **Experimental.** [Apache Arrow](https://arrow.apache.org/) dataframes, [IPC streaming format](https://arrow.apache.org/docs/format/Columnar.md#ipc-streaming-format) | +| `arrow` | `application/vnd.apache.arrow.stream` | **Experimental.** [Apache Arrow](https://arrow.apache.org/) dataframes, [IPC streaming format](https://arrow.apache.org/docs/format/Columnar.html#ipc-streaming-format) | The `csv` format accepts a formatting URL query attribute, `delimiter`, which indicates which character should be used to separate the CSV values. It defaults to comma (`,`) and cannot take any of the following values: double quote (`"`), carriage-return (`\r`) and new-line (`\n`). The tab (`\t`) can also not be used. Use the `tsv` format instead. @@ -151,7 +151,7 @@ Which returns: ### Returning localized results [esql-locale-param] -Use the `locale` parameter in the request body to return results (especially dates) formatted per the conventions of the locale. If `locale` is not specified, defaults to `en-US` (English). Refer to [JDK Supported Locales](https://www.oracle.com/java/technologies/javase/jdk17-suported-locales.md). +Use the `locale` parameter in the request body to return results (especially dates) formatted per the conventions of the locale. If `locale` is not specified, defaults to `en-US` (English). Refer to [JDK Supported Locales](https://www.oracle.com/java/technologies/javase/jdk17-suported-locales.html). Syntax: the `locale` parameter accepts language tags in the (case-insensitive) format `xy` and `xy-XY`. diff --git a/explore-analyze/query-filter/languages/esql.md b/explore-analyze/query-filter/languages/esql.md new file mode 100644 index 000000000..2b5c9a8ed --- /dev/null +++ b/explore-analyze/query-filter/languages/esql.md @@ -0,0 +1,68 @@ +--- +mapped_urls: + - https://www.elastic.co/guide/en/elasticsearch/reference/current/esql.html + - https://www.elastic.co/guide/en/elasticsearch/reference/current/esql-getting-started.html + - https://www.elastic.co/guide/en/elasticsearch/reference/current/esql-using.html + - https://www.elastic.co/guide/en/elasticsearch/reference/current/esql-examples.html + - https://www.elastic.co/guide/en/kibana/current/esql.html +--- + +# ES|QL [esql] + +% What needs to be done: Refine + +% Scope notes: everything but language reference. Merge the pages about Kibana. Add links to reference's new location + +% Use migrated content from existing pages that map to this page: + +% - [ ] ./raw-migrated-files/elasticsearch/elasticsearch-reference/esql.md +% - [ ] ./raw-migrated-files/elasticsearch/elasticsearch-reference/esql-getting-started.md +% - [ ] ./raw-migrated-files/elasticsearch/elasticsearch-reference/esql-using.md +% - [ ] ./raw-migrated-files/elasticsearch/elasticsearch-reference/esql-examples.md +% - [ ] ./raw-migrated-files/kibana/kibana/esql.md + +## What's {{esql}}? [_the_esql_compute_engine] + +**Elasticsearch Query Language ({{esql}})** is a piped query language for filtering, transforming, and analyzing data. + +You can author {{esql}} queries to find specific events, perform statistical analysis, and generate visualizations. It supports a wide range of [commands, functions and operators](https://www.elastic.co/guide/en/elasticsearch/reference/current/esql-functions-operators.html) to perform various data operations, such as filtering, aggregation, time-series analysis, and more. Today, it supports a subset of the features available in Query DSL, but it is rapidly evolving. + +::::{note} +**{{esql}}'s compute architecture** + +{{esql}} is built on top of a new compute architecture within {{es}}, designed to achieve high functional and performance requirements for {{esql}}. {{esql}} search, aggregation, and transformation functions are directly executed within Elasticsearch itself. Query expressions are not transpiled to Query DSL for execution. This approach allows {{esql}} to be extremely performant and versatile. + +The new {{esql}} execution engine was designed with performance in mind — it operates on blocks at a time instead of per row, targets vectorization and cache locality, and embraces specialization and multi-threading. It is a separate component from the existing Elasticsearch aggregation framework with different performance characteristics. +:::: + + +## How does it work? [search-analyze-data-esql] + +The {{es}} Query Language ({{esql}}) makes use of "pipes" (|) to manipulate and transform data in a step-by-step fashion. This approach allows you to compose a series of operations, where the output of one operation becomes the input for the next, enabling complex data transformations and analysis. + +You can use it: +- In your queries to {{es}} APIs, using the [`_query` endpoint](/explore-analyze/query-filter/languages/esql-rest.md) that accepts queries written in {{esql}} syntax. +- Within various {{kib}} tools such as Discover and Dashboards, to explore your data and build powerful visualizations. + +% Learn more in [Getting started with {{esql}}](/solutions/search/get-started.md), or try [our training course](https://www.elastic.co/training/introduction-to-esql). + +## Next steps + +Find more details about {{esql}} in the following documentation pages: +- [{{esql}} reference](https://www.elastic.co/guide/en/elasticsearch/reference/current/esql-language.html): + - Reference documentation for the [{{esql}} syntax](https://www.elastic.co/guide/en/elasticsearch/reference/current/esql-syntax.html), [commands](https://www.elastic.co/guide/en/elasticsearch/reference/current/esql-commands.html), and [functions and operators](https://www.elastic.co/guide/en/elasticsearch/reference/current/esql-functions-operators.html). + - Information about working with [metadata fields](https://www.elastic.co/guide/en/elasticsearch/reference/current/esql-metadata-fields.html) and [multivalued fields](https://www.elastic.co/guide/en/elasticsearch/reference/current/esql-multivalued-fields.html). + - Guidance for [data processing with DISSECT and GROK](https://www.elastic.co/guide/en/elasticsearch/reference/current/esql-process-data-with-dissect-and-grok.html) and [data enrichment with ENRICH](https://www.elastic.co/guide/en/elasticsearch/reference/current/esql-enrich-data.html). + +- Using {{esql}}: + - An overview of using the [`_query` API endpoint](/explore-analyze/query-filter/languages/esql-rest.md). + - [Using {{esql}} in {{kib}}](../../../explore-analyze/query-filter/languages/esql-kibana.md). + - [Using {{esql}} in {{elastic-sec}}](/explore-analyze/query-filter/languages/esql-elastic-security.md). + - [Using {{esql}} across clusters](/explore-analyze/query-filter/languages/esql-cross-clusters.md). + - [Task management](/explore-analyze/query-filter/languages/esql-task-management.md). + +- [Limitations](https://www.elastic.co/guide/en/elasticsearch/reference/current/esql-limitations.html): The current limitations of {{esql}}. + +- [Examples](/explore-analyze/query-filter/languages/esql.md): A few examples of what you can do with {{esql}}. + +To get started, you can also try [our ES|QL training course](https://www.elastic.co/training/introduction-to-esql). \ No newline at end of file diff --git a/explore-analyze/query-filter/languages/querydsl.md b/explore-analyze/query-filter/languages/querydsl.md index d0b95c198..fb149ea09 100644 --- a/explore-analyze/query-filter/languages/querydsl.md +++ b/explore-analyze/query-filter/languages/querydsl.md @@ -22,4 +22,167 @@ $$$filter-context$$$ $$$query-dsl-allow-expensive-queries$$$ -$$$relevance-scores$$$ \ No newline at end of file +$$$relevance-scores$$$ + +## What's Query DSL? [search-analyze-query-dsl] + +**Query DSL** is a full-featured JSON-style query language that enables complex searching, filtering, and aggregations. It is the original and most powerful query language for {{es}} today. + +The [`_search` endpoint](/solutions/search/querying-for-search-searching-with-the-search-api.md) accepts queries written in Query DSL syntax. + + +### Search and filter with Query DSL [search-analyze-query-dsl-search-filter] + +Query DSL support a wide range of search techniques, including the following: + +* [**Full-text search**](/solutions/search/full-text.md): Search text that has been analyzed and indexed to support phrase or proximity queries, fuzzy matches, and more. +* [**Keyword search**](https://www.elastic.co/guide/en/elasticsearch/reference/current/keyword.html): Search for exact matches using `keyword` fields. +* [**Semantic search**](/solutions/search/semantic-search/semantic-search-semantic-text.md): Search `semantic_text` fields using dense or sparse vector search on embeddings generated in your {{es}} cluster. +* [**Vector search**](/solutions/search/vector/knn.md): Search for similar dense vectors using the kNN algorithm for embeddings generated outside of {{es}}. +* [**Geospatial search**](https://www.elastic.co/guide/en/elasticsearch/reference/current/geo-queries.html): Search for locations and calculate spatial relationships using geospatial queries. + +You can also filter data using Query DSL. Filters enable you to include or exclude documents by retrieving documents that match specific field-level criteria. A query that uses the `filter` parameter indicates [filter context](#filter-context). + +### Analyze with Query DSL [search-analyze-data-query-dsl] + +[Aggregations](/explore-analyze/aggregations.md) are the primary tool for analyzing {{es}} data using Query DSL. Aggregations enable you to build complex summaries of your data and gain insight into key metrics, patterns, and trends. + +Because aggregations leverage the same data structures used for search, they are also very fast. This enables you to analyze and visualize your data in real time. You can search documents, filter results, and perform analytics at the same time, on the same data, in a single request. That means aggregations are calculated in the context of the search query. + +The following aggregation types are available: + +* [Metric](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-metrics.html): Calculate metrics, such as a sum or average, from field values. +* [Bucket](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket.html): Group documents into buckets based on field values, ranges, or other criteria. +* [Pipeline](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-pipeline.html): Run aggregations on the results of other aggregations. + +Run aggregations by specifying the [search API](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html)'s `aggs` parameter. Learn more in [Run an aggregation](/explore-analyze/aggregations.md#run-an-agg). + + +## How does it work? [query-dsl] + +Think of the Query DSL as an AST (Abstract Syntax Tree) of queries, consisting of two types of clauses: + +**Leaf query clauses**: Leaf query clauses look for a particular value in a particular field, such as the [`match`](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-match-query.html), [`term`](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-term-query.html) or [`range`](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-range-query.html) queries. These queries can be used by themselves. + +**Compound query clauses**: Compound query clauses wrap other leaf **or** compound queries and are used to combine multiple queries in a logical fashion (such as the [`bool`](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-bool-query.html) or [`dis_max`](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-dis-max-query.html) query), or to alter their behavior (such as the [`constant_score`](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-constant-score-query.html) query). + +Query clauses behave differently depending on whether they are used in [query context or filter context](#query-filter-context). + +$$$query-dsl-allow-expensive-queries$$$ + +**Allow expensive queries**: Certain types of queries will generally execute slowly due to the way they are implemented, which can affect the stability of the cluster. Those queries can be categorized as follows: + + - Queries that need to do linear scans to identify matches: + + - [`script` queries](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-script-query.html) + - queries on [numeric](https://www.elastic.co/guide/en/elasticsearch/reference/current/number.html), [date](https://www.elastic.co/guide/en/elasticsearch/reference/current/date.html), [boolean](https://www.elastic.co/guide/en/elasticsearch/reference/current/boolean.html), [ip](https://www.elastic.co/guide/en/elasticsearch/reference/current/ip.html), [geo_point](https://www.elastic.co/guide/en/elasticsearch/reference/current/geo-point.html) or [keyword](https://www.elastic.co/guide/en/elasticsearch/reference/current/keyword.html) fields that are not indexed but have [doc values](https://www.elastic.co/guide/en/elasticsearch/reference/current/doc-values.html) enabled + + - Queries that have a high up-front cost: + + - [`fuzzy` queries](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-fuzzy-query.html) (except on [`wildcard`](https://www.elastic.co/guide/en/elasticsearch/reference/current/keyword.html#wildcard-field-type) fields) + - [`regexp` queries](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-regexp-query.html) (except on [`wildcard`](https://www.elastic.co/guide/en/elasticsearch/reference/current/keyword.html#wildcard-field-type) fields) + - [`prefix` queries](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-prefix-query.html) (except on [`wildcard`](https://www.elastic.co/guide/en/elasticsearch/reference/current/keyword.html#wildcard-field-type) fields or those without [`index_prefixes`](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-prefixes.html)) + - [`wildcard` queries](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-wildcard-query.html) (except on [`wildcard`](https://www.elastic.co/guide/en/elasticsearch/reference/current/keyword.html#wildcard-field-type) fields) + - [`range` queries](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-range-query.html) on [`text`](https://www.elastic.co/guide/en/elasticsearch/reference/current/text.html) and [`keyword`](https://www.elastic.co/guide/en/elasticsearch/reference/current/keyword.html) fields + + - [Joining queries](https://www.elastic.co/guide/en/elasticsearch/reference/current/joining-queries.html) + - Queries that may have a high per-document cost: + + - [`script_score` queries](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-script-score-query.html) + - [`percolate` queries](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-percolate-query.html) + + +The execution of such queries can be prevented by setting the value of the `search.allow_expensive_queries` setting to `false` (defaults to `true`). + +## Query and filter context [query-filter-context] + +### Relevance scores [relevance-scores] + +By default, Elasticsearch sorts matching search results by **relevance score**, which measures how well each document matches a query. + +The relevance score is a positive floating point number, returned in the `_score` metadata field of the [search](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html) API. The higher the `_score`, the more relevant the document. While each query type can calculate relevance scores differently, score calculation also depends on whether the query clause is run in a **query** or **filter** context. + + +### Query context [query-context] + +In the query context, a query clause answers the question *How well does this document match this query clause?* Besides deciding whether or not the document matches, the query clause also calculates a relevance score in the `_score` metadata field. + +Query context is in effect whenever a query clause is passed to a `query` parameter, such as the `query` parameter in the [search](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html#request-body-search-query) API. + + +### Filter context [filter-context] + +A filter answers the binary question “Does this document match this query clause?”. The answer is simply "yes" or "no". Filtering has several benefits: + +1. **Simple binary logic**: In a filter context, a query clause determines document matches based on a yes/no criterion, without score calculation. +2. **Performance**: Because they don’t compute relevance scores, filters execute faster than queries. +3. **Caching**: {{es}} automatically caches frequently used filters, speeding up subsequent search performance. +4. **Resource efficiency**: Filters consume less CPU resources compared to full-text queries. +5. **Query combination**: Filters can be combined with scored queries to refine result sets efficiently. + +Filters are particularly effective for querying structured data and implementing "must have" criteria in complex searches. + +Structured data refers to information that is highly organized and formatted in a predefined manner. In the context of Elasticsearch, this typically includes: + +* Numeric fields (integers, floating-point numbers) +* Dates and timestamps +* Boolean values +* Keyword fields (exact match strings) +* Geo-points and geo-shapes + +Unlike full-text fields, structured data has a consistent, predictable format, making it ideal for precise filtering operations. + +Common filter applications include: + +* Date range checks: for example is the `timestamp` field between 2015 and 2016 +* Specific field value checks: for example is the `status` field equal to "published" or is the `author` field equal to "John Doe" + +Filter context applies when a query clause is passed to a `filter` parameter, such as: + +* `filter` or `must_not` parameters in [`bool`](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-bool-query.html) queries +* `filter` parameter in [`constant_score`](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-constant-score-query.html) queries +* [`filter`](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-filter-aggregation.html) aggregations + +Filters optimize query performance and efficiency, especially for structured data queries and when combined with full-text searches. + + +### Example of query and filter contexts [query-filter-context-ex] + +Below is an example of query clauses being used in query and filter context in the `search` API. This query will match documents where all of the following conditions are met: + +* The `title` field contains the word `search`. +* The `content` field contains the word `elasticsearch`. +* The `status` field contains the exact word `published`. +* The `publish_date` field contains a date from 1 Jan 2015 onwards. + +```console +GET /_search +{ + "query": { <1> + "bool": { <2> + "must": [ + { "match": { "title": "Search" }}, + { "match": { "content": "Elasticsearch" }} + ], + "filter": [ <3> + { "term": { "status": "published" }}, + { "range": { "publish_date": { "gte": "2015-01-01" }}} + ] + } + } +} +``` + +1. The `query` parameter indicates query context. +2. The `bool` and two `match` clauses are used in query context, which means that they are used to score how well each document matches. +3. The `filter` parameter indicates filter context. Its `term` and `range` clauses are used in filter context. They will filter out documents which do not match, but they will not affect the score for matching documents. + + +::::{warning} +Scores calculated for queries in query context are represented as single precision floating point numbers; they have only 24 bits for significand’s precision. Score calculations that exceed the significand’s precision will be converted to floats with loss of precision. +:::: + + +::::{tip} +Use query clauses in query context for conditions which should affect the score of matching documents (i.e. how well does the document match), and use all other query clauses in filter context. +:::: \ No newline at end of file diff --git a/explore-analyze/toc.yml b/explore-analyze/toc.yml index 484812b05..8f8033285 100644 --- a/explore-analyze/toc.yml +++ b/explore-analyze/toc.yml @@ -9,14 +9,16 @@ toc: - file: query-filter/languages.md children: - file: query-filter/languages/querydsl.md - - file: query-filter/languages/esorql.md + - file: query-filter/languages/esql.md children: + - file: query-filter/languages/esql-getting-started.md - file: query-filter/languages/esql-rest.md - file: query-filter/languages/esql-kibana.md - file: query-filter/languages/esql-elastic-security.md - file: query-filter/languages/esql-multi-index.md - file: query-filter/languages/esql-cross-clusters.md - file: query-filter/languages/esql-task-management.md + - file: query-filter/languages/esql-examples.md - file: query-filter/languages/sql.md children: - file: query-filter/languages/sql-overview.md diff --git a/raw-migrated-files/docs-content/serverless/security-about-rules.md b/raw-migrated-files/docs-content/serverless/security-about-rules.md index 095ef2624..72176598d 100644 --- a/raw-migrated-files/docs-content/serverless/security-about-rules.md +++ b/raw-migrated-files/docs-content/serverless/security-about-rules.md @@ -33,7 +33,7 @@ You can create the following types of rules: :::: * [**New terms**](../../../solutions/security/detect-and-alert/create-detection-rule.md#create-new-terms-rule): Generates an alert for each new term detected in source documents within a specified time range. You can also detect a combination of up to three new terms (for example, a `host.ip` and `host.id` that have never been observed together before). -* [**{{esql}}**](../../../solutions/security/detect-and-alert/create-detection-rule.md#create-esql-rule): Searches the defined indices and creates an alert when results match an [{{esql}} query](../../../explore-analyze/query-filter/languages/esorql.md). +* [**{{esql}}**](../../../solutions/security/detect-and-alert/create-detection-rule.md#create-esql-rule): Searches the defined indices and creates an alert when results match an [{{esql}} query](../../../explore-analyze/query-filter/languages/esql.md). :::{image} ../../../images/serverless--detections-all-rules.png :alt: Shows the Rules page diff --git a/raw-migrated-files/docs-content/serverless/security-rules-create.md b/raw-migrated-files/docs-content/serverless/security-rules-create.md index e6de25450..12e218d31 100644 --- a/raw-migrated-files/docs-content/serverless/security-rules-create.md +++ b/raw-migrated-files/docs-content/serverless/security-rules-create.md @@ -352,7 +352,7 @@ You uploaded a value list of known ransomware domains, and you want to be notifi ## Create an {{esql}} rule [create-esql-rule] -Use [{{esql}}](../../../explore-analyze/query-filter/languages/esorql.md) to query your source events and aggregate event data. Query results are returned in a table with rows and columns. Each row becomes an alert. +Use [{{esql}}](../../../explore-analyze/query-filter/languages/esql.md) to query your source events and aggregate event data. Query results are returned in a table with rows and columns. Each row becomes an alert. To create an {{esql}} rule: diff --git a/raw-migrated-files/docs-content/serverless/security-timelines-ui.md b/raw-migrated-files/docs-content/serverless/security-timelines-ui.md index fe76645bd..f91cd0144 100644 --- a/raw-migrated-files/docs-content/serverless/security-timelines-ui.md +++ b/raw-migrated-files/docs-content/serverless/security-timelines-ui.md @@ -219,7 +219,7 @@ From the **Correlation** tab, you can also do the following: ## Use {{esql}} to investigate events [esql-in-timeline] -The [Elasticsearch Query Language ({{esql}})](../../../explore-analyze/query-filter/languages/esorql.md) provides a powerful way to filter, transform, and analyze event data stored in {{es}}. {{esql}} queries use "pipes" to manipulate and transform data in a step-by-step fashion. This approach allows you to compose a series of operations, where the output of one operation becomes the input for the next, enabling complex data transformations and analysis. +The [Elasticsearch Query Language ({{esql}})](../../../explore-analyze/query-filter/languages/esql.md) provides a powerful way to filter, transform, and analyze event data stored in {{es}}. {{esql}} queries use "pipes" to manipulate and transform data in a step-by-step fashion. This approach allows you to compose a series of operations, where the output of one operation becomes the input for the next, enabling complex data transformations and analysis. You can use {{esql}} in Timeline by opening the **{{esql}}** tab. From there, you can: diff --git a/raw-migrated-files/elasticsearch/elasticsearch-reference/esql.md b/raw-migrated-files/elasticsearch/elasticsearch-reference/esql.md deleted file mode 100644 index 43faa205f..000000000 --- a/raw-migrated-files/elasticsearch/elasticsearch-reference/esql.md +++ /dev/null @@ -1,32 +0,0 @@ -# {{esql}} [esql] - -The {{es}} Query Language ({{esql}}) provides a powerful way to filter, transform, and analyze data stored in {{es}}, and in the future in other runtimes. It is designed to be easy to learn and use, by end users, SRE teams, application developers, and administrators. - -Users can author {{esql}} queries to find specific events, perform statistical analysis, and generate visualizations. It supports a wide range of commands and functions that enable users to perform various data operations, such as filtering, aggregation, time-series analysis, and more. - -The {{es}} Query Language ({{esql}}) makes use of "pipes" (|) to manipulate and transform data in a step-by-step fashion. This approach allows users to compose a series of operations, where the output of one operation becomes the input for the next, enabling complex data transformations and analysis. - - -## The {{esql}} Compute Engine [_the_esql_compute_engine] - -{{esql}} is more than a language: it represents a significant investment in new compute capabilities within {{es}}. To achieve both the functional and performance requirements for {{esql}}, it was necessary to build an entirely new compute architecture. {{esql}} search, aggregation, and transformation functions are directly executed within Elasticsearch itself. Query expressions are not transpiled to Query DSL for execution. This approach allows {{esql}} to be extremely performant and versatile. - -The new {{esql}} execution engine was designed with performance in mind — it operates on blocks at a time instead of per row, targets vectorization and cache locality, and embraces specialization and multi-threading. It is a separate component from the existing Elasticsearch aggregation framework with different performance characteristics. - -The {{esql}} documentation is organized in these sections: - -[Getting started](../../../explore-analyze/query-filter/languages/esorql.md) -: A tutorial to help you get started with {{esql}}. - -[{{esql}} reference](https://www.elastic.co/guide/en/elasticsearch/reference/current/esql-language.html) -: Reference documentation for the [{{esql}} syntax](https://www.elastic.co/guide/en/elasticsearch/reference/current/esql-syntax.html), [commands](https://www.elastic.co/guide/en/elasticsearch/reference/current/esql-commands.html), and [functions and operators](https://www.elastic.co/guide/en/elasticsearch/reference/current/esql-functions-operators.html). Information about working with [metadata fields](https://www.elastic.co/guide/en/elasticsearch/reference/current/esql-metadata-fields.html) and [multivalued fields](https://www.elastic.co/guide/en/elasticsearch/reference/current/esql-multivalued-fields.html). And guidance for [data processing with DISSECT and GROK](https://www.elastic.co/guide/en/elasticsearch/reference/current/esql-process-data-with-dissect-and-grok.html) and [data enrichment with ENRICH](https://www.elastic.co/guide/en/elasticsearch/reference/current/esql-enrich-data.html). - -[*Using {{esql}}*](../../../explore-analyze/query-filter/languages/esorql.md) -: An overview of using the [REST API](../../../explore-analyze/query-filter/languages/esql-rest.md), [Using {{esql}} in {{kib}}](../../../explore-analyze/query-filter/languages/esql-kibana.md), [Using {{esql}} in {{elastic-sec}}](../../../explore-analyze/query-filter/languages/esql-elastic-security.md), [Using {{esql}} across clusters](../../../explore-analyze/query-filter/languages/esql-cross-clusters.md), and [Task management](../../../explore-analyze/query-filter/languages/esql-task-management.md). - -[Limitations](https://www.elastic.co/guide/en/elasticsearch/reference/current/esql-limitations.html) -: The current limitations of {{esql}}. - -[Examples](../../../explore-analyze/query-filter/languages/esorql.md) -: A few examples of what you can do with {{esql}}. - diff --git a/raw-migrated-files/elasticsearch/elasticsearch-reference/query-dsl.md b/raw-migrated-files/elasticsearch/elasticsearch-reference/query-dsl.md deleted file mode 100644 index 8e8d07692..000000000 --- a/raw-migrated-files/elasticsearch/elasticsearch-reference/query-dsl.md +++ /dev/null @@ -1,39 +0,0 @@ -# Query DSL [query-dsl] - -Elasticsearch provides a full Query DSL (Domain Specific Language) based on JSON to define queries. Think of the Query DSL as an AST (Abstract Syntax Tree) of queries, consisting of two types of clauses: - -Leaf query clauses -: Leaf query clauses look for a particular value in a particular field, such as the [`match`](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-match-query.html), [`term`](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-term-query.html) or [`range`](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-range-query.html) queries. These queries can be used by themselves. - -Compound query clauses -: Compound query clauses wrap other leaf **or** compound queries and are used to combine multiple queries in a logical fashion (such as the [`bool`](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-bool-query.html) or [`dis_max`](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-dis-max-query.html) query), or to alter their behaviour (such as the [`constant_score`](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-constant-score-query.html) query). - -Query clauses behave differently depending on whether they are used in [query context or filter context](../../../explore-analyze/query-filter/languages/querydsl.md). - -$$$query-dsl-allow-expensive-queries$$$ - -Allow expensive queries -: Certain types of queries will generally execute slowly due to the way they are implemented, which can affect the stability of the cluster. Those queries can be categorised as follows: - - * Queries that need to do linear scans to identify matches: - - * [`script` queries](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-script-query.html) - * queries on [numeric](https://www.elastic.co/guide/en/elasticsearch/reference/current/number.html), [date](https://www.elastic.co/guide/en/elasticsearch/reference/current/date.html), [boolean](https://www.elastic.co/guide/en/elasticsearch/reference/current/boolean.html), [ip](https://www.elastic.co/guide/en/elasticsearch/reference/current/ip.html), [geo_point](https://www.elastic.co/guide/en/elasticsearch/reference/current/geo-point.html) or [keyword](https://www.elastic.co/guide/en/elasticsearch/reference/current/keyword.html) fields that are not indexed but have [doc values](https://www.elastic.co/guide/en/elasticsearch/reference/current/doc-values.html) enabled - - * Queries that have a high up-front cost: - - * [`fuzzy` queries](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-fuzzy-query.html) (except on [`wildcard`](https://www.elastic.co/guide/en/elasticsearch/reference/current/keyword.html#wildcard-field-type) fields) - * [`regexp` queries](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-regexp-query.html) (except on [`wildcard`](https://www.elastic.co/guide/en/elasticsearch/reference/current/keyword.html#wildcard-field-type) fields) - * [`prefix` queries](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-prefix-query.html) (except on [`wildcard`](https://www.elastic.co/guide/en/elasticsearch/reference/current/keyword.html#wildcard-field-type) fields or those without [`index_prefixes`](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-prefixes.html)) - * [`wildcard` queries](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-wildcard-query.html) (except on [`wildcard`](https://www.elastic.co/guide/en/elasticsearch/reference/current/keyword.html#wildcard-field-type) fields) - * [`range` queries](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-range-query.html) on [`text`](https://www.elastic.co/guide/en/elasticsearch/reference/current/text.html) and [`keyword`](https://www.elastic.co/guide/en/elasticsearch/reference/current/keyword.html) fields - - * [Joining queries](https://www.elastic.co/guide/en/elasticsearch/reference/current/joining-queries.html) - * Queries that may have a high per-document cost: - - * [`script_score` queries](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-script-score-query.html) - * [`percolate` queries](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-percolate-query.html) - - -The execution of such queries can be prevented by setting the value of the `search.allow_expensive_queries` setting to `false` (defaults to `true`). - diff --git a/raw-migrated-files/elasticsearch/elasticsearch-reference/query-filter-context.md b/raw-migrated-files/elasticsearch/elasticsearch-reference/query-filter-context.md deleted file mode 100644 index bd1b294f2..000000000 --- a/raw-migrated-files/elasticsearch/elasticsearch-reference/query-filter-context.md +++ /dev/null @@ -1,93 +0,0 @@ -# Query and filter context [query-filter-context] - - -## Relevance scores [relevance-scores] - -By default, Elasticsearch sorts matching search results by **relevance score**, which measures how well each document matches a query. - -The relevance score is a positive floating point number, returned in the `_score` metadata field of the [search](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html) API. The higher the `_score`, the more relevant the document. While each query type can calculate relevance scores differently, score calculation also depends on whether the query clause is run in a **query** or **filter** context. - - -## Query context [query-context] - -In the query context, a query clause answers the question *How well does this document match this query clause?* Besides deciding whether or not the document matches, the query clause also calculates a relevance score in the `_score` metadata field. - -Query context is in effect whenever a query clause is passed to a `query` parameter, such as the `query` parameter in the [search](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html#request-body-search-query) API. - - -## Filter context [filter-context] - -A filter answers the binary question “Does this document match this query clause?”. The answer is simply "yes" or "no". Filtering has several benefits: - -1. **Simple binary logic**: In a filter context, a query clause determines document matches based on a yes/no criterion, without score calculation. -2. **Performance**: Because they don’t compute relevance scores, filters execute faster than queries. -3. **Caching**: {{es}} automatically caches frequently used filters, speeding up subsequent search performance. -4. **Resource efficiency**: Filters consume less CPU resources compared to full-text queries. -5. **Query combination**: Filters can be combined with scored queries to refine result sets efficiently. - -Filters are particularly effective for querying structured data and implementing "must have" criteria in complex searches. - -Structured data refers to information that is highly organized and formatted in a predefined manner. In the context of Elasticsearch, this typically includes: - -* Numeric fields (integers, floating-point numbers) -* Dates and timestamps -* Boolean values -* Keyword fields (exact match strings) -* Geo-points and geo-shapes - -Unlike full-text fields, structured data has a consistent, predictable format, making it ideal for precise filtering operations. - -Common filter applications include: - -* Date range checks: for example is the `timestamp` field between 2015 and 2016 -* Specific field value checks: for example is the `status` field equal to "published" or is the `author` field equal to "John Doe" - -Filter context applies when a query clause is passed to a `filter` parameter, such as: - -* `filter` or `must_not` parameters in [`bool`](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-bool-query.html) queries -* `filter` parameter in [`constant_score`](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-constant-score-query.html) queries -* [`filter`](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-filter-aggregation.html) aggregations - -Filters optimize query performance and efficiency, especially for structured data queries and when combined with full-text searches. - - -## Example of query and filter contexts [query-filter-context-ex] - -Below is an example of query clauses being used in query and filter context in the `search` API. This query will match documents where all of the following conditions are met: - -* The `title` field contains the word `search`. -* The `content` field contains the word `elasticsearch`. -* The `status` field contains the exact word `published`. -* The `publish_date` field contains a date from 1 Jan 2015 onwards. - -```console -GET /_search -{ - "query": { <1> - "bool": { <2> - "must": [ - { "match": { "title": "Search" }}, - { "match": { "content": "Elasticsearch" }} - ], - "filter": [ <3> - { "term": { "status": "published" }}, - { "range": { "publish_date": { "gte": "2015-01-01" }}} - ] - } - } -} -``` - -1. The `query` parameter indicates query context. -2. The `bool` and two `match` clauses are used in query context, which means that they are used to score how well each document matches. -3. The `filter` parameter indicates filter context. Its `term` and `range` clauses are used in filter context. They will filter out documents which do not match, but they will not affect the score for matching documents. - - -::::{warning} -Scores calculated for queries in query context are represented as single precision floating point numbers; they have only 24 bits for significand’s precision. Score calculations that exceed the significand’s precision will be converted to floats with loss of precision. -:::: - - -::::{tip} -Use query clauses in query context for conditions which should affect the score of matching documents (i.e. how well does the document match), and use all other query clauses in filter context. -:::: diff --git a/raw-migrated-files/elasticsearch/elasticsearch-reference/search-analyze.md b/raw-migrated-files/elasticsearch/elasticsearch-reference/search-analyze.md index abfd973dc..3f19b0eba 100644 --- a/raw-migrated-files/elasticsearch/elasticsearch-reference/search-analyze.md +++ b/raw-migrated-files/elasticsearch/elasticsearch-reference/search-analyze.md @@ -64,7 +64,7 @@ Run aggregations by specifying the [search API](https://www.elastic.co/guide/en/ ### {{esql}} [search-analyze-data-esql] -[Elasticsearch Query Language ({{esql}})](../../../explore-analyze/query-filter/languages/esorql.md) is a piped query language for filtering, transforming, and analyzing data. {{esql}} is built on top of a new compute engine, where search, aggregation, and transformation functions are directly executed within {{es}} itself. {{esql}} syntax can also be used within various {{kib}} tools. +[Elasticsearch Query Language ({{esql}})](../../../explore-analyze/query-filter/languages/esql.md) is a piped query language for filtering, transforming, and analyzing data. {{esql}} is built on top of a new compute engine, where search, aggregation, and transformation functions are directly executed within {{es}} itself. {{esql}} syntax can also be used within various {{kib}} tools. The [`_query` endpoint](../../../explore-analyze/query-filter/languages/esql-rest.md) accepts queries written in {{esql}} syntax. @@ -82,7 +82,7 @@ The following table summarizes all available {{es}} query languages, to help you | Name | Description | Use cases | API endpoint | | --- | --- | --- | --- | | [Query DSL](../../../explore-analyze/query-filter/languages/querydsl.md) | The primary query language for {{es}}. A powerful and flexible JSON-style language that enables complex queries. | Full-text search, semantic search, keyword search, filtering, aggregations, and more. | [`_search`](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html) | -| [{{esql}}](../../../explore-analyze/query-filter/languages/esorql.md) | Introduced in **8.11**, the Elasticsearch Query Language ({{esql}}) is a piped query language language for filtering, transforming, and analyzing data. | Initially tailored towards working with time series data like logs and metrics.Robust integration with {{kib}} for querying, visualizing, and analyzing data.Does not yet support full-text search. | [`_query`](../../../explore-analyze/query-filter/languages/esql-rest.md) | +| [{{esql}}](../../../explore-analyze/query-filter/languages/esql.md) | Introduced in **8.11**, the Elasticsearch Query Language ({{esql}}) is a piped query language language for filtering, transforming, and analyzing data. | Initially tailored towards working with time series data like logs and metrics.Robust integration with {{kib}} for querying, visualizing, and analyzing data.Does not yet support full-text search. | [`_query`](../../../explore-analyze/query-filter/languages/esql-rest.md) | | [EQL](../../../explore-analyze/query-filter/languages/eql.md) | Event Query Language (EQL) is a query language for event-based time series data. Data must contain the `@timestamp` field to use EQL. | Designed for the threat hunting security use case. | [`_eql`](https://www.elastic.co/guide/en/elasticsearch/reference/current/eql-apis.html) | | [Elasticsearch SQL](../../../explore-analyze/query-filter/languages/sql.md) | Allows native, real-time SQL-like querying against {{es}} data. JDBC and ODBC drivers are available for integration with business intelligence (BI) tools. | Enables users familiar with SQL to query {{es}} data using familiar syntax for BI and reporting. | [`_sql`](https://www.elastic.co/guide/en/elasticsearch/reference/current/sql-apis.html) | | [Kibana Query Language (KQL)](../../../explore-analyze/query-filter/languages/kql.md) | {{kib}} Query Language (KQL) is a text-based query language for filtering data when you access it through the {{kib}} UI. | Use KQL to filter documents where a value for a field exists, matches a given value, or is within a given range. | N/A | diff --git a/raw-migrated-files/kibana/kibana/esql.md b/raw-migrated-files/kibana/kibana/esql.md index be29244d6..fa3369a3a 100644 --- a/raw-migrated-files/kibana/kibana/esql.md +++ b/raw-migrated-files/kibana/kibana/esql.md @@ -37,6 +37,6 @@ Use {{esql}} to retrieve important information for investigation by using lookup ## What’s next? [esql-whats-next] -The main documentation for {{esql}} lives in the [{{es}} docs](../../../explore-analyze/query-filter/languages/esorql.md). +The main documentation for {{esql}} lives in the [{{es}} docs](../../../explore-analyze/query-filter/languages/esql.md). We also have a short tutorial in the **Discover** docs: [Using {{esql}}](../../../explore-analyze/discover/try-esql.md). diff --git a/raw-migrated-files/kibana/kibana/managing-data-views.md b/raw-migrated-files/kibana/kibana/managing-data-views.md index 795042fbd..c117d6d62 100644 --- a/raw-migrated-files/kibana/kibana/managing-data-views.md +++ b/raw-migrated-files/kibana/kibana/managing-data-views.md @@ -137,7 +137,7 @@ Edit the settings for runtime fields, or remove runtime fields from data views. ::::{admonition} Deprecated in 7.13. :class: warning -Use [runtime fields](../../../manage-data/data-store/mapping/runtime-fields.md) instead of scripted fields. Runtime fields support Painless scripting and provide greater flexibility. You can also use the [Elasticsearch Query Language (ES|QL)](../../../explore-analyze/query-filter/languages/esorql.md) to compute values directly at query time. +Use [runtime fields](../../../manage-data/data-store/mapping/runtime-fields.md) instead of scripted fields. Runtime fields support Painless scripting and provide greater flexibility. You can also use the [Elasticsearch Query Language (ES|QL)](../../../explore-analyze/query-filter/languages/esql.md) to compute values directly at query time. :::: diff --git a/raw-migrated-files/kibana/kibana/search-ai-assistant.md b/raw-migrated-files/kibana/kibana/search-ai-assistant.md index 848bef5d3..9b2c02ea0 100644 --- a/raw-migrated-files/kibana/kibana/search-ai-assistant.md +++ b/raw-migrated-files/kibana/kibana/search-ai-assistant.md @@ -22,7 +22,7 @@ Refer to the [Observability documentation](../../../solutions/observability/obse 4. **Using Elasticsearch APIs**: Calls Elasticsearch APIs on your behalf if you need specific operations performed. 5. **Generating Sample Data**: Helps you create sample data for testing and development purposes. 6. **Visualizing and Analyzing Data**: Assists you in creating visualizations and analyzing your data using Kibana. -7. **Explaining ES|QL**: Explains how ES|QL works and help you convert queries from other languages to [ES|QL.](../../../explore-analyze/query-filter/languages/esorql.md) +7. **Explaining ES|QL**: Explains how ES|QL works and help you convert queries from other languages to [ES|QL.](../../../explore-analyze/query-filter/languages/esql.md) ## Requirements [ai-assistant-requirements] diff --git a/raw-migrated-files/security-docs/security/about-rules.md b/raw-migrated-files/security-docs/security/about-rules.md index 513ef0e3e..0a750b33d 100644 --- a/raw-migrated-files/security-docs/security/about-rules.md +++ b/raw-migrated-files/security-docs/security/about-rules.md @@ -27,7 +27,7 @@ You can create the following types of rules: :::: * [**New terms**](../../../solutions/security/detect-and-alert/create-detection-rule.md#create-new-terms-rule): Generates an alert for each new term detected in source documents within a specified time range. You can also detect a combination of up to three new terms (for example, a `host.ip` and `host.id` that have never been observed together before). -* [**ES|QL**](../../../solutions/security/detect-and-alert/create-detection-rule.md#create-esql-rule): Searches the defined indices and creates an alert when results match an [Elasticsearch Query Language (ES|QL)](../../../explore-analyze/query-filter/languages/esorql.md) query. +* [**ES|QL**](../../../solutions/security/detect-and-alert/create-detection-rule.md#create-esql-rule): Searches the defined indices and creates an alert when results match an [Elasticsearch Query Language (ES|QL)](../../../explore-analyze/query-filter/languages/esql.md) query. ::::{note} {{esql}} is enabled by default in {{kib}}. It can be disabled using the `enableESQL` setting from the [Advanced Settings](https://www.elastic.co/guide/en/kibana/current/advanced-options.html). This will hide the {{esql}} user interface from various applications. However, users will be able to access existing {{esql}} artifacts like saved searches and visualizations. diff --git a/raw-migrated-files/security-docs/security/rules-ui-create.md b/raw-migrated-files/security-docs/security/rules-ui-create.md index 629d3dd96..03dd68dc0 100644 --- a/raw-migrated-files/security-docs/security/rules-ui-create.md +++ b/raw-migrated-files/security-docs/security/rules-ui-create.md @@ -353,7 +353,7 @@ You uploaded a value list of known ransomware domains, and you want to be notifi ## Create an {{esql}} rule [create-esql-rule] -Use [{{esql}}](../../../explore-analyze/query-filter/languages/esorql.md) to query your source events and aggregate event data. Query results are returned in a table with rows and columns. Each row becomes an alert. +Use [{{esql}}](../../../explore-analyze/query-filter/languages/esql.md) to query your source events and aggregate event data. Query results are returned in a table with rows and columns. Each row becomes an alert. To create an {{esql}} rule: diff --git a/raw-migrated-files/security-docs/security/timelines-ui.md b/raw-migrated-files/security-docs/security/timelines-ui.md index 6c96fe9e2..45ce6e4da 100644 --- a/raw-migrated-files/security-docs/security/timelines-ui.md +++ b/raw-migrated-files/security-docs/security/timelines-ui.md @@ -219,7 +219,7 @@ From the **Correlation** tab, you can also do the following: :::: -The [Elasticsearch Query Language ({{esql}})](../../../explore-analyze/query-filter/languages/esorql.md) provides a powerful way to filter, transform, and analyze event data stored in {{es}}. {{esql}} queries use "pipes" to manipulate and transform data in a step-by-step fashion. This approach allows you to compose a series of operations, where the output of one operation becomes the input for the next, enabling complex data transformations and analysis. +The [Elasticsearch Query Language ({{esql}})](../../../explore-analyze/query-filter/languages/esql.md) provides a powerful way to filter, transform, and analyze event data stored in {{es}}. {{esql}} queries use "pipes" to manipulate and transform data in a step-by-step fashion. This approach allows you to compose a series of operations, where the output of one operation becomes the input for the next, enabling complex data transformations and analysis. You can use {{esql}} in Timeline by opening the **{{esql}}** tab. From there, you can: diff --git a/solutions/observability/incident-management/create-an-elasticsearch-query-rule.md b/solutions/observability/incident-management/create-an-elasticsearch-query-rule.md index 7f4f2c3c3..f0d8ae63d 100644 --- a/solutions/observability/incident-management/create-an-elasticsearch-query-rule.md +++ b/solutions/observability/incident-management/create-an-elasticsearch-query-rule.md @@ -49,7 +49,7 @@ When you create an {{es}} query rule, your choice of query type affects the info If you use [KQL](../../../explore-analyze/query-filter/languages/kql.md) or [Lucene](../../../explore-analyze/query-filter/languages/lucene-query-syntax.md), you must specify a data view then define a text-based query. For example, `http.request.referrer: "https://example.com"`. - If you use [ES|QL](../../../explore-analyze/query-filter/languages/esorql.md), you must provide a source command followed by an optional series of processing commands, separated by pipe characters (|). For example: + If you use [ES|QL](../../../explore-analyze/query-filter/languages/esql.md), you must provide a source command followed by an optional series of processing commands, separated by pipe characters (|). For example: ```sh FROM kibana_sample_data_logs From 51ad0595223690bd196d2255d0b637fd3824d716 Mon Sep 17 00:00:00 2001 From: florent-leborgne Date: Tue, 4 Feb 2025 14:59:14 +0100 Subject: [PATCH 4/7] Update explore-analyze/query-filter.md Co-authored-by: Liam Thompson <32779855+leemthompo@users.noreply.github.com> --- explore-analyze/query-filter.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/explore-analyze/query-filter.md b/explore-analyze/query-filter.md index fd1db0818..d06f85129 100644 --- a/explore-analyze/query-filter.md +++ b/explore-analyze/query-filter.md @@ -18,7 +18,7 @@ You’ll use a combination of an API endpoint and a query language to interact w - A number of [tools](/explore-analyze/query-filter/tools.md) are available for you to save, debug, and optimize your queries. % todo: update link to the best target -% > If you're just getting started with Elasticsearch, check [first steps with Elasticsearch](/solutions/search/get-started.md) for a hands-on example of using the `_search` endpoint, adding data to {{es}}, and running basic searches in Query DSL syntax. +If you're just getting started with Elasticsearch, try the hands-on [API quickstart](/solutions/search/elasticsearch-basics-quickstart.md) to learn how to add data and run basic searches using Query DSL and the `_search` endpoint. ## Filtering From bad71f27833540f4ab5d4ba65b79017487b5e736 Mon Sep 17 00:00:00 2001 From: Florent Le Borgne Date: Tue, 4 Feb 2025 15:25:09 +0100 Subject: [PATCH 5/7] fix img paths back to relative --- explore-analyze/query-filter/filtering.md | 17 ++++++------ .../languages/esql-getting-started.md | 12 ++++----- .../query-filter/languages/esql-kibana.md | 26 +++++++++---------- 3 files changed, 28 insertions(+), 27 deletions(-) diff --git a/explore-analyze/query-filter/filtering.md b/explore-analyze/query-filter/filtering.md index 36f8469d4..ce8fc07ec 100644 --- a/explore-analyze/query-filter/filtering.md +++ b/explore-analyze/query-filter/filtering.md @@ -32,16 +32,17 @@ Display data within a specified time range when your index contains time-based e * **Commonly used**. Select a time range from options such as **Last 15 minutes**, **Today**, and **Week to date**. * **Recently used date ranges**. Use a previously selected data range. * **Refresh every**. Specify an automatic refresh rate. - :::{image} /images/kibana-time-filter.png - :alt: Time filter menu - :width: 300px - ::: + +:::{image} ../../images/kibana-time-filter.png +:alt: Time filter menu +:width: 300px +::: 3. To set start and end times, click the bar next to the time filter. In the popup, select **Absolute**, **Relative** or **Now**, then specify the required options. - :::{image} /images/kibana-time-relative.png - :alt: Time filter showing relative time - :class: screenshot - ::: + +:::{image} ../../images/kibana-time-relative.png +:alt: Time filter showing relative time +::: The global time filter limits the time range of data displayed. In most cases, the time filter applies to the time field in the data view, but some apps allow you to use a different time field. diff --git a/explore-analyze/query-filter/languages/esql-getting-started.md b/explore-analyze/query-filter/languages/esql-getting-started.md index 0d5e16a64..2bf5d6b68 100644 --- a/explore-analyze/query-filter/languages/esql-getting-started.md +++ b/explore-analyze/query-filter/languages/esql-getting-started.md @@ -103,7 +103,7 @@ After switching to {{esql}} mode, the query bar shows a sample query. You can re To make it easier to write queries, auto-complete offers suggestions with possible commands and functions: -:::{image} /images/elasticsearch-reference-esql-kibana-auto-complete.png +:::{image} ../../../images/elasticsearch-reference-esql-kibana-auto-complete.png :alt: esql kibana auto complete ::: @@ -116,7 +116,7 @@ You can adjust the editor’s height by dragging its bottom border to your likin Each {{esql}} query starts with a [source command](https://www.elastic.co/guide/en/elasticsearch/reference/current/esql-commands.html#esql-source-commands). A source command produces a table, typically with data from {{es}}. -:::{image} /images/elasticsearch-reference-source-command.svg +:::{image} ../../../images/elasticsearch-reference-source-command.svg :alt: A source command producing a table from {es} ::: @@ -143,7 +143,7 @@ from sample_data A source command can be followed by one or more [processing commands](https://www.elastic.co/guide/en/elasticsearch/reference/current/esql-commands.html#esql-processing-commands), separated by a pipe character: `|`. Processing commands change an input table by adding, removing, or changing rows and columns. Processing commands can perform filtering, projection, aggregation, and more. -:::{image} /images/elasticsearch-reference-esql-limit.png +:::{image} ../../../images/elasticsearch-reference-esql-limit.png :alt: A processing command changing an input table ::: @@ -167,7 +167,7 @@ FROM sample_data | LIMIT 3 ### Sort a table [esql-getting-started-sort] -:::{image} /images/elasticsearch-reference-esql-sort.png +:::{image} ../../../images/elasticsearch-reference-esql-sort.png :alt: A processing command sorting an input table ::: @@ -205,7 +205,7 @@ There are many other processing commands, like [`KEEP`](https://www.elastic.co/g You can chain processing commands, separated by a pipe character: `|`. Each processing command works on the output table of the previous command. The result of a query is the table produced by the final processing command. -:::{image} /images/elasticsearch-reference-esql-sort-limit.png +:::{image} ../../../images/elasticsearch-reference-esql-sort-limit.png :alt: Processing commands can be chained ::: @@ -308,7 +308,7 @@ FROM sample_data {{esql}} enables you to [enrich](https://www.elastic.co/guide/en/elasticsearch/reference/current/esql-enrich-data.html) a table with data from indices in {{es}}, using the [`ENRICH`](https://www.elastic.co/guide/en/elasticsearch/reference/current/esql-commands.html#esql-enrich) command. -:::{image} /images/elasticsearch-reference-esql-enrich.png +:::{image} ../../../images/elasticsearch-reference-esql-enrich.png :alt: esql enrich ::: diff --git a/explore-analyze/query-filter/languages/esql-kibana.md b/explore-analyze/query-filter/languages/esql-kibana.md index 38944cddc..ed894cd73 100644 --- a/explore-analyze/query-filter/languages/esql-kibana.md +++ b/explore-analyze/query-filter/languages/esql-kibana.md @@ -47,7 +47,7 @@ Click the **ES|QL help** button to open the in-product reference documentation f To make it easier to write queries, auto-complete offers suggestions with possible commands and functions: -:::{image} /images/elasticsearch-reference-esql-kibana-auto-complete.png +:::{image} ../../../images/elasticsearch-reference-esql-kibana-auto-complete.png :alt: esql kibana auto complete ::: @@ -91,7 +91,7 @@ You can reuse your recent {{esql}} queries in the query bar. In the query bar, c You can then scroll through your recent queries: -:::{image} /images/elasticsearch-reference-esql-discover-query-history.png +:::{image} ../../../images/elasticsearch-reference-esql-discover-query-history.png :alt: esql discover query history ::: @@ -112,7 +112,7 @@ From the **Recent** tab, you can star any queries you want. In the **Starred** tab, find all the queries you have previously starred. -:::{image} /images/elasticsearch-reference-esql-discover-query-starred.png +:::{image} ../../../images/elasticsearch-reference-esql-discover-query-starred.png :alt: esql discover query starred ::: @@ -211,19 +211,19 @@ FROM kibana_sample_data_logs The resulting visualization is a bar chart showing the top 3 countries: -:::{image} /images/elasticsearch-reference-esql-kibana-bar-chart.png +:::{image} ../../../images/elasticsearch-reference-esql-kibana-bar-chart.png :alt: esql kibana bar chart ::: To make changes to the visualization, like changing the visualization type, axes and colors, click the pencil button (![esql icon edit visualization](/images/elasticsearch-reference-esql-icon-edit-visualization.svg "")). This opens an in-line editor: -:::{image} /images/elasticsearch-reference-esql-kibana-in-line-editor.png +:::{image} ../../../images/elasticsearch-reference-esql-kibana-in-line-editor.png :alt: esql kibana in line editor ::: You can save the visualization to a new or existing dashboard by clicking the save button (![esql icon save visualization](/images/elasticsearch-reference-esql-icon-save-visualization.svg "")). Once saved to a dashboard, you’ll be taken to the Dashboards page. You can continue to make changes to the visualization. Click the options button in the top-right (![esql icon options](/images/elasticsearch-reference-esql-icon-options.svg "")) and select **Edit ES|QL visualization** to open the in-line editor: -:::{image} /images/elasticsearch-reference-esql-kibana-edit-on-dashboard.png +:::{image} ../../../images/elasticsearch-reference-esql-kibana-edit-on-dashboard.png :alt: esql kibana edit on dashboard ::: @@ -232,19 +232,19 @@ You can save the visualization to a new or existing dashboard by clicking the sa You can use {{esql}} queries to create panels on your dashboards. To add a panel to a dashboard, under **Dashboards**, click the **Add panel** button and select {{esql}}. -:::{image} /images/elasticsearch-reference-esql-dashboard-panel.png +:::{image} ../../../images/elasticsearch-reference-esql-dashboard-panel.png :alt: esql dashboard panel ::: Check the {{esql}} query by clicking the Panel filters button (![Panel filters button on panel header](/images/elasticsearch-reference-dashboard_panel_filter_button.png "")): -:::{image} /images/elasticsearch-reference-esql-dashboard-panel-query.png +:::{image} ../../../images/elasticsearch-reference-esql-dashboard-panel-query.png :alt: esql dashboard panel query ::: You can also edit the {{esql}} visualization from here. Click the options button in the top-right (![esql icon options](/images/elasticsearch-reference-esql-icon-options.svg "")) and select **Edit ESQL visualization** to open the in-line editor. -:::{image} /images/elasticsearch-reference-esql-dashboard-panel-edit-visualization.png +:::{image} ../../../images/elasticsearch-reference-esql-dashboard-panel-edit-visualization.png :alt: esql dashboard panel edit visualization ::: @@ -253,19 +253,19 @@ You can also edit the {{esql}} visualization from here. Click the options button The {{esql}} [`ENRICH`](https://www.elastic.co/guide/en/elasticsearch/reference/current/esql-commands.html#esql-enrich) command enables you to [enrich](https://www.elastic.co/guide/en/elasticsearch/reference/current/esql-enrich-data.html) your query dataset with fields from another dataset. Before you can use `ENRICH`, you need to [create and execute an enrich policy](https://www.elastic.co/guide/en/elasticsearch/reference/current/esql-enrich-data.html#esql-set-up-enrich-policy). If a policy exists, it will be suggested by auto-complete. If not, click **Click to create** to create one. -:::{image} /images/elasticsearch-reference-esql-kibana-enrich-autocomplete.png +:::{image} ../../../images/elasticsearch-reference-esql-kibana-enrich-autocomplete.png :alt: esql kibana enrich autocomplete ::: Next, you can enter a policy name, the policy type, source indices, and optionally a query: -:::{image} /images/elasticsearch-reference-esql-kibana-enrich-step-1.png +:::{image} ../../../images/elasticsearch-reference-esql-kibana-enrich-step-1.png :alt: esql kibana enrich step 1 ::: Click **Next** to select the match field and enrich fields: -:::{image} /images/elasticsearch-reference-esql-kibana-enrich-step-2.png +:::{image} ../../../images/elasticsearch-reference-esql-kibana-enrich-step-2.png :alt: esql kibana enrich step 2 ::: @@ -286,7 +286,7 @@ FROM kibana_sample_data_logs You can use {{esql}} queries to create alerts. From Discover, click **Alerts** and select **Create search threshold rule**. This opens a panel that enables you to create a rule using an {{esql}} query. Next, you can test the query, add a connector, and save the rule. -:::{image} /images/elasticsearch-reference-esql-kibana-create-rule.png +:::{image} ../../../images/elasticsearch-reference-esql-kibana-create-rule.png :alt: esql kibana create rule ::: From 94552a553ed4ca9ce16707393146b56017513963 Mon Sep 17 00:00:00 2001 From: Florent Le Borgne Date: Tue, 4 Feb 2025 15:28:56 +0100 Subject: [PATCH 6/7] more img paths --- explore-analyze/query-filter/filtering.md | 4 ++-- explore-analyze/query-filter/languages/esql-kibana.md | 10 +++++----- 2 files changed, 7 insertions(+), 7 deletions(-) diff --git a/explore-analyze/query-filter/filtering.md b/explore-analyze/query-filter/filtering.md index ce8fc07ec..45fcadbbb 100644 --- a/explore-analyze/query-filter/filtering.md +++ b/explore-analyze/query-filter/filtering.md @@ -25,7 +25,7 @@ Some apps provide more options, such as [Dashboards](../dashboards.md). Display data within a specified time range when your index contains time-based events, and a time-field is configured for the selected [{{data-source}}](../find-and-organize/data-views.md). The default time range is 15 minutes, but you can customize it in [Advanced Settings](https://www.elastic.co/guide/en/kibana/current/advanced-options.html). -1. Click ![calendar icon](/images/kibana-time-filter-icon.png). +1. Click ![calendar icon](../../../images/kibana-time-filter-icon.png). 2. Choose one of the following: * **Quick select**. Set a time based on the last or next number of seconds, minutes, hours, or other time unit. @@ -54,4 +54,4 @@ To manually resubmit a search, click the **Refresh** button. This is useful when Structured filters are a more interactive way to create {{es}} queries, and are commonly used when building dashboards that are shared by multiple analysts. Each filter can be disabled, inverted, or pinned across all apps. Each of the structured filters is combined with AND logic on the rest of the query. -![Add filter popup](/images/kibana-add-filter-popup.png "") \ No newline at end of file +![Add filter popup](../../../images/kibana-add-filter-popup.png "") \ No newline at end of file diff --git a/explore-analyze/query-filter/languages/esql-kibana.md b/explore-analyze/query-filter/languages/esql-kibana.md index ed894cd73..f965e5bc7 100644 --- a/explore-analyze/query-filter/languages/esql-kibana.md +++ b/explore-analyze/query-filter/languages/esql-kibana.md @@ -99,7 +99,7 @@ You can then scroll through your recent queries: {{esql}} features in-app help and suggestions, so you can get started faster and don’t have to leave the application to check syntax. -![The ES|QL syntax reference and the autocomplete menu](/images/kibana-esql-in-app-help.png "") +![The ES|QL syntax reference and the autocomplete menu](../../../images/kibana-esql-in-app-help.png "") ### Starred queries [esql-kibana-starred-queries] @@ -215,13 +215,13 @@ The resulting visualization is a bar chart showing the top 3 countries: :alt: esql kibana bar chart ::: -To make changes to the visualization, like changing the visualization type, axes and colors, click the pencil button (![esql icon edit visualization](/images/elasticsearch-reference-esql-icon-edit-visualization.svg "")). This opens an in-line editor: +To make changes to the visualization, like changing the visualization type, axes and colors, click the pencil button (![esql icon edit visualization](../../../images/elasticsearch-reference-esql-icon-edit-visualization.svg "")). This opens an in-line editor: :::{image} ../../../images/elasticsearch-reference-esql-kibana-in-line-editor.png :alt: esql kibana in line editor ::: -You can save the visualization to a new or existing dashboard by clicking the save button (![esql icon save visualization](/images/elasticsearch-reference-esql-icon-save-visualization.svg "")). Once saved to a dashboard, you’ll be taken to the Dashboards page. You can continue to make changes to the visualization. Click the options button in the top-right (![esql icon options](/images/elasticsearch-reference-esql-icon-options.svg "")) and select **Edit ES|QL visualization** to open the in-line editor: +You can save the visualization to a new or existing dashboard by clicking the save button (![esql icon save visualization](../../../images/elasticsearch-reference-esql-icon-save-visualization.svg "")). Once saved to a dashboard, you’ll be taken to the Dashboards page. You can continue to make changes to the visualization. Click the options button in the top-right (![esql icon options](../../../images/elasticsearch-reference-esql-icon-options.svg "")) and select **Edit ES|QL visualization** to open the in-line editor: :::{image} ../../../images/elasticsearch-reference-esql-kibana-edit-on-dashboard.png :alt: esql kibana edit on dashboard @@ -236,13 +236,13 @@ You can use {{esql}} queries to create panels on your dashboards. To add a panel :alt: esql dashboard panel ::: -Check the {{esql}} query by clicking the Panel filters button (![Panel filters button on panel header](/images/elasticsearch-reference-dashboard_panel_filter_button.png "")): +Check the {{esql}} query by clicking the Panel filters button (![Panel filters button on panel header](../../../images/elasticsearch-reference-dashboard_panel_filter_button.png "")): :::{image} ../../../images/elasticsearch-reference-esql-dashboard-panel-query.png :alt: esql dashboard panel query ::: -You can also edit the {{esql}} visualization from here. Click the options button in the top-right (![esql icon options](/images/elasticsearch-reference-esql-icon-options.svg "")) and select **Edit ESQL visualization** to open the in-line editor. +You can also edit the {{esql}} visualization from here. Click the options button in the top-right (![esql icon options](../../../images/elasticsearch-reference-esql-icon-options.svg "")) and select **Edit ESQL visualization** to open the in-line editor. :::{image} ../../../images/elasticsearch-reference-esql-dashboard-panel-edit-visualization.png :alt: esql dashboard panel edit visualization From ac9748078e04a331d326fba0f20e9f118c40f2a3 Mon Sep 17 00:00:00 2001 From: Florent Le Borgne Date: Tue, 4 Feb 2025 15:31:57 +0100 Subject: [PATCH 7/7] img width --- explore-analyze/query-filter/filtering.md | 1 + 1 file changed, 1 insertion(+) diff --git a/explore-analyze/query-filter/filtering.md b/explore-analyze/query-filter/filtering.md index 45fcadbbb..9b6e62322 100644 --- a/explore-analyze/query-filter/filtering.md +++ b/explore-analyze/query-filter/filtering.md @@ -42,6 +42,7 @@ Display data within a specified time range when your index contains time-based e :::{image} ../../images/kibana-time-relative.png :alt: Time filter showing relative time +:width: 350px ::: The global time filter limits the time range of data displayed. In most cases, the time filter applies to the time field in the data view, but some apps allow you to use a different time field.