2.18 release blog.

Signed-off-by: Nate B <96254688+nateynateynate@users.noreply.github.com>

_posts/2024-11-05-get-started-with-opensearch-2-18.md

@@ -0,0 +1,92 @@
+---
+layout: post
+title:  "Get started with OpenSearch 2.18"
+authors:
+  - jamesmcintyre
+date:   2024-11-05 
+categories:
+  - releases
+meta_keywords: OpenSearch 2.18, machine learning, generative AI, vector search, search performance, vector workloads, inference processors, workspace, discover, search pipelines, rerank
+meta_description: Explore the new features and enhancements in OpenSearch 2.17, including enhanced tools for building generative AI applications, improved performance for search workloads, and ease-of-use updates to help your teams collaborate.
+---
+
+Our latest release brings a number of updates that help you build generative AI (GenAI) applications, improve performance for core search and related workloads, and advance the way your teams work together with significant ease-of-use updates for OpenSearch Dashboards. What follows is an overview of a few of the exciting features now available with [OpenSearch 2.18](https://opensearch.org/downloads.html). For a comprehensive look at what's new, check out the [release notes](https://github.com/opensearch-project/opensearch-build/blob/main/release-notes/opensearch-release-notes-2.18.0.md), and visit [OpenSearch Playground](https://playground.opensearch.org/app/home) to explore OpenSearch's visualization toolkit.
+
+
+## Vector database and GenAI
+
+OpenSearch 2.18 introduces several updates to the vector database and machine learning (ML) toolkit that help you build more flexible and performant ML and GenAI applications.
+
+**Accelerate vector search application performance**
+
+This release introduces several performance improvements for your ML and GenAI workloads. Highlights include:
+
+* More efficient memory and cache management: OpenSearch 2.18 introduces [optimizations](https://github.com/opensearch-project/k-NN/pull/2182) for better native library memory and cache management, helping ensure efficient resource utilization.
+* Improved indexing and search performance: The [integration of AVX512 SIMD](https://github.com/opensearch-project/k-NN/pull/2110) (Single Instruction, Multiple Data) support for the Faiss engine lets you take advantage of advanced hardware capabilities that deliver significant indexing and search performance improvements.
+* Index acceleration: An updated approach to [creating vector data structures](https://github.com/opensearch-project/k-NN/issues/1942) enhances index acceleration by skipping Hierarchical Navigable Small World (HNSW) indexes and avoiding overhead for smaller segments.
+
+**Improve search pipeline outputs**
+
+OpenSearch 2.18 introduces several improvements to the [ML inference search response processor](https://opensearch.org/docs/latest/search-plugins/search-pipelines/ml-inference-search-response/). You can now save predictions generated by ML models to the `ext.ml_inference` search extension in the search response rather than only writing predictions to hits, offering more flexible result handling for GenAI use cases. For reranking use cases, OpenSearch now provides the ability to pass query text to the model input dataset alongside documents, supporting predictions that are more context aware. Additionally, when combined with the rerank processor, the ML inference search response processor can now rank search hits and update scores based on model predictions. Together, these enhancements improve the integration of ML models in search pipelines, enabling more sophisticated and context-aware document ranking and result augmentation.
+
+**Rerank by field for enhanced search results**
+
+This release enhances OpenSearch's [rerank processor](https://opensearch.org/docs/latest/search-plugins/search-pipelines/rerank-processor/) by introducing a [new reranker type](https://opensearch.org/docs/latest/search-plugins/search-pipelines/rerank-processor/#the-ml_opensearch-reranker-type) called `ByField`. This allows you to perform a second-level rerank on your search results based on a specified target field. You can therefore achieve greater search relevance, particularly when applying a numerical score from a previous document search result. When a previous search response processor has added a numerical result to the search hits—such as those generated by an ML inference processor—this new rerank type lets you adjust the ranking based on that field. You can also retain your original scores, making it easy to evaluate whether the adjusted search relevance meets your expectations. This lets you rerank results by the previous score, allowing you to return to your initial starting point if desired.
+
+
+## Search
+
+This release also delivers new functionality that enhances search operations.
+
+**Define search pipelines through the Multi-search API**
+
+In earlier OpenSearch versions, users of the [Multi-search API](https://opensearch.org/docs/latest/api-reference/multi-search/) needed to define a default search pipeline for an index, which limited their ability to deploy search pipelines effectively. With OpenSearch 2.18, you can now define the `search_pipeline` parameter directly in the Multi-search API request body, offering greater flexibility and control over search operations. The following is an example of how the parameter can be defined through the Multi-search API:
+
+```
+{ "index": "test"}
+{ "query": { "match_all": {} }, "from": 0, "size": 10, "search_pipeline": "my_pipeline"}
+{ "index": "test-1", "search_type": "dfs_query_then_fetch"}
+{ "query": { "match_all": {} }, "search_pipeline": "my_pipeline1" }
+```
+
+
+## Cost, performance, and scalability
+
+This release includes updates that help you improve the cost, performance, and scalability of your OpenSearch clusters.
+
+**Reduce workload for cluster statistics and optimize troubleshooting at scale**
+
+OpenSearch 2.18 introduces new [paginated APIs](https://opensearch.org/docs/latest/api-reference/list/index) for `_cat/indices` and `_cat/shards`. `_cat APIs` are useful when troubleshooting an issue in a cluster but don't scale well in larger clusters, generating significant overhead on the node receiving the request and occasional timeouts due to larger response sizes. The new paginated APIs can be accessed as `_list/indices` and `_list/shards` APIs. The index/shard response is returned one page at a time, and you can request additional pages using the `next` token; because the index/shard statistics are collected and aggregated only for the requested page, the APIs scale significantly better in larger clusters. This release also introduces metric filters for the Cluster Stats API, which provides complete cluster statistics related to indexes, shards, and nodes. Computing these statistics in larger clusters can result in costly overhead; now the API provides the option to specify filters in the input in order to retrieve only a specific subset of cluster statistics, meaning that you can retrieve the statistics you need without overburdening the cluster.
+
+
+## Stability, availability, and resiliency
+
+OpenSearch 2.18 also adds functionality that helps you maintain the stability, availability, and resiliency of your OpenSearch deployments.
+
+**Identify resource-intensive query patterns with enhanced query grouping**
+
+Monitoring the [top N queries](https://opensearch.org/docs/latest/observing-your-data/query-insights/top-n-queries/) can help you identify the most resource-intensive queries, based on metrics like latency, CPU, and memory usage, within a given time frame. However, if one highly demanding query is repeatedly executed, it can dominate the top N slots, overshadowing other resource-intensive queries. To overcome this, grouping queries by similarity can provide insight into various high-impact query patterns rather than individual instances. As of OpenSearch 2.17, top N queries could be [grouped by similarity](https://opensearch.org/docs/latest/observing-your-data/query-insights/grouping-top-n-queries/#grouping-queries-by-similarity), grouping queries that share the same query structure. In version 2.18, this grouping functionality is enhanced to also account for field names and data types, allowing you to group queries at a finer level of detail. This allows for even more accurate query grouping, helping better identify and analyze resource-intensive query patterns across diverse query types.
+
+
+## Ease of use
+
+This release introduces new capabilities designed to transform the way you experience, interact with, and collaborate through OpenSearch. 
+
+**Create and collaborate across teams with Workspace**
+
+OpenSearch 2.18 introduces [Workspace](https://opensearch.org/docs/latest/dashboards/workspace/), a multi-tenant environment designed to increase team collaboration. We've received substantial community feedback on the value of tenant-level, "isolated" spaces for index patterns, dashboards, and saved objects in OpenSearch Dashboards. With this release, you can now manage collaboration with granular access controls, easily adding or removing collaborators to or from your workspace and controlling their permissions at the workspace level. As the creator of a workspace, you can assign role-based permissions: "read-only" for stakeholders who only need to access your workspace and view data visualizations; "read and write" for colleagues who need to create and update saved objects; and "admin" for co-owners who need full permissions to read, write, configure settings, or delete the workspace. Additionally, There are five workspace types, each tailored toward a different use case—Security Analytics, observability, search, essentials, and analytics—each with a different set of available features that provide a tailored experience. To learn how to access Workspace, visit the [documentation](https://opensearch.org/docs/latest/dashboards/workspace/).
+
+
+
+**Boost efficiency and simplify workflows with a redesigned homepage and navigation structure**
+
+OpenSearch 2.18 debuts a redesigned homepage and navigation structure that offer a richer, more user-friendly experience. The new OpenSearch homepage serves as a central hub, with clear pathways allowing users to access and create workspaces. A redesigned navigation structure moves away from complex menus, offering a cleaner, more organized navigation bar that adapts to the selected workspace and supports tailored workflows for each use case. Once a workspace is selected, users are presented with an environment suited to their specific tasks; each workspace type provides dedicated tools, views, and navigational paths to help teams work more efficiently. Streamlined "Get Started" guides simplify the process for new users and direct them to key functionalities based on their needs. Whether managing security incidents, conducting data analysis, or monitoring infrastructure, users can access necessary tools with fewer clicks. Together, these updates aim to streamline data management, analysis, and collaboration across teams, reducing complexity and driving productivity.
+
+**Query data and streamline insights with an updated Discover interface**
+
+A new experimental [Discover](https://opensearch.org/docs/latest/dashboards/discover/index-discover/) interface enhances the query experience and offers greater customization options. The interface allows analysts to collapse unused sections and streamline their workspace. Over the past year, we've gathered valuable feedback from users across OpenSearch's query interfaces, including Piped Processing Language (PPL) in Log Explorer (for observability) and SQL in Query Workbench. In response, we're adding PPL and SQL  as query options in Discover alongside Dashboards Query Language (DQL) and Lucene, allowing analysts to use their preferred query language. We've also improved the data selector and added autocomplete functionality—a highly requested feature. The new experience is available behind a feature flag, which you can enable by navigating to **Dashboards Management**, selecting **Advanced Settings**, and toggling "Enable query enhancements" to ON. We welcome your [feedback on this functionality](https://github.com/opensearch-project/OpenSearch-Dashboards/issues/8813) and will be collecting community input until version 2.21, when the new Discover experience will become generally available. Additionally, we've introduced a new pie chart visualization in VisBuilder for Vega-based visualizations. To try it out, enable "Enable vega transformation in visbuilder" in **Advanced Settings**.
+
+## Explore OpenSearch 2.18
+
+Today's release is [available for download](https://www.opensearch.org/downloads.html) and ready to explore on [OpenSearch Playground](https://playground.opensearch.org/app/home#/). For more information about this release, check out the [release notes](https://github.com/opensearch-project/opensearch-build/blob/main/release-notes/opensearch-release-notes-2.18.0.md) as well as the [documentation release notes](https://github.com/opensearch-project/documentation-website/blob/main/release-notes/opensearch-documentation-release-notes-2.18.0.md). As always, we hope you'll share your feedback on this release on our [community forum](https://forum.opensearch.org/) or on the [OpenSearch Slack workspace](https://opensearch.org/slack.html).
+