References
- Docs
- Elastic Search Python Package
- Python Elastic Search Quickstart
- Elasticsearch Crash Course
- Elasticsearch Tutorial Github
- Elasticsearch Tutorial for Beginners
The Elastic Stack is a group of open source products built by Elastic
The stack includes : Elastic Search, Kibana, Logstash, Beats, X-Pack
Elasticsearch is a distributed search and analytics engine, scalable data store, and vector database built on Apache Lucene. It's optimized for speed and relevance on production-scale workloads. Use Elasticsearch to search, index, store, and analyze data of all shapes and sizes in near real time. It is an open source analytics and full text search engine
- Full-text search: Build a fast, relevant full-text search solution using inverted indexes, tokenization, and text analysis.
- Vector database: Store and search vectorized data, and create vector embeddings with built-in and third-party natural language processing (NLP) models.
- Semantic search: Understand the intent and contextual meaning behind search queries using tools like synonyms, dense vector embeddings, and learned sparse query-document expansion.
- Hybrid search: Combine full-text search with vector search using state-of-the-art ranking algorithms.
- Build search experiences: Add hybrid search capabilities to apps or websites, or build enterprise search engines over your organization’s internal data sources.
- Retrieval augmented generation (RAG): Use Elasticsearch as a retrieval engine to supplement generative AI models with more relevant, up-to-date, or proprietary data for a range of use cases.
- Geospatial search: Search for locations and calculate spatial relationships using geospatial queries.
It is a dashboard used for analyzing and visualizing data in Elastic Search
It processes logs from application and sends them to elastic search. It is a free and open server-side data processing pipeline that ingests data from multitude of sources, transforms it and sends it to your favorite stash (kafka/elasticsearch)
Example:
new line log file event -> Logstash -> Kafka/Elasticsearch
Adds additional features to the Elasticsearch & Kibana
- Security : Adds authentication and authorization
- Monitors the performance of elastic stack and gets notified
- Enables machine learning on Kibana and elasticsearch
- Graph : Analyze relationship / relevance in data useful for recommendation. It exposes an API that we can integrate in our applications
- Elasticsearch SQL -> SQL API (SQL Query to Result) and Translate API (SQL Query to Query Domain Specific Language)
Beats is a collection of data shippers. They are light weight agents which send data from hundreds or thousands of machines and systems to Logstash or Elasticsearch.
Install Elastic Search With Docker - Click Here
- Create a new docker network
docker network create elastic- Pull the Elasticsearch Docker image
docker pull docker.elastic.co/elasticsearch/elasticsearch:8.17.0- Start an Elasticsearch container.
docker run --name es01 --net elastic -p 9200:9200 -it -m 1GB docker.elastic.co/elasticsearch/elasticsearch:8.17.0Machine learning features such as semantic search with ELSER require a larger container with more than 1GB of memory. If you intend to use the machine learning capabilities then start with
docker run --name es01 --net elastic -p 9200:9200 -it -m 6GB -e "xpack.ml.use_auto_machine_memory_percent=true" docker.elastic.co/elasticsearch/elasticsearch:8.17.0- Copy the generated elastic password and enrollment token
docker exec -it es01 /usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic
docker exec -it es01 /usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s kibana- Copy the SSL Certificate from container to local machine
docker cp es01:/usr/share/elasticsearch/config/certs/http_ca.crt .