Elasticsearch is a search engine with a JSON REST API that uses Lucene and is written in Java. A description of all the advantages of this engine is available on the official website. From here on, we will refer to Elasticsearch as ES.

Such engines are used for complex searches in document databases. For example, searches that take into account language morphology or geo-coordinates.

In this article, I will discuss the basics of ES using the example of indexing blog posts. I will demonstrate how to filter, sort, and search documents.

To avoid dependency on the operating system, I will make all requests to ES using CURL. There is also a Google Chrome plugin called sense..

Throughout the text, links to documentation and other sources are included. At the end, there are links for quick access to the documentation. Definitions of unfamiliar terms can be found in the glossary..

Installing ES

First, we will need Java. The developers recommend recommend installing Java versions newer than Java 8 update 20 or Java 7 update 55.

The ES distribution is available on the developer's website.After extracting the archive, you need to run bin/elasticsearch.Packages for apt and yum are also available, as well as the official image for Docker.. There is More about installation. After installation and startup, we will check the functionality:.

We should receive a response similar to this:

# для удобства запомним адрес в переменную
#export ES_URL=$(docker-machine ip dev):9200
export ES_URL=localhost:9200

curl -X GET $ES_URL

{ "name" : "Heimdall", "cluster_name" : "elasticsearch", "version" : { "number" : "2.2.1", "build_hash" : "d045fc29d1932bce18b2e65ab8b297fbf6cd41a1", "build_timestamp" : "2016-03-09T09:38:54Z", "build_snapshot" : false, "lucene_version" : "5.4.1" }, "tagline" : "You Know, for Search" }

Indexing

Let's add a post to ES:

server response:

# Добавим документ c id 1 типа post в индекс blog.
# ?pretty указывает, что вывод должен быть человеко-читаемым.

curl -XPUT "$ES_URL/blog/post/1?pretty" -d'
{
  "title": "Веселые котята",
  "content": "<p>A funny story about kittens<p>','
  "tags": [
    "kittens",
    "funny story"
  ],
  "published_at": "2014-09-12T20:44:42+00:00"
}'

{ "_index" : "blog", "_type" : "post", "_id" : "1", "_version" : 1, "_shards" : { "total" : 2, "successful" : 1, "failed" : 0 }, "created" : false }

ES automatically created the

index blog and type post. You can draw a conditional analogy: an index is a database, while a type is a table in that database. Each type has its own schema — mapping, just like a relational table. Mapping is automatically generated during the indexing of a document: In the server response, I've added the field values of the indexed document in comments:It is worth noting that ES does not differentiate between a single value and an array of values. For example, the title field contains just a header, while the tags field is an array of strings, though they are presented in mapping similarly.

# Получим mapping всех типов индекса blog
curl -XGET "$ES_URL/blog/_mapping?pretty"

Later, we will discuss mapping in more detail.

{
  "blog" : {
    "mappings" : {
      "post" : {
        "properties" : {
          /* "content": "<p>A funny story about kittens<p>", */
          "content" : {
            "type" : "string"
          },
          /* "published_at": "2014-09-12T20:44:42+00:00" */
          "published_at" : {
            "type" : "date",
            "format" : "strict_date_optional_time||epoch_millis"
          },
          /* "tags": ["kittens", "funny story"] */
          "tags" : {
            "type" : "string"
          },
          /*  "title": "Funny Kittens" */
          "title" : {
            "type" : "string"
          }
        }
      }
    }
  }
}

Retrieving a document by its ID:
New keys have appeared in the response:

Requests

_version

# извлечем документ с id 1 типа post из индекса blog
curl -XGET "$ES_URL/blog/post/1?pretty"

{
    "_index": "blog",
    "_type": "post",
    "_id": "1",
    "_version": 1,
    "found": true,
    "_source": {
        "title": "\u0412\u0435\u0441\u0435\u043b\u044b\u0435 \u043a\u043e\u0442\u044f\u0442\u0430",
        "content": "<p>A funny story about kittens<p>",
        "tags": [
            "\u043a\u043e\u0442\u044f\u0442\u0430",
            "\u0441\u043c\u0435\u0448\u043d\u0430\u044f \u0438\u0441\u0442\u043e\u0440\u0438\u044f"
        ],
        "published_at": "2014-09-12T20:44:42+00:00"
    }
}

_source . In general, all keys starting with and are present.In general, all keys starting with _ are related to services.

— some characteristic of the node (for example, a number). The key is needed to identify the element of the tree corresponding to this key. Example of a binary search tree: . In general, all keys starting with shows the document version. It is needed for the optimistic locking mechanism. For example, we want to change a document that has version 1. We submit the modified document and indicate that this is an edit of the document with version 1. If someone else also edited the document with version 1 and submitted changes before us, ES will not accept our changes, as it stores the document with version 2.

— some characteristic of the node (for example, a number). The key is needed to identify the element of the tree corresponding to this key. Example of a binary search tree: are present. contains the document we indexed. ES does not use this value for search operations, as indexes are used for searching. To save space, ES stores the compressed original document. If we only need the ID, not the entire original document, we can disable storing the original.

If we do not need additional information, we can obtain only the _source content:

curl -XGET "$ES_URL/blog/post/1/_source?pretty"

{
    "title": "\u0412\u0435\u0441\u0435\u043b\u044b\u0435 \u043a\u043e\u0442\u044f\u0442\u0430",
    "content": "<p>A funny story about kittens<p>",
    "tags": [
        "\u043a\u043e\u0442\u044f\u0442\u0430",
        "\u0441\u043c\u0435\u0448\u043d\u0430\u044f \u0438\u0441\u0442\u043e\u0440\u0438\u044f"
    ],
    "published_at": "2014-09-12T20:44:42+00:00"
}

You can also select only specific fields:

# извлечем только поле title
curl -XGET "$ES_URL/blog/post/1?_source=title&pretty"

{
  "_index" : "blog",
  "_type" : "post",
  "_id" : "1",
  "_version" : 1,
  "found" : true,
  "_source" : {
    "title" : "Funny Kittens"
  }
}

Let's index a few more posts and perform more complex queries.

curl -XPUT "$ES_URL/blog/post/2" -d'
{
  "title": "Playful Puppies",
  "content": "<p>A Funny Story About Puppies<p>"
  "tags": [
    "puppies",
    "funny story"
  ],
  "published_at": "2014-08-12T20:44:42+00:00"
}'

curl -XPUT "$ES_URL/blog/post/3" -d'
{
  "title": "How I Got My Kitten",
  "content": "<p>A heart-wrenching story about a poor street kitten<p>",
  "tags": [
    "kittens"
  ],
  "published_at": "2014-07-21T20:44:42+00:00"
}'

Sorting

# найдем последний пост по дате публикации и извлечем поля title и published_at
curl -XGET "$ES_URL/blog/post/_search?pretty" -d'
{
  "size": 1,
  "_source": ["title", "published_at"],
  "sort": [{"published_at": "desc"}]
}'

{
  "took" : 8,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "failed" : 0
  },
  "hits" : {
    "total" : 3,
    "max_score" : null,
    "hits" : [ {
      "_index" : "blog",
      "_type" : "post",
      "_id" : "1",
      "_score" : null,
      "_source" : {
        "title" : "Funny Kittens",
        "published_at" : "2014-09-12T20:44:42+00:00"
      },
      "sort" : [ 1410554682000 ]
    } ]
  }
}

We selected the last post. size limits the number of documents in the output. total shows the total number of documents that match the query. sort in the output contains an array of integers used for sorting. That is, the date was converted to an integer. More about sorting can be read in the documentation.

Filters and Queries

ES version 2 does not distinguish between filters and queries; instead, the concept of contexts is introduced..
The context of a query differs from the context of a filter in that the query generates a _score and is not cached. What _score is will be shown later.

Filtering by Date

We use the query range in the filter context:

# получим посты, опубликованные 1ого сентября или позже
curl -XGET "$ES_URL/blog/post/_search?pretty" -d'
{
  "filter": {
    "range": {
      "published_at": { "gte": "2014-09-01" }
    }
  }
}'

Filtering by tags

We use term query to search for document IDs containing a specific word:

# найдем все документы, в поле tags которых есть элемент 'котята'
curl -XGET "$ES_URL/blog/post/_search?pretty" -d'
{
  "_source": [
    "title",
    "tags"
  ],
  "filter": {
    "term": {
      "tags": "котята"
    }
  }
}'

{
  "took" : 9,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "failed" : 0
  },
  "hits" : {
    "total" : 2,
    "max_score" : 1.0,
    "hits" : [ {
      "_index" : "blog",
      "_type" : "post",
      "_id" : "1",
      "_score" : 1.0,
      "_source" : {
        "title" : "Funny Kittens",
        "tags" : [ "kittens", "funny story" ]
      }
    }, {
      "_index" : "blog",
      "_type" : "post",
      "_id" : "3",
      "_score" : 1.0,
      "_source" : {
        "title" : "How I Got My Kitten",
        "tags" : [ "kittens" ]
      }
    } ]
  }
}

Full-text search

Three of our documents contain the following in the content field:

A funny story about kittens
A Funny Story About Puppies
A heart-wrenching story about a poor street kitten

We use match query to search for document IDs containing a specific word:

# source: false означает, что не нужно извлекать _source найденных документов
curl -XGET "$ES_URL/blog/post/_search?pretty" -d'
{
  "_source": false,
  "query": {
    "match": {
      "content": "история"
    }
  }
}'

{
  "took" : 13,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "failed" : 0
  },
  "hits" : {
    "total" : 3,
    "max_score" : 0.11506981,
    "hits" : [ {
      "_index" : "blog",
      "_type" : "post",
      "_id" : "2",
      "_score" : 0.11506981
    }, {
      "_index" : "blog",
      "_type" : "post",
      "_id" : "1",
      "_score" : 0.11506981
    }, {
      "_index" : "blog",
      "_type" : "post",
      "_id" : "3",
      "_score" : 0.095891505
    } ]
  }
}

However, if you search for "stories" in the content field, you will not find anything, as the index contains only original words, not their stems. To enable quality search, you need to set up the analyzer.

Field _score shows relevance. If the query is performed in filter context, the _score value will always be equal to 1, indicating a complete match with the filter.

Analyzers

Analyzers are needed to transform the source text into a set of tokens.
Analyzers consist of one Tokenizer and several optional TokenFilters. The Tokenizer may precede multiple CharFilters. The Tokenizer splits the input string into tokens, for example, by spaces and punctuation marks. A TokenFilter can modify tokens, delete or add new ones, such as retaining only the root of a word, removing prepositions, or adding synonyms. CharFilter modifies the entire input string, for example, it can cut out HTML tags.

In ES there are several standard analyzers. For example, the analyzer russian.

Let's use api and let's see how the standard and russian analyzers process the string "Funny stories about kittens":

# используем анализатор standard       
# обязательно нужно перекодировать не ASCII символы
curl -XGET "$ES_URL/_analyze?pretty&analyzer=standard&text=%D0%92%D0%B5%D1%81%D0%B5%D0%BB%D1%8B%D0%B5%20%D0%B8%D1%81%D1%82%D0%BE%D1%80%D0%B8%D0%B8%20%D0%BF%D1%80%D0%BE%20%D0%BA%D0%BE%D1%82%D1%8F%D1%82"

{
  "tokens" : [ {
    "token" : "funny",
    "start_offset" : 0,
    "end_offset" : 6,
    "type" : "",
    "position" : 0
  }, {
    "token" : "stories",
    "start_offset" : 7,
    "end_offset" : 14,
    "type" : "",
    "position" : 1
  }, {
    "token" : "about",
    "start_offset" : 15,
    "end_offset" : 20,
    "type" : "",
    "position" : 2
  }, {
    "token" : "kittens",
    "start_offset" : 21,
    "end_offset" : 28,
    "type" : "",
    "position" : 3
  } ]
}

# используем анализатор russian
curl -XGET "$ES_URL/_analyze?pretty&analyzer=russian&text=%D0%92%D0%B5%D1%81%D0%B5%D0%BB%D1%8B%D0%B5%20%D0%B8%D1%81%D1%82%D0%BE%D1%80%D0%B8%D0%B8%20%D0%BF%D1%80%D0%BE%20%D0%BA%D0%BE%D1%82%D1%8F%D1%82"

{
  "tokens" : [ {
    "token" : "cheerful",
    "start_offset" : 0,
    "end_offset" : 7,
    "type" : "",
    "position" : 0
  }, {
    "token" : "story",
    "start_offset" : 8,
    "end_offset" : 15,
    "type" : "",
    "position" : 1
  }, {
    "token" : "cat",
    "start_offset" : 20,
    "end_offset" : 25,
    "type" : "",
    "position" : 3
  } ]
}

The standard analyzer splits the string at spaces and converts everything to lowercase. The Russian analyzer removes insignificant words, converts to lowercase, and retains the root of the words.

Let's look at the Tokenizers, TokenFilters, and CharFilters used by the Russian analyzer:

{
  "filter": {
    "russian_stop": {
      "type":       "stop",
      "stopwords":  "_russian_"
    },
    "russian_keywords": {
      "type":       "keyword_marker",
      "keywords":   []
    },
    "russian_stemmer": {
      "type":       "stemmer",
      "language":   "russian"
    }
  },
  "analyzer": {
    "russian": {
      "tokenizer":  "standard",
      /* TokenFilters */
      "filter": [
        "lowercase",
        "russian_stop",
        "russian_keywords",
        "russian_stemmer"
      ]
      /* CharFilters are absent */
    }
  }
}

Let's describe our analyzer based on Russian, which will strip HTML tags. We'll call it default, as an analyzer with that name will be used by default.

{
  "filter": {
    "ru_stop": {
      "type":       "stop",
      "stopwords":  "_russian_"
    },
    "ru_stemmer": {
      "type":       "stemmer",
      "language":   "russian"
    }
  },
  "analyzer": {
    "default": {
      /* adding HTML tags removal */
      "char_filter": ["html_strip"],
      "tokenizer":  "standard",
      "filter": [
        "lowercase",
        "ru_stop",
        "ru_stemmer"
      ]
    }
  }
}

First, all HTML tags will be removed from the source string, then it will be tokenized by the standard tokenizer, the resulting tokens will be converted to lowercase, insignificant words will be removed, and the remaining tokens will be reduced to their root forms.

Creating an index

Above, we described the default analyzer. It will be applied to all string fields. Our post contains an array of tags, and accordingly, the tags will also be processed by the analyzer. Since we are searching posts for an exact match of a tag, we need to disable analysis for the tags field.

Let's create the blog2 index with an analyzer and mapping where the analysis for the tags field is disabled:

curl -XPOST "$ES_URL/blog2" -d'\n{\n  "settings": {\n    "analysis": {\n      "filter": {\n        "ru_stop": {\n          "type": "stop",\n          "stopwords": "_russian_"\n        },\n        "ru_stemmer": {\n          "type": "stemmer",\n          "language": "russian"\n        }\n      },\n      "analyzer": {\n        "default": {\n          "char_filter": [\n            "html_strip"\n          ],\n          "tokenizer": "standard",\n          "filter": [\n            "lowercase",\n            "ru_stop",\n            "ru_stemmer"\n          ]\n        }\n      }\n    }\n  },\n  "mappings": {\n    "post": {\n      "properties": {\n        "content": {\n          "type": "string"\n        },\n        "published_at": {\n          "type": "date"\n        },\n        "tags": {\n          "type": "string",\n          "index": "not_analyzed"\n        },\n        "title": {\n          "type": "string"\n        }\n      }\n    }\n  }\n}'

Let's add the same 3 posts to this index (blog2). I will skip this process, as it is similar to adding documents to the blog index.

Full-text search with expression support

Let's get acquainted with another type of queries:

# найдем документы, в которых встречается слово 'истории'
# query -> simple_query_string -> query содержит поисковый запрос
# поле title имеет приоритет 3
# поле tags имеет приоритет 2
# поле content имеет приоритет 1
# приоритет используется при ранжировании результатов
curl -XPOST "$ES_URL/blog2/post/_search?pretty" -d'
{
  "query": {
    "simple_query_string": {
      "query": "истории",
      "fields": [
        "title^3",
        "tags^2",
        "content"
      ]
    }
  }
}'

Since we are using an analyzer with Russian stemming, this query will return all documents, even though they contain only the word ‘history’.

The query can contain special characters, for example:

""fried eggs" +(eggplant | potato) -frittata"

Query syntax:

+ signifies AND operation
| signifies OR operation
- negates a single token
" wraps a number of tokens to signify a phrase for searching
* at the end of a term signifies a prefix query
( and ) signify precedence
~N after a word signifies edit distance (fuzziness)
~N after a phrase signifies slop amount

# найдем документы без слова 'щенки'
curl -XPOST "$ES_URL/blog2/post/_search?pretty" -d'
{
  "query": {
    "simple_query_string": {
      "query": "-щенки",
      "fields": [
        "title^3",
        "tags^2",
        "content"
      ]
    }
  }
}'

# получим 2 поста про котиков

PS

If you're interested in similar articles or tutorials, have ideas for new articles, or have proposals for collaboration, I would be glad to hear from you via personal message or at m.kuzmin+habr@darkleaf.ru.

Source: habr.com