Elasticsearch

Page last updated:

Overview

Elasticsearch is a powerful open-source search engine for full text search.

Integrating the Service With Your App

After the creation of the service and the service key, you’ll find all required credentials in this service key, as shown here:

{
  "host": "https://d15c5caa5a3147ed9b444e4694162b3f.elasticsearch.lyra-836.appcloud.swisscom.com",
  "kibana_system_username": "username",
  "kibana_system_password": "password",
  "logstash_system_username": "username",
  "logstash_system_password": "password",
  "kibana_read_only_username": "username",
  "kibana_read_only_password": "password",
  "kibana_dashboard_only_username": "username",
  "kibana_dashboard_only_password": "password",
  "full_access_username": "username",
  "full_access_password": "password"
}

User Privileges

Each service key or binding to the elasticsearch-service comes with multiple users with different privileges.

  • kibana_read_only_username: a user which has read access to Kibana. This user should be used when giving access to your Kibana.
  • kibana_dashboard_only_username: a user which has the kibana_dashboard_only_role assigned and can only access dashboards in Kibana.
  • full_access_username: a user which has read and write access to all indices. This user is optimal for integration with your application.
  • kibana_system_username: a system user for the Kibana application. This is a system user and you should not use it (e.g. for logging in to Kibana).
  • logstash_system_username: a system user for the Logstash application. This is a system user and you should not use it.

Versions

For the Elasticsearch service multiple versions are available. Currently available versions:

  • 7.16.1

Deprecated versions:

  • 6.8.*
  • 7.8.*

To create a cluster with a specific version you must provide the version (from the list above) as a parameter. E.g.:

cf create-service elasticsearch xxsmall <my-elasticsearch> -c '{"version":"X.Y.Z"}'

Or you can update any existing service by using the update-service command. E.g.:

cf update-service <service-name> -c '{"version":"X.Y.Z"}'

Note: when using a custom Elasticsearch version you must specify the version on all cf update-service calls. Otherwise, we will use the default version.

Version Upgrade

Upgrading a Service Instance requires cf CLI v6.46.0+ and CAPI release 1.83.0+

Elasticsearch supports upgrading service instances to newer versions. Upgrades to specific versions of Elasticsearch are described below.

To upgrade your service instances, do the following:

  1. Confirm that an upgrade is available by running cf services and reviewing the upgrade available column:

    $ cf services
    Getting services in org acceptance / space dev as admin...
    
    name      service         plan     bound apps   last operation     broker         upgrade available
    mydb      elasticsearch   small                 create succeeded   ece-broker     yes
    otherdb   elasticsearch   medium                create succeeded   ece-broker     no
    
  2. Upgrade the service instance using the --upgrade flag:

    $ cf update-service mydb --upgrade
    You are about to update mydb.
    Warning: This operation may be long running and will block further operations on the service until complete.
    Really update service mydb? [yN]: y
    OK
    

We recommend creating a backup before evey upgrade, to prevent possible data loss. The preferred method is using backman.

Upgrade to Specific Version

The upgrade to the specific versions is done through the CF CLI by running the update service command.

cf update-service <service-name> -c '{"version":"X.Y.Z"}'

We recommend creating a backup before evey elasticsearch upgrade, to prevent possible data loss. The preferred method is using backman.

Upgrade to Elasticsearch version 7.x

Before you can upgrade to the latest major release of Elasticsearch (version 7) you must check for any incompatibilities of your existing indices. We recommend first upgrading to the latest minor release of Elasticsearch version 6 (currently version 6.8.2) and then upgrade to version 7.x. Currently the upgrade assistant is not available.

Plan Upgrade

Elasticsearch service can be upgraded to larger plans (downgrading to smaller plans is not possible).

The upgrade to a larger plan is done through the CF CLI by running the update service command.

cf update-service <service-name> -p <new-plan-name>

Note: if you use a different Elasticsearch version than the default you must specify that version as well. Otherwise, we will use the default version.

cf update-service <service-name> -p <new-plan-name> -c '{"version":"X.Y.Z"}'

X-Pack

Our Elasticsearch service comes with X-Pack preinstalled and activated. The X-Pack license fee is included in the price.

Kibana

We have deprecated the Kibana buildpack with the release of cflinuxfs3 (Ubuntu 18).

You can find a guide on how to set up Kibana with Docker here: Kibana Docker

Index Lifecycle Management

To clean up some old data and free up space in your elasticsearch service, you have to set up index lifecycle policies. This is necessary, because otherwise your instance would get overloaded over time and stop working properly.

The corresponding guide can be found here: Index Lifecycle Management

Logstash

We have deprecated the Logstash buildpack with the release of cflinuxfs3 (Ubuntu 18).

You can find a guide on how to set up Logstash with Docker here: Logstash Docker

Curator

Current versions of ElasticSearch (6.7+) offer Index Lifecycle Management as alternative to Curator.

For older versions, use cf-curator-cron as workaround to periodically clean up your Indices.

High Availability

The Elasticsearch cluster consists of 3 nodes in 3 different availability zones. By default, each index in Elasticsearch has 5 primary shards and 1 replica allocated to it. If you don’t want to use the default settings, make sure to configure the number of shards & replicas accordingly in the index templates. Swisscom cannot guarantee the availability of your data unless you configure at least one replica.

Backups

If you want to create backups there is an app available that allows for automatic scheduled backups: backman - the appcloud backup manager. Any Elasticsearch service instance bound to this app will be automatically backed up, can be downloaded and also restored on-demand.

Further reading

Official ElasticSearch Documentation

View the source for this page in GitHub