ProHoster > Blog > Administration > Riak Cloud Storage. Part 1. Configuring Riak KV

Riak Cloud Storage. Part 1. Configuring Riak KV

Riak CS (Cloud Storage) – an easy-to-use software for organizing object storage that operates on top of Riak KV. Riak (KV) is a distributed NoSQL key-value database. Riak CS is designed to provide simplicity, availability, and distribution of cloud storage at any scale, and can be used to build cloud architectures—whether public or private—or as infrastructure storage for high-load applications and services. The Riak CS API is compatible with Amazon S3 and supports report generation for various scenarios.

Riak Cloud Storage. Part 1. Configuring Riak KV
This article is a free translation of the official Riak CS system guide version 2.1.1.

In the Riak CS storage system, three components work together, meaning that each component must be configured to work with the others:

  • Riak (KV) – a database system that acts as the ultimate system.
  • Riak CS — a cloud storage layer on top of Riak that provides storage and API capabilities, stores files and metadata in Riak, and then delivers them to end users.
  • Stanchion — manages requests involving globally unique entities, such as buckets and users in a Riak instance. For example, creating users, creating or deleting buckets.

Additionally, you can also configure an S3 client to communicate with the Riak CS system.

You should plan for one Riak node for each Riak CS node in your system. Riak and Riak CS nodes can run on different physical machines, but in most cases, it is preferable to run one Riak node and one Riak CS node on the same physical machine. Assuming a single physical machine has enough power to meet the needs of both Riak and Riak CS nodes, you will generally see better performance due to reduced network latency.

If your system consists of multiple nodes, the configuration primarily involves setting up communication between the components. Other settings, such as where log files will be stored, have default values and only need to be changed if you want to use non-standard values.

System component configuration. Configuring Riak KV for CS

Since Riak CS is an application built on top of Riak, it's crucial to pay attention to your Riak configuration when running Riak CS. This document serves both as a guide for configuring Riak and as a reference document describing important configuration parameters.

Before configuring, ensure that Riak KV and Riak CS are installed on each node in your cluster. Stanchion, on the other hand, should be installed on only one node in the entire cluster.

Riak Cloud Storage. Part 1. Configuring Riak KV

Backends for Riak CS

By default, the backend used by Riak is Bitcask, but the Riak CS package includes a special backend that should be used by the Riak cluster as part of the Riak CS system. The standard version has the standard Multi backend that comes with Riak.

The same Riak buckets used inside Riak CS utilize secondary indices, which now require the LevelDB backend. Other components of the Riak CS system can benefit from using the Bitcask backend. The exemplary Multi backend is included in Riak CS to take advantage of both backends for the best combination of performance and functionality. The following section explains how to properly configure Riak to use this Multi backend.

Backend — is what Riak will use to store data. Riak KV has several backends: Bitcask, LevelDB, Memory, and Multi.

Additionally, the data storage calculation system utilizes Riak MapReduce to summarize files into buckets. This means you must inform all Riak nodes where to look for prepared Riak CS files before calculating storage.

Several other parameters need to be changed to configure the Riak node as part of the Riak CS system, such as IP address and the IP address and port for messaging over Protocol Buffers. Other settings can be modified if necessary. The following sections describe how to configure the Riak node to work as part of the Riak CS system.

Configuring the Riak backend

First, the riak.conf or advanced.config/app.config configuration files are edited. These files may be located in the /etc/riak or /opt/riak/etc directories. By default, Riak uses the Bitcask backend. The first thing we need to do is modify the configuration file by removing the following line:

RIAK.CONF

## Delete this line:
storage_backend = bitcask

ADVANCED.CONFIG

{riak_kv,
 [ %% Delete this line: 
{storage_backend, riak_kv_bitcask_backend},
 ]}

APP.CONFIG

{riak_kv, 
  [ %% Delete this line:
    {storage_backend, riak_kv_bitcask_backend},
]}

Next, we need to indicate the need for RiakCS modules for Riak and specify Riak to use the configured backend that provides Riak CS. We need to use the advanced.config or app.config file and add the following options:

ADVANCED.CONFIG

{eleveldb, [
    {total_leveldb_mem_percent, 30}
    ]},
{riak_kv, [
    %% Other configs
    {add_paths, ["/usr/lib/riak-cs/lib/riak_cs-2.1.1/ebin"]},
    {storage_backend, riak_cs_kv_multi_backend},
    {multi_backend_prefix_list, [{

APP.CONFIG

{eleveldb, [
    {total_leveldb_mem_percent, 30}
    ]},
{riak_kv, [
    %% Other configs
    {add_paths, ["/usr/lib/riak-cs/lib/riak_cs-2.1.1/ebin"]},
    {storage_backend, riak_cs_kv_multi_backend},
    {multi_backend_prefix_list, [{

It is very important to note that many of these values will depend on directory variations specific to your operating system, so follow instructions accordingly. For example, the add_paths parameter assumes that Riak CS is installed in /usr/lib/riak-cs, while the data_root parameter assumes that Riak is installed in /var/lib. (Note: In my case, it was add_paths — /usr/lib64/riak-cs/).

This configuration assumes that Riak CS is installed on the same machine as Riak. If not, the package needs to be copied to a separate host.

Configuring Sibling Creation

Now, we need to set the allow_mult parameter to true. We can add a line to the riak.conf configuration file or the riak_core section in advanced.config or app.config.

RIAK.CONF

buckets.default.allow_mult = true

ADVANCED.CONFIG

{riak_core, [
    %% Other configs
    {default_bucket_props, [{allow_mult, true}]},
    %% Other configs
]}

APP.CONFIG

{riak_core, [
    %% Other configs
    {default_bucket_props, [{allow_mult, true}]},
    %% Other configs
]}

This will allow Riak to create siblings, which are necessary for the functioning of Riak CS. If you are connecting to Riak CS using a client library, don't worry: you won't have to resolve conflicts, as all Riak CS operations are strictly resolved according to their definition.

Sibling — is a way of storing multiple objects under one key so that the object can have different values across different nodes.

Note: allow_mult
Any Riak node that also supports Riak CS will always have the allow_mult parameter set to true. Riak CS will fail to start if the value is false.

Configuring the node name and IP address

Each Riak node has a name that can be specified in riak.conf under the nodename option. If you are using the app.config configuration file, you need to create a file called vm.args in the same directory as the app.config file and specify the node name using the -name flag. We recommend naming nodes in the format @. So if you have three running nodes on one host 100.0.0.1, you can name them riak1@100.0.0.1, riak2@100.0.0.1, and riak3@100.0.0.1, or you can give them more specific names like test_cluster1@100.0.0.1, user_data3@100.0.0.1, and so on. The example below demonstrates changing the node name to riak1@127.0.0.1, which will work on the local host.

RIAK.CONF

 nodename = riak1@127.0.0.1

VM.ARGS

 -name riak1@127.0.0.1

You must set names for all nodes before starting and adding them to the cluster.

Test Configuration

Now that all necessary node configurations have been made, we can try to start Riak:

SHELL

 riak start

Note: The response in my case is:

Riak Cloud Storage. Part 1. Configuring Riak KV

Here you need to wait a bit. Then you can start testing the running node.

SHELL

 riak ping

If the response is pong, then Riak is running: if the response is Node not responding to pings, then something went wrong.

Note: The response in my case is:

Riak Cloud Storage. Part 1. Configuring Riak KV

If the node started incorrectly, check the erlang.log.1 file in the /log directory of the node if the problem can be identified. One of the most common errors is invalid_storage_backend, which indicates that the path to the Riak CS library in advanced.config or app.config is incorrect (or Riak CS is not installed on the server). Despite this error, ensure you have not changed from riak_cs_kv_multi_backend to riak_kv_multi_backend.

Configuring Riak to use protocol buffers

Riak protocol buffer settings are located in riak.conf or the riak_api section in the advanced.config or app.config files, which are located in the /etc/riak/ directory. By default, the host has an IP address of 127.0.0.1 and port 8087. You need to change these if you plan to run Riak and Riak CS not in a local environment. Replace the IP address 127.0.0.1 of the Riak node and port 8087 with the appropriate ones.

RIAK.CONF

 listener.protobuf.internal = 10.0.2.10:10001

ADVANCED.CONF

{riak_api, [
    %% Other configs
    {pb, ["10.0.2.10", 10001]},
    %% Other configs
]}

APP.CONFIG

riak_api, [
    %% Other configs
    {pb, ["10.0.2.10", 10001]},
    %% Other configs
]}

Note:The value of the listener.protobuf.internal parameter in the riak.conf (or pb parameter in advanced.conf / app.config) file must match the values for riak_host in the Riak CS riak-cs.config and Stanchion stanchion.conf (or riak_host respectively in advanced.config/app.config) files.

Note about the port number
A different port number may be required if the port conflicts with ports used by another application or if you are using a load balancer or proxy server.

It is also recommended that users ensure that the size of Riak protobuf.backlog (or in advanced.config/app.config files, this is pb_backlog) is equal to or larger than the size of pool.request.size specified for Riak CS in riak-cs.config (or request_pool_size in advanced.config/app.conf files).

If the value of pool.request.size in Riak CS has been changed, the value of protobuf.backlog in Riak should also be updated.

Other Riak settings

The riak.conf and advanced.config files include other settings that configure log file creation and the paths for their storage. These settings have default values and should work in most cases. For more information, we recommend reading our documentation on configuration files.

Configuring the IP address for Riak

When setting up an IP address for Riak, you must ensure that Riak nodes have unique IP addresses, whether you are working with a single node or adding additional nodes to the system. The Riak IP address is contained in riak.conf or — if you are using the app.config file — in the vm.args configuration file located in the /etc/riak directory (or in /opt/riak/etc/ in other operating systems).

Initially, the line containing the Riak IP address points to the local host at this location:

RIAK.CONF

 nodename = riak@127.0.0.1

VM.ARGS

 -name riak@127.0.0.1

Replace 127.0.0.1 with the preferred IP address or hostname of the Riak node.

Performance and throughput settings

For performance reasons, we strongly recommend adding values to the configuration files Riak riak.conf or vm.args located in the /etc/riak/ or /opt/riak/etc directories.

RIAK.CONF

 erlang.max_ports = 65536

VM.ARGS

## This setting should already be present for recent Riak installs.
 -env ERL_MAX_PORTS 65536

Disabling JavaScript MapReduce

It is recommended not to use the deprecated JavaScript MapReduce in conjunction with any version of Riak CS. For performance reasons, you should disable the virtual machine that executes JavaScript MapReduce operations by configuring it in the riak.conf configuration file, or in the riak_kv section of advanced.conf or app.config:

RIAK.CONF

 javascript.map_pool_size = 0
 javascript.reduce_pool_size = 0
 javascript.hook_pool_size = 0

ADVANCED.CONFIG

{riak_kv, [
    %% Other configs
    {map_js_vm_count, 0},
    {reduce_js_vm_count, 0},
    {hook_js_vm_count, 0}
    %% Other configs
]}

APP.CONFIG

{riak_kv, [
    %% Other configs
    {map_js_vm_count, 0},
    {reduce_js_vm_count, 0},
    {hook_js_vm_count, 0}
    %% Other configs
]}

Next, we need to configure the remaining components of the Riak CS system.

Original manual.

Source: habr.com

Buy reliable website hosting with DDoS protection, VPS VDS servers 🔥 Buy reliable website hosting with DDoS protection, VPS VDS servers | ProHoster