🥇Vela → smart cache for time series and more

In fintech, we often have to process large volumes of currency exchange rate data. We obtain data from various sources, each with its own representation of how to extrapolate values for tomorrow, the day after tomorrow, the next month, and even the next three years. If someone could predict exchange rates, correctly, it would be time to close the business and just simply exchange money back and forth. Some sources are more trustworthy, while others deliver mostly trash, with rare instances of almost correct values, but only for exotic pairs. Our job is to sift through these tens of thousands of values per second and determine what exactly to show clients. We need to filter out the single correct value from tons of dirt and sludge, just like flamingos do during lunch.

A distinctive feature of flamingos is their massive downturned beak, which they use to filter food from water or mud.
— Wikipedia

Thus, the library was born Vela, which stores state cache for multiple values over specified time intervals. Under the hood, it automatically filters out bad and outdated data, while also providing access to the latest N validated values for each key (currency pairs, in our case).

Let's say we are collecting rates for three currency pairs. The simplest definition Vela for storing the current state might look something like this:

defmodule Pairs do
  use Vela,
    eurusd: [sorter: &Kernel. 0
end

Updating values

Vela.put/3 The function will sequentially do the following:

call validator on the value, if defined (see the section Validation below);
add the value either to the list of good values if validation is successful, or to the service list :__errors__ otherwise;
call sorting if sorter is defined for the given key, or simply place the value at the head of the list (LIFO, see the section Sorting below);
trim the list according to the parameter :limit passed during creation;
return the updated structure Vela.

iex|1 > pairs = %Pairs{}
iex|2 > Vela.put(pairs, :eurcad, 1.0)
#⇒ %Pairs{..., eurcad: [1.0], ...}
iex|3 > Vela.put(pairs, :eurcad, -1.0)
#⇒ %Pairs{__errors__: [eurcad: -1.0], ...}
iex|4 > pairs |> Vela.put(:eurusd, 2.0) |> Vela.put(:eurusd, 1.0)
#⇒ %Pairs{... eurusd: [1.0, 2.0]}

Also Vela implements Access, so any of the standard functions for deep updating structures from the arsenal can be used to update values Kernel: Kernel.get_in/2, Kernel.put_in/3, Kernel.update_in/3, Kernel.pop_in/2, and Kernel.get_and_update_in/3.

Validation

A validator can be defined as:

an external function with one argument (&MyMod.my_fun/1), it will receive only the value for validation;
an external function with two arguments, &MyMod.my_fun/2, it will receive the pair serie, value for validation;
a module implementing Vela.Validator;
configuration parameter threshold, and — optionally — compare_by, see the section Comparison below.

If validation is successful, the value is added to the list under the corresponding key; otherwise, the tuple {serie, value} is sent to :__errors_.

Comparison

Values stored in these series can be anything. To teach Vela them to compare, you need to pass compare_by a parameter in the series definition (unless the values can be compared by standard Kernel.</2); this parameter must be of type (Vela.value() -> number()). By default, it is simply & &1.

Also, you can pass the parameter comparator for computing delta values (min/max); for example, by passing Date.diff/2 as a comparator, you can obtain accurate deltas for dates.

Another convenient way to work is to pass a parameter threshold, which defines the maximum allowable ratio of the new value to {min, max} interval. Since it is specified in percentages, the check does not use comparator, but still uses compare_by. For example, to specify a threshold for date times, you need to specify compare_by: &DateTime.to_unix/1 (to get an integer value) and threshold: 1, resulting in new values being permitted only if they fall within ±band interval from the current values.

Finally, you can use Vela.equal?/2 for comparing two caches. If the values define a function equal?/2 or compare/2, then these functions will be used for comparison; otherwise, we simply use ==/2.

Getting values

Processing the current state typically begins with a call to Vela.purge/1, which removes outdated values (if validator tied to timestamps). Then you can call Vela.slice/1, which will return keyword with row names as keys and the first, current values.

You can also use get_in/2/pop_in/2 for low-level access to values in each row.

Application

Vela can be extremely useful as a time series cache in the state of a process like GenServer/Agent. We want to never use outdated exchange rate values, and for that, we simply maintain a process with state managed Vela, with the validator shown below.

@impl Vela.Validator
def valid?(_key, %Rate{} = rate),
  do: Rate.age(rate) < @death_age

and Vela.purge/1 quietly removes all outdated values every time we need the data. To access the current values, we simply call Vela.slice/1, and when a bit of history is needed (the entire series), we just return it — already sorted — with validated values.

Happy caching of time series!

Source: habr.com