Module cloudi_crdt

CloudI CRDT (Conflict-free Replicated Data Type)

This module provides a POLog CRDT implementation with an Erlang map data type for use in internal CloudI services. Usage of the module handles the replication of Erlang map state between Erlang processes to provide an eventually consistent data store among internal CloudI service processes.

.

Copyright © 2017-2018 Michael Truog

Version: 1.7.5 Dec 17 2018 18:07:36 ------------------------------------------------------------------------

Authors: Michael Truog (mjtruog at protonmail dot com).

Description

CloudI CRDT (Conflict-free Replicated Data Type)

This module provides a POLog CRDT implementation with an Erlang map data type for use in internal CloudI services. Usage of the module handles the replication of Erlang map state between Erlang processes to provide an eventually consistent data store among internal CloudI service processes.

The bootstrap functionality and the clean_vclocks functionality are not described in the POLog papers and are likely unique to this implementation. This additional functionality allows CloudI service processes that utilize cloudi_crdt to start, restart or fail (a crash, netsplit, etc.) without affecting other instances of cloudi_crdt that are configured with the same service name and manage the same data.

The cloudi_crdt functions that may be called within cloudi_service_init/4 are events_subscribe/3, events_subscribe/4, events_clear/3, new/1 and new/2. A CloudI service that uses cloudi_crdt should have a destination refresh method that is immediate.

The papers related to this implementation of the POLog CRDT are:

Carlos Baquero, Paulo Sérgio Almeida, Ali Shoker. Pure Operation-Based Replicated Data Types. 2017. https://arxiv.org/abs/1710.04469

Georges Younes, Ali Shoker, Paulo Sérgio Almeida, and Carlos Baquero. Integration Challenges of Pure Operation-based CRDTs in Redis. In First Workshop on Programming Models and Languages for Distributed Computing (PMLDC '16). ACM, New York, NY, USA, Article 7, 2016. http://haslab.uminho.pt/cbm/files/pmldc-2016-redis-crdts.pdf

Carlos Baquero, Paulo Sérgio Almeida, and Ali Shoker. Making operation-based crdts operation-based. In Proceedings of the First Workshop on Principles and Practice of Eventual Consistency, page 7. ACM, 2014. http://haslab.uminho.pt/ashoker/files/opbaseddais14.pdf

Mattern, Friedemann. "Virtual Time and Global States of Distributed Systems". Workshop on Parallel and Distributed Algorithms: pp. 215-226 (1988). http://homes.cs.washington.edu/~arvind/cs425/doc/mattern89virtual.pdf

Lamport, Leslie. "Time, clocks, and the ordering of events in a distributed system". Communications of the ACM. 21 (7): 558–565. (1978) http://research.microsoft.com/en-us/um/people/lamport/pubs/time-clocks.pdf

Data Types

bootstrap_state()

bootstrap_state() = {VClockAvg::float(), NodeId::node_id(), VClock::vclock(), VClocks::vclocks(), POLogMode::polog_mode(), POLog::polog(), Data::data()}

data()

data() = #{key() := value()}

event_id()

event_id() = cloudi_service:trans_id() | any()

event_type()

event_type() = assign | clear | decr | incr | put | update

events()

events() = #{key() := [event_type()]}

key()

key() = any()

node_id()

node_id() = {node(), cloudi_service:source()}

operation_write()

operation_write() = {assign, Id::event_id(), Key::key(), Value::value()} | {incr, Id::event_id(), Key::key(), Value::value()} | {decr, Id::event_id(), Key::key(), Value::value()} | {update, Id::event_id(), Key::key(), ModuleVersion::list(), Module::module(), Function::atom()} | {update, Id::event_id(), Key::key(), ModuleVersion::list(), Module::module(), Function::atom(), Argument1::any()} | {update_assign, Id::event_id(), Key::key(), Value::value(), ModuleVersion::list(), Module::module(), Function::atom()} | {update_assign, Id::event_id(), Key::key(), Value::value(), ModuleVersion::list(), Module::module(), Function::atom(), Argument1::any()} | {put, Id::event_id(), Key::key(), Value::value()} | {clear, Id::event_id(), Key::key()} | {clear_all, Id::event_id()}

options()

options() = [{service_name, string()} | {clean_vclocks, seconds()} | {clean_vclocks_failure, float() | 1..100} | {retry, non_neg_integer()} | {retry_delay, non_neg_integer()} | {timeout_default, cloudi_service:timeout_milliseconds()} | {priority_default, cloudi_service:priority()} | {priority_default_offset, -255..255 | undefined}]

polog()

polog() = [{vclock(), operation_write()}]

polog_mode()

polog_mode() = bootstrap | normal

seconds()

seconds() = 1..4294967

state()

state() = #cloudi_crdt{service_name_full = cloudi_service:service_name(), clean_vclocks_interval = seconds(), clean_vclocks_failure = number(), queue = cloudi_queue:state(), word_size = pos_integer(), node_id = node_id(), node_ids = [node_id()], vclock = vclock(), vclocks = vclocks(), polog_mode = polog_mode(), bootstrap_node_id = undefined | node_id(), bootstrap_states = [bootstrap_state()], bootstrap_requests = non_neg_integer(), polog = polog(), data = data(), events = events()}

value()

value() = any()

vclock()

vclock() = #{node_id() := non_neg_integer()}

vclocks()

vclocks() = #{node_id() := vclock()}

Function Index

assign/4

Assign a value iff none exists in the CloudI CRDT.

.
assign_id/5

Assign a value iff none exists in the CloudI CRDT with an event_id.

.
byte_size/2

Return the size of the CloudI CRDT in bytes.

.
clear/2

Clear the CloudI CRDT.

.
clear/3

Clear a key in the CloudI CRDT.

.
clear_id/3

Clear the CloudI CRDT with an event_id.

.
clear_id/4

Clear a key in the CloudI CRDT with an event_id.

.
decr/3

Decrement a numerical value by 1 in the CloudI CRDT.

.
decr/4

Decrement a numerical value in the CloudI CRDT.

.
decr_id/4

Decrement a numerical value by 1 in the CloudI CRDT with an event_id.

.
decr_id/5

Decrement a numerical value in the CloudI CRDT with an event_id.

.
events_clear/3

Clear a subscription to events from the CloudI CRDT.

.
events_subscribe/3

Subscribe to events from the CloudI CRDT.

.
events_subscribe/4

Subscribe to specific events from the CloudI CRDT.

.
find/3

Find a value in the CloudI CRDT.

.
fold/4

Fold a function over the CloudI CRDT.

.
get/3

Get a value from the CloudI CRDT.

.
handle_info/3

Handle all info messages related to the CloudI CRDT.

Must be called from the cloudi_service_handle_info/3 callback function.
handle_request/11

Handle a CRDT service request.

Must be called from the cloudi_service_handle_request/11 callback function.
incr/3

Increment a numerical value by 1 in the CloudI CRDT.

.
incr/4

Increment a numerical value in the CloudI CRDT.

.
incr_id/4

Increment a numerical value by 1 in the CloudI CRDT with an event_id.

.
incr_id/5

Increment a numerical value in the CloudI CRDT with an event_id.

.
is_key/3

Check if a key is in the CloudI CRDT.

.
keys/2

Get all keys in the CloudI CRDT.

.
new/1

Create a CloudI CRDT.

.
new/2

Create a CloudI CRDT.

.
put/4

Put a value into the CloudI CRDT.

.
put_id/5

Put a value into the CloudI CRDT with an event_id.

.
size/2

Get the size of the CloudI CRDT.

.
update/5

Update a value iff it exists in the CloudI CRDT.

Function Module:Function/1 must exist with the same version for every CloudI service process that shares this CloudI CRDT.
update/6

Update a value iff it exists in the CloudI CRDT.

Function Module:Function/2 must exist with the same version for every CloudI service process that shares this CloudI CRDT.
update_assign/6

Update a value or assign a value in the CloudI CRDT.

Function Module:Function/1 must exist with the same version for every CloudI service process that shares this CloudI CRDT.
update_assign/7

Update a value or assign a value in the CloudI CRDT.

Function Module:Function/2 must exist with the same version for every CloudI service process that shares this CloudI CRDT.
update_assign_id/7

Update a value or assign a value in the CloudI CRDT with an event_id.

Function Module:Function/1 must exist with the same version for every CloudI service process that shares this CloudI CRDT.
update_assign_id/8

Update a value or assign a value in the CloudI CRDT with an event_id.

Function Module:Function/2 must exist with the same version for every CloudI service process that shares this CloudI CRDT.
update_id/6

Update a value iff it exists in the CloudI CRDT with an event_id.

Function Module:Function/1 must exist with the same version for every CloudI service process that shares this CloudI CRDT.
update_id/7

Update a value iff it exists in the CloudI CRDT with an event_id.

Function Module:Function/2 must exist with the same version for every CloudI service process that shares this CloudI CRDT.
values/2

Get all values in the CloudI CRDT.

.
zero/3

Put a zero value in the CloudI CRDT.

.
zero_id/4

Put a zero value in the CloudI CRDT with an event_id.

.

Function Details

assign/4

assign(Dispatcher::cloudi_service:dispatcher(), Key::key(), Value::value(), State::state()) -> state()

Assign a value iff none exists in the CloudI CRDT.

assign_id/5

assign_id(Dispatcher::cloudi_service:dispatcher(), Key::key(), Value::value(), Id::event_id(), State::state()) -> state()

Assign a value iff none exists in the CloudI CRDT with an event_id.

byte_size/2

byte_size(Dispatcher::cloudi_service:dispatcher(), State::state()) -> non_neg_integer()

Return the size of the CloudI CRDT in bytes.

clear/2

clear(Dispatcher::cloudi_service:dispatcher(), State::state()) -> state()

Clear the CloudI CRDT.

clear/3

clear(Dispatcher::cloudi_service:dispatcher(), Key::key(), State::state()) -> state()

Clear a key in the CloudI CRDT.

clear_id/3

clear_id(Dispatcher::cloudi_service:dispatcher(), Id::event_id(), State::state()) -> state()

Clear the CloudI CRDT with an event_id.

clear_id/4

clear_id(Dispatcher::cloudi_service:dispatcher(), Key::key(), Id::event_id(), State::state()) -> state()

Clear a key in the CloudI CRDT with an event_id.

decr/3

decr(Dispatcher::cloudi_service:dispatcher(), Key::key(), State::state()) -> state()

Decrement a numerical value by 1 in the CloudI CRDT.

decr/4

decr(Dispatcher::cloudi_service:dispatcher(), Key::key(), Value::number(), State::state()) -> state()

Decrement a numerical value in the CloudI CRDT.

decr_id/4

decr_id(Dispatcher::cloudi_service:dispatcher(), Key::key(), Id::event_id(), State::state()) -> state()

Decrement a numerical value by 1 in the CloudI CRDT with an event_id.

decr_id/5

decr_id(Dispatcher::cloudi_service:dispatcher(), Key::key(), Value::number(), Id::event_id(), State::state()) -> state()

Decrement a numerical value in the CloudI CRDT with an event_id.

events_clear/3

events_clear(Dispatcher::cloudi_service:dispatcher(), Key::key(), State::state()) -> state()

Clear a subscription to events from the CloudI CRDT.

events_subscribe/3

events_subscribe(Dispatcher::cloudi_service:dispatcher(), Key::key(), State::state()) -> state()

Subscribe to events from the CloudI CRDT.

events_subscribe/4

events_subscribe(Dispatcher::cloudi_service:dispatcher(), Key::key(), EventTypes::[event_type()], State::state()) -> state()

Subscribe to specific events from the CloudI CRDT.

find/3

find(Dispatcher::cloudi_service:dispatcher(), Key::key(), State::state()) -> {ok, Value::value()} | error

Find a value in the CloudI CRDT.

fold/4

fold(Dispatcher::cloudi_service:dispatcher(), F::fun((Key::key(), Value::value(), AccIn::any()) -> AccOut::any()), AccInit::any(), State::state()) -> AccFinal::any()

Fold a function over the CloudI CRDT.

get/3

get(Dispatcher::cloudi_service:dispatcher(), Key::key(), State::state()) -> Value::value()

Get a value from the CloudI CRDT.

handle_info/3

handle_info(Request::any(), State::state(), Dispatcher::cloudi_service:dispatcher()) -> {ok, StateNew::state()} | {{error, Reason::cloudi_service:error_reason()}, StateNew::state()} | {ignored, State::state()}

Handle all info messages related to the CloudI CRDT.

Must be called from the cloudi_service_handle_info/3 callback function.

handle_request/11

handle_request(Type::cloudi_service:request_type(), Name::cloudi_service:service_name(), Pattern::cloudi_service:service_name_pattern(), RequestInfo::cloudi_service:request_info(), Request::cloudi_service:request(), Timeout::cloudi_service:timeout_value_milliseconds(), Priority::cloudi_service:priority_value(), TransId::cloudi_service:trans_id(), Pid::cloudi_service:source(), State::state(), Dispatcher::cloudi_service:dispatcher()) -> {ok, StateNew::state()} | {ignored, State::state()}

Handle a CRDT service request.

Must be called from the cloudi_service_handle_request/11 callback function.

incr/3

incr(Dispatcher::cloudi_service:dispatcher(), Key::key(), State::state()) -> state()

Increment a numerical value by 1 in the CloudI CRDT.

incr/4

incr(Dispatcher::cloudi_service:dispatcher(), Key::key(), Value::number(), State::state()) -> state()

Increment a numerical value in the CloudI CRDT.

incr_id/4

incr_id(Dispatcher::cloudi_service:dispatcher(), Key::key(), Id::event_id(), State::state()) -> state()

Increment a numerical value by 1 in the CloudI CRDT with an event_id.

incr_id/5

incr_id(Dispatcher::cloudi_service:dispatcher(), Key::key(), Value::number(), Id::event_id(), State::state()) -> state()

Increment a numerical value in the CloudI CRDT with an event_id.

is_key/3

is_key(Dispatcher::cloudi_service:dispatcher(), Key::key(), State::state()) -> boolean()

Check if a key is in the CloudI CRDT.

keys/2

keys(Dispatcher::cloudi_service:dispatcher(), State::state()) -> [key()]

Get all keys in the CloudI CRDT.

new/1

new(Dispatcher::cloudi_service:dispatcher()) -> state()

Create a CloudI CRDT.

new/2

new(Dispatcher::cloudi_service:dispatcher(), Options::options()) -> state()

Create a CloudI CRDT.

put/4

put(Dispatcher::cloudi_service:dispatcher(), Key::key(), Value::value(), State::state()) -> state()

Put a value into the CloudI CRDT.

put_id/5

put_id(Dispatcher::cloudi_service:dispatcher(), Key::key(), Value::value(), Id::event_id(), State::state()) -> state()

Put a value into the CloudI CRDT with an event_id.

size/2

size(Dispatcher::cloudi_service:dispatcher(), State::state()) -> non_neg_integer()

Get the size of the CloudI CRDT.

update/5

update(Dispatcher::cloudi_service:dispatcher(), Key::key(), Module::module(), Function::atom(), State::state()) -> state()

Update a value iff it exists in the CloudI CRDT.

Function Module:Function/1 must exist with the same version for every CloudI service process that shares this CloudI CRDT. If the function does not execute to return the same result (when given the same value) for each instance of the CloudI CRDT, it can create inconsistencies in the Erlang map that is used for all read operations (inconsistencies which would only be resolvable manually).

update/6

update(Dispatcher::cloudi_service:dispatcher(), Key::key(), Module::module(), Function::atom(), Argument1::any(), State::state()) -> state()

Update a value iff it exists in the CloudI CRDT.

Function Module:Function/2 must exist with the same version for every CloudI service process that shares this CloudI CRDT. If the function does not execute to return the same result (when given the same value) for each instance of the CloudI CRDT, it can create inconsistencies in the Erlang map that is used for all read operations (inconsistencies which would only be resolvable manually).

update_assign/6

update_assign(Dispatcher::cloudi_service:dispatcher(), Key::key(), Value::value(), Module::module(), Function::atom(), State::state()) -> state()

Update a value or assign a value in the CloudI CRDT.

Function Module:Function/1 must exist with the same version for every CloudI service process that shares this CloudI CRDT. If the function does not execute to return the same result (when given the same value) for each instance of the CloudI CRDT, it can create inconsistencies in the Erlang map that is used for all read operations (inconsistencies which would only be resolvable manually).

update_assign/7

update_assign(Dispatcher::cloudi_service:dispatcher(), Key::key(), Value::value(), Module::module(), Function::atom(), Argument1::any(), State::state()) -> state()

Update a value or assign a value in the CloudI CRDT.

Function Module:Function/2 must exist with the same version for every CloudI service process that shares this CloudI CRDT. If the function does not execute to return the same result (when given the same value) for each instance of the CloudI CRDT, it can create inconsistencies in the Erlang map that is used for all read operations (inconsistencies which would only be resolvable manually).

update_assign_id/7

update_assign_id(Dispatcher::cloudi_service:dispatcher(), Key::key(), Value::value(), Module::module(), Function::atom(), Id::event_id(), State::state()) -> state()

Update a value or assign a value in the CloudI CRDT with an event_id.

Function Module:Function/1 must exist with the same version for every CloudI service process that shares this CloudI CRDT. If the function does not execute to return the same result (when given the same value) for each instance of the CloudI CRDT, it can create inconsistencies in the Erlang map that is used for all read operations (inconsistencies which would only be resolvable manually).

update_assign_id/8

update_assign_id(Dispatcher::cloudi_service:dispatcher(), Key::key(), Value::value(), Module::module(), Function::atom(), Argument1::any(), Id::event_id(), State::state()) -> state()

Update a value or assign a value in the CloudI CRDT with an event_id.

Function Module:Function/2 must exist with the same version for every CloudI service process that shares this CloudI CRDT. If the function does not execute to return the same result (when given the same value) for each instance of the CloudI CRDT, it can create inconsistencies in the Erlang map that is used for all read operations (inconsistencies which would only be resolvable manually).

update_id/6

update_id(Dispatcher::cloudi_service:dispatcher(), Key::key(), Module::module(), Function::atom(), Id::event_id(), State::state()) -> state()

Update a value iff it exists in the CloudI CRDT with an event_id.

Function Module:Function/1 must exist with the same version for every CloudI service process that shares this CloudI CRDT. If the function does not execute to return the same result (when given the same value) for each instance of the CloudI CRDT, it can create inconsistencies in the Erlang map that is used for all read operations (inconsistencies which would only be resolvable manually).

update_id/7

update_id(Dispatcher::cloudi_service:dispatcher(), Key::key(), Module::module(), Function::atom(), Argument1::any(), Id::event_id(), State::state()) -> state()

Update a value iff it exists in the CloudI CRDT with an event_id.

Function Module:Function/2 must exist with the same version for every CloudI service process that shares this CloudI CRDT. If the function does not execute to return the same result (when given the same value) for each instance of the CloudI CRDT, it can create inconsistencies in the Erlang map that is used for all read operations (inconsistencies which would only be resolvable manually).

values/2

values(Dispatcher::cloudi_service:dispatcher(), State::state()) -> [value()]

Get all values in the CloudI CRDT.

zero/3

zero(Dispatcher::cloudi_service:dispatcher(), Key::key(), State::state()) -> state()

Put a zero value in the CloudI CRDT.

zero_id/4

zero_id(Dispatcher::cloudi_service:dispatcher(), Key::key(), Id::event_id(), State::state()) -> state()

Put a zero value in the CloudI CRDT with an event_id.


Generated by EDoc