NAME

Storage back ends - how to add back ends to the HammerServer

PURPOSE

This document describes how storage back ends can be added to the HammerServer. Currently three storage back ends are implemented: based on Postgresql, on flat file storage, and on CouchDb.

HOW TO WRITE A BACK END

Back ends are typically modules in the directory mod/db/. E.g., you will find mod/db/Pg.pm, mod/db/File.pm and mod/CouchDB.pm.

Methods

Every back end module must define the following methods:

Initialization:

 use mod::db::MyModule;
 my $b = MyModule->new();

Method new() can inspect any necessary configuration values using Program::arg(). It would typically connect to a database, or prepare files, and then return a blessed object reference.

Data Insertion:

 $b->insert($id, $data, $stamp, $hash, $previd, $sig);

The stated values are taken as one record and committed to storage. The fields id and previd are numerical fields, and not the external amorphous key (which higher modules will return to client applications). Also, these fields may be a ``padded'' numeric field, e.g., 000001 to denote 1, and so on. Back end that deal with ID's must be prepared to make numeric conversions on this field where appropriate.

All other fields (data, stamp, hash, sig) can be treated by the back end as string data. I.e., higher-level modules make sure that $sig is a valid signature for this record. The back end modules never need to use cryptography to compute hashes or generate signatures.

ID Accessors

 my $newid  = $b->getnextid();
 my $lastid = $b->getlastid();

The first method must return the next available ID, and must reserve it for usage. Typically it will increment a sequence. The second method must return the last-used id (e.g., a relational database system may request a max(id) selection from the database).

Back ends may pad the ID's that higher-level modules request. E.g., the next ID may be returned as 000013 instead of 13. When a back end module pads ID's, then such padding is maintained and passed to other calls (e.g., the ID of the insert() call will also be padded).

Hash Access

 my $h = $b->gethashbyid($id);

This method must return a previously stored hash value at the stated id.

Retrieval of Data

 my ($data, $stamp, $hash, $previd, $signature) =
   $b->getbyid($id);

This method must return the stated columns (plain-text data, timestamp of insertion, hash of the data, previous id, and signature), given the stated id. The return value is an array of 5 scalar variables.

Searching by Substrings

 my @result = $b->getbyterm($term);
 for my $d (@result) {
   my ($stamp, $id, $data) = @{ $d };
 }

This method must search the storage and perform a substring matching. For the matched records an array reference is constructed from the stamp, id and data. An array of such array references is returned.

Searching by Timestamp

 my @result = $b->getbyinterval($start, $end);
 for my $d (@result) {
   my ($id, $data, $stamp, $hash, $previd, $signature) = @{ $d };
 }

This method must return an array of references that represent matching records between two timestamps. Each element in the array must be an array reference to the shown fields.

Deleting by Timestamp

 $b->deletebyinterval($start, $end);

This method may clean up the storage and delete all records between the stated timestamps. It is called during offloading. Some storage back ends don't need to (or can't) delete data; in this case, the function must still be implemented, but do nothing.

Locking

If required, a database back end may establish locks to avoid concurrency-related problems. An object of a back end class will only exist for the duration of one request. Therefore, new() could establish a lock, and DESTROY() might release it. (See the methods new() and DESTROY() in mod/db/File.pm for an example.)

HOW TO ENABLE A NEW BACK END

Writing a new module

In order to make the back end available to the HammerServer, a new module must be added to the directory mod/db and of course all above methods must be implemented.

Registering in class Datastore

The class Datastore must be made 'aware' of the new module. To do this, the file mod/Datastore.pm must be edited. The following actions are necessary:

The list of modules is extended. E.g., to add a new module MyModule, the ``use'' directive is added:
```
 use mod::db::MyModule;
```
The module's constructor is called in the function init(), similar to Pg, File, CouchDB and so on. By convention, the first string in the configuration setting of db indicates the back end type (e.g., db = file:..... indicates flat file storage). The new back end must therefore use an identifier stating its type.

Enabling the module in etc/hammerserver

The configuration db must be changed in the configuration file so that HammerServer's class Datastore activates the class.

Back end initializations

If the new back end requires initializing, bin/hs-init-backend can be modified to perform these.