The Quartz database backend

Status

WARNING: This document describes a piece of code which is still in development. While we try hard to update this document correctly to reflect the current state of the code, it's possible discrepancies may exist. If you notice any, please alert us to the problem using the Mailing Lists.

Why Quartz?

What is this thing called Quartz? How does it fit in with the Xapian library?

Xapian can access information stored in various different formats. For example, there are the legacy formats which can be read by Xapian but not written to (at present, the Muscat 3.6 formats "DA" and "DB"), and the InMemory format which is held in memory.

Each of these formats is comprised by a set of classes providing an interface to a Database object and several other related objects (PostList, TermList, etc...).

Quartz is simply the name of only currently supported disk-based backend which allows databases to be dynamically updated. The design of Quartz draws on all our past experience to satisfy the following criteria:

Different backends can be optionally compiled into the Xapian library (by specifying appropriate options to the configure script). Quartz is compiled by default.

Why do we call it Quartz - where does the name come from?

Well, we had to call it something, and Quartz was simply the first name we came up with which we thought we could live with...

Tables

A Quartz database consists of several tables, each of which stores a different type of information: for example, one table stores the user-defined data associated with each document, and another table stores the posting lists (the lists of documents which particular terms occur in).

These tables consist of a set of key-tag pairs, which I shall often refer to these as items or entries. Items may be accessed randomly by specifying a key and reading the item pointed to, or in sorted order by creating a cursor pointing to a particular item. The sort order is a lexicographical ordering based on the contents of the keys. Only one instance of a key may exist in a single table - inserting a second item with the same key as an existing item will overwrite the existing item.

Positioning of cursors may be performed even when a full key isn't known, by attempting to access an item which doesn't exist: the cursor will then be set to point to the first item with a key before that requested.

The QuartzTable class defines the standard interface to tables. This has two subclasses - the QuartzDiskTable and QuartzBufferedTable interface. The former provides direct access to the table as stored on disk, and the latter provides access via a large buffer in order to use the memory as a write cache and greatly speed the process of indexing.

The contents of the tables

We shall worry about the implementation of the tables later, but first we shall look at what is stored within each table.

There are six tables comprising a quartz database.

Representation of integers, strings, etc

It is well known that in modern computers there are many, many CPU cycles for each disk read, or even memory read. It is therefore important to minimise disk reads, and can be advantageous to do so even at the expense of a large amount of computation. In other words, Compression is good.

The current implementation uses simple compression - we're investigating more effective schemes - these are (FIXME: this is slightly out of date now):

PostLists and chunks

Posting lists can grow to be very large - some terms occur in a very large proportion of the documents, and their posting lists can represent a significant fraction of the size of the whole database. Therefore, we do not wish to read an entire posting list into memory at once. (Indeed, we'd rather only read a small portion of it at all, but that's a different story - see the documentation on optimisations performed by the matcher).

To deal with this, we store posting lists in small chunks, each the right size to be stored in a single B-tree block, and hence to be accessed with a minimal amount of disk latency.

The key for the first chunk in a posting list is the term name of the term whose posting list it is. The key in subsequent chunks is the term name followed by the document ID of the first document in the chunk. This allows the cursor methods to be used to scan through the chunks in order, and also to jump to the chunk containing a particular document ID.

It is quite possible that the termlists and position lists would benefit from being split into chunks in this way.

Btree implementation

The tables are currently all implemented as B-trees (actually a form of B-tree sometimes known as a B+ tree). (For some tables, the use of a different structure could be more appropriate - perhaps a hashing scheme might provide more space and time efficient access. This is an area for future investigation).

A B-tree is a fairly standard structure for storing this kind of data, so I will not describe it in detail - see a reference book on database design and algorithms for that. The essential points are that it is a block-based multiply branching tree structure, storing keys in the internal blocks and key-tag pairs in the leaf blocks.

Our implementation is fairly standard, except for its revision scheme, which allows modifications to be applied atomically whilst other processes are reading the database. This scheme involves copying each block in the tree which is involved in a modification, rather than modifying it in place, so that a complete new tree structure is built up whilst the old structure is unmodified (although this new structure will typically share a large number of blocks with the old structure). The modifications can then be atomically applied by writing the new root block and making it active.

After a modification is applied successfully, the old version of the table is still fully intact, and can be accessed. The old version only becomes invalid when a second modification is attempted (and it becomes invalid whether or not that second modification succeeds).

There is no need for a process which is writing the database to know whether any processes are reading previous versions of the database. As long as only one update is performed before the reader closes (or reopens) the database, no problem will occur. If more than one update occurs whilst the table is still open, the reader will notice that the database has been changed whilst it has been reading it by comparing a revision number stored at the start of each block with the revision number it was expecting. An appropriate action can then be taken (for example, to reopen the database and repeat the operation).

An alternative approach would be to obtain a read-lock on the revision being accessed. A write would then have to wait until no read-locks existed on the old revision before modifying the database.

Buffered tables

If each change to a table (ie, modification of a key-tag pair) was immediately written to disk, there would be two problems.
  1. The system would be very slow. If a disk read and then a write was required each time an item was changed, the indexing process would spend most of its time waiting for the disk's write head to seek to the appropriate block. By buffering up a large set of changes, and then writing them all out in a sorted order, seeking is minimised.
  2. Operations which involve modifying more than one key-tag pair would not be atomic. For example, when adding a document to a database the record table has three items updated - one containing the data stored in the document, one which stores the next document ID to allocate, and one which stores the sum of all the document lengths in the database. These three items must be updated at the same time, so that a consistent state is always seen by other processes, and a consistent state remains if the system is terminated unexpectedly.

As a result, the QuartzBufferedTable object is available. This simply stores a set of modified entries in memory, applying them to disk only when the apply method is called.

In fact, a QuartzBufferedTable object has to have two handles open on the table - one for reading and one for writing. This is simply because the interface for writing a table is more limited than that for reading a table (in particular, cursor operations are not available).

Applying changes to all the tables simultaneously

This is all wonderful: we have tables storing arbitrary bits of useful data, we can update bits of them as we like, and we can then call a method and have all the modifications applied to the table atomically. Unfortunately, we need more than that - we need to be able to apply modifications as a single atomic transaction across multiple tables, so that the tables are always accessed in a mutually consistent state.

The revisioning scheme described earlier comes to the rescue! By carefully making sure that we open all the tables at the same revision, and by ensuring that at least one such consistent revision always exists, we can extend the scope of atomicity to cover all the tables. In detail:

This scheme guarantees that modifications are atomic across all the tables - essentially we have made the modification get committed only when the final table is committed.

Items to be added to this document

Endnote

The system as described could, no doubt, be improved in several ways. If you can think of such ways then suggest it to us, so we can have a discussion of the improvement to see whether it would help: if it would we will add it to the design (and eventually the code) - if not, we'll add a discussion about it to this document.

The Btree Implementation

[Note: the Btree code used to be object-like C, evolved through a strange mishmash of this and C++, and is now pretty much "normal" C++. These docs may not be fully up-to-date with these changes (FIXME - check)]

I'm not sure about the name 'Btree' that runs through all this, since the fact that it is all implemented as a B-tree is surely irrelevant. I have not been able to think of a better name though ...

Some of the constants mentioned below depend upon a byte being 8 bits, but this assumption is not built into the code.

Keys and tags

Thinking of 'byte' having type 'unsigned char', a key and a tag are both sequences of bytes. The B-tree is a repository for key-tag pairs. A key can be looked up to find its corresponding tag. If a key is deleted, the corresponding tag is deleted. And in the B-tree keys are unique, so if a key-tag pair is added in and the key is already in the Btree, the tag to be added in replaces the tag in the B-tree.

In the B-tree key-tag pairs are ordered, and the order is the ASCII collating order of the keys. Very precisely, if key1 and key2 point to keys with lengths key1_len, key2_len, key1 is before/equal/after key2 according as the following procedure returns a value less than, equal to or greater than 0,

static int compare_keys(const byte * key1, int key1_len,
			const byte * key2, int key2_len)
{
    int smaller = key1_len < key2_len ? key1_len : key2_len;
    for (int i = 0; i < smaller; i++) {
        int diff = (int) key1[i] - key2[i];
        if (diff != 0) return diff;
    }
    return key1_len - key2_len;
}

[This is okay, but none of the code fragments below have been checked.]

Any large-scale operation on the B-tree will run very much faster when the keys have been sorted into ASCII collating order. This fact is critical to the performance of the B-tree software.

A key-tag pair is called an 'item'. The B-tree consists therefore of a list of items, ordered by their keys:

    I1  I2  ...  Ij-1  Ij  Ij+1  ...  In-1  In

Item Ij has a 'previous' item, Ij-1, and a 'next' item, Ij+1.

When the B-tree is created, a single item is added in with null key and null tag. This is the 'null item'. The null item may be searched for, and it's possible, although perhaps not useful, to replace the tag part of the null item. But the null item cannot be deleted, and an attempt to do so is merely ignored.

A key must not exceed 252 bytes in length.

A tag may have length zero. There is an upper limit on the length of a tag, but it is quite high. Roughly, the tag is divided into items of size L - kl, where L is a a few bytes less than a quarter of the block size, and kl is length of its key. You can then have 64K such items. So even with a block size as low as 2K and key length as large as 100, you could have a tag of 2.5 megabytes. More realistically, with a 16K block size, the upper limit on the tag size is about 256 megabytes.

Revision numbers

The B-tree has a revision number, and each time it is updated, the revision number increases. In a single transaction on the B-tree, it is first opened, its revision number, R is found, updates are made, and then the B-tree is closed with a supplied revision number. The supplied revision number will typically be R+1, but any R+k is possible, where k > 0.

If this sequence fails to complete for some reason, revision R+k of the B-tree will not, of course, be brought into existence. But revision R will still exist, and it is that version of the B-tree that will be the starting point for later revisions.

If this sequence runs to a successful termination, the new revision, R+k, supplants the old revision, R. But it is still possible to open the B-tree at revision R. After a successful revision of the B-tree, in fact, it will have two valid versions: the current one, revision R+k, and the old one, revision R.

You might want to go back to the old revision of a B-tree if it is being updated in tandem with second B-tree, and the update on the second B-tree fails. Suppose B1 and B2 are two such B-trees. B1 is opened and its latest revision number is found to be R1. B2 is opened and its latest revision number is found to be R2. If R1 > R2, it must be the case that the previous transaction on B1 succeeded and the previous transaction on B2 failed. Then B1 needs to opened at its previous revision number, which must be R1.

The calls using revision numbers described below are intended to handle this type of contingency.

The files

The B-tree has three associated files. DB contains the data proper of the B-tree. The revision numbers, other administrative information, and a bitmap are held in two files, baseA and baseB.

When the B-tree is opened without any particular revision number being specified, the later of baseA and baseB is chosen as the opening base, and as soon as a write to the file DB occurs, the earlier of baseA or baseB is deleted. On closure, the new revision number is written to baseB if baseA was the opening base, and to baseA if baseB was the opening base. If the B-tree update fails for some reason, only one base will usually survive.

The bitmap stored in each base file will have bit n set if block n is in use in the corresponding revision of the B-tree.

The API

Most of the procedures described below can return with an error condition. Error conditions are described later. Meanwhile, the code examples given here ignore the possibility of errors. Bad practice, but it makes them more readable.
void Btree::create(const string & name, int block_size);
Creates a new B-tree with the given name and block size. On error, throws an exception.
        Btree::create("/home/martin/develop/btree/", 8192);

        Btree::create("X-", 6000); /* files will be X-baseA, X-DB etc */
The block size must be less than 64K, where K = 1024. It is unwise to use a small block size (less than 1024 perhaps) - quartz enforces a minimum block size of 2K.

Thereafter there are two modes for accessing the B-tree: update and retrieval.

Update mode

void Btree::open_to_write(const string & name);
The name is the same as the one used in creating. Throws an exception in case of error. E.g:
        Btree::open_to_write("/home/martin/develop/btree/");
string::size_type Btree::max_key_len;
This gives the upper limit of the size of a key that may be presented to the B-tree (252 bytes with the present implementation).
uint4 Btree::revision_number;
This gives the revision number of the B-tree. (The type uint4 is an unsigned integral type at least 4 bytes in size).
uint4 get_latest_revision_number() const;
This gives the higher of the revision numbers held in the base files of the B-tree, or just the revision number if there's only one base file.
bool Btree::open_to_write(const string & name, unsigned long revision);
Like Btree::open_to_write, but open at the given revision number, and returns false if the database couldn't be opened at the requested revision.
bool Btree::add(const string &key, const string &tag);
Adds the given key-tag pair to the B-tree. e.g.
        ok = B->add("TODAY", "Mon 9 Oct 2000");

        ok = B->add(string(k + 1, k[0]), string(t + 1, t[0]));
The key and tag are C++ strings.

If key and tag are empty, then the null item is replaced. If key.length() exceeds the the limit on key sizes an error condition occurs. The result is false if the key is already in the B-tree, otherwise true. Treated as an integer, the result also measures the increase in the total number of keys in the B-tree.

bool Btree::del(const string &key);
If key.empty() nothing happens, and the result is false.

Otherwise this deletes the key and its tag from the B-tree, if found. e.g.

        ok = B->del("TODAY")

        ok = B->del(string(k + 1, k[0]));
The result is then false if the key is not in the B-tree, true if it is. Treated as an integer, the result also measures the decrease in the total number of keys in the B-tree.
bool Btree::find_key(const string &key);
The result is true iff the specified key is found.
bool Btree::find_tag(const string &key, string * tag);
The same result, but when the key is found the tag is copied to tag. If the key is not found tag is left unchanged. e.g.
        string t;
        B->find_tag("TODAY", &t); /* get today's date */
int4 Btree::item_count;
This returns the number of items in the B-tree, not including the ever-present item with null key.

Retrieval mode

In retrieval mode there is no opportunity for updating the B-tree, but access is considerably elaborated by the use of cursors.
void Btree::open_to_read(const string & name);
uint4 Btree::revision_number;
void Btree::open_to_read(const string & name, uint4 revision);
These are the same as for update mode, except that that the opened B-tree is not modifiable.
Bcursor::Bcursor(Btree *B);
Creates a cursor, which can be used to remember a position inside the B-tree. The position is simply the item (key and tag) to which the cursor points. A cursor is either positioned or unpositioned, and is initially unpositioned.

NB: You must make sure that the Bcursor is destroyed before the Btree it is attached to.

bool Bcursor::find_key(const string & key);
The result is true iff the specified key is found in the Btree.

If found, the cursor is made to point to the item with the given key, and if not found, it is made to point to the last item in the B-tree whose key is <= the key being searched for, The cursor is then set as 'positioned'. Since the B-tree always contains a null key, which precedes everything, a call to Bcursor::find_key always results in a valid key being pointed to by the cursor.

bool Bcursor::next();
If cursor BC is unpositioned, the result is simply false.

If cursor BC is positioned, and points to the very last item in the Btree the cursor is made unpositioned, and the result is false. Otherwise the cursor BC is moved to the next item in the B-tree, and the result is true.

Effectively, Bcursor::next() loses the position of BC when it drops off the end of the list of items. If this is awkward, one can always arrange for a key to be present which has a rightmost position in a set of keys, e.g.

        B->add("\xFF", "");
        /* all other keys have first char < xF0, and a fortiori < xFF */
 
bool Bcursor::prev();
This is like Bcursor::next, but BC is taken to the previous rather than next item.
 
bool Bcursor::get_key(string * key);
If cursor BC is unpositioned, the result is simply false.

If BC is positioned, the key of the item is copied into key and the result is then true.

For example,

 
        Bcursor BC(&B);
        string key;

        /* Now we'll print all the keys in the B-tree (assuming they
           have a simple form */

        BC.find_key(""); // must give result true

        while (BC.next()) { // skipping the null item
	    BC.get_key(&key);
	    cout << key << endl;
        }
 
bool Bcursor::get_tag(string * tag);
If cursor BC is unpositioned, the result is false.

If BC is positioned, the tag of the item at cursor BC is copied into tag. BC is then moved to the next item as if Bcursor::next() had been called - this may leave BC unpositioned. The result is true iff BC is left positioned.

For example,

 
        Bcursor BC(&B);
	string key;
        string tag;

        /* Now do something to each key-tag pair in the Btree */
           have a simple form */

        BC.find_key(""); // must give result true

        while (BC.get_key(&key)) {
            BC.get_tag(&tag);
            do_something_to(key, tag);
        }

        /* when BC is unpositioned by Bcursor::get_tag, Bcursor::get_key
           gives result false the next time it called
        */
 
Bcursor::~Bcursor();
Loses cursor BC.

Checking the B-tree

The following static method is provided in btreecheck.h:
 
void BtreeCheck::check(const string & name, int opts);
BtreeCheck::check(s, opts) is essentially equivalent to
        Btree B;
	B.open_to_write(s);
        {
            // do a complete integrity check of the B-tree,
            // reporting according to the bitmask opts
        }
The option bitmask may consist of any of the following values |-ed together: The options control what is reported - the entire B-tree is always checked as well as reporting the information.

Full compaction

As the B-tree grows, items are added into blocks, and, when a block is full, it splits into two (amoeba-like) and one of the new blocks accommodated the new entry. Blocks are therefore between 50% and 100% full during growth, or 75% full on average.

Let us say an item is 'new' if it is presented for addition to the B-tree and its key is not already in the B-tree. Then presenting a long run of new items ordered by key causes the B-tree updating process to switch into a mode where much higher compaction than 75% is achieved - about 90%. This is called 'sequential' mode. It is possible to force an even higher compaction rate with the procedure

 
void Btree::full_compaction(bool parity);
So
 
    B.full_compaction(true);
switches full compaction on, and
 
    B.full_compaction(false);
switches it off. Full compaction may be switched on or off at any time, but it only affects the compaction rate of sequential mode. In sequential mode, full compaction gives around 98-99% block usage - it is not quite 100% because keys are not split across blocks.

The downside of full compaction is that block splitting will be heavy on the next update. However, if a B-tree is created with no intention of being updated, full compaction is very desirable.

Full compaction with revision 1

Retrieval mode is faster when the B-tree has revision number 1 than for higher revision numbers. This is because there are no unused blocks in the B-tree and the blocks are in a special order, and this enables the Bcursor::prev and Bcursor::next procedures, and the other procedures which use them implicitly, to have more efficient forms.

To make a really fast structure for retrieval therefore, create a new B-tree, open it for updating, set full compaction mode, and add all the items in a single transaction, sorted on keys. After closing, do not update further. Further updates can be prevented quite easily by deleting (or moving) the bitmap files. These are required in update mode but ignored in retrieval mode.

Here is a program fragment to unload B-tree B/ and reform it in Bnew/ as a fully compact B-tree with revision number 1:

 
    {
	Btree B;
	B.open_to_read("B/");
        Bcursor BC(&B);
        string key, tag;

	Btree::create("Bnew/", 8192);

	Btree new_B;
	new_B.open_to_write("Bnew/");
	new_B.set_full_compaction(true);

	BC.find_key("");

	while (BC.get_key(&key)) {
	    BC.get_tag(&tag);
	    new_B.add(key, tag);
	}
	new_B.commit(1);
    }
For a complete program which does this, see the quartzcompact utility.

Notes on space requirements

The level of the B-tree is the distance of the root block from a leaf block. At minimum this is zero. If a B-tree has level L and block size B, then update mode requires space for 2(LB + b1 + b2) bytes, where b1 and b2 are the size of the two bitmap files. Of course, L, b1 and b2 may grow during an update on the B-tree. If the revision number is greater than one, then retrieval mode requires (L - 2 + 2c)B bytes, where c is the number of active cursors. If however the revision number is one, it only requires (L - 2 + c)B bytes.

This may change in the future with code redesign, but meanwhile not that a K term query that needs k <= K cursors open at once to process, will demand 2KB bytes of memory in the B-tree manager.

Updating during retrieval

The B-tree cannot be updated by two separate processes at the same time. The user of the B-tree software should establish a locking mechanism to ensure that this never happens.

It is possible to do retrieval while the B-tree is being updated. If the updating process overwrites a part of the B-tree required by the retrieval process, then a Xapian::DatabaseModifiedError exception is thrown.

This should be handled, and suitable action taken - either the operation aborted, or the Btree reopened at the latest revision and the operation retried. Here is a model scheme:

 
static Btree * reopen(Btree * B)
{
    uint4 revision = B->revision_number;

    // Get the revision number. This will return the correct value, even when
    // B->overwritten is detected during opening

    while (true) {
	try {
	    delete B;  /* close the B-tree */
	    B = new Btree;
	    B->open_to_read(s); /* and reopen */
	    break;
	} catch (const Xapian::DatabaseModifiedError &) {
	}
    }

    if (revision == B->revision_number) {
        // The revision number ought to have gone up from last time,
        // so if we arrive here, something has gone badly wrong ...
        printf("Possible database corruption!\n");
        exit(1);
    }
    return B;
}


    ....

    char * s = "database/";
    Btree * B = 0;
    uint4 revision = 0;

    /* open the B-tree */
    while (true) {
	try {
	    delete B;  /* close the B-tree */
	    B = new Btree;
	    B->open_to_read(s); /* and reopen */
	    break;
	} catch (const Xapian::DatabaseModifiedError &) {
	}
    }

    string t;
    while (true) {
	try {
	    B->find_tag("brunel", &t); /* look up some keyword */
	    break;
	} catch (const Xapian::DatabaseModifiedError &) {
	    B = reopen(s);
	}
    }

    ...
If the overwritten condition were detected in updating mode, this would mean that there were two updating processes at work, or the database has become corrupted somehow. If this is detected, a Xapian::DatabaseCorruptError is thrown. There's not much which can usefully be done to automatically handle this condition.

In retrieval mode, the following can cause Xapian::DatabaseModifiedError to be thrown:

    Btree::open_to_read(name);
    Btree::open_to_read(name, revision);
    Bcursor::next();
    Bcursor::prev();
    Bcursor::find_key(const string &key);
    Bcursor::get_tag(string * tag);
The following can not:
   Bcursor::Bcursor(Btree * B);
   Bcursor::get_key(string * key);
Note particularly that opening the B-tree can cause it, but Bcursor::get_key(..) can't.