Metadata

Understanding Metadata

YapDatabase supports storing both an 'object' & 'metadata' for every (collection, key) tuple.

- (void)setObject:(nullable id)object
           forKey:(NSString *)key
     inCollection:(nullable NSString *)collection
     withMetadata:(nullable id)metadata;

In other words, the underlying database table has a dedicated column for metadata:

| collection(text) | key(text) | object(blob) | metadata(blob) |

In contrast to the 'object', metadata is completely optional. That is, if you attempt to store a nil 'object' to a row, then YapDatabase will delete that entire row. If you attempt to store a nil 'metadata' to a row, that's entirely OK.

Some people have no use for metadata. Others rely heavily on it. Both are supported by YapDatabase, and your own usage is entirely a product of your application's needs.

Motivation

So when would one use metadata? What is it good for?

Simply put, the metadata field allows you to store extra information that may not be included in the object itself. A concrete example may be instructive.

Let's say your application is downloading JSON objects from a server. You have a bunch of different model classes that already support mapping from/to the JSON structure. You've taken advantage of this for your serialization into YapDatabase. Thus, creating an object from a downloaded JSON object uses the same code path as deserializing a blob from YapDatabase (JSON -> object).

There's just one problem. You also want to store the date the object was downloaded. This will allow you to see if the object is stale, and should be refreshed from the server. But you really don't want to force this information into the object. It's not part of the JSON, and forcing it to become part of the object doesn't feel right. Enter metadata !

Since each row has an extra metadata column, you're free to store whatever you like right next to the object within the database. So you might simply store the download timestamp. Or maybe you get more advanced, and you store both the download timestamp & the eTag (to make refreshes even faster)!

Again, there's absolutely no requirement that you store this information within the metadata column. You could just as easily add these properties directly to the object itself. It all has to do with your existing architecture.

Some other examples of when metadata might be really useful:

• You're dealing with existing legacy classes, and it's a pain to change them.
• You're getting your objects from a 3rd-party framework, and it's not really possible/desirable to change them.
• It's just not possible to change the objects. (E.g. you're storing raw images in the object column)

And when this happens, metadata comes to your rescue.

Core API

Metadata is a first class citizen, and is supported by all extensions. But let's start with the basics. You've probably seen a lot of code examples that look like this:

[databaseConnection readWriteWithBlock:^(YapDatabaseReadWriteTransaction *transaction){

    [transaction setObject:obj forKey:key inCollection:collection];
}];

Where's the metadata?

A lot of the documentation glosses over metadata purely to keep things simple. Everybody understands key/value storage. And collection/key/value storage isn't a hard adjustment. But the truth is that YapDatabase is really collection/key/value/value, and that's a bit more confusing for many people. So we just kinda ignore that second value most of the time, at least in the docs. But now you know the truth.

The setObject:forKey:inCollection: method is really just short-hand for calling setObject:forKey:inCollection:withMetadata:, and passing a nil value for the metadata parameter.

Here's a quick overview of the set/update API's (from YapDatabaseReadWriteTransaction):

/**
 * Invokes setObject:forKey:inCollection:withMetadata:,
 * and passes a nil value for the metadata parameter.
**/
- (void)setObject:(nullable id)object
           forKey:(NSString *)key
     inCollection:(nullable NSString *)collection;

/**
 * If you call this method with a nil object, then it will delete the row.
 * (Equivalent to calling removeObjectForKey:inCollection:)
 * 
 * Otherwise, this method inserts/updates the row,
 * and sets BOTH the object & metadata columns to the given values.
**/
- (void)setObject:(nullable id)object
           forKey:(NSString *)key
     inCollection:(nullable NSString *)collection
     withMetadata:(nullable id)metadata;

/**
 * If a row with the given collection/key already exists,
 * then this method updates ONLY the object value.
 * The metadata value for the row isn't touched. (It remains whatever it was before.)
 * 
 * Again, it's not possible to have a nil object for a row.
 * So if you try to set the object to nil, this is just going to delete the row.
**/
- (void)replaceObject:(nullable id)object
               forKey:(NSString *)key
         inCollection:(nullable NSString *)collection;

/**
 * If a row with the given collection/key already exists,
 * then this method updates ONLY the metadata value.
 * The object value for the row isn't touched. (It remains whatever it was before.)
**/
- (void)replaceMetadata:(nullable id)metadata
                 forKey:(NSString *)key
           inCollection:(nullable NSString *)collection;

You can also customize how your metadata objects are serialized & deserialized. This is because YapDatabase has separate serializer/deserializer properties for objects vs metadata:

From YapDatabase.h

@property (nonatomic, strong, readonly) YapDatabaseSerializer objectSerializer;
@property (nonatomic, strong, readonly) YapDatabaseDeserializer objectDeserializer;

@property (nonatomic, strong, readonly) YapDatabaseSerializer metadataSerializer;
@property (nonatomic, strong, readonly) YapDatabaseDeserializer metadataDeserializer;

You'll find various init methods that allow you to set the the metadata options separate from the object options.

Similarly, there are separate caches for objects vs metadata:

From YapDatabaseConnection.h

@property (atomic, assign, readwrite) BOOL objectCacheEnabled;
@property (atomic, assign, readwrite) NSUInteger objectCacheLimit;

@property (atomic, assign, readwrite) BOOL metadataCacheEnabled;
@property (atomic, assign, readwrite) NSUInteger metadataCacheLimit;

This gives you the flexibility to achieve your performance goals. For example, you might access the metadata of an object often in order to check to see if its expired. Thus, you may wish to crank up the metadataCacheLimit in order to speed up these enumerations.

Extensions API

Metadata is fully supported in all extensions as well. For example, here's an excerpt from YapDatabaseViewTypes:

@interface YapDatabaseViewGrouping : NSObject

typedef id YapDatabaseViewGroupingBlock; // One of the YapDatabaseViewGroupingX types below.

typedef NSString* _Nullable (^YapDatabaseViewGroupingWithKeyBlock)
             (YapDatabaseReadTransaction *transaction, NSString *collection, NSString *key);

typedef NSString* _Nullable (^YapDatabaseViewGroupingWithObjectBlock)
             (YapDatabaseReadTransaction *transaction, NSString *collection, NSString *key, id object);

typedef NSString* _Nullable (^YapDatabaseViewGroupingWithMetadataBlock)
             (YapDatabaseReadTransaction *transaction, NSString *collection, NSString *key, _Nullable id metadata);

typedef NSString* _Nullable (^YapDatabaseViewGroupingWithRowBlock)
             (YapDatabaseReadTransaction *transaction, NSString *collection, NSString *key, id object, _Nullable id metadata);

+ (instancetype)withKeyBlock:(YapDatabaseViewGroupingWithKeyBlock)block;
+ (instancetype)withObjectBlock:(YapDatabaseViewGroupingWithObjectBlock)block;
+ (instancetype)withMetadataBlock:(YapDatabaseViewGroupingWithMetadataBlock)block;
+ (instancetype)withRowBlock:(YapDatabaseViewGroupingWithRowBlock)block;

So, if you wanted to, you could create a view that groups/sorts entirely based on your metadata values. Or a combination of your object & metadata values. And you'll notice that other extensions work the same way. So metadata is always supported.