db8

db8 is an addition to the webOS JavaScript Framework's current storage methods that is designed to meet the needs of robust, high-performance applications. db8 is a service—com.palm.db—available on the device bus that interfaces to an embedded JSON database. JavaScript applications have two options for interfacing to this service:

  1. JavaScript Wrapper API -- API calls provided with the JavaScript Foundation libraries to interface with the db8 service.

  2. Using serviceRequest -- Call the db8 service using the "serviceRequest" API. See the Services Overview for more information on using this call.

Note: It is assumed the reader has basic knowledge of database concepts and SQL operations. This article focuses on information specific to db8.

In this section:

See also:

 


db8 Features

db8 features include:

  • High speed access and queries.
  • Stores JSON objects. JSON (JavaScript Object Notation) is derived from JavaScript and is the de facto web app standard for persisting data (more compact than XML).
  • Designed for syncing with cloud computing and apps such as Facebook or Google.
  • App-aware access control.
  • Paging support for queries that can retrieve up to 500 objects at a time.
  • Change notification support.
  • Optional backing up of data storage. If the user moves to another device, saved app data can be restored.
  • Data object validation through schema enforcement.

Note: Though a vast improvement on current webOS storage mechanisms, db8 is not a fully-featured database supporting a rich set of SQL or SQL-like commands.

 


Basic Steps

If you decide to use db8 storage in your code, here are the basic steps you need to follow to set this up:

  1. Determine what you are going to store -- see db8 Stores JSON Objects.

  2. Determine what your indexes are going to be. This is very important, as the indexes you create determine the kind of queries you can do. See Queries and Indexing.

  3. Decide if you want to use the JavaScript Wrapper API or the serviceRequest API to set up and access your database.

  4. Use one of the API's putKind calls to create a kind. Once you have done that, your app can store data objects of that kind.

  5. Determine what other apps or services can access your data, such as the new Just Type service. See Allowing Other Apps or Services to Access Your db8 Data.

  6. Start storing your data with the put call.

 


db8 Stores JSON objects

JSON is a lightweight data-interchange text format based on a subset of the JavaScript programming language that is easy for humans to read and write and for machines to parse and generate. Like XML, JSON contains name/value pair collections and ordered list of values (i.e., arrays).

db8 stores two types of JSON objects you need to know about:

  1. Kind objects -- JSON objects that define the owner, schema, and indexes for JSON data objects. The indexes determine the type of queries that can be made. In db8, these objects are known as kinds. Before a data object can be saved, its defining kind object must be stored and registered with the "putKind" call.

  2. Data objects -- JSON objects of a particular kind that contain data. Similar to objects that are instances of classes in object-oriented programming.

For more information about JSON, see the following websites:

Consult the two links above for information on formatting JSON objects. However, in brief:

  • Curly brackets ('{', '}') are used to enclose objects.
  • Angle brackets ('[', ']') are used to enclose arrays.
  • Colons (':') are used as name/value delimiters.
  • Quotes (") are used for string values. Numeric and boolean values do not require quotes.

Example JSON data object

The following is a fully-formatted JSON data object for a hypothetical contacts database:

{
  "displayName"  : "John Doe",
  "name"         : {"familyName":"Doe", "givenName":"John"},
  "nickname"     : "Skippy",
  "emails"       : [ "jdoe@gmail.com", "john.doe@palm.com", "jdoe99@gmail.com" ]
}

This documentation uses a simplified form of defining JSON objects that, for the above object, would look like this:

{
  "displayName" : string,
  "name"        : {
      "familyName" : string,
      "givenName"  : string
  },
  "nickname"    : string,
  "emails"      : string array
}

Before you could store this data object, your app would need to create its kind object with the "putKind" API. For the example data object, the parameters to this call would look something like this:

{
  "id"      :  "com.palm.contact:1",
  "owner"   :  "com.palm.contacts",
  "extends" :  ["PimObject:1"],
  "indexes" :  [{"name":"dname", "props":[{"name":"displayName"}]}]
}

Note the following about this example:

  • The id property has a version number. This is required and is appended to the end (i.e., ":1") of the property value field. Initially, this number should be "1" and incremented whenever a version of this kind is updated that is not backward-compatible with previous versions.

  • It extends another kind object (PimObject:1). Kinds can extend other kinds.

  • The owner property is the bus address of the service's appId of the app that owns this kind. This is the only caller that has permission to modify the kind.

  • The indexes property defines the displayName property as the one index property. How you define indexes for your kind objects determines the kind of queries your app can do. See Queries and Indexing for more information.

 

Special Properties for db8 JSON Objects

There are some special properties that db8 assigns to objects. These are delineated with an underscore ('_') based on JSON DB conventions. Because of this, applications cannot create properties that start with an underscore.

Currently, the following special properties are assigned to stored objects:

  • _id -- An object's globally unique identifier. This is assigned even if the object is contained in another object. If not already assigned, the database assigns one. Meant to be an opaque object within a JSON object and it is, therefore, advised your code not make any assumptions about this field.

  • _rev -- Each database instance maintains a globally-unique revision counter that is incremented after every object update or change. An object is assigned this counter each time it is modified. No two objects can ever have the same value for this field. You can use this number as a base point for syncing and in your queries, i.e., "Give me all the changes that were made to contacts since the revision number from yesterday's update."

  • _kind -- Applications are required to assign this string kind identifier (without the underscore).

Here is a sample contacts object once it has been assigned special properties:

{
  "_id"          : "2+n4",
  "_rev"         : 276,
  "_kind"        : "com.palm.contact:1",
  "displayName"  : "John Doe",
  "name"         : {"familyName":"Doe", "givenName":"John"},
  "nickname"     : "Skippy",
  "emails"       : [ "jdoe@gmail.com", "john.doe@palm.com", "jdoe99@gmail.com" ]
}

 


Queries and Indexing

db8 queries take the following form:

{
  "select"  : string array,
  "from"    : string,
  "where"   : [ {
      "prop"    : string,
      "op"      : string,
      "val"     : any
  } ]
  "orderBy" : string,
  "desc"    : boolean,
  "incDel"  : boolean,
  "limit"   : integer,
  "page"    : string
}

See the Query object documentation for field descriptions.

For example: (Give me the display name and state for all contacts living in California, order by display name, return in descending order, include records that have been deleted, return up to 40 records, and use the "next page" key returned from the previous query.)

{
  "select"  : ["displayName", "state"],
  "from"    : "com.mystuff.contacts:1",
  "where"   : [{"prop":"state","op":"=","val":"CA"}],
  "orderBy" : "displayName",
  "desc"    : true,
  "incDel"  : true,
  "limit"   : 40,
  "page"    : "BGhlbGxvAAs3"
}

Queries allow you to do the following:

  • Specify the following filter operators:

    =, <, <=, >, >=, !=, %, ? (equals, less than, greater than, greater than or equal, not equal, wildcard, and full-text search)
    
    

    Though there is no specific range filter operator, you could get a range using other filter operators (i.e., '>' and '<') in two where clauses.

    The "%" operator (aka - the prefix operator) is a type of wildcard -- it will return all matches beginning with the value you specify. For example:

    "where":[{"prop":"displayName","op":"%","val":"J"}]
    
    

    This where clause returns all objects whose displayName field begins with 'J'.

    In SQL, this JSON formatted where clause would look like this:

    WHERE displayName  LIKE 'J%'
    
    

    Note: In db8, all where clauses are implicitly AND'd; OR is not supported.

  • Order objects on a specified field in ascending or descending order.

  • Limit the number of objects returned (maximum is 500).

  • Get the number of objects that would have been returned if there had not been a limit.

  • Include objects that have been deleted. Objects are not fully deleted until an administrative purge operation has occurred. Until then, they are simply marked as deleted.

  • Use a page identifier (key) returned from one query to get the next page of objects.

 

All Queries Must be on Indexed Fields

It is important to note that all queries must be on indexed fields and the indexes you create determine the kind of queries you can make. This means two things:

  1. You can only query on an indexed property

  2. Your queries are limited to those that can be answered with contiguous results from the index table(s)

You define indexes with IndexClause objects when you are creating a kind with the "putKind" call. You can create multiple indexes for your kind, each of which can consist of multiple properties. Each entry in an index is an array of values. So, for instance, if you create a multiple property index composed first of a "state" property and then a "displayName" property, the entries would look like this:

["CA", "John Doe"], ["CA", "Mabel Syrup"], ["OR", "Don Juan"]...

The indexes are ordered first by state, then displayName. This means you could get results if you queried on state (i.e., state = "CA"), since state values are contiguously ordered in the index, but not if you queried on displayName.

Not only do results have to be contiguously ordered, but they have to be guaranteed to be contiguously ordered in advance. What does this mean? Let's take an example; given the following [state, displayName] index:

["CA", "Helen Wheels"],  ["CA", "Jerry Seinfeld"], ["CA", "John Doe"],
  ["CA", "Mabel Syrup"],  ["CA", "Tony Soprano"] 

You could do a search on a single displayName entry, i.e., "where" :[{"prop":"displayName", "op":"=", "val":"Tony Soprano"}] that would return one result. It would meet the criteria of being contiguously ordered, but there is no way of guaranteeing that in advance. Property values are not guaranteed to be unique and there could be multiple "Tony Soprano" entries in different states. To remedy this, you could create an additional index consisting of displayName first and, optionally, other properties after that.

You need to keep the above information in mind when you are creating your indexes and how your data is going to be accessed.

 


Atomicity and Optimistic Concurrency

In an object update, atomicity means that either all of it takes place and is committed to the database, or none of it is.

In db8, all updates (ie., putting or deleting multiple objects) done in a single API call are atomic. Batch operations that combine multiple operations atomically are not supported.

Application-level transactions are not supported. Doing so would allow database locks to be held for a long period of time, potentially blocking other applications, including those the user is waiting on.

Users can implement a form of optimistic concurrency to ensure atomicity. This would involve the following steps:

  1. Get an object and note its revision value (_rev).
  2. Modify the object.
  3. Perform a conditional merge operation with a query that specifies the object's id and revision values (_id and _rev).
  4. If the revision number has changed, indicating it has been modified, repeat the cycle and try again.

 


Schema Enforcement

db8 provides a mechanism for validating JSON data objects when stored with the put call. Your app or service can do this when creating the data object's kind with the putKind call using the schema property. For this field, enter the object's JSON schema according to official JSON standards.

The following is a simple example of how this works using the serviceRequest API.

  1. Create a kind with a schema:

         this.controller.serviceRequest("palm://com.palm.db/", {
            method: "putKind",
            parameters: { "id":"com.palm.schema.test:1",
                          "schema": {"id":"com.palm.schema.test:1", 
                                     "type": "object", 
                                     "properties" : { "_kind" : {"type": "string",
                                                                 "value":"com.palm.schema.test:1"}, 
                                                        "foo": {"type": "string",
                                                                "description": "foo string"}, 
                                                        "bar": {"type": "string", 
                                                                "description": "bar string"},
                                                        "isMember": {"type": "boolean", 
                                                                     "description" : "Is member flag" }
                                                    }
                                    },
                        "owner":"com.palm.dbtest",
                        "indexes": [ {"name":"foo",
                                      "props":[{"name":"foo"}]},
                                    {"name":"barfoo",
                                      "props":[{"name":"bar"},{"name":"foo"}] }]
            },
            onSuccess: function() { Mojo.Log.info("Schema test, putKind success!");},
            onFailure: function(e) { Mojo.Log.info("Schema test, putKind failure! Err = " + JSON.stringify(e));}
     });
     
    

    Output:

     Schema test, putKind success!
    
  2. Store an object with all required fields:

       var GoodObj = {"_kind":"com.palm.schema.test:1", "foo":"myFoo", "bar":"myBar", "isMember": true};
       var objs = [GoodObj];
       this.controller.serviceRequest("palm://com.palm.db/", {
            method: "put",
            parameters: {"objects": objs },
            onSuccess: function(e) { Mojo.Log.info("Schema test, put success! e=" + JSON.stringify(e));},
            onFailure: function(e) { Mojo.Log.info("Schema test, put failure! Err = " + JSON.stringify(e));}
     }); 
     
    

    Output:

     Schema test, put success! e={"returnValue":true,"results":[{"id":"++HG_CzxbvBR_CAe","rev":8946}]}
     
    
  3. Try to store an object with a missing required field:

       var BadObj = {"_kind":"com.palm.schema.test:1", "foo":"myFoo", "bar":"myBar"};
       objs = [BadObj];
       this.controller.serviceRequest("palm://com.palm.db/", {
            method: "put",
            parameters: {"objects": objs },
            onSuccess: function(e) { Mojo.Log.info("Schema test, put success! e=" + JSON.stringify(e));},
            onFailure: function(e) { Mojo.Log.info("Schema test, put failure! Err = " + JSON.stringify(e));}
     });  
     
    

    Output:

     Schema test, put failure! Err = {"errorCode":-985,"errorText":"schema validation failed for kind 'com.palm.schema.test:1': required property not found - 'isMember'","returnValue":false}
     
    
  4. Try to store an object with a field that has a wrong type value:

       var BadObj2 = {"_kind":"com.palm.schema.test:1", "foo":"myFoo", "bar":"myBar", "isMember":"true"};
       objs = [BadObj2];
       this.controller.serviceRequest("palm://com.palm.db/", {
            method: "put",
            parameters: {"objects": objs },
            onSuccess: function(e) { Mojo.Log.info("Schema test, put success! e=" + JSON.stringify(e));},
            onFailure: function(e) { Mojo.Log.info("Schema test, put failure! Err = " + JSON.stringify(e));}
     }); 
     
    

    Output:

     Schema test, put failure! Err = {"errorCode":-985,"errorText":"schema validation failed for kind 'com.palm.schema.test:1': invalid type for property 'isMember'","returnValue":false},
     
    

 


Change Notifications

The "find" call in both the JS Wrapper API and service API supports change notification. The service API also has the "search" call that supports this. When the results returned from the initial query change, the asynchronous callback provided is invoked. See the example code provided with the "find" call documentation to see how this works.

 

Using Revision Sets

If your app or service wants to be notified only when a subset of an object's properties are updated, and you are using the Service API, then you can use revision sets. For example, a sync engine might want to be notified only when a contact's phone number changes. A revision set creates a property that is only updated when one of the set's properties is updated.

For example, the revision set with the name "phoneRev" on the property "phoneNumber" creates a property "phoneRev" with an integer value that is set to the current value of "_rev" whenever the "phoneNumber" property is updated. This allows an app to create a watch using a query of the form "where phoneRev \> X" to be notified when a phone number is updated.

Revision sets are specified at kind creation with the putKind API. For example, the following creates a revision set for the "state" property:

//**
//** Note here that the revision set field ("stateRev") is added to the indexes so that we can query on it later.
//**
var indexes = [{"name":"state", "props":[{"name":"state"}]}, {"name":"stateRev", "props":[{"name":"stateRev"}]} ];
var revSets = [{"name":"stateRev", "props":[{"name":"state"}]}];
this.controller.serviceRequest("palm://com.palm.db/", {
  method: "putKind",
  parameters: { "id":"com.palm.sample:1",
      "owner":"com.palm.dbtest",
      "indexes": indexes,
      "revSets": revSets
  },
  onSuccess: function()  { Mojo.Log.info("putKind success!");},
  onFailure: function(e) { Mojo.Log.info("putKind failure! Err = " + JSON.stringify(e));}
});

After you "put" a record, your app can do a "get" to see the revision set number:

var contact1 = {"_kind":"com.palm.sample:1", "name":"Mabel Syrup", "state":"CA"};
var objs = [contact1];
this.controller.serviceRequest("palm://com.palm.db/", {
  method: "put",
  parameters: {"objects": objs },
  onSuccess: function(e) { Mojo.Log.info("put success! e=" + JSON.stringify(e)); },
  onFailure: function(e) { Mojo.Log.info("put failure! Err = " + JSON.stringify(e));}
});

var id1 = "++HEIviIqT+9MYkj";
var ids = [id1];
this.controller.serviceRequest("palm://com.palm.db/", {
  method: "get",
  parameters: { "ids": ids },
  onSuccess: function(e) { Mojo.Log.info("get success!, results ="+ JSON.stringify(e));},
  onFailure: function(e) { Mojo.Log.info("get failure! Err = " + JSON.stringify(e));}
});

Example Output

get success!, results ={
  "returnValue":true,
  "results":[
      {
          "_id":"++HEIviIqT+9MYkj",
          "_kind":"com.palm.sample:1",
          "_rev":4881,
          "name":"Mabel Syrup",
          "state":"CA",
          "stateRev":4881
      }
  ]
}

Using the revision set number you can the do a "watch" to be notified when the number gets modified (the "fired" flag is true in the results). Note that if you want to do a query on a revision set, then there has to be an index for it (see the putKind example above).

Example:

var fquery = {"from":"com.palm.sample:1", "where":[{"prop":"stateRev","op":">","val":4881}]};
this.controller.serviceRequest("palm://com.palm.db/", {
  method: "watch",
  parameters: { "query": fquery},
  onSuccess: function(e)
  {
      if (!e.fired)
          Mojo.Log.info("watch initial response success, results= " + JSON.stringify(e));
      else
          Mojo.Log.info("watch results changed");
  },
  onFailure: function(e) { Mojo.Log.info("watch failure! Err = " + JSON.stringify(e));}
});

If the "state" field is subsequently updated, the watch fires. For example, the following "merge" does this:

var mprops = { "state":"MA"};
var mquery = { "from":"com.palm.sample:1","where":[{"prop":"state","op":"%","val":"C"}] };
this.controller.serviceRequest("palm://com.palm.db/", {
  method: "merge",
  parameters: { "query": mquery, "props": mprops },
  onSuccess: function(e) { Mojo.Log.info("merge success!, count= " +e.count);},
  onFailure: function(e) { Mojo.Log.info("merge failure! Err = " + JSON.stringify(e));}
});

 


Allowing Other Apps or Services to Access Your db8 Data

If your app or service creates objects that other apps or services need to access, then you need to create a permissions file in "/etc/palm/db/permissions/". A single file can have permissions for multiple kinds, and it is not necessary to create permissions for the kind owner as it gets all permissions implicitly.

Example 1

This file allows the caller all possible permissions to "com.palm.foo.bar:1".

[
  {
      "type": "db.kind",
      "object": "com.palm.foo.bar:1",
      "caller": "com.palm.fooApp",
      "operations": {
          "create": "allow",
          "read": "allow",
          "update": "allow",
          "delete": "allow"
      }
  }
]

Note that wildcards could also be used for the caller. For example:

"caller": "com.palm.*",

Example 2 - Just Type

For Just Type (formerly known as "Universal Search"), you would have to grant read permissions to "com.palm.launcher" (pardon the somewhat unintuitive name). This allows Just Type to scan your db8 data storage for entries matching the user-entered text. This also involves configuring your app's appinfo.json file. See Just Type for more information.

[
  {
      "type": "db.kind",
      "object": "com.palm.sample:1",
      "caller": "com.palm.launcher",
      "operations": {
          "read": "allow"
      }
  }
]

Note: It is likely that 3rd party developers are NOT going to get direct access to this directory, but, in JavaScript, you can implement db8 permissions programmatically with the following "serviceRequest" call:

var permObj =[{"type":"db.kind","object":'com.palm.sample:1', 
               "caller":"com.palm.launcher", "operations":{"read":"allow"}}];

this.controller.serviceRequest("palm://com.palm.db/", {
        method: "putPermissions",
        parameters: {"permissions":permObj},
        onSuccess: function() { Mojo.Log.info("DB permission granted successfully!");},
        onFailure: function() { Mojo.Log.error("DB failed to grant permissions!");}
 });

 


db8 FAQ

  • Are there any size restrictions for a device's db8 database?

    There is a shared 135MB limit for all apps or services, but no per-app or service limit.

  • How is a single app kept from maxing out the entire 135MB?

    Currently, nothing—the user can allocate all the space to one app that they really care about. In the future, there will be a UI that allows users to see and manage how much space apps are using.

  • How does an app know if there is storage available and how much is left?

    There is not currently an API to do this. In general, this is true for all device storage resources. There may be minimum guarantees in the future, but we are not committed to it, since this would reduce the total amount of usable space. In the next few months, we will closely observe how these issues affect users and developers and respond accordingly. Making device-wide space allocation more dynamic and flexible is a high priority.

  • What happens when an app tries to save to the DB when the max size has been reached?

    It gets an "out-of-space" error.

  • Is data automatically culled based on, say, date?

    Currently, deleted objects are purged two weeks after being marked as deleted. In the future, deletion will be more aggressive and not time-based.

  • Where is the db8 database file stored? Is it on "/media/internal", so the user can back it up?

    It is stored on "/var/db". By setting "sync":true when calling "putKind", you can have Palm back up data objects of that kind. Data is backed up on a daily basis.

  • Can multiple db8 databases be open at the same time?

    There is only one shared db8 database. Apps or services can register multiple kinds for that database.

  • How does it handle non-English characters in sorting and in queries (upper/lowercase)?

    Depends on the system locale and the collation strength defined on the index. With no collate property set, it sorts by utf8 codepoint. With "collate":"primary" set, it does a case and accent insensitive collation using locale-specific rules (this is usually what you want when doing search). With "collate":"secondary" set, it takes acccents into account in locales where accent-dependent collation is expected (usually what you want when displaying a sorted list). Read about collation concepts for more information on collation strengths. Note that the collate property defined on a query must match the collate property defined on an index.