JSON and DB2 By Joe Geller (JoeDB2@aol.com) Among the many exciting enhancements with DB2 10.

5 BLU for LUW and DB2 11 for z/OS (coming soon and DB2 10 also via PTF), flying a little under the radar is the announced support for JSON within DB2.

What is JSON? JSON stands for JavaScript Object Notation. It is a data exchange format and is a subset of javascript, but you can think of it as a light weight XML. It is: Name/Value pairs (think XML nodes and node values or attributes). Structured (i.e. you can have a hierarchy of name/values, and you can have arrays of values). There are no tags. JSON is not as self-documenting as XML is. No XML schema (schemas are optional in pureXML, and XML in general).

JSON is just a structured set of name value pairs. For example: {custid: 123788, ordernum: 4563, shipmethod: USPS, items: [ {itemnum: 45, qty: 4, unitprice: 12.95} , {itemnum: 52, qty: 6, unitprice: 349, color:blue) ] }

This document has attributes custid, ordernum and shipmentmethod, and an array (items) each element of which consists of a substructure with several attributes itemnum, qty, price and color. Note that one item has the attribute color and the other one does not. Why JSON? It is lighter in weight than XML. There are no tags for example . Since it takes fewer bytes than XML it takes less storage space, and more important, there are fewer bytes to send over the network. It is very simple in format. It is quicker to parse and process than XML. It is very useful for object persistence from various object oriented languages such as Java and is being heavily used for that purpose and for use with web pages that use AJAX to dynamically update those pages. As with XML, it has a very flexible schema (i.e., the schema is not required). Different documents of the same type can evolve and can have differing content, including optional fields. This enhances rapid development and can easily accommodate change.

Why DB2? JSON is a popular format for data persistence This has led to the success of so-called NOSQL databases. Of these, the most widely used is MongoDB DB2 has a robust industrial strength infrastructure. By using DB2 for persisting your JSON objects, you get the most secure, reliable DBMS on the market. DB2 provides multi-statement transactions, security, backup/recovery and high availability. DB2 has partnered with MongoDB and has implemented the MongoDB API. This allows applications to be easily ported to DB2. These applications can be written in any language that supports that API. By storing JSON in DB2, you can integrate your relational data with your JSON data. You could of course create a name/value store yourself within DB2. Many shops have such a store, often called an Attribute table. Even a hierarchy could be done with some clever design and coding. But, these tables will have many more rows and therefore will be slower and require more complex coding than the use of JSON within DB2.

Just as the growing popularity of XML led to IBM developing a best of breed solution for the storage and retrieval of XML data, it is important for IBM to do the same with JSON. JSON use is increasing. It has many of the benefits of XML, with better performance and a simpler model. If DB2 doesnt support JSON, shops will be looking elsewhere for their database needs.

DB2 Support for JSON

The initial support is a first pass and is referred to as a Technology Preview. In some ways that is like the XML extender. It provided support, but pureXML was lightyears beyond. JSON data is called a document. Documents are stored in collections. Collections do not impose structure to the documents, but do make finding and processing similar documents much easier. UDFs to support the creation and retrieval of documents. An API for manipulating the data. Indexes can be defined .

At this point, JSON in DB2 is not a first class data type like XML, but I think it is likely that this will change in the future. Some features not yet supported include accessing relational and JSON data in the same query; SQL access to JSON data; construction functions to enable you to build JSON documents from relational data as with the XML TABLE function; joins between documents in different collections and with relational tables; temporal support for JSON data. These are all in IBMs future plans for JSON. With the initial release, there may be some small differences between DB2 LUW and DB2 z/OS in syntax and data typing rules, but they should be fairly close.

Accessing and Manipulating JSON Data in DB2 There are 3 ways to access the JSON datastore in DB2: 1) A Java API any Java application can interact with JSON data via this API. There are many similarities to JDBC access. 2) A Wire Listener that interprets the MongoDB wire protocol (API). This enables an existing MongoDB application written in languages such as PHP, Ruby, C++ and NodeJS (a server side javascript framework) to be directly ported to DB2. 3) A command line processor (similar to DB2 LUWs CLP) to manipulate JSON via the command line.

JSON Data Storage in DB2 JSON data can be stored in an existing DB2 database that also contains relational tables. You have to enable this database to support JSON (some tables and UDFs are created). This is done with the enable(true) command. This example is for DB2 LUW. Since the concept of a database is different in DB2 z/OS, the process may be different there. Using the command line processor (db2nosql) you would first connect to the database: db2nosql -db sample user testuser password testpwd nosql > enable(true)

JSON namespaces are implemented as DB2 schemas (qualifiers). To use an existing schema named myschema, you would enter: nosql > use myschema Documents and Collections Documents of a particular type are organized into a collection. The documents in a collection do not have to actually be of the same type of data. However, having similar documents in one collection makes search and retrieval easier. Even with only one type of document (e.g. customer_info), the individual documents can have differing structure and optional fields. The application and data can evolve over time and newer documents can have new attributes and structure, with no need to modify existing documents (depending of course on application requirements). A collection is automatically created the first time you insert a document into it. Each document must have a unique identifier. If it has an attribute named _id, that will be used. If not, DB2 will generate a unique id of type VARCHAR(12) FOR BIT DATA. To insert an order into the ORDERS collection you can issue: nosql > db.ORDERS.insert( { custid:123788, ordernum:4563, shipmethod:USPS } )

You can also create the collection explicitly as follows: nosql > db.createCollection(ORDERS)

Java Access 1) Connect to the database DB db = NoSQLClient.getDB(databaseurl, user, password, namespace); DB is a database object. Namespace is the schema you will be using. 2) Get a handle to the collection DBCollection col = db.getCollection(ORDERS); 3) Setup the document object

BasicDBObject doc1 = new BASICDBObject(); doc1.append(custid, 123788); doc1.append(ordernum,4563); doc1.append(shipmethod,USPS); col.insert(doc1); 4) Retrieving documents. You can select all documents in the collection or you can specify predicates (like a Where clause). As is SQL, you can get all attributes of the document or a subset. In Java you define a cursor and then iterate through it BASICDBObject where = new BASICDBObject(); where.append(custid,123788); BASICDBObject collist = new BASICDBObject(); collist.append(ordernum,1); collist.append(shipmethod,1); DBcursor c1 = col.find(where,collist); try { while (c1.hasNext()) { DBOBJect.obj = c1.next(); System.out.println(ORDER: + obj) } } finally { c1.close(); }

DB2 Storage of Collections and Documents DB2 creates a table to store the documents of a collection. The table schema is the namespace you are using. The table name is the collection name. This name is case sensitive. Each document is a row in that table. The table has 2 columns. ID has the ID of the document. The document is stored in a BLOB column (it is a BLOB and not a CLOB because the data is stored in BSON format, which is a binary JSON format).

Indexes Index definitions are stored in the table SYSTOOLS.SYSJSON_INDEX which is created when you enable the database for JSON. There is an index automatically created on the _id attribute. You can create additional indexes:

nosql> db.ORDERS.ensureIndex({custid:1}) The 1 means ascending. You would use -1 for descending.

Data Design and Rapid Application Development In todays Internet world, organizations must develop and evolve applications rapidly in order to compete successfully. As part of that rapid development, they are minimizing or eliminating the development of database schemas and data models. Developers design and write their programs and need to store the program objects in a persistent store. Changing requirements necessitate a data storage scheme that is flexible and can handle change without requiring (or minimizing) modification to the existing data. This is the reality we face, but we have to recognize the drawbacks too. As database professionals we understand the importance of a data model that reflects the business and the real world objects. A data design that is driven primarily by the process design of a program is likely to result in data redundancy and inconsistency. The data structures that best support a particular program (and support it today) are not likely to best support of the rest of the application system, or the needs of this program in the future. Flexibility and data evolution are critical, along with rapid development. But we need to continue to stress the importance of having a data-centric emphasis working in conjunction with the processing requirements to identify the data design that best meets the current and future needs. NoSQL and JSON provide a means for that flexible and evolving data store, but NoSQL does not mean NoAnalysis and NoDesign. But, we have to recognize the need for speed and continuous integration in application development today.

Summary Just as XML has become an important part of DB2 (important to both customers and to IBM), we are going to see the same for JSON. In many areas JSON use is replacing XML for data interchange and document storage. What better place is there to store it than in DB2?

Learn More A good starting point is a set of Developerworks articles starting with http://www.ibm.com/developerworks/data/library/techarticle/dm-1306nosqlforjson1/ Parts 2 -4 of the above article: http://www.ibm.com/developerworks/data/library/techarticle/dm-1306nosqlforjson2/

http://www.ibm.com/developerworks/data/library/techarticle/dm-1307nosqlforjson3/ http://www.ibm.com/developerworks/data/library/techarticle/dm-1306nosqlforjson4/

This article will have links to a number of other sources of information on JSON, DB2s support for JSON and MongoDB. The Java API is in the form of Javadoc. It can be found at http://publib.boulder.ibm.com/infocenter/db2luw/v10r5/index.jsp?topic=%2Fcom.ibm.swg.im.dbclient. json.doc%2Fjavadoc%2Findex.html&com/ibm/nosql/json/util/JSON.html More about MongoDB - http://www.ibm.com/developerworks/library/?search_by=MongoDB About the author : Joe Geller has been a DB2 consultant since Version 1.2. Hes been a Data Architect, DBA, Data Modeler and Software Engineer, often simultaneously. He feels that the best way to learning one aspect of a system is to work on all aspects. Joe specializes in performance tuning. He once rewrote a PeopleSoft view and improved the query by a factor of 70,000. Joe is the author of two books on database systems DB2 Performance and Development Guide and IMS Administration, Programming and Data Base Design. Currently Joe is working as a consultant for The Fillmore Group.