[[titan-io-format]]
Titan IO Formats
----------------

This section documents InputFormat and OutputFormat implementations for Titan databases in Cassandra and HBase.

//[[http://thinkaurelius.github.com/titan/images/titan-logo.png|width=400px]]

* Cassandra
** *InputFormat*:{nbsp}`com.thinkaurelius.titan.hadoop.formats.cassandra.TitanCassandraInputFormat`
** *OutputFormat*:{nbsp}`com.thinkaurelius.titan.hadoop.formats.cassandra.TitanCassandraOutputFormat`
* HBase
** *InputFormat*:{nbsp}`com.thinkaurelius.titan.hadoop.formats.hbase.TitanHBaseInputFormat`
** *OutputFormat*:{nbsp}`com.thinkaurelius.titan.hadoop.formats.hbase.TitanHBaseOutputFormat`

Titan-Hadoop can read and write a Titan database backed by Cassandra or HBase.

The read paths build on the MapReduce InputFormats built and maintained by the Cassandra and HBase projects: `ColumnFamilyInputFormat` and `TableInputFormat`, respectively.  Titan uses these input formats to efficiently read raw byte data out of the storage backend, then builds deserialization and type management logic atop using components from Titan's core.

The write path is more complicated.  It's typically desirable to set `batch-loading=true` when writing data from Titan-Hadoop to Cassandra or HBase for efficiency reasons.  However, `batch-loading=true` prevents Titan from creating new types.  If the Titan-Hadoop job in question created new types in the course of its execution, then `batch-loading=true` could present a problem.

The most efficient way to avoid this problem is to create any types needed by a Titan-Hadoop job prior to execution as documented in <<schema>>.  However, for ease of use, Titan-Hadoop provides an alternative.  When `infer-schema=true` is set (see <<hadoop-config-ref>>), Titan-Hadoop will automatically insert an SchemaInferencer job just prior to the job that writes to Titan.  The SchemaInferencer job scans the graph output headed for Titan and compiles a list of all types mentioned in that output.  It then creates any types that don't already exist in Titan.  Then, Titan can write with `batch-loading=true`, since the SchemaInferencer job has already created any new types that might be needed.

Writing data to Titan is spread across three MapReduce tasks:

. A Mapper writes all vertices.  This mapper writes a pair of (temporary ID, permanent Titan-assigned ID) for each vertex.
. A Reducer reads the temporary-permanent vertex ID pairs and pushes it to all edge pairs, creating a read-only distributed hash map.
. A Mapper writes all edges and output is complete.

These writes are carried out inside Titan transactions configured to persist their entire mutation set in a single batch request.  The mechanism to control the size of a task for both the mappers and reducers is `mapred.max.split.size`. Be sure that the size of the split is not so large that OOMEs occur or so small that coordination overhead costs dominate actual work. Furthermore, some backends prefer larger TX sizes (Cassandra: `20-30MB`) and others prefer smaller TX sizes (HBase: `5-10MB`). It is best to play with these TX sizes/split-sizes; the ideal size can vary depending on the backend and network capacity, the topology of the graph, and the storage configuration (e.g. replication factor, atomic batch writes, etc.). Finally, it is best to have `mapred.job.reuse.jvm.num.tasks=-1` (or a large number like `1000`) as transactions are typically small and JVM start up time may be costly in the long run.

*NOTE*: For those importing into Titan using an external index such as http://www.elasticsearch.org[Elasticsearch], be sure to provide the appropriate properties referencing the Elasticsearch cluster. 

// http://thinkaurelius.github.com/titan/[Titan] is a distributed graph database developed by "Aurelius":http://thinkaurelius.com/ and provided under the "Apache 2":http://www.apache.org/licenses/LICENSE-2.0.html license. Titan is backend agnostic and is currently deployed with support for Apache "Cassandra":http://cassandra.apache.org/ and Apache "HBase":http://hbase.apache.org/ (see "The Benefits of Titan":https://github.com/thinkaurelius/titan/wiki/The-Benefits-of-Titan and "Storage Backend Overview":https://github.com/thinkaurelius/titan/wiki/Storage-Backend-Overview). Titan-Hadoop can be used to bulk load, incremental load, and read data to and from a Titan cluster.

Cassandra IO Formats
~~~~~~~~~~~~~~~~~~~~

TitanCassandraInputFormat
^^^^^^^^^^^^^^^^^^^^^^^^^

This input format encapsulates a Cassandra `ColumnFamilyInputFormat`.  This encapsulated input format generates input splits and reads data from Cassandra. `TitanCassandraInputFormat` is configured through the `input` meta-namespace as described in <<hadoop-config-ref>>.  It allows any of the configuration keys documented in <<titan-config-ref>> under the `input` meta-namespace.  However, since it relies entirely on Cassandra's `ColumnFamilyInputFormat`, only a subset of TitanGraph config keys that have meaningful equivalents in `ColumnFamilyInputFormat` are actually effective:

* `storage.hostname`
* `storage.port`
* `storage.username`
* `storage.password`
* `storage.cassandra.keyspace`

[NOTE]
The configuration file should also set `cassandra.input.partitioner.class` to the full package and class name of the Cassandra partitioner in use.  `ColumnFamilyInputFormat` needs to know the partitioner class to function.

Other keys under the `input` meta-namespace have no effect.

Cassandra Input Example
+++++++++++++++++++++++

First, let's load the _Graph of the Gods_ into a Titan instance backed by Cassandra and Elasticsearch.  Then we'll read it back out using TitanCassandraInputFormat.

[source,gremlin]
----
gremlin> g = TitanFactory.open('conf/titan-cassandra-es.properties')
==>titangraph[cassandrathrift:[127.0.0.1]]
gremlin> GraphOfTheGodsFactory.load(g)
==>null
gremlin> g.shutdown()
==>null
----

Now we'll read data from Cassandra using the following `conf/hadoop/titan-cassandra-input.properties` config that ships with Titan:

[source,properties]
----
# input graph parameters
titan.hadoop.input.format=com.thinkaurelius.titan.hadoop.formats.cassandra.TitanCassandraInputFormat
titan.hadoop.input.conf.storage.backend=cassandrathrift
titan.hadoop.input.conf.storage.hostname=localhost
titan.hadoop.input.conf.storage.port=9160
titan.hadoop.input.conf.storage.cassandra.keyspace=titan
cassandra.input.partitioner.class=org.apache.cassandra.dht.Murmur3Partitioner

# output and side effects
titan.hadoop.sideeffect.format=org.apache.hadoop.mapreduce.lib.output.TextOutputFormat
titan.hadoop.output.format=com.thinkaurelius.titan.hadoop.formats.graphson.GraphSONOutputFormat
----

[source,gremlin]
----
gremlin> g = HadoopFactory.open('conf/hadoop/titan-cassandra-input.properties')      
==>titangraph[hadoop:titancassandrainputformat->graphsonoutputformat]
gremlin> g.V.count()
...
INFO  org.apache.hadoop.mapreduce.Job  - Job job_local808130446_0001 completed successfully
...
==>12
----

//////

// this needs to be rewritten, particularly to remove the imaginary/awkwardly-placed words (inoculous, non-excepting) and make it less misleading about split size vs frame size exceeded on a wide row

*NOTE*: When using Titan/Cassandra as a data source, and if there are vertices with a large number of edges (i.e. a very wide row in Cassandra), an inoculous exception may occur warning that the thrift frame size has been exceeded. While the `cassandra.yaml` can be updated and the following job properties added `cassandra.thrift.framed.size_mb`/`cassandra.thrift.message.max_size_mb`, typically, the easiest way to solve this is to add the following property to the `HadoopGraph` being worked with: `cassandra.input.split.size=512` (see `bin/titan-cassandra-input.properties`). The value `512` is how many kilobytes to make the input split size and this value can be adjusted higher or lower to ensure performant, non-excepting behavior.

//////

TitanCassandraOutputFormat
^^^^^^^^^^^^^^^^^^^^^^^^^^

As mentioned in the introduction to this chapter, `TitanCassandraOutputFormat` opens a `TitanGraph` via `TitanFactory.open` and writes data in much the same way as a `TitanGraph` opened from the Gremlin REPL.

The configuration passed to `TitanFactory.open` is controlled by the `output` meta-namespace as described in <<hadoop-config-ref>>.  In contrast to the Cassandra input format, every option listed in <<titan-config-ref>> is valid and effective with this output format.

[NOTE]
Using Elasticsearch?  Remember to include your ES configuration properties under the `output` meta-namespace.  This way, Titan can update Elasticsearch index records as it writes data to Cassandra.  Writing data to Cassandra via Titan-Hadoop without specifying ES configuration properties could lead to missing or outdated records in index results.

Cassandra Output Example
++++++++++++++++++++++++

This example reads the _Graph of the Gods_ from a GraphSON file and writes it to Cassandra using the output format.  It also shows how a Titan-Hadoop side-effect step can mutate data as it passes through MapReduce.

[source,properties]
----
# input graph parameters
titan.hadoop.input.format=com.thinkaurelius.titan.hadoop.formats.graphson.GraphSONInputFormat
titan.hadoop.input.location=examples/graph-of-the-gods.json

# output data (graph or statistic) parameters
titan.hadoop.output.format=com.thinkaurelius.titan.hadoop.formats.cassandra.TitanCassandraOutputFormat
titan.hadoop.output.conf.storage.backend=cassandrathrift
titan.hadoop.output.conf.storage.hostname=localhost
titan.hadoop.output.conf.storage.port=9160
titan.hadoop.output.conf.storage.cassandra.keyspace=titan
titan.hadoop.output.conf.storage.batch-loading=false
# Schema inference enabled -- no need to create types before loading data
titan.hadoop.output.infer-schema=true
mapred.max.split.size=5242880
mapred.job.reuse.jvm.num.tasks=-1
titan.hadoop.sideeffect.format=org.apache.hadoop.mapreduce.lib.output.TextOutputFormat
----

[source,gremlin]
----
gremlin> g = HadoopFactory.open('conf/hadoop/titan-cassandra-output.properties') 
==>titangraph[hadoop:graphsoninputformat->titancassandraoutputformat]
gremlin> g.V.sideEffect('{it.roman = true}') 
INFO  org.apache.hadoop.mapreduce.Job  - Job job_local1610039367_0001 completed successfully
...
INFO  org.apache.hadoop.mapreduce.Job  - Job job_local1411283528_0002 completed successfully
...
// Besides the normal graph-of-the-gods properties,
// the Titan-Cassandra DB now contains the roman property
// added by the sideEffect step above
gremlin> t = TitanFactory.open('conf/titan-cassandra-es.properties')
==>titangraph[cassandrathrift:[127.0.0.1]]
gremlin> t.V.map
==>{name=alcmene, roman=true, type=human}
==>{name=pluto, roman=true, type=god}
==>{name=hercules, roman=true, type=demigod}
==>{name=nemean, roman=true, type=monster}
==>{name=jupiter, roman=true, type=god}
==>{name=cerberus, roman=true, type=monster}
==>{name=sea, roman=true, type=location}
==>{name=tartarus, roman=true, type=location}
==>{name=hydra, roman=true, type=monster}
==>{name=sky, roman=true, type=location}
==>{name=saturn, roman=true, type=titan}
==>{name=neptune, roman=true, type=god}
----

HBase IO Formats
~~~~~~~~~~~~~~~~

TitanHBaseInputFormat
^^^^^^^^^^^^^^^^^^^^^

This input format encapsulates an HBase `TableInputFormat`.  This encapsulated input format generates input splits and reads data from HBase. `TitanHBaseInputFormat` is configured through the `input` meta-namespace as described in <<hadoop-config-ref>>.  It allows any of the configuration keys documented in <<titan-config-ref>> under the `input` meta-namespace.  However, since it relies entirely on HBase's `TableInputFormat`, only a subset of TitanGraph config keys that have meaningful equivalents in `TableInputFormat` are actually effective:

* `storage.hostname`
* `storage.port`
* `storage.hbase.table`
* `storage.hbase.short-cf-names`

Other keys under the `input` meta-namespace have no effect.

HBase Input Example
+++++++++++++++++++

This example is superficially similar to its Cassandra counterpart example earlier in the chapter.  It first loads the _Graph of the Gods_, then reads it with Titan-Hadoop. 

_The Graph of the Gods_ dataset deployed with Titan can be loaded into Titan/HBase using http://gremlin.tinkerpop.com[Gremlin] (see <<getting-started>>).

[source,gremlin]
----
gremlin> g = TitanFactory.open('conf/titan-hbase-es.properties')
==>titangraph[hbase[titan@127.0.0.1:2181]]
gremlin> GraphOfTheGodsFactory.load(g)
==>null
gremlin> g.shutdown()
==>null
----

Now we'll read data from HBase using the following `conf/hadoop/titan-hbase-input.properties` config that ships with Titan:

[source,properties]
----
# input graph parameters
titan.hadoop.input.format=com.thinkaurelius.titan.hadoop.formats.hbase.TitanHBaseInputFormat
titan.hadoop.input.conf.storage.backend=hbase
titan.hadoop.input.conf.storage.hostname=localhost
titan.hadoop.input.conf.storage.port=2181
titan.hadoop.input.conf.storage.hbase.table=titan
# hbase.mapreduce.scan.cachedrows=1000

# output data (graph or statistic) parameters
titan.hadoop.output.format=com.thinkaurelius.titan.hadoop.formats.graphson.GraphSONOutputFormat
titan.hadoop.sideeffect.format=org.apache.hadoop.mapreduce.lib.output.TextOutputFormat
----

[source,gremlin]
----
gremlin> g = HadoopFactory.open('conf/hadoop/titan-hbase-input.properties')      
==>titangraph[hadoop:titanhbaseinputformat->graphsonoutputformat]
gremlin> g.V.count()
...
INFO  org.apache.hadoop.mapreduce.Job  - Job job_local808130446_0001 completed successfully
...
==>12
----

Please follow the link below for more information on streaming data out of HBase.

* http://gbif.blogspot.com/2012/02/performance-evaluation-of-hbase.html

TitanHBaseOutputFormat
^^^^^^^^^^^^^^^^^^^^^^

As mentioned in the introduction to this chapter, `TitanHBaseOutputFormat` opens a `TitanGraph` via `TitanFactory.open` and writes data in much the same way as a `TitanGraph` opened from the Gremlin REPL.

The configuration passed to `TitanFactory.open` is controlled by the `output` meta-namespace as described in <<hadoop-config-ref>>.  In contrast to the HBase input format, every option listed in <<titan-config-ref>> is valid and effective with this output format.

[NOTE]
Using Elasticsearch?  Remember to include your ES configuration properties under the `output` meta-namespace.  This way, Titan can update Elasticsearch index records as it writes data to HBase.  Writing data to HBase via Titan-Hadoop without specifying ES configuration properties could lead to missing or outdated records in index results.

HBase Output Example
++++++++++++++++++++

[source,properties]
----
# input graph parameters
titan.hadoop.input.format=com.thinkaurelius.titan.hadoop.formats.graphson.GraphSONInputFormat
titan.hadoop.input.location=examples/graph-of-the-gods.json

# output data (graph or statistic) parameters
titan.hadoop.output.format=com.thinkaurelius.titan.hadoop.formats.hbase.TitanHBaseOutputFormat
titan.hadoop.output.conf.storage.backend=hbase
titan.hadoop.output.conf.storage.hostname=localhost
titan.hadoop.output.conf.storage.port=2181
titan.hadoop.output.conf.storage.hbase.table=titan
titan.hadoop.output.conf.storage.batch-loading=true
titan.hadoop.output.infer-schema=true
# controls size of transaction
mapred.max.split.size=5242880
# mapred.reduce.tasks=10
mapred.job.reuse.jvm.num.tasks=-1

titan.hadoop.sideeffect.format=org.apache.hadoop.mapreduce.lib.output.TextOutputFormat
----

///////////////////

// This commented block is not accurate

[source,properties]
----
faunus.graph.output.titan.ids.num-partitions=5 // typically the number of region servers
faunus.graph.output.titan.ids.partition=true
----

Because Titan/HBase does not randomly distribute the data around the cluster it is good to tell Titan to generate random partitions of the ID space so that data is written in (as best as possible) a round robin fashion so no single region server is burdened with data writes.

///////////////////

[source,gremlin]
----
gremlin> g = HadoopFactory.open('conf/hadoop/titan-hbase-output.properties') 
==>titangraph[hadoop:graphsoninputformat->titanhbaseoutputformat]
gremlin> g.V.sideEffect('{it.roman = true}') 
INFO  org.apache.hadoop.mapreduce.Job  - Job job_local1610039367_0001 completed successfully
...
INFO  org.apache.hadoop.mapreduce.Job  - Job job_local1411283528_0002 completed successfully
...
// Besides the normal graph-of-the-gods properties,
// the Titan-HBase DB now contains the roman property
// added by the sideEffect step above
gremlin> t = TitanFactory.open('conf/titan-hbase-es.properties')
==>titangraph[hbase[titan@127.0.0.1:2181]]
gremlin> t.V.map
==>{name=alcmene, roman=true, type=human}
==>{name=pluto, roman=true, type=god}
==>{name=hercules, roman=true, type=demigod}
==>{name=nemean, roman=true, type=monster}
==>{name=jupiter, roman=true, type=god}
==>{name=cerberus, roman=true, type=monster}
==>{name=sea, roman=true, type=location}
==>{name=tartarus, roman=true, type=location}
==>{name=hydra, roman=true, type=monster}
==>{name=sky, roman=true, type=location}
==>{name=saturn, roman=true, type=titan}
==>{name=neptune, roman=true, type=god}
----

Incremental Loading using TitanOutputFormat
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

By default, TitanOutputFormat assumes the graph which receives its output is empty.  In this mode, each connected component in the graph must be completely loaded in a single job, and any duplicate data in the input will be passed through TitanOutputFormat (unless a unique index has been defined to detect the duplication).

However, TitanOutputFormat also supports incremental loading.  In this mode, the graph is not assumed to be empty, and duplicate data in the input can be detected and resolved before Titan-Hadoop writes data to the graph.

The setting controlling incremental loading is `titan.hadoop.output.loader-script-file`.  When this is set, it points to a Gremlin-Groovy script defining some methods to which Titan delegates vertex, edge, and vertex-property resolution.  These methods, implemented by the user, define TitanOutputFormat's strategy for incrementally updating graph data across several jobs and for detecting and resolving duplicate data in the input.

Incremental Loading Method Contracts
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

When `loader-script-file` is set, TitanOutputFormat checks the script for each of the following methods:
  
`getOrCreateVertex(FaunusVertex, TitanGraph, TaskInputOutputContext, Logger)`::
The first parameter (of type `FaunusVertex`) is the vertex which Titan-Hadoop is attempting to write to `TitanGraph`.  This method must either retrieve an existing `Vertex` from the `TitanGraph` parameter which corresponds to the `FaunusVertex` parameter or a new `Vertex` in the `TitanGraph` parameter if no existing vertices correspond to the `FaunusVertex` parameter.
+
This method must return an object of type `TitanVertex`.
+
If this method throws an exception or returns null, then Titan-Hadoop may drop the `FaunusVertex`.
+
The last two parameters are conveniences to support Hadoop counter updates and logging (at the implementation's option).  The last two parameters are not required to implement the method contract and can safely be ignored without affecting correctness.

`getOrCreateVertexProperty(TitanProperty, TitanVertex, TitanGraph, TaskInputOutputContext, Logger)`::
The first parameter (of type `TitanProperty`) is a vertex property which Titan-Hadoop is attempting to set on the second parameter (of type `TitanVertex`).  The vertex belongs to the `TitanGraph` parameter.  If the previous method was defined, then it will be the same TitanVertex object returned from the previous method.  However, the property parameter is not from the `TitanGraph`.  It's from the Titan-Hadoop job input.  This method is responsible for checking whether the `TitanVertex` parameter already has a property corresponding to the `TitanProperty` parameter, creating said property on the `TitanVertex` parameter if necessary.
+
Implementations should return the value associated with the property, but this is not required.  The return value of this method is currently ignored.
+
The last two parameters are conveniences to support Hadoop counter updates and logging (at the implementation's option).  The last two parameters are not required to implement the method contract and can safely be ignored without affecting correctness.

`getOrCreateEdge(FaunusEdge, TitanVertex, TitanVertex, TitanGraph, TaskInputOutputContext, Logger)`::
The first parameter is an edge from the Titan-Hadoop job input.  The second and third parameters are the out/tail and in/head vertices of the edge.  Both the second and third parameters are vertices belonging to the fourth parameter of type `TitanGraph`.  This method must either retrieve an existing edge or create a new edge corresponding to the first parameter.
+
This method must return an object of type `TitanEdge`.
+
The last two parameters are conveniences to support Hadoop counter updates and logging (at the implementation's option).  The last two parameters are not required to implement the method contract and can safely be ignored without affecting correctness.

The script must define at least one of the methods listed above.  If TitanOutputFormat can't load and compile at least one of the methods listed above using the supplied file, then an exception is thrown.  There will also be some debugging output citing compilation failures just prior to the exception.  Only the parameter types and method names matter; the parameter names are ignored.  TitanOutputFormat carries out the following steps when checking for each method:

. Load the script file into a string builder
. Append a newline
. Append `metaClass.getMetaMethod(mname, ...) != null`, where `mname` is the method name and `...` is the comma-separated list of argument types
. Append another newline
. Compile and invoke the resulting string using GremlinGroovyScriptEngine

If the compiled script string evaluates to true, then the process is repeated with the final `metaClass...` line changed to an invocation of the method itself.  This is compiled and then invoked by TitanOutputFormat as needed.

Example Incremental Loading Script
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

The sample incremental loading script below was tested with the <<getting-started,Graph of the Gods>> dataset.  It assumes the property key "name" exists on all vertices and has unique values among all vertices.

[source,groovy]
----
def TitanVertex getOrCreateVertex(faunusVertex, graph, context, log) {
    String uniqueKey = "name";
    Object uniqueValue = faunusVertex.getProperty(uniqueKey);
    Vertex titanVertex;
    if (null == uniqueValue)
      throw new RuntimeException(faunusVertex + " has no value for key " + uniqueKey);
  
    Iterator<Vertex> itty = graph.query().has(uniqueKey, uniqueValue).vertices().iterator();
    if (itty.hasNext()) {
      titanVertex = itty.next();
      if (itty.hasNext())
        log.info("The key {} has duplicated value {}", uniqueKey, uniqueValue);
    } else {
      titanVertex = graph.addVertex(faunusVertex.getId());
    }
    return titanVertex;
}

def TitanEdge getOrCreateEdge(faunusEdge, inVertex, outVertex, graph, context, log) {
    final String label = faunusEdge.getLabel();

    log.debug("outVertex:{} label:{} inVertex:{}", outVertex, label, inVertex);

    final Edge titanEdge = !outVertex.out(label).has("id", inVertex.getId()).hasNext() ?
        graph.addEdge(null, outVertex, inVertex, label) :
        outVertex.outE(label).as("here").inV().has("id", inVertex.getId()).back("here").next();

    return titanEdge;
}

def Object getOrCreateVertexProperty(faunusProperty, vertex, graph, context, log) {

    final com.thinkaurelius.titan.core.PropertyKey pkey = faunusProperty.getPropertyKey();
    if (pkey.getCardinality().equals(com.thinkaurelius.titan.core.Cardinality.SINGLE)) {
        vertex.setProperty(pkey.getName(), faunusProperty.getValue());
    } else {
        vertex.addProperty(pkey.getName(), faunusProperty.getValue());
    }

    log.debug("Set property {}={} on {}", pkey.getName(), faunusProperty.getValue(), vertex);

    return faunusProperty.getValue();
}
----

