[[script-io-format]]
Script IO Format
----------------

// [[images/script-format-logo.png|width=350px]]

* *InputFormat*:{nbsp}`com.thinkaurelius.titan.hadoop.formats.script.ScriptInputFormat`
* *OutputFormat*:{nbsp}`com.thinkaurelius.titan.hadoop.formats.script.ScriptOutputFormat`

`ScriptInputFormat` and `ScriptOutputFormat` delegate the formatting of an adjacency list to a user-defined http://gremlin.tinkerpop.com/[Gremlin] script.  This is useful when reading or writing adjacency list files in an unusual format.

`ScriptInputFormat` extends Hadoop's standard `FileInputFormat`, and likewise, `ScriptOutputFormat` extends `FileOutputFormat`.

ScriptInputFormat
~~~~~~~~~~~~~~~~~

ScriptInputFormat extends FileInputFormat.  It uses two configuration parameters:

`titan.hadoop.input.location`:: Path(s) of adjacency list file(s) to parse.  The value of this setting is passed to `FileInputFormat.setInputPaths`.
`titan.hadoop.input.conf.script-file`:: Path to a Gremlin script defining a `boolean read(FaunusVertex vertex, String line)` method.  Input splits will be broken into line strings and passed, one at a time, into this method.  The method should parse its line parameter and modify vertex parameter to reflect the parse result.  Each invocation gets a new, just-instantiated vertex object.  This method should return true when it successfully parses a vertex and false otherwise.

ScriptInputFormat Example
^^^^^^^^^^^^^^^^^^^^^^^^^

The data below represent an adjacency list representation of an untyped, directed graph. First line reads, "vertex 0 has no outgoing edges." The second line reads, "vertex 1 has an outgoing edge to vertices 4, 3, 2, and 0."

[source,text]
----
0:
1:4,3,2,0
2:5,3,1
3:11,6,1,2
4:
5:
6:
7:8,9,10,11,1
8:
9:
10:
11:6
----

There is no corresponding `InputFormat` that can parse this particular file (or some adjacency list variant of it), but `ScriptInputFormat` can parse it when provided with the follwing Gremlin file:

[source,groovy]
----
import com.thinkaurelius.titan.hadoop.FaunusVertex
import com.tinkerpop.blueprints.Direction

def boolean read(FaunusVertex v, String line) {
    parts = line.split(':');
    v.setId(Long.valueOf(parts[0]));
    if (parts.length == 2) {
        parts[1].split(',').each {
            v.addEdge(Direction.OUT, 'linkedTo', Long.valueOf(it));
        }
    }
    return true;
}
----

Both of the files listed are in the Titan distribution archive's `examples` directory.

The following Titan-Hadoop configuration ties this example together.  This file is `conf/hadoop/script-input.properties` in the Titan distribution archive.

[source,properties]
----
# input graph parameters
titan.hadoop.input.format=com.thinkaurelius.titan.hadoop.formats.script.ScriptInputFormat
titan.hadoop.input.location=examples/graph-of-the-gods.id
titan.hadoop.input.conf.script-file=examples/ScriptInput.groovy
titan.hadoop.input.edge-copy-direction=OUT

# output data parameters
titan.hadoop.output.format=com.thinkaurelius.titan.hadoop.formats.graphson.GraphSONOutputFormat

titan.hadoop.sideeffect.format=org.apache.hadoop.mapreduce.lib.output.TextOutputFormat
----

Here's how it works in a Gremlin REPL session:

[source,gremlin]
----
// Copy files to HDFS (unnecessary when using LocalFileSystem)
gremlin> hdfs.copyFromLocal('examples/ScriptInput.groovy','examples/ScriptInput.groovy')
==>null
gremlin> hdfs.copyFromLocal('examples/graph-of-the-gods.id','examples/graph-of-the-gods.id')
==>null
gremlin> hdfs.ls('examples')
==>rw-r--r-- marko supergroup 868 ScriptInput.groovy
==>rw-r--r-- marko supergroup 69 graph-of-the-gods.id
// Open HadoopGraph
gremlin> g = HadoopFactory.open('conf/hadoop/script-input.properties')
==>titangraph[hadoop:scriptinputformat->graphsonoutputformat]
gremlin> g._()
...
INFO  org.apache.hadoop.mapreduce.Job  - Job job_local902401374_0001 completed successfully
...
gremlin> hdfs.head('jobs')
==>{"_id":0,"_inE":[{"_id":null,"_outV":1,"_label":"linkedTo"}]}
==>{"_id":1,"_outE":[{"_id":null,"_inV":4,"_label":"linkedTo"},{"_id":null,"_inV":2,"_label":"linkedTo"},{"_id":null,"_inV":0,"_label":"linkedTo"},{"_id":null,"_inV":3,"_label":"linkedTo"}],"_inE":[{"_id":null,"_outV":2,"_label":"linkedTo"},{"_id":null,"_outV":7,"_label":"linkedTo"},{"_id":null,"_outV":3,"_label":"linkedTo"}]}
==>{"_id":2,"_outE":[{"_id":null,"_inV":1,"_label":"linkedTo"},{"_id":null,"_inV":5,"_label":"linkedTo"},{"_id":null,"_inV":3,"_label":"linkedTo"}],"_inE":[{"_id":null,"_outV":1,"_label":"linkedTo"},{"_id":null,"_outV":3,"_label":"linkedTo"}]}
==>{"_id":3,"_outE":[{"_id":null,"_inV":2,"_label":"linkedTo"},{"_id":null,"_inV":1,"_label":"linkedTo"},{"_id":null,"_inV":6,"_label":"linkedTo"},{"_id":null,"_inV":11,"_label":"linkedTo"}],"_inE":[{"_id":null,"_outV":1,"_label":"linkedTo"},{"_id":null,"_outV":2,"_label":"linkedTo"}]}
==>{"_id":4,"_inE":[{"_id":null,"_outV":1,"_label":"linkedTo"}]}
==>{"_id":5,"_inE":[{"_id":null,"_outV":2,"_label":"linkedTo"}]}
==>{"_id":6,"_inE":[{"_id":null,"_outV":3,"_label":"linkedTo"},{"_id":null,"_outV":11,"_label":"linkedTo"}]}
==>{"_id":7,"_outE":[{"_id":null,"_inV":1,"_label":"linkedTo"},{"_id":null,"_inV":11,"_label":"linkedTo"},{"_id":null,"_inV":8,"_label":"linkedTo"},{"_id":null,"_inV":10,"_label":"linkedTo"},{"_id":null,"_inV":9,"_label":"linkedTo"}]}
==>{"_id":8,"_inE":[{"_id":null,"_outV":7,"_label":"linkedTo"}]}
==>{"_id":9,"_inE":[{"_id":null,"_outV":7,"_label":"linkedTo"}]}
==>{"_id":10,"_inE":[{"_id":null,"_outV":7,"_label":"linkedTo"}]}
==>{"_id":11,"_outE":[{"_id":null,"_inV":6,"_label":"linkedTo"}],"_inE":[{"_id":null,"_outV":7,"_label":"linkedTo"},{"_id":null,"_outV":3,"_label":"linkedTo"}]}
----

Consider the setting `titan.hadoop.input.edge-copy-direction=OUT` in the configuration used above.  This setting generates `_inE` entries mirroring the `_outE` entries.  Here's how the output looks without the `edge-copy` step:

[source,gremlin]
----
...
gremlin> hdfs.head('jobs')
==>{"_id":0}
==>{"_id":1,"_outE":[{"_id":null,"_inV":4,"_label":"linkedTo"},{"_id":null,"_inV":2,"_label":"linkedTo"},{"_id":null,"_inV":0,"_label":"linkedTo"},{"_id":null,"_inV":3,"_label":"linkedTo"}]}
==>{"_id":2,"_outE":[{"_id":null,"_inV":1,"_label":"linkedTo"},{"_id":null,"_inV":5,"_label":"linkedTo"},{"_id":null,"_inV":3,"_label":"linkedTo"}]}
==>{"_id":3,"_outE":[{"_id":null,"_inV":2,"_label":"linkedTo"},{"_id":null,"_inV":1,"_label":"linkedTo"},{"_id":null,"_inV":6,"_label":"linkedTo"},{"_id":null,"_inV":11,"_label":"linkedTo"}]}
==>{"_id":4}
==>{"_id":5}
==>{"_id":6}
==>{"_id":7,"_outE":[{"_id":null,"_inV":1,"_label":"linkedTo"},{"_id":null,"_inV":11,"_label":"linkedTo"},{"_id":null,"_inV":8,"_label":"linkedTo"},{"_id":null,"_inV":10,"_label":"linkedTo"},{"_id":null,"_inV":9,"_label":"linkedTo"}]}
==>{"_id":8}
==>{"_id":9}
==>{"_id":10}
==>{"_id":11,"_outE":[{"_id":null,"_inV":6,"_label":"linkedTo"}]}
----

ScriptOutputFormat
~~~~~~~~~~~~~~~~~~

`titan.hadoop.output.conf.script-file`:: Path to a Gremlin script defining a `void write(FaunusVertex vertex, DataOutput output)` method.  This method is invoked once per vertex.  The method should serialize the vertex parameter using the `DataOutput` parameter.

The output path is controlled by Titan-Hadoop.  See `jobdir` in <<hadoop-config-ref>> for information on configuring Titan-Hadoop's output path.

ScriptOutputFormat Example
^^^^^^^^^^^^^^^^^^^^^^^^^^

The following Gremlin script writes vertices in the same adjacency list format shown in the previous section.  The vertex ID appears first on each line, then a colon, then a comma-separated list of edges represented by the ids of the in-vertices.

[source,groovy]
----
import com.thinkaurelius.titan.hadoop.FaunusVertex
import com.tinkerpop.blueprints.Edge

import static com.tinkerpop.blueprints.Direction.IN
import static com.tinkerpop.blueprints.Direction.OUT

def void write(FaunusVertex vertex, DataOutput output) {
    output.writeUTF(vertex.getId().toString() + ':');
    Iterator<Edge> itty = vertex.getEdges(OUT).iterator()
    while (itty.hasNext()) {
        output.writeUTF(itty.next().getVertex(IN).getId().toString());
        if (itty.hasNext())
            output.writeUTF(',');
    }
    output.writeUTF('\n');
}
----

The following Gremlin REPL session shows how they work in practice:

[source,gremlin]
----
// Copy files to HDFS (not necessary when using LocalFileSystem)
gremlin> hdfs.copyFromLocal('examples/graph-of-the-gods.json','examples/graph-of-the-gods.json')
==>null
gremlin> hdfs.copyFromLocal('examples/ScriptOutput.groovy','examples/ScriptOutput.groovy')
==>null
gremlin> g = HadoopFactory.open('conf/hadoop/script-output.properties')
==>titangraph[hadoop:graphsoninputformat->scriptoutputformat]
gremlin> g._()
...
INFO  org.apache.hadoop.mapreduce.Job  - Running job: job_local1168433758_0001
...
INFO  org.apache.hadoop.mapreduce.Job  - Job job_local1970636407_0001 completed successfully
...
gremlin> hdfs.head('jobs')
==>0:
==>1:4,3,2,0
==>2:5,3,1
==>3:11,6,2,1
==>4:
==>5:
==>6:
==>7:8,9,10,11,1
==>8:
==>9:
==>10:
==>11:6
----
