RedisGraph protocol v2 --compact (#4)

* RedisGraph protocol v2 --compact

* Maintain support for RedisGraph protocol v1.
* Add support for RedisGraph protocol v2.
  * locally cache labels, relationshipTypes, propertyKeys.
  * use --compact form for all graph queries.

* RedisGraph protocol v2 --compact

* Add additional specs from quickstart at the following location:
  https://oss.redislabs.com/redisgraph/
* Modify parsing to pass added specs.
  * aggregate
  * resultset w/o propertyKeys de-reference

* RedisGraph protocol v2 --compact

* Fix aggregate function support.
  * on comparison to RedisGraph protocol v1, found
    aggregate function resultset was incorrectly
    not returning an array of values for the row.

* RedisGraph protocol v2 --compact

* Remove backwards compatibility for v1.
  * Provide README guidance wrt previous version in a subsection under
    compatibility.
  * Specify the docker image which contains v2.
  * Update the gem version to 2.0.0 .
* Refactor RedisGraph::Metadata to remove circular dependency between
  RedisGraph and QueryResult.
* Remove eager load of Metadata.
  * Only invalidate Metadata on observing labels, propertyKeys, and
    relationshipTypes while reading.
  * Do not parse query to determine query type; this is performed by
    the module.
* Refactor and reorder parsing of the resultset columns.
  * Header contains type information.
  * Rows contain compact references to Metadata, so use the first
    row to combine both vectors of information into an array of
    columns.
* Make the specs more isolated using `described_class` w/i the graph
  name.
* Make the specs more readily run individually as well as isolated
  (rspec can/will run in random order) by moving graph creation into
  the before.

* RedisGraph protocol v2 --compact

* Remove remnant compact/call_compact? from when supporting v1.
* Add raise on incompatible version on connect.
* Modify QueryResult columns and result_set to match expected.
  * columns is as in the query.
  * result_set cell by type:
    * scalar - value
    * node - array of properties { key => value }
    * edge - array of properties { key => value }
* Revert begin/end wrap of rescue w/i rspec before(:all).

* RedisGraph protocol v2 --compact

* Add mapping of scalar values to supported scalar types.

* RedisGraph protocol v2 --compact

* Add mapping of scalar value of NULL type.
* Add TODO'd mapping of scalar value of Array type.
  * Module support for this was added 3d ago, but is
    not yet in the distro package or docker image.

Author: paegun

README.md

@@ -9,10 +9,17 @@
 `redisgraph-rb` is a Ruby gem client for the [RedisGraph](https://github.com/RedisLabsModules/RedisGraph) module. It relies on `redis-rb` for Redis connection management and provides support for graph QUERY, EXPLAIN, and DELETE commands.
 
 ## RedisGraph compatibility
-`redisgraph-rb` is currently compatible with RedisGraph versions <= 1.2.2.
+`redisgraph-rb` is currently compatible with RedisGraph versions >= 1.99 (module version: 19900)
 
 The result set structure introduced by RedisGraph 2.0 requires some modifications to this client. If you are interested in using this client with the latest RedisGraph, please inform us by commenting on [the corresponding issue](https://github.com/RedisGraph/redisgraph-rb/issues/1)!
 
+### Previous Version
+For RedisGraph versions >= 1.0 and < 2.0 (ie module version: 10202), instead use and refer to
+the redisgraph gem version ~> 1.0.0
+
+which corresponds to the following docker image
+`docker run -p 6379:6379 -it --rm redislabs/redisgraph:1.2.2`
+
 ## Installation
 To install, run:
 
@@ -62,7 +69,7 @@ To ensure prerequisites are installed, run the following:
 These tests expect a Redis server with the Graph module loaded to be available at localhost:6379
 
 The currently compatible version of the RedisGraph module may be run as follows:
-`docker run -p 6379:6379 -it --rm redislabs/redisgraph:1.2.2`
+`docker run -p 6379:6379 -it --rm redislabs/redisgraph:2.0-edge`
 
 A simple test suite is provided, and can be run with:
 `rspec`

lib/redisgraph.rb

@@ -8,37 +8,73 @@
 class RedisGraph
   attr_accessor :connection
   attr_accessor :graphname
+  attr_accessor :metadata
+
+  class Metadata
+    def initialize(opts = {})
+      @graphname = opts[:graphname]
+      @connection = opts[:connection]
+
+      # cache semantics around these labels, propertyKeys, and relationshipTypes
+      # defers first read and is invalidated when changed.
+      @labels_proc =  -> { call_procedure('db.labels') }
+      @property_keys_proc = -> { call_procedure('db.propertyKeys') }
+      @relationship_types_proc = -> { call_procedure('db.relationshipTypes') }
+    end
+
+    def invalidate
+      @labels = @property_key = @relationship_types
+    end
+
+    def labels
+      @labels ||= @labels_proc.call
+    end
+
+    def property_keys
+      @property_keys ||= @property_keys_proc.call
+    end
+
+    def relationship_types
+      @relationship_types ||= @relationship_types_proc.call
+    end
+
+    def call_procedure(procedure)
+      res = @connection.call("GRAPH.QUERY", @graphname, "CALL #{procedure}()")
+      res[1].flatten
+    rescue Redis::CommandError => e
+      raise CallError, e
+    end
+  end
 
   # The RedisGraph constructor instantiates a Redis connection
   # and validates that the graph module is loaded
   def initialize(graph, redis_options = {})
     @graphname = graph
     connect_to_server(redis_options)
+    @metadata = Metadata.new(graphname: @graphname,
+                             connection: @connection)
   end
 
   # Execute a command and return its parsed result
   def query(command)
-    begin
-    resp = @connection.call("GRAPH.QUERY", @graphname, command)
-    rescue Redis::CommandError => e
-      raise QueryError, e
-    end
-
-    QueryResult.new(resp)
+    resp = @connection.call("GRAPH.QUERY", @graphname, command, '--compact')
+    QueryResult.new(resp,
+                    metadata:   @metadata)
+  rescue Redis::CommandError => e
+    raise QueryError, e
   end
 
   # Return the execution plan for a given command
   def explain(command)
-    begin
-    resp = @connection.call("GRAPH.EXPLAIN", @graphname, command)
-    rescue Redis::CommandError => e
-      raise QueryError, e
-    end
+    @connection.call("GRAPH.EXPLAIN", @graphname, command)
+  rescue Redis::CommandError => e
+    raise ExplainError, e
   end
 
   # Delete the graph and all associated keys
   def delete
-    resp = @connection.call("GRAPH.DELETE", @graphname)
+    @connection.call("GRAPH.DELETE", @graphname)
+  rescue Redis::CommandError => e
+    raise DeleteError, e
   end
 end
-

lib/redisgraph/connection.rb

@@ -1,17 +1,19 @@
 class RedisGraph
   def connect_to_server(options)
     @connection = Redis.new(options)
-    self.verify_module()
+    @module_version = module_version()
+    raise ServerError, "RedisGraph module not loaded." if @module_version.nil?
+    raise ServerError, "RedisGraph module incompatible, expecting >= 1.99." if @module_version < 19900
   end
 
   # Ensure that the connected Redis server supports modules
   # and has loaded the RedisGraph module
-  def verify_module()
+  def module_version()
     redis_version = @connection.info["redis_version"]
     major_version = redis_version.split('.').first.to_i
     raise ServerError, "Redis 4.0 or greater required for RedisGraph support." unless major_version >= 4
     modules = @connection.call("MODULE", "LIST")
     module_graph = modules.detect { |_name_key, name, _ver_key, _ver| name == 'graph' }
-    raise ServerError, "RedisGraph module not loaded." if module_graph.nil?
+    module_graph[3] if module_graph
   end
 end

lib/redisgraph/errors.rb

@@ -1,10 +1,10 @@
 class RedisGraph
-  class RedisGraphError < RuntimeError
-  end
+  class RedisGraphError < RuntimeError; end
 
-  class ServerError < RedisGraphError
-  end
+  class ServerError < RedisGraphError; end
 
-  class QueryError < RedisGraphError
-  end
+  class CallError < RedisGraphError; end
+  class QueryError < RedisGraphError; end
+  class ExplainError < RedisGraphError; end
+  class DeleteError < RedisGraphError; end
 end

lib/redisgraph/query_result.rb

@@ -3,6 +3,23 @@ class QueryResult
   attr_accessor :resultset
   attr_accessor :stats
 
+  def initialize(response, opts = {})
+    # The response for any query is expected to be a nested array.
+    # If compact (RedisGraph protocol v2)
+    # The resultset is an array w/ three elements:
+    # 0] Node/Edge key names w/ the ordinal position used in [1]
+    #    en lieu of the name to compact the result set.
+    # 1] Node/Edge key/value pairs as an array w/ two elements:
+    #  0] node/edge name id from [0]
+    #  1..matches] node/edge values
+    # 2] Statistics as an array of strings
+
+    @metadata = opts[:metadata]
+
+    @resultset = parse_resultset(response)
+    @stats = parse_stats(response)
+  end
+
   def print_resultset
     pretty = Terminal::Table.new headings: columns do |t|
       resultset.each { |record| t << record }
@@ -11,49 +28,111 @@ def print_resultset
   end
 
   def parse_resultset(response)
+    # In the v2 protocol, CREATE does not contain an empty row preceding statistics
+    return unless response.length > 1
+
     # Any non-empty result set will have multiple rows (arrays)
-    return nil unless response[0].length > 1
-    # First row is return elements / properties
-    @columns = response[0].shift
-    # Subsequent rows are records
-    @resultset = response[0]
+
+
+    # First row is header describing the returned records, corresponding
+    # precisely in order and naming to the RETURN clause of the query.
+    header = response[0]
+    @columns = header.map { |(_type, name)| name }
+
+    # Second row is the actual data returned by the query
+    # note handling for encountering an id for propertyKey that is out of
+    # the cached set.
+    data = response[1].map do |row|
+      i = -1
+      header.reduce([]) do |agg, (type, _it)|
+        i += 1
+        el = row[i]
+
+        case type
+        when 1 # scalar
+          agg << map_scalar(el[0], el[1])
+        when 2 # node
+          props = el[2]
+          agg << props.sort_by { |prop| prop[0] }.map { |prop| map_prop(prop) }
+        when 3 # relation
+          props = el[4]
+          agg << props.sort_by { |prop| prop[0] }.map { |prop| map_prop(prop) }
+        end
+
+        agg
+      end
+    end
+
+    data
+  end
+
+  def map_scalar(type, val)
+    map_func = case type
+               when 1 # null
+                 return nil
+               when 2 # string
+                 :to_s
+               when 3 # integer
+                 :to_i
+               when 4 # boolean
+                 # no :to_b
+                 return val == "true"
+               when 5 # double
+                 :to_f
+               # TODO: when in the distro packages and docker images,
+               #   the following _should_ work
+               # when 6 # array
+               #   val.map { |it| map_scalar(it[0], it[1]) }
+               end
+    val.send(map_func)
+  end
+
+  def map_prop(prop)
+    # maximally a single @metadata.invalidate should occur
+
+    property_keys = @metadata.property_keys
+    prop_index = prop[0]
+    if prop_index > property_keys.length
+      @metadata.invalidate
+      property_keys = @metadata.property_keys
+    end
+    { property_keys[prop_index] => map_scalar(prop[1], prop[2]) }
   end
 
   # Read metrics about internal query handling
   def parse_stats(response)
-    return nil unless response[1]
+    # In the v2 protocol, CREATE does not contain an empty row preceding statistics
+    stats_offset = response.length == 1 ? 0 : 2
 
+    return nil unless response[stats_offset]
+
+    parse_stats_row(response[stats_offset])
+  end
+
+  def parse_stats_row(response_row)
     stats = {}
 
-    response[1].each do |stat|
+    response_row.each do |stat|
       line = stat.split(': ')
-      val = line[1].split(' ')[0]
+      val = line[1].split(' ')[0].to_i
 
       case line[0]
       when /^Labels added/
-        stats[:labels_added] = val.to_i
+        stats[:labels_added] = val
       when /^Nodes created/
-        stats[:nodes_created] = val.to_i
+        stats[:nodes_created] = val
       when /^Nodes deleted/
-        stats[:nodes_deleted] = val.to_i
+        stats[:nodes_deleted] = val
       when /^Relationships deleted/
-        stats[:relationships_deleted] = val.to_i
+        stats[:relationships_deleted] = val
       when /^Properties set/
-        stats[:properties_set] = val.to_i
+        stats[:properties_set] = val
       when /^Relationships created/
-        stats[:relationships_created] = val.to_i
+        stats[:relationships_created] = val
       when /^Query internal execution time/
-        stats[:internal_execution_time] = val.to_f
+        stats[:internal_execution_time] = val
       end
     end
     stats
   end
-
-  def initialize(response)
-    # The response for any query is expected to be a nested array.
-    # The only top-level values will be the result set and the statistics.
-    @resultset = parse_resultset(response)
-    @stats = parse_stats(response)
-  end
 end
-

lib/redisgraph/version.rb

@@ -1,3 +1,3 @@
 class RedisGraph
-  VERSION = '1.0.0'
+  VERSION = '2.0.0'
 end

spec/redisgraph_quickstart_spec.rb

@@ -0,0 +1,73 @@
+require 'helper.rb'
+
+require_relative '../lib/redisgraph.rb'
+
+# based on queries extracted from
+describe RedisGraph do
+  before(:all) do
+    begin
+      @r = RedisGraph.new("#{described_class}_test")
+      create_graph
+    rescue Redis::BaseError => e
+      $stderr.puts(e)
+      exit 1
+    end
+  end
+
+  after(:all) do
+    @r.delete if @r
+  end
+
+  def create_graph()
+    q = "CREATE (:Rider {name:'Valentino Rossi'})-[:rides]->(:Team {name:'Yamaha'})," \
+      "(:Rider {name:'Dani Pedrosa'})-[:rides]->(:Team {name:'Honda'})," \
+      "(:Rider {name:'Andrea Dovizioso'})-[:rides]->(:Team {name:'Ducati'})"
+
+    res = @r.query(q)
+
+    expect(res.resultset).to be_nil
+    stats = res.stats
+    expect(stats).to include(:internal_execution_time)
+    stats.delete(:internal_execution_time)
+    expect(stats).to eq({
+      labels_added:          2,
+      nodes_created:         6,
+      properties_set:        6,
+      relationships_created: 3
+    })
+  end
+
+  context 'quickstart' do
+    it 'should query relations, with a predicate' do
+      q = "MATCH (r:Rider)-[:rides]->(t:Team) WHERE t.name = 'Yamaha' RETURN r.name, t.name"
+
+      res = @r.query(q)
+
+      expect(res.columns).to eq(["r.name", "t.name"])
+      expect(res.resultset).to eq([["Valentino Rossi", "Yamaha"]])
+    end
+
+    # not in the quickstart, but demonstrates multiple rows
+    it 'should query relations, without a predicate' do
+      q = "MATCH (r:Rider)-[:rides]->(t:Team) RETURN r.name, t.name ORDER BY r.name"
+
+      res = @r.query(q)
+
+      expect(res.columns).to eq(["r.name", "t.name"])
+      expect(res.resultset).to eq([
+        ["Andrea Dovizioso", "Ducati"],
+        ["Dani Pedrosa", "Honda"],
+        ["Valentino Rossi", "Yamaha"]
+      ])
+    end
+
+    it 'should query relations, with an aggregate function' do
+      q = "MATCH (r:Rider)-[:rides]->(t:Team {name:'Ducati'}) RETURN count(r)"
+
+      res = @r.query(q)
+
+      expect(res.columns).to eq(["count(r)"])
+      expect(res.resultset).to eq([[1]])
+    end
+  end
+end