Non-token aware queries graph (and gauge) is broken.

[michoecho]

Panel Name: Non-Token Aware Queries
Dashboard Name: CQL
Scylla-Monitoring Version: master
Scylla-Version: master

There are two problems with this graph:

1. The combination of Remove counters and LWT from non-token aware #1304 and set_skip_when_empty is broken.
   With set_skip_when_empty, the used scylla_storage_proxy_replica_received_counter_updates metric isn't equal 0, but doesn't exist at all. So the entire formula evaluates to nothing and the entire graph is empty, even if non-token aware queries are actually present.
2. The formula (scylla_cql_reads - scylla_storage_proxy_coordinator_reads_local_node) is wrong. A coordinator might choose not to serve a replica read even if it belongs to the replica set. (This happens due to heat weighted load balancing — even if the query is token-aware, the coordinator prefers other replicas while its cache is still cold).

But Scylla has metrics which track non-token-aware queries directly. See scylladb/scylladb#4338.

So instead, consider using this for the graph:

rate(scylla_storage_proxy_coordinator_reads_coordinator_outside_replica_set[...]) + rate(scylla_storage_proxy_coordinator_writes_coordinator_outside_replica_set[...])

(with whatever change is needed to make it work even if one of these doesn't exist due to set_skip_when_empty).