Passing in variables for attribute query

[jamesnyika]

Hi @tonsky
I am just loving this library and I am trying to get something rather generic going

Challenge

Call this function and pass in a keyword that is one of my attributes. in this case :event/name

(defn eventsearch [attr_pk ds]
  (d/q '[:find [?e ...]
         :in $
         :where
          [?e :event/name] ;; THIS WORKS - HARDCODED
          ;;[?e attr_pk]  ;; NOTE THIS DOES NOT WORK. DYNAMIC RESOLUTION FAILS
          [?e :event/active true]] ds))

Exception

Datascript says it expected a pattern .. etc.

I would expect this would work but for some reason it does not seem to. I am on v0.18.4

Thanks for your help

[claj]

Since your query is quoted ('[:find ...]) you cannot resolve attr_pk this way.

Instead use query variables:

(defn eventsearch [attr_pk ds]
  (d/q '[:find [?e ...]
         :in $ ?attr_pk ;; << additional query argument
         :where
          [?e ?attr_pk] ;; << note ?attr_pk
          [?e :event/active true]] ds attr_pk)) ;;<< attr_pk added as extra query arg

The reason for this is, at least in datomic, is that the queries are compiled and that compilation is cached somehow. The inlining in the queries would make that caching harder.

[jamesnyika]

@claj @tonsky
Thank you for this. It works perfectly. But I suspect that my problem was not actually the query but rather it is probably that I am having a problem with :advanced compilation. This query, when executed in a repl or in dev works. But if you pass in a keyword to attr_pk it breaks under :advanced. By breaks, I do not get any exceptions, I just get no results back because the query fails to get any results. If I hard code the actual value of attr_pk then it just works!!! . It is ridiculous!

Again, I think this is what is happening because when I searched for the keyword I pass, it is nowhere to be found in the index.js that is created. The keyword I send in is :event/name and I suspect it is being changed by :advanced compilation. What do you guys think ?

How do I protect the attr_pk from being munged ? Is there a way to store keywords in extern files ?

Or there could be something else going on that I am not aware of.

[tonsky]

No, keywords are not munged. Probably something else is going on. Why don’t you describe in full what are you doing, what are you expecting and what are you getting out of it?

[jamesnyika]

@tonsky @claj Here is the description. And thank you in advance for taking your precious time to look at this

Terminology

By PROD here I just mean that I am compiling with shadow-cljs, to get a JS file that is part of my react native application. It uses advanced compilation of course and that is the only thing I could think of that would be different when I run it on the device vs. in the simulator

Problem

I am using re-frame and would like to use a subscription to generically pull data from datascript. But rather than write a different subscription for every single type of entity, I want to create 2-3 generic ones Right now I am working to load data from datascript for a dashboard. Once i get this working generically, I can load lots of other data using this single subscription.

;; loads the dashboard for this app
  (reg-sub
   :data/dashboard
 
   (fn [db _]
       (let [ds @(:re-conn db)  ;; get the datasource
 
            ;; pull the relevant entity IDS by running a query from this map with attribute passed in, and 
            ;; datasource as well. 
            ;; I get a function name from a map, then execute it passing in an attribute key and the 
            ;; datasource
             eids ((first (:display/dashboard dq/querytree)) :event/name ds)

             ;; then run a pull many function to pull the attributes for those entity ids. 
             events ((second (:display/dashboard dq/querytree)) eids ds)
            
            ;; other processing that I do to prepare the data for viewing... not relevant to the problem.
             enrichedevents (sp/transform [sp/ALL] #(u/addEventGroupingKey :grpkey :event/startdate "yyyy-MM-dd" %) events)
             groupedevents (dq/group :grpkey enrichedevents)
            ]
          
            ;; return the groupedevents
            groupedevents
       )
     ))

The functions executed to pull the Entity IDs (eids) and the events are here below (dq/querytree)
The idea is to use the query tree as a way of defining pipelines of queries to run to pull all the data for a given screen (eg. dashboard). Each element of the pipeline is just a function call. But since the pattern used is similar (query for entity ids then do a pull-many) I wanted to make that generic so that all I need to change is the attribute key that I pass to the function call.

(ns datum.data.query
  (:require [datascript.core :as d]
       [datum.util :as u]))

;; I hope to make this function generic and not specific to events
(defn eventsearch [attribute ds]

  ;; this does not work in prod. But works in DEV mode
  ;(d/q '[:find [?entity ...]
  ;     :in $ [[?attr [[?aprop ?avalue] ...]] ...] ?attribute
  ;     :where [(= ?attr ?attribute)]
  ;            [?entity ?attr ?value]]
  ;   ds (:schema ds) attribute))

  ;; similar to the one above, does NOT work in prod but work in DEV mode.
  (d/q '[:find [?e ...]
         :in $ ?attribute
         :where
         ; [?e :event/name] ;; IF I USE THIS, IT WORKS. BUT THIS IS HARD CODED. NOT DESIRED
          [?e ?attribute]  ;; I WANT TO DO THIS BUT IT DOES NOT WORK IN PROD, ONLY DEV
          [?e :event/active true]] ds attribute))

;; generic pull
(defn entities-pullone [id ds] (d/pull ds '[*] id))
(defn entities-pullmany [ids ds]  (d/pull-many ds '[*] ids))

 ;; here is the querytree referred to in the subscription above. 
(def querytree
    { ;; maps a command to a query
      ;===========
      ; Authentication Related
      ;; NOTE YOU MUST FULLY QUALIFY THESE FUNCTION NAMES> OTHERWISE RESOLUTION WILL FAIL
      :display/dashboard [datum.data.query/eventsearch datum.data.query/entities-pullmany]
      :display/# [datum.data.query/entities-pullone]

    }
  )

## Results
In DEV, the queries work flawlessly. I get the EIDS and the pull many works perfectly. But in production, the failure point is the datum.data.query/eventsearch  call. I can confirm that the call is indeed executed but for some reason, the results back are []. Hence the following call to pull many returns nothing and there are no major errors. In prod (ie. on the mobile device) I cannot get the query to run if I do not hard code ````  ; [?e :event/name] ```` in the eventsearch query. Using ```` 
 [?e ?attribute] ```` does not seem to work and I cannot figure out why but it is the desired behaviour because it would allow me to pass in whatever attribute keyword I want for ?attribute 

 

Environment

React Native, Expo and Re-frame mobile application.
Database is Datascript

[tonsky]

Just a wild guess: take a look at #298 (comment)

[jamesnyika]

Confirming as well that this query

(defn eventsearch [attribute ds]
(d/q '[:find [?e ...]
         :in $ ?attribute
         :where
          ;[?e :event/name] ;; I USED TO HAVE TO DO THIS
          [?e ?attribute]  ;; BUT NOW I CAN INSTEAD DO THIS
          [?e :event/active true]] ds attribute))
)

WORKS just by adding this to my shadow-cljs.edn file

...
 :compiler-options {:infer-externs :auto
                      :externs ["datascript/externs.js"]} ;; <--- just add this in

I got the answer as @tonsky rightly pointed out in issue #298
Thank you @tonsky @claj

I kindly request you or I add this to your wiki or docs for those using datascript with shadow-cljs as @Lokeh had suggested.

[jamesnyika]

I added a section under the wiki Tips & Tricks section.