In an attempt to create a more singular description of our data, we will move
the specs into the Datascript schema. This won't actually work, but it'll
define a goal:
(def schema
{:person/id {:db/unique :db.unique/identity
:schema/spec (s/conformer uuid)}
:person/name {:schema/spec string?}
:person/age {:schema/spec ::number}
:person/entity {:schema/spec (s/keys :req [:person/id :person/name :person/age])}
:movie/id {:db/unique :db.unique/identity
:schema/spec (s/conformer uuid)}
:movie/title {:schema/spec string?}
:movie/description {:schema/spec string?}
:movie/release-date {:schema/spec ::number}
:movie/people {:db/valueType :db.type/ref
:db/cardinality :db.cardinality/many
:schema/spec (s/coll-of :person/entity)}
:movie/entity {:schema/spec (s/keys :req [:movie/id :movie/title :movie/description
:movie/release-date :movie/people])}})
schema is a custom namespace that I just introduced. We'll make use of it
shortly. Because the schema now has extraneous junk in it, Datascript will no
longer eat it raw. We'll need a function that turns this back into a pure
Datascript schema. Since we'll need a supporting function anyway, let's see if
we can make some more improvements while we're at it.
When we reviewed our original specs, we concluded that they described everything
about our data except for uniqueness constraints. It just so happens that spec
has APIs to work with defined specs as data, allowing us to extract e.g. keys
from a (s/keys) spec and more. That means we no longer need :db/valueType or
:db/cardinality - the former can be induced from (s/keys) specs (those will
be references) and the latter from (s/coll-of) specs (collection -
:db.cardinality/many).
This leaves us with this leaner representation:
(def schema
{:person/id {:db/unique :db.unique/identity
:schema/spec (s/conformer uuid)}
:person/name {:schema/spec string?}
:person/age {:schema/spec ::number}
:person/entity {:schema/spec (s/keys :req [:person/id :person/name :person/age])}
:movie/id {:db/unique :db.unique/identity
:schema/spec (s/conformer uuid)}
:movie/title {:schema/spec string?}
:movie/description {:schema/spec string?}
:movie/release-date {:schema/spec ::number}
:movie/people {:schema/spec (s/coll-of :person/entity)}
:movie/entity {:schema/spec (s/keys :req [:movie/id :movie/title :movie/description
:movie/release-date :movie/people])}})
Before we can extract the Datascript schema, we'll need to define the specs so
we can mine them for data. cljs.spec.alpha/def is a macro, and in order to
call it correctly on behalf of the schema definition, we need a macro to define
the schema as well.
The macro goes into unified-schema/macros.cljc:
(ns unified-schema.macros
#?(:cljs (:require [cljs.spec.alpha])))
(defmacro defschema [name schema]
(apply list 'do (concat
(for [[attr attr-def] schema]
`(cljs.spec.alpha/def ~attr ~(:schema/spec attr-def)))
`[(def ~name ~schema)])))
...and can be used like so:
(ns unified-schema.example
(:require [unified-schema.macros :refer-macros [defschema]])
(defschema example-schema
{:person/id {:db/unique :db.unique/identity
:schema/spec (s/conformer uuid)}
:person/name {:schema/spec string?}
:person/age {:schema/spec ::number}
:person/entity {:schema/spec (s/keys :req [:person/id :person/name :person/age])}
:movie/id {:db/unique :db.unique/identity
:schema/spec (s/conformer uuid)}
:movie/title {:schema/spec string?}
:movie/description {:schema/spec string?}
:movie/release-date {:schema/spec ::number}
:movie/people {:schema/spec (s/coll-of :person/entity)}
:movie/entity {:schema/spec (s/keys :req [:movie/id :movie/title :movie/description
:movie/release-date :movie/people])}})
Now the specs will be defined, and example-schema will refer to our schema
data.
To extract the schema, we'll start by simply preserving all the keys in the db
namespace:
(defn select-namespaced-keys [m ns]
(->> (keys m)
(filter #(= (namespace %) ns))
(select-keys m)))
(defn extract-schema [attributes]
(->> attributes
(map (fn [[k v]] [k (select-namespaced-keys v "db"]))
(into {})))
We will now add :db.cardinality/many to any attribute that has a s/coll-of
spec, and :db.type/ref to any attribute that has a s/keys spec. You can
inspect the underlying data structure of a spec with s/form:
(s/form :person/entity)
To devise a generalized solution, we'd also like to support this spec:
(s/def :person/entity
(s/and (s/keys :req [:user/name])
(s/or :email (s/keys :req [:user/email])
:phone (s/keys :req [:user/tel]))))
(s/form :person/entity)
To work with this data, I wrote two helper functions (see all the way to the
bottom if you're interested):
coll-of, which returns the first s/coll-of specspecced-keys, which returns a set of all keys, required and optional, for a
map spec (#{:user/name :user/email :user/tel} in the above example).
Using these two functions, we can add the cardinality and ref types to the
relevant attributes:
(defn- schema-attrs [attributes attr-key attr-def]
(let [coll-type (coll-of attr-key)]
(-> (select-namespaced-keys attributes "db")
(assoc-non-nil :db/cardinality (when coll-type :db.cardinality/many))
(assoc-non-nil :db/valueType (when (or (seq (specced-keys attr-key))
(seq (specced-keys coll-type)))
:db.type/ref)))))
(defn extract-schema [attributes]
(->> attributes
(map (fn [[k v]] [k (schema-attrs attributes k v]))
(into {})))
Finally, we'll evict all attribute definitions that don't have any descriptors
in the db namespace:
(defn extract-schema [attributes]
(let [schema (->> attributes
(map (fn [[k v]] [k (schema-attrs attributes k v)]))
(into {}))]
(->> (keys attributes)
(filter #(not (seq (select-namespaced-keys (% attributes) "db"))))
(apply dissoc schema))))
We've unified the Datascript schema and specs. What about the mapping from API
data to schema data? It would be neat if we could achieve that declaratively as
well. In this particular case the mapping was quite straight forward: attributes
have different names, and we want to pass values through our specs. The
declarative bit of the solution could look like this:
(defschema example-schema
{:person/id {:db/unique :db.unique/identity
:schema/spec (s/conformer uuid)
:schema/source :id}
:person/name {:schema/spec string? :schema/source :name}
:person/age {:schema/spec ::number :schema/source :age}
:person/entity {:schema/spec (s/keys :req [:person/id :person/name :person/age])}
:movie/id {:db/unique :db.unique/identity
:schema/spec (s/conformer uuid)
:schema/source :id}
:movie/title {:schema/spec string? :schema/source :title}
:movie/description {:schema/spec string? :schema/source :description}
:movie/release-date {:schema/spec ::number :schema/source :release_date}
:movie/people {:schema/spec (s/coll-of :person/entity) :schema/source :people}
:movie/entity {:schema/spec (s/keys :req [:movie/id :movie/title :movie/description
:movie/release-date :movie/people])}})
Frequently, your schema will contain namespaced versions of API data keys (e.g.
:person/name vs "name"). Because this particular mapping is so common, we'll
just bolt it into the converter, and can leave them out of our schema.
The implementation of convert-data looks like this:
(defn convert-data [attributes api-data key]
(let [{:keys [schema/source]} (attributes key)
data (if source
(get api-data source) (get api-data key (get api-data (keyword (name key)) api-data))) collection-type (coll-of key)
keys (specced-keys key)]
(cond
(seq keys) (->> keys
(map (fn [k] [k (convert-data attributes api-data k)]))
(into {}))
collection-type (map #(convert-data attributes % collection-type) api-data)
:default (if (s/valid? key api-data)
(s/conform key api-data)
(s/assert key api-data)))))
There is one more feature that could be useful in this utility: the ability to
provide a custom mapper for a value. We do already have this, through the key
specs. However, sometimes we need to break down data in ways that are unsuitable
for conformers to handle, such as converting "SomeLabel_23" into
{:thing/label "SomeLabel" :thing/id 23} before continuing processing. This
could be fixed with a :schema/mapper function that receives the current data
and produces the data to process.
