ajet/df-research
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

init research

Browse Source
This commit is contained in:
Adam Jeniski
2026-02-08 11:20:43 -10:00
commit bdf064f54d
3041 changed files with 1592200 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
malli/docs/cljs-instrument-development.md
Vendored
+19
View File
@@ -0,0 +1,19 @@
To work on the function instrumentation for ClojureScript clone the malli repository and:
```bash
npm i
./node_modules/.bin/shadow-cljs watch instrument
```
Open an nREPL connection from your favorite editor to the port located in `.shadow-cljs/nrepl.port`
Open a browser to `http://localhost:8000`
the port is set in the `shadow-cljs.edn` file should you wish to change it.
In your editor evaluate:
`(shadow/repl :instrument)`
The dev-time code is located in the file: `app/malli/instrument_app.cljs`.
malli/docs/clojurescript-function-instrumentation.md
Vendored
+122
View File
@@ -0,0 +1,122 @@
# ClojureScript Function Instrumentation
Function instrumentation is also supported when developing ClojureScript browser applications.
The implementation works by collecting function schemas using a ClojureScript macro which registers the function schemas
available in the application.
Instrumentation happens at runtime in a JavaScript runtime. The functions to instrument are replaced by instrumented versions.
# Dev Setup
For the best developer experience make sure you install the latest version of binaryage/devtools and use a Chromium or Firefox based browser:
https://clojars.org/binaryage/devtools
if you are using shadow-cljs just ensure this library is on the classpath.
For an application that uses React.js such as Reagent you will typically declare an entry namespace and init function in your `shadow-cljs.edn` config like so:
```clojure
{...
:modules {:app {:entries [your-app.entry-ns]
:init-fn your-app.entry-ns/init}}
...}
```
All development-time code should ideally live in a [preload](https://shadow-cljs.github.io/docs/UsersGuide.html#_preloads) namespace.
Add a preload namespace to your codebase to enable the malli instrumentation:
```clojure
(ns com.myapp.dev-preload
{:dev/always true}
(:require
your-app.entry-ns ; <---- make sure you include your entry namespace
[malli.dev.cljs :as dev]))
(dev/start!)
```
By including your entry namespace in the `:require` form the compiler will ensure that the preload namespace is compiled
after your application. This will ensure your schemas are up-to-date when `(malil.dev.cljs/start!)` is evaluated.
We also add `{:dev/always true}` metadata to the namespace so that the compiler will never cache this file.
Add this preload to your shadow-cljs config:
```clojure
{...
:modules {:app {:entries [your-app.entry-ns]
:preloads [com.myapp.dev-preload]
:init-fn your-app.entry-ns/init}}
...}
```
If you want to get clj-kondo static checking from your function schemas add the `malli.dev.cljs-kondo-preload` namespace
to the preloads vector:
```clojure
{...
:modules {:app {:entries [your-app.entry-ns]
:preloads [com.myapp.dev-preload
malli.dev.cljs-kondo-preload ;; <----
]
:init-fn your-app.entry-ns/init}}
...}
```
Now while you develop clj-kondo config will be written under the `.clj-kondo` directory to enable static type checking
on any functions that have schemas.
## Errors in the browser console
When you get a schema validation error and instrumentation is on you will see an exception in the browser devtools.
A validation error looks like this:
<img src="img/cljs-instrument/cljs-instrument-error-collapsed.png"/>
If you click the arrow that is highlighted in the above image you will see the error message:
<img src="img/cljs-instrument/cljs-instrument-error-expanded.png"/>
and if you click the arrow highlighted in this above image you will see the stracktrace:
<img src="img/cljs-instrument/cljs-instrument-stacktrace-expanded.png"/>
the instrumented function is the one with the red rectangle around it in the image above.
If you click the filename (`instrument_app.cljs` in this example) the browser devtools will open a file viewer at the problematic call-site.
# Release builds
If you follow the strategy outlined above using a `preload` to include all development-time tooling then you don't
have to make any changes to prevent that code from ending up in a release build.
One other technique you may want to use to optimize even further is to have a separate malli registry for development
and one for your release build. The dev build may include schemas that are only used for instrumenting functions or for
attaching generators to schemas which can increase bundle size significantly.
You can use a configuration like so using the `:ns-aliases` feature of shadow-cljs to switch malli registry namespaces
without needing to update your codebase:
```clojure
{:target :browser
:output-dir "resources/public/js/main"
:asset-path "js/main"
:dev {:modules {:main {:init-fn com.my-org.client.dev-entry/init}}}
:release {:modules {:main {:init-fn com.my-org.client.release-entry/init}}
:build-options {:ns-aliases
{com.my-org.client.malli-registry com.my-org.client.malli-registry-release}}}
:closure-defines {malli.registry/type "custom"}
,,,
```
This example also demonstrates how if you really need to, you can also have a completely separate entry namespace for a release.
It should be noted also that metadata declared on function vars in ClojureScript is excluded by default by the ClojureScript
compiler. The function var metadata is only included in a build if you explicitly invoke a static call to it such as:
```clojure
(meta (var com.my-org.some.ns/a-fn))
```
Unless you explicitly ask for it, function var metadata will not be included in your compiled JS, so annotating functions
with malli schemas in metadata adds no cost on your builds.
malli/docs/function-schemas.md
Vendored
+885
View File
@@ -0,0 +1,885 @@
# Function Schemas
* [Functions](#functions)
* [Predicate Schemas](#predicate-schemas)
* [Function Schemas](#function-schemas)
* [Generative Testing](#generative-testing)
* [Function Guards](#function-guards)
* [Generating Functions](#generating-functions)
* [Multi-arity Functions](#multi-arity-functions)
* [Instrumentation](#instrumentation)
* [Flat Arrow Function Schemas](#flat-arrow-function-schemas)
* [Defn Schemas](#defn-schemas)
* [Defining Function Schemas](#defining-function-schemas)
* [Function Schema Annotations](#function-schema-annotations)
* [Function Schema Metadata](#function-schema-metadata)
* [Function Inline Schemas](#function-inline-schemas)
* [Defn Instrumentation](#defn-instrumentation)
* [Defn Checking](#defn-checking)
* [Development Instrumentation](#development-instrumentation)
* [ClojureScript Support](#clojurescript-support)
* [Static Type Checking](#static-type-checking)
* [Pretty Errors](#pretty-errors)
* [Defn Schemas via metadata](#defn-schemas-via-metadata)
* [TL;DR](#tldr)
## Functions
In Clojure, functions are first-class. Here's a simple function:
```clojure
(defn plus [x y]
(+ x y))
(plus 1 2)
;; => 3
```
## Predicate Schemas
Simplest way to describe function values with malli is to use predefined predicate schemas `fn?` and `ifn?`:
```clojure
(require '[malli.core :as m])
(m/validate fn? plus)
;; => true
(m/validate ifn? plus)
;; => true
```
Note that `ifn?` also accepts many data-structures that can be used as functions:
```clojure
(m/validate ifn? :kikka)
;; => true
(m/validate ifn? {})
;; => true
```
But, neither of the predefined function predicate schemas can validate function arity, function arguments or return values. As it stands, [there is no robust way to programmatically check function arity at runtime](https://stackoverflow.com/questions/1696693/clojure-how-to-find-out-the-arity-of-function-at-runtime).
Enter, function schemas.
## Function Schemas
Function values can be described with `:=>` and `:function` schemas. They allows description of both function arguments (as [sequence schemas](https://github.com/metosin/malli#sequence-schemas)) and function return values.
Examples of function definitions:
```clojure
;; no args, no return
[:=> :cat :nil]
;; int -> int
[:=> [:cat :int] :int]
;; x:int, xs:int* -> int
[:=> [:catn
[:x :int]
[:xs [:+ :int]]] :int]
;; arg:int -> ret:int, arg > ret
(defn guard [[arg] ret]
(> arg ret))
[:=> [:cat :int] :int [:fn guard]]
;; multi-arity function
[:function
[:=> [:cat :int] :int]
[:=> [:cat :int :int [:* :int]] :int]]
```
What is that `:cat` all about in the input schemas? Wouldn't it be simpler without it? Sure, check out [Flat Arrow Function Schema](#flat-arrow-function-schemas).
Function definition for the `plus` looks like this:
```clojure
(def =>plus [:=> [:cat :int :int] :int])
```
Let's try:
```clojure
(m/validate =>plus plus)
;; => true
```
But, wait, as there was no way to know the function arity & other information at runtime, so how did the validation work? Actually, it didn't. By default. `:=>` validation just checks that it's a `fn?`, so this holds too:
```clojure
(m/validate =>plus str)
;; => true
```
Bummer.
Enter, generative testing.
### Generative Testing
Like [clojure.spec](https://clojure.org/about/spec) demonstrated, we can use [test.check](https://github.com/clojure/test.check) to check the functions at runtime. For this, there is `:malli.core/function-checker` option.
```clojure
(require '[malli.generator :as mg])
(def =>plus
(m/schema
[:=> [:cat :int :int] :int]
{::m/function-checker mg/function-checker}))
(m/validate =>plus plus)
;; => true
(m/validate =>plus str)
;; => false
```
Explanation why it is not valid:
```clojure
(m/explain =>plus str)
;{:schema [:=> [:cat :int :int] :int],
; :value #object[clojure.core$str],
; :errors ({:path [],
; :in [],
; :schema [:=> [:cat :int :int] :int],
; :value #object[clojure.core$str],
; :check {:total-nodes-visited 0,
; :depth 0,
; :pass? false,
; :result false,
; :result-data nil,
; :time-shrinking-ms 1,
; :smallest [(0 0)],
; :malli.generator/explain-output {:schema :int,
; :value "00",
; :errors ({:path []
; :in []
; :schema :int
; :value "00"})}}})}
```
Smallest failing invocation is `(str 0 0)`, which returns `"00"`, which is not an `:int`. Looks good.
But, why `mg/function-checker` is not enabled by default? The reason is that it uses generative testing, which is orders of magnitude slower than normal validation and requires an extra dependency to `test.check`, which would make `malli.core` much heavier. This would be especially bad for CLJS bundle size.
### Function Guards
`:=>` accepts optional third child, a guard schema that is used to validate a vector of function arguments and return value.
```clojure
;; function schema of arg:int -> ret:int, where arg < ret
;; with generative function checking always enabled
(def arg<ret
(m/schema
[:=>
[:cat :int]
:int
[:fn {:error/message "argument should be less than return"}
(fn [[[arg] ret]] (< arg ret))]]
{::m/function-checker mg/function-checker}))
(m/explain arg<ret (fn [x] (inc x)))
;; => nil
(m/explain arg<ret (fn [x] x))
;{:schema ...
; :value #object[user$eval19073$fn__19074],
; :errors ({:path [],
; :in [],
; :schema ...,
; :value #object[user$eval19073$fn__19074],
; :check {:total-nodes-visited 1,
; :result false,
; :result-data nil,
; :smallest [(0)],
; :time-shrinking-ms 0,
; :pass? false,
; :depth 0,
; :malli.core/result 0}},
; {:path [2],
; :in [],
; :schema [:fn
; #:error{:message "argument should be less than return"}
; (fn [[[arg] ret]] (< arg ret))],
; :value [(0) 0]})}
(require '[malli.error :as me])
(me/humanize *1)
; ["invalid function" "argument should be less than return"]
```
Identical schema using the Schema AST syntax:
```clojure
(m/from-ast
{:type :=>
:input {:type :cat
:children [{:type :int}]}
:output {:type :int}
:guard {:type :fn
:value (fn [[[arg] ret]] (< arg ret))
:properties {:error/message "argument should be less than return"}}}
{::m/function-checker mg/function-checker})
```
### Generating Functions
We can also generate function implementations based on the function schemas. The generated functions check the function arity and arguments at runtime and return generated values.
<!-- :test-doc-blocks/skip -->
```clojure
(def plus-gen (mg/generate =>plus))
(plus-gen 1)
; =throws=> :malli.core/invalid-arity {:arity 1, :arities #{{:min 2, :max 2}}, :args [1], :input [:cat :int :int], :schema [:=> [:cat :int :int] :int]}
(plus-gen 1 "2")
; =throws=> :malli.core/invalid-input {:input [:cat :int :int], :args [1 "2"], :schema [:=> [:cat :int :int] :int]}
(plus-gen 1 2)
; => -1
```
### Multi-arity Functions
Multi-arity functions can be composed with `:function`:
<!-- :test-doc-blocks/skip -->
```clojure
;; multi-arity fn with function checking always on
(def =>my-fn
(m/schema
[:function {:registry {::small-int [:int {:min -100, :max 100}]}}
[:=> [:cat ::small-int] :int]
[:=> [:cat ::small-int ::small-int [:* ::small-int]] :int]]
{::m/function-checker mg/function-checker}))
(m/validate
=>my-fn
(fn
([x] x)
([x y & z] (apply - (- x y) z))))
; => true
(m/validate
=>my-fn
(fn
([x] x)
([x y & z] (str x y z))))
; => false
(m/explain
=>my-fn
(fn
([x] x)
([x y & z] (str x y z))))
;{:schema [:function
; {:registry {::small-int [:int {:min -100, :max 100}]}}
; [:=> [:cat ::small-int] :int]
; [:=> [:cat ::small-int ::small-int [:* ::small-int]] :int]],
; :value #object[malli.core_test$eval27255$fn__27256],
; :errors ({:path [],
; :in [],
; :schema [:function
; {:registry {::small-int [:int {:min -100, :max 100}]}}
; [:=> [:cat ::small-int] :int]
; [:=> [:cat ::small-int ::small-int [:* ::small-int]] :int]],
; :value #object[malli.core_test$eval27255$fn__27256],
; :check ({:total-nodes-visited 2,
; :depth 1,
; :pass? false,
; :result false,
; :result-data nil,
; :time-shrinking-ms 0,
; :smallest [(0 0)],
; :malli.generator/explain-output {:schema :int,
; :value "00",
; :errors ({:path []
; :in []
; :schema :int
; :value "00"})}})})}
```
Generating multi-arity functions:
<!-- :test-doc-blocks/skip -->
```clojure
(def my-fn-gen (mg/generate =>my-fn))
(my-fn-gen)
; =throws=> :malli.core/invalid-arity {:arity 0, :arities #{1 :varargs}, :args nil, :input nil, :schema [:function {:registry {::small-int [:int {:min -100, :max 100}]}} [:=> [:cat ::small-int] :int] [:=> [:cat ::small-int ::small-int [:* ::small-int]] :int]]}
(my-fn-gen 1)
; => -3237
(my-fn-gen 1 2)
; => --543
(my-fn-gen 1 2 3 4)
; => -2326
```
### Instrumentation
Besides testing function schemas as values, we can also instrument functions to enable runtime validation of arguments and return values.
Simplest way to do this is to use `m/-instrument` which takes an options map and a function and returns an instrumented function. Valid options include:
| key | description |
| ----------|-------------|
| `:schema` | function schema
| `:scope` | optional set of scope definitions, defaults to `#{:input :output}`
| `:report` | optional side-effecting function of `key data -> any` to report problems, defaults to `m/-fail!`
| `:gen` | optional function of `schema -> schema -> value` to be invoked on the args to get the return value
Instrumenting a function with input & return constraints:
<!-- :test-doc-blocks/skip -->
```clojure
(def pow
(m/-instrument
{:schema [:=> [:cat :int] [:int {:max 6}]]}
(fn [x] (* x x))))
(pow 2)
; => 4
(pow "2")
; =throws=> :malli.core/invalid-input {:input [:cat :int], :args ["2"], :schema [:=> [:cat :int] [:int {:max 6}]]}
(pow 4)
; =throws=> :malli.core/invalid-output {:output [:int {:max 6}], :value 16, :args [4], :schema [:=> [:cat :int] [:int {:max 6}]]}
(pow 4 2)
; =throws=> :malli.core/invalid-arity {:arity 2, :arities #{{:min 1, :max 1}}, :args [4 2], :input [:cat :int], :schema [:=> [:cat :int] [:int {:max 6}]]}
```
Example of a multi-arity function with instrumentation scopes and custom reporting function:
```clojure
(def multi-arity-pow
(m/-instrument
{:schema [:function
[:=> [:cat :int] [:int {:max 6}]]
[:=> [:cat :int :int] [:int {:max 6}]]]
:scope #{:input :output}
:report println}
(fn
([x] (* x x))
([x y] (* x y)))))
(multi-arity-pow 4)
;; =stdout=> :malli.core/invalid-output {:output [:int {:max 6}], :value 16, :args [4], :schema [:=> [:cat :int] [:int {:max 6}]]}
;; => 16
(multi-arity-pow 5 0.1)
;; =stdout=> :malli.core/invalid-input {:input [:cat :int :int], :args [5 0.1], :schema [:=> [:cat :int :int] [:int {:max 6}]]}
;; :malli.core/invalid-output {:output [:int {:max 6}], :value 0.5, :args [5 0.1], :schema [:=> [:cat :int :int] [:int {:max 6}]]}
;; => 0.5
```
With `:gen` we can omit the function body. Here's an example to generate random values based on the return schema:
<!-- :test-doc-blocks/skip -->
```clojure
(def pow-gen
(m/-instrument
{:schema [:function
[:=> [:cat :int] [:int {:max 6}]]
[:=> [:cat :int :int] [:int {:max 6}]]]
:gen mg/generate}))
(pow-gen 10)
; => -253
(pow-gen 10 20)
; => -159
(pow-gen 10 20 30)
; =throws=> :malli.core/invalid-arity {:arity 3, :arities #{1 2}, :args (10 20 30), :input nil, :schema [:function [:=> [:cat :int] [:int {:max 6}]] [:=> [:cat :int :int] [:int {:max 6}]]]}
```
### Flat Arrow Function Schemas
Function schema `:=>` requires input arguments to be wrapped in `:cat` or `:catn`. Since `0.16.2` there is also flat arrow schema: `:->` that allows input schema to be defined as flat sequence:
```clojure
;; no args, no return
[:-> :nil]
;; int -> int
[:-> :int :int]
;; arg:int -> ret:int, arg > ret
(defn guard [[arg] ret]
(> arg ret))
[:-> {:guard guard} :int :int]
;; multi-arity function
[:function
[:-> :int :int]
[:-> :int :int [:* :int] :int]]
```
Technically `:->` is implemented as a proxy to `:=>`. To get the actual schema:
```clojure
(m/deref [:-> :int :int])
; [:=> [:cat :int] :int]
```
This can be seen also in explain results:
```clojure
(m/explain
[:-> :int :int]
(fn [x] (str x))
{::m/function-checker mg/function-checker})
;{:schema [:-> :int :int],
; :value #object[...],
; :errors ({:path [:malli.core/in],
; :in [],
; :schema [:=> [:cat :int] :int],
; :value #object[...],
; :check {:total-nodes-visited 0,
; :result false,
; :result-data nil,
; :smallest [(0)],
; :time-shrinking-ms 0,
; :pass? false,
; :depth 0,
; :malli.core/result "0"}}
; {:path [:malli.core/in 1]
; :in [], :schema :int
; :value "0"})}
```
## Defn Schemas
### Defining Function Schemas
There are three ways to add function schemas to function Vars (e.g. `defn`s):
1. Function Schema Annotation with `m/=>`
2. Function Schema Metadata via `:malli/schema`
3. Function Inline Schemas with `mx/defn`
#### Function Schema Annotations
`m/=>` macro takes the Var name and the function schema and stores the var -> schema mappings in a global registry.
```clojure
(def small-int [:int {:max 6}])
(defn plus1 [x] (inc x))
(m/=> plus1 [:=> [:cat :int] small-int])
```
The order doesn't matter, so this also works:
```clojure
(m/=> plus1 [:=> [:cat :int] small-int])
(defn plus1 [x] (inc x))
```
Listing the current accumulation of function (Var) schemas:
```clojure
(m/function-schemas)
;{user {plus1 {:schema [:=> [:cat :int] [:int {:max 6}]]
; :ns user
; :name plus1}}}
```
Without instrumentation turned on, there is no schema enforcement:
```clojure
(plus1 10)
;; => 11
```
Turning instrumentation on:
<!-- :test-doc-blocks/skip -->
```clojure
(require '[malli.instrument :as mi])
(mi/instrument!)
; =stdout=> ..instrumented #'user/plus1
(plus1 10)
; =throws=> :malli.core/invalid-output {:output [:int {:max 6}], :value 11, :args [10], :schema [:=> [:cat :int] [:int {:max 6}]]}
```
Note that vars already containing a primitive JVM function will not be instrumented.
#### Function Schema Metadata
`defn` schemas can be defined with standard Var metadata. It allows `defn` schema documentation and instrumentation without dependencies to malli itself from the functions. It's just data.
```clojure
(defn minus
"a normal clojure function, no dependencies to malli"
{:malli/schema [:=> [:cat :int] small-int]}
[x]
(dec x))
```
To collect instrumentation for the `defn`, we need to call `mi/collect!`. It reads all public vars from a given namespace and registers function schemas from `:malli/schema` metadata.
```clojure
(mi/collect!)
; => #{#'user/minus}
(m/function-schemas)
;{user {plus1 {:schema [:=> [:cat :int] [:int {:max 6}]]
; :ns user
; :name plus1},
; minus {:schema [:=> [:cat :int] [:int {:min 6}]]
; :ns user
; :name minus}}}
```
We'll also have to reinstrument the new var:
<!-- :test-doc-blocks/skip -->
```clojure
(mi/instrument!)
; =stdout=> ..instrumented #'user/plus1
; =stdout=> ..instrumented #'user/minus
(minus 6)
; =throws=> :malli.core/invalid-output {:output [:int {:min 6}], :value 5, :args [6], :schema [:=> [:cat :int] [:int {:min 6}]]}
```
All Var metadata keys with `malli` namespace are used. The list of relevant keys:
| key | description |
| ----------------|-------------|
| `:malli/schema` | function schema
| `:malli/scope` | optional set of scope definitions, defaults to `#{:input :output}`
| `:malli/report` | optional side-effecting function of `key data -> any` to report problems, defaults to `m/-fail!`
| `:malli/gen` | optional value `true` or function of `schema -> schema -> value` to be invoked on the args to get the return value
Setting `:malli/gen` to `true` while function body generation is enabled with `mi/instrument!` allows body to be generated, to return valid generated data.
#### Function Inline Schemas
Malli also supports [Plumatic Schema -style](https://github.com/plumatic/schema#beyond-type-hints) schema hints via `malli.experimental` ns:
```clojure
(require '[malli.experimental :as mx])
(mx/defn times :- :int
"x times y"
[x :- :int, y :- small-int]
(* x y))
```
Function schema is registered automatically:
```clojure
(m/function-schemas)
;{user {plus1 {:schema [:=> [:cat :int] [:int {:max 6}]]
; :ns user
; :name plus1},
; minus {:schema [:=> [:cat :int] [:int {:max 6}]]
; :ns user
; :name minus},
; times {:schema [:=> [:cat :int [:int {:max 6}]] :int]
; :ns user
; :name times}}}
```
... but not instrumented:
```clojure
(times 10 10)
;; => 100
```
You can enable instrumentation with `mi/instrument!`:
<!-- :test-doc-blocks/skip -->
```clojure
(mi/instrument!)
; =stdout=> ..instrumented #'user/plus1
; =stdout=> ..instrumented #'user/minus
; =stdout=> ..instrumented #'user/times
(times 10 10)
; =throws=> :malli.core/invalid-input {:input [:cat :int [:int {:max 6}]], :args [10 10], :schema [:=> [:cat :int [:int {:max 6}]] :int]}
```
... or by using metadata `^:malli/always`:
```clojure
(mx/defn ^:malli/always times :- :int
"x times y"
[x :- :int, y :- small-int]
(* x y))
```
<!-- :test-doc-blocks/skip -->
```clojure
user=> (times 10 5)
50
user=> (times 10 10)
Execution error (ExceptionInfo) at malli.core/-exception (core.cljc:138).
:malli.core/invalid-input
```
### Defn Instrumentation
The function (Var) registry is passive and doesn't do anything by itself. To instrument the Vars based on the registry, there is the `malli.instrument` namespace. Var instrumentation is intended for development time, but can also be used for production builds.
```clojure
(require '[malli.instrument :as mi])
```
Vars can be instrumented with `mi/instrument!` and the instrumentation can be removed with `mi/unstrument!`.
<!-- :test-doc-blocks/skip -->
```clojure
(m/=> power [:=> [:cat :int] [:int {:max 6}]])
(defn power [x] (* x x))
(power 6)
; => 36
;; instrument all registered vars
(mi/instrument!)
(power 6)
; =throws=> :malli.core/invalid-output {:output [:int {:max 6}], :value 36, :args [6], :schema [:=> [:cat :int] [:int {:max 6}]]}
(mi/unstrument!)
(power 6)
; => 36
```
Instrumentation can be configured with the same options as `m/-instrument` and with a set of `:filters` to select which Vars should be instrumented.
<!-- :test-doc-blocks/skip -->
```clojure
(mi/instrument!
{:filters [;; everything from user ns
(mi/-filter-ns 'user)
;; ... and some vars
(mi/-filter-var #{#'power})
;; all other vars with :always-validate meta
(mi/-filter-var #(-> % meta :always-validate))]
;; scope
:scope #{:input :output}
;; just print
:report println})
(power 6)
; =stdout=> :malli.core/invalid-output {:output [:int {:max 6}], :value 36, :args [6], :schema [:=> [:cat :int] [:int {:max 6}]]}
; => 36
```
### Defn Checking
We can also check the defn schemas against their function implementations using `mi/check`. It takes same options as `mi/instrument!`.
Checking all registered schemas:
```clojure
(mi/check)
;{user/plus1 {:schema [:=> [:cat :int] [:int {:max 6}]],
; :value #object[user$plus1],
; :errors ({:path [],
; :in [],
; :schema [:=> [:cat :int] [:int {:max 6}]],
; :value #object[user$plus1],
; :check {:total-nodes-visited 12,
; :depth 4,
; :pass? false,
; :result false,
; :result-data nil,
; :time-shrinking-ms 0,
; :smallest [(6)],
; :malli.generator/explain-output {:schema [:int {:max 6}],
; :value 7,
; :errors ({:path [],
; :in [],
; :schema [:int {:max 6}],
; :value 7})}}})}}
```
It reports that the `plus1` is not correct. It accepts `:int` but promises to return `[:int {:max 6}]`. Let's fix the contract by constraining the input values.
<!-- :test-doc-blocks/skip -->
```clojure
(m/=> plus1 [:=> [:cat [:int {:max 5}]] [:int {:max 6}]])
(mg/check)
; => nil
```
All good! But, it's still wrong as the actual implementation allows invalid inputs resulting in invalid outputs (e.g. `6` -> `7`). We could enable instrumentation for the function to fail on invalid inputs at runtime - or write similar range checks ourselves into the function body.
A pragmatically correct schema for `plus1` would be `[:=> [:cat :int] [:int]]`. It also checks, but would fail on `Long/MAX_VALUE` as input. Fully correct schema would be `[:=> [:cat [:int {:max (dec Long/MAX_VALUE)}] [:int]]]`. Generative testing is best effort, not a silver bullet.
We redefined `plus1` function schema and the instrumentation is now out of sync. We have to call `mi/instrument!` to re-instrument it correctly.
<!-- :test-doc-blocks/skip -->
```clojure
;; the old schema & old error
(plus1 6)
; =throws=> :malli.core/invalid-output {:output [:int {:max 6}], :value 9, :args [8], :schema [:=> [:cat :int] [:int {:max 6}]]}
(mi/instrument!)
;; the new schema & new error
(plus1 6)
; =throws=> :malli.core/invalid-input {:input [:cat [:int {:max 5}]], :args [6], :schema [:=> [:cat [:int {:max 5}]] [:int {:max 6}]]}
```
This is not good developer experience.
We can do much better.
## Development Instrumentation
For better DX, there is `malli.dev` namespace.
```clojure
(require '[malli.dev :as dev])
```
It's main entry points is `dev/start!`, taking same options as `mi/instrument!`. It runs `mi/instrument!` and `mi/collect!` (for all loaded namespaces) once and starts watching the function registry for changes. Any change that matches the filters will cause automatic re-instrumentation for the functions. `dev/stop!` removes all instrumentation and stops watching the registry.
<!-- :test-doc-blocks/skip -->
```clojure
(defn plus1 [x] (inc x))
(m/=> plus1 [:=> [:cat :int] [:int {:max 6}]])
(dev/start!)
; malli: instrumented 1 function var
; malli: dev-mode started
(plus1 "6")
; =throws=> :malli.core/invalid-input {:input [:cat :int], :args ["6"], :schema [:=> [:cat :int] [:int {:max 6}]]}
(plus1 6)
; =throws=> :malli.core/invalid-output {:output [:int {:max 6}], :value 9, :args [8], :schema [:=> [:cat :int] [:int {:max 6}]]}
(m/=> plus1 [:=> [:cat :int] :int])
; =stdout=> ..instrumented #'user/plus1
(plus 6)
; => 7
(dev/stop!)
; malli: unstrumented 1 function vars
; malli: dev-mode stopped
```
## ClojureScript support
See the document: [docs/clojurescript-function-instrumentation.md](clojurescript-function-instrumentation.md)
### Static Type Checking
Running `malli.dev` instrumentation also emits [clj-kondo](https://github.com/metosin/malli#clj-kondo) type configs for all `defn`s, enabling basic static type checking/linting for the instrumented functions.
Here's the above code in [Cursive IDE](https://cursive-ide.com/) with [clj-kondo](https://github.com/clj-kondo/clj-kondo) enabled:
<img src="img/clj-kondo-instrumentation.png">
### Pretty Errors
For prettier runtime error messages, we can swap the default error printer / thrower.
```clojure
(require '[malli.dev.pretty :as pretty])
```
<!-- :test-doc-blocks/skip -->
```clojure
(defn plus1 [x] (inc x))
(m/=> plus1 [:=> [:cat :int] [:int {:max 6}]])
(dev/start! {:report (pretty/reporter)})
(plus1 "2")
; =stdout=>
; -- Schema Error ----------------------------------- malli.demo:13 --
;
; Invalid function arguments:
;
; ["2"]
;
; Input Schema:
;
; [:cat :int]
;
; Errors:
;
; {:in [0],
; :message "should be an integer",
; :path [0],
; :schema :int,
; :type nil,
; :value "2"}
;
; More information:
;
; https://cljdoc.org/d/metosin/malli/LATEST/doc/function-schemas
;
; --------------------------------------------------------------------
; =throws=> Execution error (ClassCastException) at malli.demo/plus1 (demo.cljc:13).
; java.lang.String cannot be cast to java.lang.Number
```
To throw the prettified error instead of just printint it:
```clojure
(dev/start! {:report (pretty/thrower)})
```
Pretty printer uses [fipp](https://github.com/brandonbloom/fipp) under the hood and has lot of configuration options:
```clojure
(dev/start! {:report (pretty/reporter (pretty/-printer {:width 80
:print-length 30
:print-level 2
:print-meta true}))})
```
### TL;DR
Example of annotating function with var meta-data and using `malli.dev` for dev-time function instrumentation, pretty runtime exceptions and clj-kondo for static checking:
<!-- :test-doc-blocks/skip -->
```clojure
(ns malli.demo)
(defn plus1
"Adds one to the number"
{:malli/schema [:=> [:cat :int] :int]}
[x] (inc x))
;; instrument, clj-kondo + pretty errors
(require '[malli.dev :as dev])
(require '[malli.dev.pretty :as pretty])
(dev/start! {:report (pretty/reporter)})
(plus1 "123")
(comment
(dev/stop!))
```
Here's the same code in [Cursive IDE](https://cursive-ide.com/) with [clj-kondo](https://github.com/clj-kondo/clj-kondo) enabled:
<img src="img/defn-schema.png"/>
## Future work
* [support Schema defn syntax](https://github.com/metosin/malli/issues/125)
* better integration with [clj-kondo](https://github.com/clj-kondo/clj-kondo) and [clojure-lsp](https://github.com/clojure-lsp/clojure-lsp) for enhanced DX.
malli/docs/img/bats-in-the-attic.png
Vendored
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 206 KiB

malli/docs/img/clj-kondo-instrumentation.png
Vendored
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 128 KiB

malli/docs/img/clj-kondo.png
Vendored
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 190 KiB

malli/docs/img/cljs-instrument/cljs-instrument-error-collapsed.png
Vendored
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 88 KiB

malli/docs/img/cljs-instrument/cljs-instrument-error-expanded.png
Vendored
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 52 KiB

malli/docs/img/cljs-instrument/cljs-instrument-stacktrace-expanded.png
Vendored
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 78 KiB

malli/docs/img/defn-schema.png
Vendored
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 312 KiB

malli/docs/img/dot.png
Vendored
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 47 KiB

malli/docs/img/malli-defn.png
Vendored
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 376 KiB

malli/docs/img/malli-error.png
Vendored
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 134 KiB

malli/docs/img/malli.png
Vendored
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 257 KiB

malli/docs/img/plantuml.png
Vendored
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 29 KiB

malli/docs/img/pretty-coerce.png
Vendored
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 253 KiB

malli/docs/img/pretty-explain.png
Vendored
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 330 KiB

malli/docs/img/var-deserialization-error.png
Vendored
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 220 KiB

malli/docs/jmh.md
Vendored
+15
View File
@@ -0,0 +1,15 @@
# Benchmarking with JMH
## Build
Requires tools build
```sh
clj -T:build all
```
## Run
```sh
clojure -M:jmh '{:output "jmh-report.edn"}'
```
malli/docs/releasing.md
Vendored
+22
View File
@@ -0,0 +1,22 @@
# Making a Malli release
1. Make sure `CHANGELOG.md` mentions all relevant unreleased changes
2. Recommended: update dependencies `clj -M:outdated --upgrade`
* Gotchas:
* The script tries to upgrade :clojure-11 alias as well. This
should not be done, as it is used to test the compatibility with
clojure 11
* If shadow-cljs is updgraded, remember to update package.json as well
* Make a PR out of this to get the CI to run all the tests!
3. Pick a new version number. Remember: we use [BreakVer](https://www.taoensso.com/break-versioning)
4. Set the version number
* Add a title to `CHANGELOG.md`
* Change the `<version>` and `<tag>` fields in `pom.xml`
5. Push to `master`
6. Create a release via the [GitHub UI](https://github.com/metosin/malli/releases/new)
* Use the version number as the tag name, eg. `0.22.33`
* Copypaste the changelog from `CHANGELOG.md` to the text field
7. Once the release is published, the `release` GitHub Action will build a release and deploy it to Clojars.
* See progress here: https://github.com/metosin/malli/actions/workflows/release.yml
8. Check that the release is listed on clojars: https://clojars.org/metosin/malli
9. Navigate to the cljdoc of the new release to trigger cljdoc build: https://cljdoc.org/versions/metosin/malli
malli/docs/reusable-schemas.md
Vendored
+228
View File
@@ -0,0 +1,228 @@
# Reusable Schemas
Malli currently has two ways for re-using schemas (instances):
1. Schemas as Vars - *the [plumatic](https://github.com/plumatic/schema) way*
2. Schemas via Global Registry - *the [spec](https://clojure.org/about/spec) way*
3. Schemas via Local Registries
## Schemas as Vars
We can define Schemas using `def`:
```clojure
(require '[malli.core :as m])
(def UserId :uuid)
(def Address
[:map
[:street :string]
[:lonlat [:tuple :double :double]]])
(def User
[:map
[:id UserId]
[:name :string]
[:address Address]])
(def user
{:id (random-uuid)
:name "Tiina"
:address {:street "Satakunnunkatu 10"
:lonlat [61.5014816, 23.7678986]}})
(m/validate User user)
;; => true
```
All subschemas as inlined as values:
```clojure
(m/schema User)
;[:map
; [:id :uuid]
; [:name :string]
; [:address [:map
; [:street :string]
; [:lonlat [:tuple :double :double]]]]]
```
## Schemas via Global Registry
To support spec-like mutable registry, we'll define the registry and a helper function to register a schema:
```clojure
(require '[malli.registry :as mr])
(defonce *registry (atom {}))
(defn register! [type ?schema]
(swap! *registry assoc type ?schema))
(mr/set-default-registry!
(mr/composite-registry
(m/default-schemas)
(mr/mutable-registry *registry)))
```
Registering Schemas:
```clojure
(register! ::user-id :uuid)
(register! ::address [:map
[:street :string]
[:lonlat [:tuple :double :double]]])
(register! ::user [:map
[:id ::user-id]
[:name :string]
[:address ::address]])
(m/validate ::user user)
;; => true
```
By default, reference keys are used instead of values:
```clojure
(m/schema ::user)
; :user/user
```
We can recursively deref the Schema to get the values:
```clojure
(m/deref-recursive ::user)
;[:map
; [:id :uuid]
; [:name :string]
; [:address [:map
; [:street :string]
; [:lonlat [:tuple :double :double]]]]]
```
### Decomplect Maps, Keys and Values
Clojure Spec declared [map specs should be of keysets only](https://clojure.org/about/spec#_map_specs_should_be_of_keysets_only). Malli supports this too:
```clojure
;; (╯°□°)╯︵ ┻━┻
(reset! *registry {})
(register! ::street :string)
(register! ::latlon [:tuple :double :double])
(register! ::address [:map ::street ::latlon])
(register! ::id :uuid)
(register! ::name :string)
(register! ::user [:map ::id ::name ::address])
(m/deref-recursive ::user)
;[:map
; [:user/id :uuid]
; [:user/name :string]
; [:user/address [:map
; [:user/street :string]
; [:user/latlon [:tuple :double :double]]]]]
;; data has a different shape now
(m/validate ::user {::id (random-uuid)
::name "Maija"
::address {::street "Kuninkaankatu 13"
::latlon [61.5014816, 23.7678986]}})
;; => true
```
## Schemas via Local Registries
Schemas can be defined as a `ref->?schema` map:
```clojure
(def registry
{::user-id :uuid
::address [:map
[:street :string]
[:lonlat [:tuple :double :double]]]
::user [:map
[:id ::user-id]
[:name :string]
[:address ::address]]})
```
Using registry via Schema properties:
```clojure
(m/schema [:schema {:registry registry} ::user])
; => :user/user
```
Using registry via options:
```clojure
(m/schema ::user {:registry (merge (m/default-schemas) registry)})
```
Works with both:
<!-- :test-doc-blocks/skip -->
```clojure
(m/deref-recursive *1)
;[:map
; [:id :uuid]
; [:name :string]
; [:address [:map
; [:street :string]
; [:lonlat [:tuple :double :double]]]]]
```
# Which one should I use?
Here's a comparison matrix of the two different ways:
| Feature | Vars | Global Registry | Local Registry |
|----------------------------------|:----:|:---------------:|:--------------:|
| Supported by Malli | ✅ | ✅ | ✅ |
| Explicit require of Schemas | ✅ | ❌ | ✅ |
| Support Recursive Schemas | ✅ | ✅ | ✅ |
| Decomplect Maps, Keys and Values | ❌ | ✅ | ✅ |
You should pick the way what works best for your project.
[My](https://gist.github.com/ikitommi) personal preference is the Var-style - it's simple and Plumatic proved it works well even with large codebases.
# Future Work
1. Could we also decomplect the Maps, Keys and Values with the Var Style?
2. Utilities for transforming between inlined and referenced models (why? why not!)
<!-- :test-doc-blocks/skip -->
```clojure
(-flatten-refs
[:schema {:registry {::user-id :uuid
::address [:map
[:street :string]
[:lonlat [:tuple :double :double]]]
::user [:map
[:id ::user-id]
[:name :string]
[:address ::address]]}}
::user])
;[:map {:id :user/user}
; [:id [:uuid {:id :user/user-id}]]
; [:name :string]
; [:address [:map {:id :user/address}
; [:street :string]
; [:lonlat [:tuple :double :double]]]]]
(-unflatten-refs *1)
;[:schema {:registry {::user-id :uuid
; ::address [:map
; [:street :string]
; [:lonlat [:tuple :double :double]]]
; ::user [:map
; [:id ::user-id]
; [:name :string]
; [:address ::address]]}}
; ::user]
```
malli/docs/tips.md
Vendored
+549
View File
@@ -0,0 +1,549 @@
# Tips
## Accessing both schema and value in transformation
```clojure
(require '[malli.core :as m])
(require '[malli.transform :as mt])
(def Address
[:map
[:id :string]
[:tags [:set :keyword]]
[:address [:map
[:street :string]
[:city :string]]]])
(def lillan
{:id "Lillan"
:tags #{:artesan :coffee :hotel}
:address {:street "Ahlmanintie 29"
:city "Tampere"}})
(m/decode
Address
lillan
(mt/transformer
{:default-decoder
{:compile (fn [schema _]
(fn [value]
(prn [value (m/form schema)])
value))}}))
;[{:id "Lillan", :tags #{:coffee :artesan :hotel}, :address {:street "Ahlmanintie 29", :city "Tampere"}} [:map [:id :string] [:tags [:set :keyword]] [:address [:map [:street :string] [:city :string]]]]]
;["Lillan" [:malli.core/val :string]]
;["Lillan" :string]
;[#{:coffee :artesan :hotel} [:malli.core/val [:set :keyword]]]
;[#{:coffee :artesan :hotel} [:set :keyword]]
;[:coffee :keyword]
;[:artesan :keyword]
;[:hotel :keyword]
;[{:street "Ahlmanintie 29", :city "Tampere"} [:malli.core/val [:map [:street :string] [:city :string]]]]
;[{:street "Ahlmanintie 29", :city "Tampere"} [:map [:street :string] [:city :string]]]
;["Ahlmanintie 29" [:malli.core/val :string]]
;["Ahlmanintie 29" :string]
;["Tampere" [:malli.core/val :string]]
;["Tampere" :string]
;; => {:id "Lillan", :tags #{:coffee :artesan :hotel}, :address {:street "Ahlmanintie 29", :city "Tampere"}}
```
## Removing Schemas based on a property
Schemas can be walked over recursively using `m/walk`:
```clojure
(require '[malli.core :as m])
(def Schema
[:map
[:user :map]
[:profile :map]
[:tags [:vector [:int {:deleteMe true}]]]
[:nested [:map [:x [:tuple {:deleteMe true} :string :string]]]]
[:token [:string {:deleteMe true}]]])
(m/walk
Schema
(fn [schema _ children options]
;; return nil if Schema has the property
(when-not (:deleteMe (m/properties schema))
;; there are two syntaxes: normal and the entry, handle separately
(let [children (if (m/entries schema) (filterv last children) children)]
;; create a new Schema with the updated children, or return nil
(try (m/into-schema (m/type schema) (m/properties schema) children options)
(catch #?(:clj Exception, :cljs js/Error) _))))))
;[:map
; [:user :map]
; [:profile :map]
; [:nested :map]]
```
In the example, `:tags` key was removed as it's contents would have been an empty `:vector`, which is not legal Schema syntax. Empty `:map` is ok.
## Trimming strings
Example how to trim all `:string` values using a custom transformer:
```clojure
(require '[malli.transform :as mt])
(require '[malli.core :as m])
(require '[clojure.string :as str])
;; a decoding transformer, only mounting to :string schemas with truthy :string/trim property
(defn string-trimmer []
(mt/transformer
{:decoders
{:string
{:compile (fn [schema _]
(let [{:string/keys [trim]} (m/properties schema)]
(when trim #(cond-> % (string? %) str/trim))))}}}))
;; trim me please
(m/decode [:string {:string/trim true, :min 1}] " kikka " string-trimmer)
;; => "kikka"
;; no trimming
(m/decode [:string {:min 1}] " " string-trimmer)
;; => " "
;; without :string/trim, decoding is a no-op
(m/decoder :string string-trimmer)
; => #object[clojure.core$identity]
```
## Decoding collections
Transforming a comma-separated string into a vector of ints:
```clojure
(require '[malli.core :as m])
(require '[malli.transform :as mt])
(require '[clojure.string :as str])
(m/decode
[:vector {:decode/string #(str/split % #",")} int?]
"1,2,3,4"
(mt/string-transformer))
;; => [1 2 3 4]
```
Using a custom transformer:
```clojure
(defn query-decoder [schema]
(m/decoder
schema
(mt/transformer
(mt/transformer
{:name "vectorize strings"
:decoders
{:vector
{:compile (fn [schema _]
(let [separator (-> schema m/properties :query/separator (or ","))]
(fn [x]
(cond
(not (string? x)) x
(str/includes? x separator) (into [] (.split ^String x separator))
:else [x]))))}}})
(mt/string-transformer))))
(def decode
(query-decoder
[:map
[:a [:vector {:query/separator ";"} :int]]
[:b [:vector :int]]]))
(decode {:a "1", :b "1"})
;; => {:a [1], :b [1]}
(decode {:a "1;2", :b "1,2"})
;; => {:a [1 2], :b [1 2]}
```
## Normalizing properties
Returning a Schema form with `nil` in place of empty properties:
```clojure
(require '[malli.core :as m])
(defn normalize-properties [?schema]
(m/walk
?schema
(fn [schema _ children _]
(if (vector? (m/form schema))
(into [(m/type schema) (m/properties schema)] children)
(m/form schema)))))
(normalize-properties
[:map
[:x :int]
[:y [:tuple :int :int]]
[:z [:set [:map [:x [:enum 1 2 3]]]]]])
;; => [:map nil
;; [:x nil :int]
;; [:y nil [:tuple nil :int :int]]
;; [:z nil [:set nil
;; [:map nil
;; [:x nil [:enum nil 1 2 3]]]]]]
```
## Default value from a function
The `mt/default-value-transformer` can fill default values if the `:default` property is given. It
is possible though to calculate a default value with a given function providing custom transformer
derived from `mt/default-value-transformer`:
```clojure
(defn default-fn-value-transformer
([]
(default-fn-value-transformer nil))
([{:keys [key] :or {key :default-fn}}]
(let [add-defaults
{:compile
(fn [schema _]
(let [->k-default (fn [[k {default key :keys [optional]} v]]
(when-not optional
(when-some [default (or default (some-> v m/properties key))]
[k default])))
defaults (into {} (keep ->k-default) (m/children schema))
exercise (fn [x defaults]
(reduce-kv (fn [acc k v]
; the key difference compare to default-value-transformer
; we evaluate v instead of just passing it
(if-not (contains? x k)
(-> (assoc acc k ((m/eval v) x))
(try (catch Exception _ acc)))
acc))
x defaults))]
(when (seq defaults)
(fn [x] (if (map? x) (exercise x defaults) x)))))}]
(mt/transformer
{:decoders {:map add-defaults}
:encoders {:map add-defaults}}))))
```
Example 1: if `:secondary` is missing, same its value to value of `:primary`
```clojure
(m/decode
[:map
[:primary :string]
[:secondary {:default-fn '#(:primary %)} :string]]
{:primary "blue"}
(default-fn-value-transformer))
```
Example 2: if `:cost` is missing, try to calculate it from `:price` and `:qty`:
```clojure
(def Purchase
[:map
[:qty {:default 1} number?]
[:price {:optional true} number?]
[:cost {:default-fn '(fn [m] (* (:qty m) (:price m)))} number?]])
(def decode-autonomous-vals
(m/decoder Purchase (mt/transformer (mt/string-transformer) (mt/default-value-transformer))))
(def decode-interconnected-vals
(m/decoder Purchase (default-fn-value-transformer)))
(-> {:qty "100" :price "1.2"} decode-autonomous-vals decode-interconnected-vals)
;; => {:price 1.2, :qty 100.0, :cost 120.0}
(-> {:price "1.2"} decode-autonomous-vals decode-interconnected-vals)
;; => {:qty 1, :price 1.2, :cost 1.2}
(-> {:prie "1.2"} decode-autonomous-vals decode-interconnected-vals)
;; => {:prie "1.2", :qty 1}
```
## Walking Schema and Entry Properties
1. walk entries on the way in
2. unwalk entries on the way out
```clojure
(defn walk-properties [schema f]
(m/walk
schema
(fn [s _ c _]
(m/into-schema
(m/-parent s)
(f (m/-properties s))
(cond->> c (m/entries s) (map (fn [[k p s]] [k (f p) (first (m/children s))])))
(m/options s)))
{::m/walk-entry-vals true}))
```
Stripping all swagger-keys:
```clojure
(defn remove-swagger-keys [p]
(not-empty
(reduce-kv
(fn [acc k _]
(cond-> acc (some #{:swagger} [k (-> k namespace keyword)]) (dissoc k)))
p p)))
(walk-properties
[:map {:title "Organisation name"}
[:ref {:swagger/description "Reference to the organisation"
:swagger/example "Acme floor polish, Houston TX"} :string]
[:kikka [:string {:swagger {:title "kukka"}}]]]
remove-swagger-keys)
;[:map {:title "Organisation name"}
; [:ref :string]
; [:kikka :string]]
```
## Allowing invalid values on optional keys
e.g. don't fail if the optional keys hava invalid values.
1. create a helper function that transforms the schema swapping the actual schema with `:any`
2. done.
```clojure
(require '[malli.util :as mu])
(defn allow-invalid-optional-values [schema]
(m/walk
schema
(m/schema-walker
(fn [s]
(cond-> s
(m/entries s)
(mu/transform-entries
(partial map (fn [[k {:keys [optional] :as p} s]] [k p (if optional :any s)]))))))))
(allow-invalid-optional-values
[:map
[:a :string]
[:b {:optional true} :int]
[:c [:maybe
[:map
[:d :string]
[:e {:optional true} :int]]]]])
;[:map
; [:a :string]
; [:b {:optional true} :any]
; [:c [:maybe [:map
; [:d :string]
; [:e {:optional true} :any]]]]]
(m/validate
[:map
[:a :string]
[:b {:optional true} :int]]
{:a "Hey" :b "Nope"})
;; => false
(m/validate
(allow-invalid-optional-values
[:map
[:a :string]
[:b {:optional true} :int]])
{:a "Hey" :b "Nope"})
;; => true
```
## Collecting inlined reference definitions from schemas
By default, one can inline schema reference definitions with `:map`, like:
```clojure
(def User
[:map
[::id :int]
[:name :string]
[::country {:optional true} :string]])
```
It would be nice to be able to simplify the schemas into:
```clojure
[:map
::id
[:name :string]
[::country {:optional true}]]
```
Use cases:
* Simplify large schemas
* Finding differences in semantics
* Refactoring multiple schemas to use a shared registry
Naive implementation (doesn't look up the local registries):
```clojure
(require '[malli.registry :as mr])
(defn collect-references [schema]
(let [acc* (atom {})
->registry (fn [registry]
(->> (for [[k d] registry]
(if (seq (rest d))
(m/-fail! ::ambiguous-references {:data d})
[k (first (keys d))]))
(into {})))
schema (m/walk
schema
(fn [schema path children _]
(let [children (if (= :map (m/type schema)) ;; just maps
(->> children
(mapv (fn [[k p s]]
;; we found inlined references
(if (and (m/-reference? k) (not (m/-reference? s)))
(do (swap! acc* update-in [k (m/form s)] (fnil conj #{}) (conj path k))
(if (seq p) [k p] k))
[k p s]))))
children)
;; accumulated registry, fail on ambiguous refs
registry (->registry @acc*)]
;; return simplified schema
(m/into-schema
(m/-parent schema)
(m/-properties schema)
children
{:registry (mr/composite-registry (m/-registry (m/options schema)) registry)}))))]
{:registry (->registry @acc*)
:schema schema}))
```
In action:
```clojure
(collect-references User)
;{:registry {:user/id :int,
; :user/country :string}
; :schema [:map
; :user/id
; [:name :string]
; [:user/country {:optional true}]]}
```
<!-- :test-doc-blocks/skip -->
```clojure
(collect-references
[:map
[:user/id :int]
[:child [:map
[:user/id :string]]]])
; =throws=> :user/ambiguous-references {:data {:string #{[:child :user/id]}, :int #{[:user/id]}}}
```
## Getting error-values into humanized result
```clojure
(require '[malli.error :as me])
(-> [:map
[:x :int]
[:y [:set :keyword]]
[:z [:map
[:a [:tuple :int :int]]]]]
(m/explain {:x "1"
:y #{:a "b" :c}
:z {:a [1 "2"]}})
(me/humanize {:wrap #(select-keys % [:value :message])}))
;; => {:x [{:value "1"
; :message "should be an integer"}],
; :y #{[{:value "b"
; :message "should be a keyword"}]},
; :z {:a [nil [{:value "2"
; :message "should be an integer"}]]}}
```
## Dependent String Schemas
A schema for a string made of two components `a` and `b` separated by a `/` where the schema of `b`
depends on the value of `a`. The valid values of a are known in advance.
For instance:
* When `a` is "ip" , `b` should be a valid ip
* When `a` is "domain", `b` should be a valid domain
Here are a few examples of valid and invalid data:
* `"ip/127.0.0.1"` is valid
* `"ip/111"` is not valid
* `"domain/cnn.com"` is valid
* `"domain/aa"` is not valid
* `"kika/aaa"` is not valid
```clojure
(def domain #"[a-zA-Z0-9][-a-zA-Z0-9]{0,62}(\.[a-zA-Z0-9][-a-zA-Z0-9]{0,62})+")
(def ipv4 #"^(?:[0-9]{1,3}\.){3}[0-9]{1,3}$")
;; a multi schema describing the values as a tuple
;; includes transformation guide to and from a string domain
(def schema [:multi {:dispatch first
:decode/string #(str/split % #"/")
:encode/string #(str/join "/" %)}
["domain" [:tuple [:= "domain"] domain]]
["ip" [:tuple [:= "ip"] ipv4]]])
;; define workers
(def validate (m/validator schema))
(def decode (m/decoder schema mt/string-transformer))
(def encode (m/encoder schema mt/string-transformer))
(decode "ip/127.0.0.1")
;; => ["ip" "127.0.0.1"]
(-> "ip/127.0.0.1" (decode) (encode))
;; => "ip/127.0.0.1"
(map (comp validate decode)
["ip/127.0.0.1"
"ip/111"
"domain/cnn.com"
"domain/aa"
"kika/aaa"])
;; => (true false true false false)
```
It is also possible to use a custom transformer instead of `string-transformer` (for example, in order to avoid `string-transformer` to perform additional encoding and decoding):
```clojure
(def schema [:multi {:dispatch first
:decode/my-custom #(str/split % #"/")
:encode/my-custom #(str/join "/" %)}
["domain" [:tuple [:= "domain"] domain]]
["ip" [:tuple [:= "ip"] ipv4]]])
(def decode (m/decoder schema (mt/transformer {:name :my-custom})))
(decode "ip/127.0.0.1")
;; => ["ip" "127.0.0.1"]
```
## Converting Schemas
Example utility to convert schemas recursively:
```clojure
(defn schema-mapper [m]
(fn [s] ((or (get m (m/type s)) ;; type mapping
(get m ::default) ;; default mapping
(constantly s)) ;; nop
s)))
(m/walk
[:map
[:id :keyword]
[:size :int]
[:tags [:set :keyword]]
[:sub
[:map
[:kw :keyword]
[:data [:tuple :keyword :int :keyword]]]]]
(m/schema-walker
(schema-mapper
{:keyword (constantly :string) ;; :keyword -> :string
:int #(m/-set-properties % {:gen/elements [1 2]}) ;; custom :int generator
::default #(m/-set-properties % {::type (m/type %)})}))) ;; for others
;[:map {::type :map}
; [:id :string]
; [:size [:int {:gen/elements [1 2 3]}]]
; [:tags [:set {::type :set} :string]]
; [:sub [:map {::type :map}
; [:kw :string]
; [:data [:tuple {::type :tuple}
; :string
; [:int {:gen/elements [1 2 3]}]
; :string]]]]]
```
malli/docs/value-transformation.md
Vendored
+15
View File
@@ -0,0 +1,15 @@
# Value Transformation
## Terminology
| Term | Description
|-------------------------|------------
| transformation function | a function of A->B, e.g. conversion strings to dates
| decoding | a process of transforming (invalid) values into potentially valid ones (IN), `m/decoder` & `m/decode`
| encoding | a process of transforming (valid) values into something else (OUT), `m/encoder` & `m/encode`
| transformer | a top-level component that maps Schemas with transformation functions (e.g. “json-transformer transforms strings to dates, but not strings to numbers”). Needed in encoding and decoding, `mt/transformer`
| named transformer | If a transformer has `:name` defined, Schemas can define their transformation functions (for both encoding & decoding) using Schema properties
| interceptor | a component that bundles transforming functions into *transforming phases*
| transforming phase | either `:enter` or `:leave`, timing when a transformation function is applied in the chain (before or after the fact)
| interceptor chain | a sequence of interceptors that is used to run the (optimized sequence of) transformation functions from interceptors in correct order
| transformation chain | transformers compose too: `(mt/transformer {:name :before} mt/json-transformer {:name :after})`