Introducing Farseer: the JSON RPC server, the client and utilities

(This is a copy of the readme file from the repository.)

Farseer is a set of modules for JSON RPC. It includes a transport-independent handler, Ring HTTP handler, Jetty server, HTTP client, local stub, documentation, and more.

What and Why is JSON RPC?
- Benefits
- Disadvantages
The Structure of this Project
- Installation
RPC Handler
Ring HTTP Handler
Jetty Server
HTTP Stub
HTTP Client
Documentation Builder
Ideas & Further Development

What and Why is JSON RPC?

Briefly, JSON RPC is a protocol based on HTTP & JSON. When calling the server, you specify the method (procedure) name and its parameters. The parameters could be either a map or a vector. The server returns a JSON response with the result or error fields. For example:

Request:

{"jsonrpc": "2.0", "method": "sum", "params": [1, 2], "id": 3}

Response:

{"jsonrpc": "2.0", "result": 3, "id": 3}

Pay attention: the protocol depends on neither HTTP method, nor query params, nor HTTP headers and so on. Although looking a bit primitive, this schema eventually appears to be robust, scalable and reliable.

Benefits

RPC protocol brings significant changes into your API, namely:

There is single API endpoint on the server, for example /api. You don’t need to concatenate strings manually to build the paths like /post/42/comments/52352 in REST.
All the data is located in one place. There is no need to parse the URI, query params, check out the method and so on. You don’t need to guess which HTTP method to pick (PUT, PATCH) for an operation when several entities change.
RPC grows horizontally with ease. Once you’ve set it up, you only extend it. Technically it means adding a new key into a map.
RPC doesn’t depend on transport. You can save the payload in Cassandra or push to Kafka. Later on, you can replay the sequence as it has everything you need.
RPC is a great choice for interaction between internal services. When all the services follow the same protocol, it’s easy to develop and maintain them. When protected with authentication, RPC can be provided to the end customers as well.

Disadvantages

The only disadvantage of RPC protocol is that it’s free from caching. On the other hand, we rarely want to get cached data. Most often, it’s important to get actual data on each request. If you share some public data that update rarely, perhaps you should organize ordinary GET endpoints.

The Structure of this Project

Farseer consists from several minor projects that complement each other. Every sub-project requires its own dependencies. If it was a single project, you would download lots of stuff you don’t really need. Instead, with sub-projects, you install only those parts (and transient dependencies) you really need in your project.

The root project is named com.github.igrishaev/farseer-all. It unites all the sub-projects listed below:

com.github.igrishaev/farseer-common: dependency-free parts required by other sub-projects;
com.github.igrishaev/farseer-handler: a transport-free implementation of RPC handler;
com.github.igrishaev/farseer-http: HTTP Ring handler for RPC;
com.github.igrishaev/farseer-jetty: HTTP Jetty server for RPC;
com.github.igrishaev/farseer-stub: an HTTP stub for RPC server, useful for tests;
com.github.igrishaev/farseer-client: HTTP RPC client based on clj-http;
com.github.igrishaev/farseer-doc: RPC documentation builder.

Installation

The “-all” bundle:

Lein:

[com.github.igrishaev/farseer-all "0.1.1"]

Deps.edn

com.github.igrishaev/farseer-all {:mvn/version "0.1.1"}

Maven

<dependency>
  <groupId>com.github.igrishaev</groupId>
  <artifactId>farseer-all</artifactId>
  <version>0.1.1</version>
</dependency>

Alternatevely, install only what you need:

Lein:

[com.github.igrishaev/farseer-http "0.1.1"]
[com.github.igrishaev/farseer-client "0.1.1"]

and so on (see the list of packages on Clojars).

RPC Handler

The com.github.igrishaev/farseer-handler package provides basic implementation of RPC protocol. It has no any transport layer, only a handler that serves RPC requests no matter where the data comes from. Other packages provide HTTP layer for this handler. You can develop another transport layer as well.

First, add the package to the project:

[com.github.igrishaev/farseer-handler ...]

Here is the minimal usage example. Prepare a namespace:

(ns demo
  (:require
   [farseer.handler :refer [make-handler]]))

Create a method handler and the config:

(defn rpc-sum
  [_ [a b]]
  (+ a b))

(def config
  {:rpc/handlers
   {:math/sum
    {:handler/function rpc-sum}}})

Now declare a handler and call it:

(def handler
  (make-handler config))

(handler {:id 1
          :method :math/sum
          :params [1 2]
          :jsonrpc "2.0"})

;; {:id 1, :jsonrpc "2.0", :result 3}

Method handlers

The rpc-sum function is a handler for the :math/sum method. The function takes exactly two arguments. The first argument is the context map which we’ll discuss later. The second is the parameters passed to the method in request. They might be either a map or a vector. If a method doesn’t accept parameters, the arguments will be nil.

The function might be defined in another namespace. In this case, you import it and pass to the map as usual:

(ns demo
  (:require
   [com.project.handlers.math :as math]))

(def config
  {:rpc/handlers
   {:math/sum
    {:handler/function math/sum-handler}}})

It’s useful to pass the functions as vars using the #' syntax:

(def config
  {:rpc/handlers
   {:math/sum
    {:handler/function #'rpc-sum}}})

In this case, you can update the function by evaling its defn form in REPL, and the changes come into play without re-creating the RPC handler. For example, we change plus to minus in the rpc-sum function:

(defn rpc-sum
  [_ [a b]]
  (- a b))

Then we go to the closing bracket of the defn form and perform cider-eval-last-sexp. Now we make a new RPC call and get a new result:

(handler {:id 1
          :method :math/sum
          :params [1 2]
          :jsonrpc "2.0"})

{:id 1, :jsonrpc "2.0", :result -1}

Only the methods declared in the config are served by the RPC handler. If you specify a non-existing one, you’ll get a negative response:

(handler {:id 1
          :method :system/rmrf
          :params [1 2]
          :jsonrpc "2.0"})

{:error
 {:code -32601, :message "Method not found", :data {:method :system/rmrf}},
 :id 1,
 :jsonrpc "2.0"}

Specs

The code above doesn’t validate the incoming parameters and thus is dangerous to execute. If you pass something like ["one" nil] instead of two numbers, you’ll end up with NPE, which is not good.

For each handler, you can specify a couple of specs, the input and output ones. The input spec validates the incoming params field, and the output spec is for the result of the function call with these parameters.

Input Spec

To protect our :math/sum handler from weird data, you declare the specs:

(s/def :math/sum.in
  (s/tuple number? number?))

(s/def :math/sum.out
  number?)

Then you add these specs to the method config. Their keys are :handler/spec-in and :handler/spec-out:

(def config
  {:rpc/handlers
   {:math/sum
    {:handler/function #'rpc-sum
     :handler/spec-in :math/sum.in
     :handler/spec-out :math/sum.out}}})

Now if you pass something wrong to the handler, you’ll get a negative response:

(handler {:id 1
          :method :math/sum
          :params ["one" nil]
          :jsonrpc "2.0"})

{:id 1
 :jsonrpc "2.0"
 :error {:code -32602
         :message "Invalid params"
         :data {:explain "<spec explain string>"}}}

The :data field of the :error object has an extra explain field. Inside it, there is standard explain string produced by the s/explain-str function. This kind of message looks noisy sometimes, and in the future, most likely Farseer will use Expound.

According to the RPC specification, the :params field might be either a map or a vector. Thus, for the input spec, you probably use s/tuple or s/keys specs. Our :math/sum method accepts vector params. Let’s rewrite it and the specs such that they work with a map:

;; a new handler
(defn rpc-sum
  [_ {:keys [a b]}]
  (+ a b))

;; new input spec
(s/def :sum/a number?)
(s/def :sum/b number?)

(s/def :math/sum.in
  (s/keys :req-un [:sum/a :sum/a]))

The output spec and the config are still the same:

(s/def :math/sum.out
  number?)

(def config
  {:rpc/handlers
   {:math/sum
    {:handler/function #'rpc-sum
     :handler/spec-in :math/sum.in
     :handler/spec-out :math/sum.out}}})

Now we pass a map, not vector:

(handler {:id 1
          :method :math/sum
          :params {:a 1 :b 2}
          :jsonrpc "2.0"})

{:id 1, :jsonrpc "2.0", :result 3}

Output Spec

If the result of the function doesn’t match the output spec, it triggers an internal error. Let’s reproduce this scenario by spoiling the spec:

(s/def :math/sum.out
  string?)

(handler {:id 1
          :method :math/sum
          :params {:a 1 :b 2}
          :jsonrpc "2.0"})


{:id 1,
 :jsonrpc "2.0"
 :error {:code -32603,
         :message "Internal error",
         :data {:method :math/sum}}}

You’ll see the following log entry:

10:18:31.256 ERROR farseer.handler - RPC result doesn't match the output spec,
             id: 1, method: :math/sum, code: -32603, message: Internal error

There is no the s/explain message, because sometimes it’s huge and also contains private data.

In Production

You can turn off checking the input or the output specs globally in the configuration (see the “Configuration” section below). In real projects, we always validate the input data. Regarding the output, we validate it only in tests to save time in production.

More on Context

Summing numbers is good for tutorial but makes no sense in real projects. We’re rather interested in networking IO and database access. Until now, it wasn’t clear how a function can reach Postgres or Kafka clients, especially if the project relies on a system framework (e.g. Component or Integrant).

In OOP languages, the environment for the RPC method usually comes from the this parameter. It’s an instance of some RPCHadler class that has fields for the database connection, message queue client and so on. In Clojure, we act almost like this, but instead of this object, we use context.

A context is a map that carries the data needed by the method handler in runtime. This is the first argument of a function from the :handler/function key. By default, the context has the current id and method of the RPC call. If you print the first argument of the function, you’ll see:

(defn rpc-sum
  [context {:keys [a b]}]
  (println context)
  (+ a b))

#:rpc{:id 1, :method :math/sum}

Both fields are prefixed with the :rpc/ namespace to prevent the keys from clashing, e.g. :id for the RPC call and :id for the current user. Instead, the framework passes the :rpc/id field, and you should pass :user/id one.

There are two ways of passing context: a static and dynamic ones.

Static Context

When you call the make-handler function to build an RPC handler, the second argument might be a context map. This map will be available in all the RPC functions. For example:

(def handler
  (make-handler
   config
   {:db (open-db-connection {...})
    :version (get-app-version)}))

We assume the open-db-connection returns a connection pool, which is available to all the RPC functions as the :db field of the context. The :version field is the application version that is fetched from a text file.

Now, if we had an RPC method that fetches a user by id, it would look like this:

(defn get-user-by-id
  [{:keys [db]}        ;; context
   {:keys [user-id]}]  ;; params
  (jdbc/get-by-id db :users user-id))

(s/def :user/id pos-int?)

(s/def :user/user-by-id.in
  (s/keys :req-un [:user/id]))

(s/def :user/user-by-id.out
  (s/nilable map?))

The config:

(def config
  {:rpc/handlers
   {:user/get-by-id
    {:handler/function #'get-user-by-id
     :handler/spec-in :user/user-by-id.in
     :handler/spec-out :user/user-by-id.out}}})

The call:

(handler {:id 1
          :method :user/get-by-id
          :params {:id 5}
          :jsonrpc "2.0"})

{:id 1, :jsonrpc "2.0", :result {:id 5 :name "Test"}}

Dynamic Context

Use dynamic context to pass a value needed only for the current RPC request or you don’t have a value yet when building a handler. In that case, pass the context map as the second argument to the function made by the make-handler. The example with the database would look like this:

(def handler (make-handler config))

;; dynamic context, the second arg
(handler {:id 1
          :method :user/get-by-id
          :params {:id 5}
          :jsonrpc "2.0"}
         {:db hikari-cp-pool})

You can use both ways to pass the context. Most likely the database is needed by all the RPC functions, so its place in the global context. Some minor fields might be passes on demand for certain calls:

(def handler
  (make-handler config {:db (make-db ...)}))

(handler {:id 1 :method ...} {:version "0.2.1"})

The context maps are always merged, so from the function’s point of view, there is only a single map.

The local context map gets merged into the global one. It gives you an opportunity to override the default values from the context. Let’s say, if the method :user/get-by-id needs a special (read-only) database, we can override it like this:

(def handler
  (make-handler config {:db (make-db ...)}))

(handler {:id 1 :method :user/get-by-id ...}
         {:db read-only-db})

Request & Response Formats

Request

An RPC request is a map of the following fields:

:id is either a number or a string value representing this request. The handler must return the same id in response unless it was a notification (see below).
:method is either a string or a keyword (the latter is preferred) that specify the RPC method. If a method was a string, it gets coerced to the keyword anyway. We recommend using the full qualified keywords with namespaces. The namespaces help to group methods by semantic.
:params is either a map of [keyword?, any?] pairs, or a vector of any? values (sequential?, if more precisely). This field is optional because some methods don’t require arguments.
:jsonrpc: a string with exact value "2.0", the required one.

Examples:

;; all the fields
{:id 1
 :method :math/sum
 :params [1 2]
 :jsonrpc "2.0"}

;; no params
{:id 2
 :method :app/version
 :jsonrpc "2.0"}

;; no id (notification)
{:method :user/delete-by-id
 :params {:id 3}
 :jsonrpc "2.0"}

The RPC request might be of a batch form then it’s a vector of such maps. Batch is useful to perform multiple actions per one call. See the “Batch Requests” section below.

Response

The response is map with the :id and :jsonrpc fields. The ID is the same you passed in the request so you can match the request and the response by ID. If the response was positive, its :result field carries the value that the RPC function returned:

{:id 1, :jsonrpc "2.0", :result 3}

A negative response has no the :result field but the :error one instead. The error node consists from the :code and :message fields which are the numeric code representing an error and a text message explaining it. In addition, there might be the :data fields which is an arbitrary map with some extra context. The library adds the :method field to the context automatically.

{:id 1
 :jsonrpc "2.0"
 :error {:code -32603
         :message "Internal error"
         :data {:method :math/div}}}

Error Codes

-32700 Parse error: Used then the server gets a non-JSON/broken payload.
-32600 Invalid Request: The payload is JSON but has a wrong shape.
-32601 Method not found: No such RPC method.
-32602 Invalid params: The parameters do not match the input spec.
-32603 Internal error: Either uncaught exception or the result doesn’t match the output spec.
-32000 Authentication failure: Something is wrong with auth/credentials.

Find more information about the error codes on this page.

Notifications

Sometimes, you’re not interested in the response from an RPC server. Say, if you delete a user, there is nothing for you to return. In this case, you send a notification rather than a request. Notifications are formed similar but have no the :id field. The server sends nothing back for a notification. For example:

(handler {:method :math/sum
          :params [1 2]
          :jsonrpc "2.0"})

nil

Notifications are useful to trigger some side effects on the server.

Remember, if you pass a missing method or wrong input data (or any other error occurs), you’ll get a negative response anyway despite the fact it was a notification:

(handler {:method :math/sum
          :params [1 "a"]
          :jsonrpc "2.0"})

{:error
 {:code -32602
  :message "Invalid params"}
 :jsonrpc "2.0"}

Batch Requests

Batch requests is the main feature of JSON RPC. It allows you to send multiple request maps in one call. The server executes the requests and returns a list of result maps. For example, you have a method user/get-by-id which takes a single ID and returns a user map from the database. Now you got ten IDs. With ordinary REST API, you would run a cycle and performed ten HTTP calls. With RPC, you make a batch call.

In our example, if we want to solve several math expressions at once, we do:

(handler [{:id 1
           :method :math/sum
           :params [1 2]
           :jsonrpc "2.0"}
          {:id 2
           :method :math/sum
           :params [3 4]
           :jsonrpc "2.0"}
          {:id 3
           :method :math/sum
           :params [5 6]
           :jsonrpc "2.0"}])

The result:

({:id 1 :jsonrpc "2.0" :result 3}
 {:id 2 :jsonrpc "2.0" :result 7}
 {:id 3 :jsonrpc "2.0" :result 11})

If some of the tasks fail, they won’t affect the others:

(handler [{:id 1
           :method :math/sum
           :params [1 2]
           :jsonrpc "2.0"}
          {:id 2
           :method :math/sum
           :params [3 "aaa"]  ;; bad input
           :jsonrpc "2.0"}
          {:id 3
           :method :math/missing ;; wrong method
           :params [5 6]
           :jsonrpc "2.0"}])

The result:

({:id 1 :jsonrpc "2.0" :result 3}
 {:error
  {:code -32602
   :message "Invalid params"
   :data
   {:explain ...
    :method :math/sum}}
  :id 2
  :jsonrpc "2.0"}
 {:error
  {:code -32601 :message "Method not found" :data {:method :math/missing}}
  :id 3
  :jsonrpc "2.0"})

You can mix ordinary RPC tasks with notifications in a batch. There will be no response maps for notifications in the result vector:

(handler [{:id 1
           :method :math/sum
           :params [1 2]
           :jsonrpc "2.0"}
          {:method :math/sum ;; no ID
           :params [3 4]
           :jsonrpc "2.0"}])

[{:id 1 :jsonrpc "2.0" :result 3}]

Note on Parallelism

By default, Farseer uses the standard pmap function to deal with multiple tasks. It executes the tasks in semi-parallel way. Maybe in the future, we could use a custom fixed thread executor for more control.

Configuring & Limiting Batch Requests

The following options help you to control batch requests:

:rpc/batch-allowed? (default is true): whether or not to allow batch requests. If you set this to false and someone performs a batch call, they will get an error:

(def config
  {:rpc/batch-allowed? false
   :rpc/handlers ...})

(def handler
  (make-handler config))


(handler [{:id 1
           :method :math/sum
           :params [1 2]
           :jsonrpc "2.0"}
          {:id 2
           :method :math/sum
           :params [3 4]
           :jsonrpc "2.0"}])

{:error {:code -32602, :message "Batch is not allowed"}}

:rpc/batch-max-size (default is 25): the max number of tasks in a single batch request. Sending more tasks than is allowed in one request would lead to an error:

(def config
  {:rpc/batch-allowed? true
   :rpc/batch-max-size 2
   :rpc/handlers ...})

(def handler
  (make-handler config))


(handler [{...} {...} {...}])

{:error {:code -32602, :message "Batch size is too large"}}

:rpc/batch-parallel? (default is true): whether or not to prefer pmap over the standard mapv for tasks processing. When false, the tasks get executed one by one.

Errors & Exceptions

Runtime (Unexpected) Errors

The RPC handler wraps the whole logic into try/catch form with the Throwable class. It means you’ll get a negative response even if something weird happens inside it. Here is an example of unsafe division what might lead to exception:

(defn rpc-div
  [_ [a b]]
  (/ a b))

(def config
  {:rpc/handlers
   {:math/div
    {:handler/function #'rpc-div}}})

(def handler
  (make-handler config))

(handler {:id 1
          :method :math/div
          :params [1 0]
          :jsonrpc "2.0"})

{:id 1
 :jsonrpc "2.0"
 :error {:code -32603
         :message "Internal error"
         :data {:method :math/div}}}

All the unexpected exceptions end up with the “Internal error” response with the code -32603. In the console, you’ll see the the logged exception:

10:19:35.948 ERROR farseer.handler - Divide by zero, id: 1, method: :math/div, code: -32603, message: Internal error
java.lang.ArithmeticException: Divide by zero
	at clojure.lang.Numbers.divide(Numbers.java:188)
	at demo$rpc_div.invokeStatic(form-init9886809666544152192.clj:190)
	at demo$rpc_div.invoke(form-init9886809666544152192.clj:188)
    ...

RPC (Expected) Errors

Sometimes, you know that you cannot serve the current request, and it must be failed. The easiest way to end up the request is to throw an exception. But to get a proper RPC response, there should be a special exception with the fields that take place in the response. The namespace farseer.error provides several functions for such exceptions.

When an RPC handler catches an exception, it fetches its data using the ex-data function. Then it looks for some special fields to compose the response. Namely, these fields are:

:rpc/code: a number representing the error. When specified, it becomes the code field of the error response.
:rpc/message: a string explaining the error. Becomes the message field of the error response.
:rpc/data: a map with arbitrary data sent to the client. Becomes the data field of the error response.
:log/level: a keyword representing the logging level of this error. Valid values are those that the functions from clojure.tools.logging package accept, e.g. :debug, :info, :warn, :error.
:log/stacktrace?: boolean, whether to log the entire stack trace or the message only. Useful for “method not found” or “wrong input” cases because there is no need for the full stack trace in such cases.

The data fetched from the exception instance gets merged with the default error map declared in the internal-error variable:

(def internal-error
  {:log/level       :error
   :log/stacktrace? true
   :rpc/code        -32603
   :rpc/message     "Internal error"})

Thus, if you didn’t specify some of the fields, they would come from this map.

Raising Exceptions

There are some shortcut functions to simplify raising exceptions, namely:

parse-error!
invalid-request!
not-found!
invalid-params!
internal-error!
auth-error!

Examples:

JSON parse error:

(farseer.error/parse-error!)

RPC Method is not found:

(farseer.error/not-found!
  {:rpc/message "I don't have such method"})

Wrong input parameters:

(farseer.error/invalid-params!
  {:rpc/data {:spec-explain "..."}})

Internal error:

(farseer.error/internal-error! nil caught-exception)

The signature of all these functions is [& [data e]] meaning that you can call a function even without arguments. Each function has its own default data map that gets merged to the data you passed. For example, these are default values for the invalid-params! function:

(def invalid-params
  {:log/level       :info
   :log/stacktrace? false
   :rpc/code        -32602
   :rpc/message     "Invalid params"})

The logging level is :info because this is expected behaviour. We also we don’t log the whole stack trace for the same reason.

Configuration

Ring HTTP Handler

The Ring package creates an HTTP handler from an RPC configuration. The HTTP handler follows the official Ring protocol: it’s a function that takes an HTTP request map and returns a response map. The handler uses JSON format for transport. It’s already wrapped with Ring JSON middleware that decodes and encodes the payload. You can pass other middleware stack to use something other that JSON, say MessagePack or EDN.

Add the package:

;; deps
[com.github.igrishaev/farseer-http "..."]

;; module
(ns ...
  (:require
   [farseer.http :as http]))

The package reuses the same config we wrote above. All the HTTP-related fields have default values, so you can just pass the config to the make-app function:

(def app
  (http/make-app config))

Now let’s compose the HTTP request for the app:

(def rpc
  {:id 1
   :jsonrpc "2.0"
   :method :math/sum
   :params [1 2]})

(def request
  {:request-method :post
   :uri "/"
   :headers {"content-type" "application/json"}
   :body (-> rpc json/generate-string .getBytes)})

and call it like an HTTP server:

(def response
  (-> (app request)
      (update :body json/parse-string true)))

{:status 200
 :body {:id 1 :jsonrpc "2.0" :result 3}
 :headers {"Content-Type" "application/json; charset=utf-8"}}

Negative Responses

A quick example of how would the handler behave in case of an error:

(def rpc
  {:id 1
   :jsonrpc "2.0"
   :method :math/missing ;; wrong method
   :params [nil "a"]})

(def request
  {:request-method :post
   :uri "/"
   :headers {"content-type" "application/json"}
   :body (-> rpc json/generate-string .getBytes)})

(def response
  (-> (app request)
      (update :body json/parse-string true)))

{:status 200
 :body
 {:error
  {:code -32601 :message "Method not found" :data {:method "math/missing"}}
  :id 1
  :jsonrpc "2.0"}
 :headers {"Content-Type" "application/json; charset=utf-8"}}

Pay attention that the server always responds with the status code 200. This is the main deference from the REST approach. In RPC, HTTP is nothing else than just a transport layer. Its purpose is only to deliver messages without interfering into the pipeline. It’s up to you how to check if the RPC response was correct or not. However, the HTTP client package (see below) provides an option to raise an exception in case of error response.

Batch Requests in HTTP

If your configuration allows batch requests, you can send them via HTTP. For this, replace the rpc variable above with the vector of RPC maps. The result will be a vector of response maps.

(def rpc
  [{:id 1
    :jsonrpc "2.0"
    :method :math/sum
    :params [1 2]}
   {:id 2
    :jsonrpc "2.0"
    :method :math/sum
    :params [3 4]}])

(def request
  ...)

(def response
  ...)

{:status 200
 :headers {"Content-Type" "application/json; charset=utf-8"}
 :body ({:id 1 :jsonrpc "2.0" :result 3}
        {:id 2 :jsonrpc "2.0" :result 7})}

Everything said above for batch requests also apply to HTTP as well.

Configuration

Here is a list of HTTP options the library supports:

:http/method (default is :post) is an HTTP method to listen. POST is the one recommended by the RPC specification.
:http/path (default is "/") URI path to listen. You may specify something like "/api", "/rpc" or similar.
:http/health? (default is true) whether or not the health endpoint is available. When true, GET /health or GET /healthz requests receive an empty 200 OK response. This is useful for monitoring your server.
:http/middleware (default is the farseer.http/default-middleware vector) a list of HTTP middleware that get applied to the HTTP handler. See the next section.

Middleware & Authorization

By default, the handler gets wrapped into a couple of middleware. These are the standard wrap-json-body and wrap-json-response from the ring.middleware.json package. The first one is set up such that passing an incorrect JSON payload will return a proper RPC response (pay attention to the status 200):

(app {:request-method :post
      :uri "/"
      :headers {"content-type" "application/json"}
      :body (.getBytes "1aaa-")})

{:status 200,
 :headers {"Content-Type" "application/json"},
 :body "{\"jsonrpc\":\"2.0\",\"error\":{\"code\":-32700,\"message\":\"Invalid JSON was received by the server.\"}}"}

Note: we use the wrap-json-body middleware but not wrap-json-params to make it work with batch requests. As the payload is not a map, it cannot be merged with the :params field.

The :http/middleware parameter must be a vector of middleware. Each middleware is either a function or a vector of [function, arg2, arg3, ...]. In the second case, it will be applied to the handler as (apply function handler arg2, arg3, ...). For example, if a middleware takes additional params, you specify it as a vector [middleware, params].

You can bring your own logic into the HTTP pipeline by overriding the :http/middleware field. Here is a quick example how you protect the handler with Basic Auth:

;; ns imports
[farseer.http :as http]
[ring.middleware.basic-authentication :refer [wrap-basic-authentication]]

;; preparing a middleware stack
(let [fn-auth?
      (fn [user pass]
        (and (= "foo" user)
             (= "bar" pass)))

      middleware-auth
      [wrap-basic-authentication fn-auth? "auth" http/non-auth-response]

      middleware-stack [middleware-auth
                        http/wrap-json-body
                        http/wrap-json-resp]

      config*
      (assoc config :http/middleware middleware-stack)]
  ...)

Also, you can replace JSON middleware with the one that uses some other format like MessagePack, Transient or whatever.

HTTP Context

The function http/make-app also takes an additional context map. This map will be merged with the data the RPC function accepts when being called.

(def app
  (make-app config {:app/version "0.0.1"}))

(defn rpc-func [context params]
  {:message (str "The version is " (:app/version context))})

The HTTP handler adds the :http/request item into the context. This is the instance of the request map that the handler accepted. Having the request, you can handle some extra logic in your function. For example, some middleware supplements the request with the :user field, and you check if the user has permissions.

(defn some-rpc [context params]
  (let [{:http/keys [request]} context
        {:keys [user]} request]
    (when-not request
      (throw ...))))

Jetty Server

The Jetty sub-package allows you to run an RPC server using Jetty Ring adapter. Add it to the project:

;; deps
[com.github.igrishaev/farseer-jetty "..."]

;; require
(require '[farseer.jetty :as jetty])

All the Jetty config fields have default values, so we pass a minimal config we’ve been using so far.

(def server
  (jetty/start-server config))

;; #object[org.eclipse.jetty.server.Server 0x3e82fe49 "Server@3e82fe49{STARTED}[9.4.12.v20180830]"]

The default port is 8080. Now that your server is being run, test it with cURL:

curl -X POST 'http://127.0.0.1:8080/' \
  --data '{"id": 1, "jsonrpc": "2.0", "method": "math/sum", "params": [1, 2]}' \
  -H 'content-type: application/json' | jq

{
  "id": 1,
  "jsonrpc": "2.0",
  "result": 3
}

Pay attention to the content-type header. Without it, the request payload won’t be decoded and the call will fail.

To stop the sever, pass it to the stop-server function:

(jetty/stop-server server)

The start-server function also accepts a second optional argument which is a context map.

Configuration

:jetty/port (8080 by default) is the port to listen to.
:jetty/join? (false by default) is whether or not to wait for the server being stopped. When it’s true, the main thread hangs until you press Ctrl-C.
any other Jetty-related keys with the :jetty/ namespace, for example: :jetty/ssl-context, :jetty/max-threads and so on. The library will scan the config for the :jetty/-prefixed keys, select them, unqualify and pass to the run-jetty function.

With-server macro

The macro with-server temporary spawns an RPC server. It accepts a config, an optional context map and a block of code to execute.

(jetty/with-server [config {:foo 42}]
  (println 1 2 3))

Component

The Jetty package also provides a component object for use with the Component library. The jetty/component function creates a component. It takes a config and an optional context map.

(def jetty
  (jetty/component config {:some "field"}))

Now that you have an initiated component, you can start and stop it with functions from the Component library.

(require '[com.stuartsierra.component :as component])

(def jetty-started
  (component/start jetty))

;; The server starts working

(def jetty-stopped
  (component/stop jetty-started))

;; Now it's shut down.

Of course, it’s better to place the component into a system. One more benefit of a system is, all the dependencies will become a context map. For example, if your Jetty component depends on the database, cache, Kafka, and waterier else, you’ll have them all in the context map.

(defn make-system
  [rpc-config db-config cache-config]
  (component/system-using
   (component/system-map
    :cache (cache-component cache-config)
    :db-pool (db-component db-config)
    :rpc-server (jetty/component rpc-config))
   {:rpc-server [:db-pool :cache]}))

The function above will make a new system which consists from the RPC server, cache and database pooling connection. Once the system gets started, the context map of all RPC functions will have the :db-pool and :cache keys. Here is how you reach them in your RPC function:

(defn rpc-user-get-by-id
  [{:keys [db-pool cache]} [user-id]]
  (or (get-user-from-cache cache user-id)
      (get-user-from-db db-pool user-id)))

HTTP Stub

The Stub package provides a couple of macros for making HTTP RPC stubs. These are local HTTP servers that run on your machine. The difference with the Jetty package is that a stub returns a pre-defined data which is useful for testing.

Imagine you have a piece of code that interacts with two RPC endpoints. To make this code well tested, you need to cover the cases:

both sources work fine;
the first one works, the second returns an error;
the first one is unavailable, the second one works;
neither of them work.

The package provides the with-stub macro which accepts a config map and a block of code. The config must have the :stub/handlers field which is a map of method => result. For example:

(def config
  {:stub/handlers
   {:user/get-by-id {:name "Ivan"
                     :email "test@test.com"}
    :math/sum 42}})

As the Stub package works on top of Jetty, it takes into account all the Jetty keys. To specify the port number, pass the :jetty/port field to the config:

(def config
  {:jetty/port 18080
   :stub/handlers {...}})

In the example above, we defined the handlers such that the methods :user/get-by-id and :math/sum would always return the same response.

To run a server out from this config, there is the macro with-stub:

(stub/with-stub config
  ;; Execute any expressions
  ;; while the RPC server is running.
  )

While the server is running, you can reach it as you normally do with any HTTP client. If you send either :user/get-by-id or :math/sum requests to it, you’ll get the result you defined in the config. Quick check with cURL:

curl -X POST 'http://127.0.0.1:8080/' \
  --data '{"id": 1, "jsonrpc": "2.0", "method": "math/sum", "params": [1, 2]}' \
  -H 'content-type: application/json' | jq

{
  "id": 1,
  "jsonrpc": "2.0",
  "result": 42
}

Multiple Stub

There is a multiple version of this macro called with-stub. It’s useful when you interact with more than one RPC server at once. The macro takes a vector of config maps. For each one, it runs a local HTTP Stub. All of them get stopped once you exit the macro.

(stub/with-stubs [config1 config1 ...]
  ...)

Tests

To test you application with stubs, you need:

define a port for the HTTP stub, e.g. 18080;
pass this port to the stub config: :jetty/port ...;
wrap the testing code with the with-stub macro;
Aim the code which interacts with RPC at the local address like this one: http://127.0.0.1:18080/....

Having everything said above, you can easily check how does your application behave when getting a positive or a negative responses from an RPC server. You can check out the source code of the testing module as an example.

Negative Responses

The result of a method can be not only regular data but also a function. Inside it, you can raise an exception or even trigger something weird to imitate a disaster. For example, to divide by zero:

(def config
  {:stub/handlers
   {:some/failure
    (fn [& _]
      (/ 0 0))}})

This would lead to a real exception on the server side. Another way of triggering a negative response is to pass one of the predefined functions:

stub/invalid-request
stub/not-found
stub/invalid-params
stub/internal-error
stub/auth-error

Passing them will return an RPC error result. If you want to play the scenario when a user is not authenticated on the server, compose the config:

(def config
  {:stub/handlers
   {:user/get-by-id stub/auth-error}})

HTTP Client

The Client package is to communicate with an RPC Server by HTTP protocol. It relies on clj-http library for making HTTP requests. Add it to the project:

;; deps
[com.github.igrishaev/farseer-client ...]

;; module
[farseer.client :as client]

To reach an RPC server, first you create an instance of the client. This is done with the make-client function which accepts the config map:

(def config-client
  {:http/url "http://127.0.0.1:18080/"})

(def client
  (client/make-client config-client))

There is only one mandatory field in the config: the :http/url one which is the endpoint of the server. Other fields have default values.

For further experiments we will spawn a local Jetty RPC server and will work with it using the client.

(def config
  {:jetty/port 18080
   :rpc/handlers
   {:math/sum
    {:handler/function #'rpc-sum
     :handler/spec-in :math/sum.in
     :handler/spec-out :math/sum.out}}})

(def server
  (jetty/start-server config))

Once you have the client, make a request with the client/call function. It accepts the client, method, and optional parameters.

(def response
  (client/call client :math/sum [1 2]))

;; {:id 81081, :jsonrpc "2.0", :result 3}

The parameters might be either a vector or map. If the method doesn’t accept parameters, you may omit them.

;; map params
(client/call client :user/create
             {:name "Ivan" :email "test@test.com"})

;; no params
(client/call client :some/side-effect)

An example of a negative response:

(client/call client :math/sum [nil "a"])

{:error
 {:code -32602,
  :message "Invalid params",
  :data
  {:explain
   "nil - failed: number? in: [0] at: [0] spec: :math/sum.in\n\"a\" - failed: number? in: [1] at: [1] spec: :math/sum.in\n",
   :method "math/sum"}},
 :id 73647,
 :jsonrpc "2.0"}

You won’t get an exception; the result shown above is just data. If you prefer exceptions, you can adjust the client configuration (see below).

Configuration

The following fields affect the client’s behaviour.

:rpc/fn-before-send (default is identity) a function which is called before the HTTP request gets sent to the server. It accepts the Clj-http request map and should return it as well. The function useful for signing requests, authentication and so on.
:rpc/fn-id (default is :id/int) determines an algorithm for generating IDs. The :id/int value means an ID will be a random integer; :id/uuid stands for a random UUID. You can also pass a custom function of no arguments that must return either an integer or a string.
:rpc/ensure? (default is false) when false, return the body of the response as is. When true, either return the :result field of the body or throw an exception if the :error field presents (see below).
:http/method (default is :post) an HTTP method for request.
:http/headers (default is {:user-agent "farseer.client"}) a map of HTTP headers.
:http/as (default is :json) how to treat the response body.
:http/content-type (:json) how to encode the request body.

The HTTP package takes into account all the keys prefixed with the :http/ namespace. These are the standard Clj-http keys, .e.g :http/socket-timeout, :http/throw-exceptions? and others, so you configure the HTTP part as you want. When making a request, the client scans the config for the :http/-prefixed keys, selects them, removes the namespace and passes to the clj-http/request function as a map.

The :conn-mgr/ keys specify options for the connection manager:

:conn-mgr/timeout
:conn-mgr/threads
:conn-mgr/default-per-route
:conn-mgr/insecure?

and others. They have default values copied from Clj-http. The connection manager is not created by default. You need to setup it manually (see below).

Handling Responses

By default, calling the server just returns the body of the HTTP response. It’s up to you how to handle the :result and :error fields. Sometimes, the good old exception-based approach is convenient: you either get a result or an error pops up.

The :rpc/ensure? option is exactly for that. When it’s false, you get a parsed body of the HTTP response. When it’s true, the following logic takes control:

for a positive response (no :error field) you’ll get the content of the :result field. For example:

(def config-client
  {:rpc/ensure? true
   :http/url "http://127.0.0.1:18080/"})

(def client
  (client/make-client config-client))

(client/call client :math/sum [1 2])
;; 3

For a negative response, you’ll get an exception:

(client/call client :math/sum [1 "two"])

17:04:34.780 INFO  farseer.handler - RPC error, id: 94415, method: math/sum, code: -32602, message: Invalid params

Unhandled clojure.lang.ExceptionInfo
RPC error, id: 94415, method: :math/sum, code: -32602, message: Invalid
params
#:rpc{:id 94415,
      :method :math/sum,
      :code -32602,
      :message "Invalid params",
      :data
      {:explain "\"two\" - failed: number? in: [1] at: [1] spec: :math/sum.in\n",
       :method "math/sum"}}
             client.clj:  122  farseer.client/ensure-handler
             client.clj:   97  farseer.client/ensure-handler
             client.clj:  148  farseer.client/make-request
             client.clj:  128  farseer.client/make-request
             client.clj:  187  farseer.client/call
             client.clj:  179  farseer.client/call

At the moment, the :rpc/ensure? option doesn’t affect batch requests (see below).

Auth

Handling authentication for the client is simple. Clj-http already covers most of the authentication types, so you only need to pass proper options to the config. If the server is protected with Basic auth, you extend the config with the :http/basic-auth field:

(def config-client
  {:http/url "http://127.0.0.1:18080/"
   :http/basic-auth ["user" "password"]})

For oAuth2, you pass another key:

(def config-client
  {:http/url "http://127.0.0.1:18080/"
   :http/oauth-token "***********"})

If the server requires a constant token, you put it directly into headers:

(def config-client
  {:http/url "http://127.0.0.1:18080/"
   :http/headers {"authorization" "Bearer *********"}})

Finally, the :rpc/fn-before-send parameter allows your to do everything with the request before it gets sent to the server. There might be a custom function which supplements the request with additional headers that are calculated on the fly. For example:

(defn sign-request
  [{:as request :keys [body]}]
  (let [body-hash (calc-body-hash body)
        sign (sign-body-hash body-hash "*******")
        header (str "Bearer " sign)]
    (assoc-in request [:headers "authorization"] header)))

(def config-client
  {:http/url "http://127.0.0.1:18080/"
   :rpc/fn-before-send sign-request})

Notifications

A notification is when you’re not interested in the response from the server. To send a notification, use the client/notify function. Its signature looks the same: the client, method, and optional params. The result will be nil.

(client/notify client :math/sum [1 2])
;; nil

Batch Requests

To send batch requests, there is the client/batch function. It takes the client and a vector of tasks. Each task is a pair of (method, params).

(client/batch client
              [[:math/sum [1 2]]
               [:math/sum [2 3]]
               [:math/sum [3 4]]])

[{:id 51499 :jsonrpc "2.0" :result 3}
 {:id 45992 :jsonrpc "2.0" :result 5}
 {:id 84590 :jsonrpc "2.0" :result 7}]

Some important notes on batches:

There will be only one HTTP request.
The order of the result maps always match the order of the tasks.
If one of the tasks fails, you’ll get a negative map for it. The whole request won’t fail.

(client/batch client
              [[:math/sum [1 2]]
               [:math/sum ["aa" nil]]
               [:math/sum [3 4]]])

[{:id 75623 :jsonrpc "2.0" :result 3}
 {:error
  {:code -32602
   :message "Invalid params"
   :data
   {:explain "\"aa\" - failed: number? in: [0] at: [0] spec: :math/sum.in\nnil - failed: number? in: [1] at: [1] spec: :math/sum.in\n"
    :method "math/sum"}}
  :id 43075
  :jsonrpc "2.0"}
 {:id 13160 :jsonrpc "2.0" :result 7}]

The :rpc/ensure? option doesn’t apply to batch requests (which is a subject to change in the future).

Sometimes, you want one of the tasks in a batch to be a notification. To make a task a notification, prepend its vector with the ^:rpc/notify metadata tag:

(client/batch client
              [[:math/sum [1 2]]
               ^:rpc/notify [:math/sum [2 3]]
               [:math/sum [3 4]]])

[{:id 54810 :jsonrpc "2.0" :result 3}
 {:id 34377 :jsonrpc "2.0" :result 7}]

Connection Manager (Pool)

Clj-http offers a connection manager for HTTP requests. It’s a pool of open TCP connections. Sending requests within a pool is much faster then opening and closing connections every time. The package provides some bits to handle connection manager for the client.

The function client/start-conn-mgr takes a client and returns it with the new connection manager associated under the :http/connection-manager key. If you pass the new client to the client/call function, it will take the manager into account, and the request will work faster.

The function considers the keys which start with the :conn-mgr/ namespace. These keys become a map of standard parameters for connection manager.

(def config-client
  {:conn-mgr/timeout 5
   :conn-mgr/threads 4
   :http/url "http://127.0.0.1:18080/"})

(def client
  (-> config-client
      client/make-client
      client/start-conn-mgr))

The opposite function client/stop-conn-mgr stops the manager (if present) and returns the client without the key.

(client/stop-conn-mgr client)

The macro client/with-conn-mgr enables the connection manager temporary. It takes a binding form and a block of code to execute. Inside the macro, the client is bound to the first symbol from the vector form.

;; a client without a pool
(def client
  (client/make-client config-client))

;; temporary assing a pool
(client/with-conn-mgr [client-mgr client]
  (client/call client-mgr :math/sum [1 2]))

Component

Since the client might have a state (a connection manager), you can put it into the system. There is a function client/component which returns an HTTP client charged with the start and stop methods. These methods turn on and off connection pool for the client.

;; no pool yet
(def client
  (client/component config-client))

;; enabling the pool
(def client-started
  (component/start client))

;; closing the pool
(component/stop client-started)

Documentation Builder

The config map for the server has enough data to be rendered as a document. It would be nice to pass it into a template and generate a file each time you build or the application. The Docs package serves exactly for this purpose.

Add the com.github.igrishaev/farseer-doc library into your project:

;; deps
[com.github.igrishaev/farseer-doc ...]

;; ns
[farseer.doc :as doc]

Pay attention that generating a docfile is usually a separate task, but not a part of business logic. That’s why the application must not include that library in production. The :dev-specific dependencies would be a better place for this package.

Configuration

To generate a doc file, you extend the server config with the keys that have :doc/ namespace. Here is an example:

(def config
  {:doc/title "My API"
   :doc/description "Long API Description"

   :rpc/handlers
   {:user/delete
    {:doc/title "Delete a user by ID"
     :doc/description "Long text for deleting a user."
     :handler/spec-in pos-int?
     :handler/spec-out (s/keys :req-un [:api/message])}

    :user/get-by-id
    {:doc/title "Get a user by ID"
     :doc/description "Long text for getting a user."
     :doc/ignore? false
     :doc/resource "docs/user-get-by-id.md"
     :handler/spec-in int?
     :handler/spec-out
     (s/map-of keyword? (s/or :int int? :str string?))}

    :hidden/api
    {:doc/title "Non-documented API"
     :doc/ignore? true
     :handler/spec-in any?
     :handler/spec-out any?}}})

The the list of the fields used for documentation:

:doc/title (string). A title of an API or an RPC method.
:doc/description (string). A description of an API or an RPC method.
:doc/resource (string). A path to a resource with the detailed text with examples, edge cases and so on. Useful for large chunks of text.
:doc/endpoint (string). An URL of this RPC server.
:doc/ignore? (boolean, false by default). When true, the method is not included into the documentation.
:doc/sorting (keyword, :method or :title). How to sort RPC methods. The :method keyword means to sort by machine names, e.g. :user/get-by-id. The :title means to sort by the :doc/title field, e.g. “Get user by ID”.

Building

Once you have a documentation-powered config, render it with the generate-doc function:

(doc/generate-doc
   config
   "templates/farseer/default.md"
   "dev-resources/default-out.md")

This function takes a config map, a resource template and a path of the output file. The Doc package provides the default Markdown template which can be found by the path "templates/farseer/default.md".

In your project, most likely you create a dev namespace with this function that builds the documentation file. Every time the application gets run on CI, you generate a file and host it somewhere.

Demo

You can checkout a real demo generated by the test module. The file lists all the non-ignored methods and their specs. The specs are put under collapsible items as sometimes they might be huge.

Selmer & Context

The Doc package uses the great Selmer library which is inspired by Django Templates. You can pass your own template, and not only Markdown one, but HTML, AsciiDoc, or LaTeX. The template might have any graphic elements, your logo, JavaScript, and so on.

The Doc package passes the config not directly but with transformation. Here is an example of the context map that you have when rendering a template. Note that all the keys are free from namespaces. The :handlers field is not a map but a vector of maps sorted according to the :doc/sorting option.

{:title "My API"
 :description "Long API Description"
 :resource nil
 :handlers
 ({:method "user/delete"
   :title "Delete a user by ID"
   :description "Long text for deleting a user."
   :resource nil
   :spec-in {:type "integer" :format "int64" :minimum 1}
   :spec-out
   {:type "object"
    :properties {"message" {:type "string"}}
    :required ["message"]}}
  {:method "user/get-by-id"
   :title "Get a user by ID"
   :description "Long text for getting a user."
   :resource "\n### Get user by ID examples\n\n........"
   :spec-in {:type "integer" :format "int64"}
   :spec-out
   {:type "object"
    :additionalProperties
    {:anyOf [{:type "integer" :format "int64"} {:type "string"}]}}})}

Rendering Specs

The :handler/spec-in and :handler/spec-out fields get transformed to JSON Schema using the Spec-tools library. You may see the result of transformation in the context map above. For more control of transformation, check out the manual page from the Spec-tools repository.

To render the spec in a template, use the json-pretty filter. It turns the Clojure data into a JSON string being well printed. To prevent quoting some symbols, add the safe filter to the end. Everything together gives the following snippet:

Pay attention to the empty lines before and after the JSON code block. Without them, GitHub renders the content in a weird way.

Ideas & Further Development

It would be nice to:

Keep the entire server config in an EDN file. The functions should be resolved by their full symbols.
Provide a nested map like method => overrides. With this map, one could specify custom options for specific methods. For example, to enable batch requests in common, but disallow them for specific methods.

Develop a browser version of the client. The module would rely on Fetch API.
Create a wrapper for re-frame. Instead of calling functions, one triggers events.

← Сага о DRY и зависимостях

И ещё немного книг →