Writing Clojure living-cookbooks

I've been thinking about documentation recently: as our company backend repository grows, there are patterns and usage emerging, and lot of things to document to ease the new developers onboarding, the sharing of knowledge and the standardization of practices.

There are some existing approaches to answer this problem:

• docstrings in code (namespaces, functions), with optional html generation (Codox for example)
• manually written documentation (ex: some markdown files in a docs folder)
• notebooks (a mix of html and interactive code blocks with support for custom visualizations)

But I've been quite dissatisfied : docstrings are useful when you code, but do not help grasping the big picture, manually written documentations quickly suffer from drifts when the code evolves, and notebooks are great but I didn't find a quick and easy way to introduce them in our dev environment yet (I had good hope with clerk but had too much problems with its cache system).

So I ended up creating a small script to generate markdown cookbooks based on compiled Clojure code.

What I wanted

• markdown-based
• always valid cookbook code (compilation should pass)
• automatic injection of evaluation result for each code block, For example, if I write (assoc {} :k "v"), I would get the following documentation

(assoc {} :k "v")
; =>
{:k "v"}

Here an example of such a cookbook, the Clojure version:

(ns cookbooks.maps
  (:require [clj-time.core :as t]))

;; # Clojure maps walkthrough
;; ## Basics
;; Maps are a central piece in Clojure programs
(def m {:name "mapz" "key" "val" {} "keys can be anything"})
;; You can add new keys
(assoc m :updated-at (t/now))
;; Note that maps are **immutable**
m

The rendered markdown on Github:

"Show me the code"

I've used rewrite-clj to parse the Clojure cookbook files, with a zipper to walk through the comments and Clojure forms in the file in order to evaluate and render them as markdown.

Note that to be able to evaluate the Clojure forms in the cookbook namespace (and not in the script one), I bind the *ns* var (the current namespace) to the cookbook one.

Then: (compile-cookbooks "dev/cookbooks" "doc/cookbooks")

At the end, it's very basic but already useful, and it's only 62 lines of code 🙌
Feel free to reuse this code if you find it useful.

Bonus: it happens to work with Babashka too

(ns dev.docgen
  (:require [clojure.pprint :refer [pprint]]
            [rewrite-clj.zip :as z]
            [clojure.string :as str]
            [clojure.java.io :as io])
  (:import (java.io File)))

(defn -form->markdown
  "Clojure forms are evaluated and the result is automatically inserted in the generated markdown.
  There is a special handling for 'def, we show their var binding instead"
  [zloc]
  (str "```clj" \newline
       (z/string zloc)
       \newline "; =>" \newline
       (if (= 'def (z/sexpr (z/down zloc)))
         ;; evaluate the def and prints its binding instead of the var
         (do (load-string (z/string zloc))
             (with-out-str (pprint (load-string (z/string (as-> (z/down zloc) def-zloc
                                                                ;; go to the var name
                                                                (z/find-next def-zloc #(symbol? (z/sexpr %)))))))))
         ;; not a def, evaluate the form as is
         (with-out-str (pprint (load-string (z/string zloc)))))
       "```" \newline))

(defn -comment->markdown
  "comments are in the shape:
  ;; Some optional **markdown** formatting
  So simply remove the starting '; chars and whitespaces to convert to markdown"
  [zloc]
  (str/replace (z/string zloc) #"^;+[ \t]+" ""))

(defn file->markdown [file]
  (let [zloc (z/of-file file {:track-position? true})
        ;; get the top namespace to bind the current ns
        file-ns (when (= 'ns (z/sexpr (z/down zloc)))
                  (some-> (z/find-next (z/down zloc) #(symbol? (z/sexpr %)))
                          z/sexpr))]
    (assert file-ns "the file should start with a ns form")
    (binding [*ns* (create-ns file-ns)]
      (->> zloc
           (iterate z/right*)
           (take-while (complement z/end?))
           (map (fn [zloc]
                  (try
                    (cond
                      (z/sexpr-able? zloc) (-form->markdown zloc)
                      (z/whitespace-or-comment? zloc) (-comment->markdown zloc)
                      :else (z/string zloc))
                    (catch Exception ex
                      (println "error at position" (z/position zloc)
                               (.getMessage ex)
                               (some-> (.getCause ex) .getMessage)
                               (z/string zloc))))))
           (str/join)))))

(defn compile-cookbooks [input-path output-path]
  (assert (.exists (io/file input-path)) (format "the folder %s does not exist" input-path))
  (doseq [f (file-seq (io/file input-path))
          :when (.isFile f)
          :let [markdown (file->markdown (.getPath f))
                output-name (str/replace (.getName ^File f) #".clj[cs]?$" ".md")]]
    (spit (str output-path "/" output-name) markdown)))