Partial pages are nice, but being able to write in markdown would be even
better. Cegdown is a Clojure wrapper
for Pegdown, a popular Java library for rendering markdown. Add it to your
project.clj:
:dependencies [[org.clojure/clojure "1.5.1"]
[stasis "1.0.0"]
[ring "1.2.1"]
[hiccup "1.0.5"]
[me.raynes/cegdown "0.1.1"]]
Now, add it to the namespace form in src/cjohansen_no/web.clj. While we're at
it, we will require the Clojure string library as well (we'll use it shortly):
(ns cjohansen-no.web
(:require [clojure.java.io :as io]
[clojure.string :as str]
[hiccup.page :refer [html5]]
[me.raynes.cegdown :as md]
[stasis.core :as stasis]))
Now we will add a function to render every page in resources/md as
markdown. It will be very similar to the partials we did before, but now with
markdown rendering as well. Because we don't want ".md" as part of the URL for
these pages, we will map the keys as well.
(defn markdown-pages [pages]
(zipmap (map #(str/replace % #"\.md$" "") (keys pages))
(map #(layout-page (md/to-html %)) (vals pages))))
The #( ) form is a function literal. Inside it, %
refers to the first argument. In the above example, the following are identical:
#(str/replace % #"\.md$" "")
(fn [path] (str/replace path #"\.md$" ""))
I wrote more on the anonymous function literal in
a separate post.
The final step is to add the new page source to our map:
(defn get-pages []
(stasis/merge-page-sources
{:public
(stasis/slurp-directory "resources/public" #".*\.(html|css|js)$")
:partials
(partial-pages (stasis/slurp-directory "resources/partials" #".*\.html$"))
:markdown
(markdown-pages (stasis/slurp-directory "resources/md" #"\.md$"))}))
Add the following to resources/md/my-first-post.md:
# My first post
It's pretty short for now.
Restart the server again (we added more new dependencies, remember?) When you
have, /my-first-post should present you with your brief, but lovely blog post.
Any self-respecting tech blog needs nice syntax highlighting for code blocks.
When it comes to syntax highlighting, Pygments is the
bee's knees. It supports just about any language you can think of, there's a
bunch of color themes around and it is stable and resillient. It is also the
library used to highlight code on GitHub.
Clygments is a Clojure interface to it
(which uses Jython; Pygments is a Python library).
We will add syntax highlighting as a post-processing step for HTML. This way, we
can support syntax highlighting for full static pages in the public directory,
partial pages and pages rendered from markdown. To do this, we will use another
templating library for Clojure, enlive.
Actually, enlive is more than just a templating library. As you will see, it can
be used to transform documents in various interesting ways.
Let's add a
fenced code block
to our markdown file:
# My first post
It's pretty short for now. Here's our project.clj:
```clj
(defproject cjohansen-no "0.1.0-SNAPSHOT"
:description "cjohansen.no source code"
:url "http://cjohansen.no"
:license {:name "BSD 2 Clause"
:url "http://opensource.org/licenses/BSD-2-Clause"}
:dependencies [[org.clojure/clojure "1.5.1"]
[stasis "1.0.0"]
[ring "1.2.1"]
[hiccup "1.0.5"]
[me.raynes/cegdown "0.1.1"]]
:ring {:handler cjohansen-no.web/app}
:profiles {:dev {:plugins [[lein-ring "0.8.10"]]}})
```
In order for this work, we need to inform cegdown that we want to
enable the fenced code blocks extension. While we're at it, we'll enable a
couple of other useful extensions as well:
(def pegdown-options [:autolinks :fenced-code-blocks :strikethrough])
(defn render-markdown-page [page]
(layout-page (md/to-html page pegdown-options)))
(defn markdown-pages [pages]
(zipmap (map #(str/replace % #"\.md$" "") (keys pages))
(map render-markdown-page (vals pages))))
Reloading the blog post will show you how the fenced code blocks are rendered:
<pre><code class="clj">...</code></pre>
We will now use enlive to extract this piece of markup and replace it with the
version highlighted by clygments. First off, add the new dependencies to
project.clj (remember to restart the server!)
:dependencies [[org.clojure/clojure "1.5.1"]
[stasis "1.0.0"]
[ring "1.2.1"]
[hiccup "1.0.5"]
[me.raynes/cegdown "0.1.1"]
[enlive "1.1.5"] [clygments "0.1.1"]]
Add a new namespace (i.e., file) to the project. Copy the following code into
src/cjohansen_no/highlight.clj:
(ns cjohansen-no.highlight
(:require [clojure.java.io :as io]
[clygments.core :as pygments]
[net.cgrand.enlive-html :as enlive]))
Enlive has many tricks up its sleave. Perhaps the most interesting one is the
somewhat confusingly named sniptest. It takes some HTML as a
string, and selector/function pairs. It will then;
- Parse the HTML
- Find all nodes matching the selector
- Call the corresponding function once for every match
- Replace the node with the result of calling the function
- Return the transformed HTML as a string
(defn highlight-code-blocks [page]
(enlive/sniptest page
[:pre :code] highlight
[:pre :code] #(assoc-in % [:attrs :class] "codehilite")))
This function will find every code element inside a
pre element and pass it through the highlight function
(to be shown). Then, it will add a class name to the same elements. In practice,
you would probably do both in the highlight function, but this
allows me to illustrate how you can perform multiple transformations in one go.
The "codehilite" class name just happens to be class name used by the Pygments
CSS themes available here (we
will include this later). Add the second function:
(defn- highlight [node]
(let [code (->> node :content (apply str))
lang (->> node :attrs :class keyword)]
(pygments/highlight code lang :html)))
The dash in defn- means that this function is private, and only
referrable within the current namespace. The nodes that enlive operate on are
maps like this:
{:tag :code
:attrs {:class "clj"}
:content [...]}
The content is a list of new nodes and/or strings.
->> is the
thread-last macro.
It takes any number of arguments, threads values from left to right; given
(->> a (b 1 2) c), it will take the value a,
pass it as the last argument to b (i.e. (b 1 2 a)),
pass the return value of that expression as the last argument to c,
and finally return the result of that. So the above line:
(->> node :content (apply str))
Is the same as this:
(apply str (:content node))
I wrote more on threading macros in a separate post.
Using a keyword as a function is one way to look up that key in a map. The apply
call means to call str with the following list of arguments, as individual
arguments, not one list (e.g. (fn a b c), not (fn [a b c])).
To preview the highlighting in the browser, we need to make some changes to the
web namespace. Start by updating the namespace form to pull in our new
dependency:
(ns cjohansen-no.web
(:require [cjohansen-no.highlight :refer [highlight-code-blocks]] [clojure.java.io :as io]
[clojure.string :as str]
[hiccup.page :refer [html5]]
[me.raynes.cegdown :as md]
[stasis.core :as stasis]))
Syntax highlighting should apply to all pages. A good place to do it is between
our old get-pages function and Stasis' rendering. We will do this
by adding a prepare-pages function:
(defn prepare-pages [pages]
(zipmap (keys pages)
(map #(highlight-code-blocks %) (vals pages))))
Again, we use zipmap to produce a new map where the keys are untouched, but the
values have been mapped. To make Stasis run through this, rename
get-pages to get-raw-pages, and add a new get-pages:
(defn get-raw-pages []
(stasis/merge-page-sources
{:public
(stasis/slurp-directory "resources/public" #".*\.(html|css|js)$")
:partials
(partial-pages (stasis/slurp-directory "resources/partials" #".*\.html$"))
:markdown
(markdown-pages (stasis/slurp-directory "resources/md" #"\.md$"))}))
(defn prepare-pages [pages]
(zipmap (keys pages)
(map #(fn [req] (highlight-code-blocks %)) (vals pages))))
(defn get-pages []
(prepare-pages (get-raw-pages)))
Reloading the markdown page should show you that what we've done so far both
kinda worked and kinda didn't.
We are getting highlighted code. However, Pygments includes some unwanted
wrapping markup. As we're already inside a pre element, the result
is not quite as desired. To fix this we will use enlive once more to massage the
output from Pygments. The output includes a wrapper div and a pre, let's extract
just the code:
(defn- extract-code
[highlighted]
(-> highlighted
java.io.StringReader.
enlive/html-resource
(enlive/select [:pre])
first
:content))
(defn- highlight [node]
(let [code (->> node :content (apply str))
lang (->> node :attrs :class keyword)]
(assoc node :content (-> code
(pygments/highlight lang :html)
extract-code))))
Enlive's select function selects elements from a document, but
unlike sniptest, it does not accept a string. Instead, we must go
through its html-resource function, which only accepts input
streams. The end result is that we do a select on what Pygments gives us in
order to get just the highlighted code. Refreshing the blog post shows that it
works as expected.
The -> is the thread-first macro. It works like thread-last,
except it threads values as the first argument to the next function. The above
example could be written in either of these ways, but most people find the
threading form to be the easiest on the eyes:
(-> highlighted
java.io.StringReader.
enlive/html-resource
(enlive/select [:pre])
first
:content)
(:content (first (enlive/select (enlive/html-resource
(java.io.StringReader. highlighted))
[:pre])))
To add some styling, pick a CSS file from
the suggested themes repo, and
load it onto the page. Update the layout-page in web.clj to look
like this:
(defn layout-page [request page]
(html5
[:head
[:meta {:charset "utf-8"}]
[:meta {:name "viewport"
:content "width=device-width, initial-scale=1.0"}]
[:title "Tech blog"]
[:link {:rel "stylesheet" :href "/pygments-css/autumn.css"}]]
[:body
[:div.logo "cjohansen.no"]
[:div.body page]]))
As we're adding more features to our site, it is becoming apparent that
processing all the pages to completion on every request isn't ideal. Fixing this
is quite easy with Stasis, because we can give Stasis functions instead of
strings, and then Stasis will call the function to build a particular page only
when it needs to render that specific page.
To make our pages lazy, update prepare-pages to replace the values
with functions instead of strings of highlighted HTML. The function should take
one argument, the request map.
(defn prepare-pages [pages]
(zipmap (keys pages)
(map #(fn [req] (highlight-code-blocks %)) (vals pages))))
By having the function literal return a new function that takes one argument, we
have significantly improved performance for our development server.
Now that we have a blog with syntax highlighting, we need to start thinking
about delivery. Fast webpages beat slow ones on all sorts of metrics. One way to
make our site faster is by employing various frontend asset optimization
techniques. For this purpose, there is (among others)
Optimus. We will use it to:
- Concatenate CSS and JavaScript files
- Minify CSS and JavaScript files
- Serve CSS and JavaScript from cache-friendly URLs
A "cache friendly" URL is one that is unique every time the contents of the URL
changes. This way we can serve assets with aggressive cache headers, and users
will only need to download them once. The next time we deploy, if the assets
have changed, they will have a new URL. Optimus facilitates this by providing
some functions to help us link to assets.
We will start by adding Optimus as a dependency in project.clj. Remember to
restart the server after doing this.
(defproject cjohansen-no "0.1.0-SNAPSHOT"
:description "cjohansen.no source code"
:url "http://cjohansen.no"
:license {:name "BSD 2 Clause"
:url "http://opensource.org/licenses/BSD-2-Clause"}
:dependencies [[org.clojure/clojure "1.5.1"]
[stasis "1.0.0"]
[ring "1.2.1"]
[hiccup "1.0.5"]
[me.raynes/cegdown "0.1.1"]
[enlive "1.1.5"]
[clygments "0.1.1"]
[optimus "0.14.2"]] :ring {:handler cjohansen-no.web/app}
:profiles {:dev {:plugins [[lein-ring "0.8.10"]]}})
Now update the web.clj namespace form to require some functions from Optimus:
(ns cjohansen-no.web
(:require [optimus.assets :as assets] [optimus.optimizations :as optimizations] [optimus.prime :as optimus] [optimus.strategies :refer [serve-live-assets]] [cjohansen-no.highlight :refer [highlight-code-blocks]]
[clojure.java.io :as io]
[clojure.string :as str]
[hiccup.page :refer [html5]]
[me.raynes.cegdown :as md]
[stasis.core :as stasis]))
Instead of having Stasis serve the files in public, we will hand them to Optimus
as assets. We will define a separate function for these assets, as it makes for
a natural place to add further assets and/or bundles of assets later:
(defn get-assets []
(assets/load-assets "public" [#".*"]))
If your CSS files use @import, Optimus will (by default) take care
to inline the import, so there is no need to define bundles at this point. Refer
to the Optimus readme for more
details.
To make our app use the new assets, we will change the app
function:
(def app
(optimus/wrap (stasis/serve-pages get-pages)
get-assets
optimizations/all
serve-live-assets))
The call to stasis/serve-pages returns a function (a Ring app,
remember?) optimus/wrap returns another function with the same
signature that wraps the original one. We pass it the function to get all our
assets, optimization rules (a function) and a strategy for serving the assets
(also a function). optimizations/all is a grab bag of every trick
Optimus knows:
- Minify JavaScript
- Minify CSS
- Inline CSS imports
- Concatenate bundles
- Add cache-bust expires headers (replace URL references with generated unique ones)
- Add last-modified headers
You are free to pick and choose from this list if you want, but for most cases,
optimizations/all is what you want.
Lastly, we employed Optimus' serve-live-assets strategy, which
means that Optimus will read assets from disk on every request. This is useful
in development mode, but in a production setting, you would typically use one
that's less resource intensive, like serve-frozen-assets.
Create a CSS file and make sure it gets included from the page layout.
import url(../pygments-css/autumn.css);
body {
font: 16px Helvetica, arial, freesans, clean, sans-serif;
line-height: 1.5;
margin: 0 10px;
}
Refreshing the blog in the browser should display the same page as before.
However, if you hit the CSS file directly, you will find that Optimus has done
what it can to optimize serving it.
There is one final thing to take care of. In production, we can configure our
web server to serve assets with a far future expires header. But in order for
that to be safe, we need distinct URLs for every change to the file. Let's add
another require to the web namespace form:
(ns cjohansen-no.web
(:require [optimus.assets :as assets]
[optimus.link :as link] [optimus.optimizations :as optimizations]
[optimus.prime :as optimus]
[optimus.strategies :refer [serve-live-assets]]
[cjohansen-no.highlight :refer [highlight-code-blocks]]
[clojure.java.io :as io]
[clojure.string :as str]
[hiccup.page :refer [html5]]
[me.raynes.cegdown :as md]
[stasis.core :as stasis]))
With this in place, we can use Optimus to generate the link to the CSS file.
However, to do that, it needs access to the request map, so we need to change a
few things. We will start with the layout-page function:
(defn layout-page [request page]
(html5
[:head
[:meta {:charset "utf-8"}]
[:meta {:name "viewport"
:content "width=device-width, initial-scale=1.0"}]
[:title "Tech blog"]
[:link {:rel "stylesheet" :href (link/file-path request "/styles/main.css")}]]
[:body
[:div.logo "cjohansen.no"]
[:div.body page]]))
Both the partial-pages and markdown-pages need to pass
the request to layout-page. If we change them to return functions,
Stasis will call those functions with the request.
(defn partial-pages [pages]
(zipmap (keys pages)
(map #(fn [req] (layout-page req %)) (vals pages))))
Remember that #( ) is a function literal, so the mapping function
here is a function that returns another function (which takes a request map as
its only argument). The markdown generation is similar, but includes the
additional step of running the content through cegdown:
(defn markdown-pages [pages]
(zipmap (map #(str/replace % #"\.md$" "") (keys pages))
(map #(fn [req] (layout-page req (md/to-html % pegdown-options)))
(vals pages))))
Previously these maps contained strings, so we need to update their use now that
they're functions. We start with a new function:
(defn prepare-page [page req]
(-> (if (string? page) page (page req))
highlight-code-blocks))
This function takes a page and a request. Because every page will go through
this function, some will be strings, and some will be functions. If the page is
a string, we leave it untouched, and if it's a function, we call it with the
request map and pipe the result through a series of post-processing steps.
There's currently only one step, but the threading macro has set us up for
easily adding more steps later. The final piece of the puzzle is to update the
prepare-pages function:
(defn prepare-pages [pages]
(zipmap (keys pages)
(map #(partial prepare-page %) (vals pages))))
Again, we use the function literal #( ). We also use
partial. This returns a new function that knows the first argument
to pass to prepare-page. When you call this new function with one
argument (a request), the prepare-page function will be called with
a page and a request. Update the page in the browser, view source and note that
Optimus has now given our CSS file a nice and unique URL.
So far we've only surfed the server version, but the whole point of this
exercise was to create something that can work as a static site. To dump the
file to disk, start by adding a custom Leiningen build alias in project.clj:
(defproject cjohansen-no "0.1.0-SNAPSHOT"
:description "cjohansen.no source code"
:url "http://cjohansen.no"
:license {:name "BSD 2 Clause"
:url "http://opensource.org/licenses/BSD-2-Clause"}
:dependencies [[org.clojure/clojure "1.5.1"]
[stasis "1.0.0"]
[ring "1.2.1"]
[hiccup "1.0.5"]
[me.raynes/cegdown "0.1.1"]
[enlive "1.1.5"]
[clygments "0.1.1"]
[optimus "0.14.2"]]
:ring {:handler cjohansen-no.web/app}
:aliases {"build-site" ["run" "-m" "cjohansen-no.web/export"]} :profiles {:dev {:plugins [[lein-ring "0.8.10"]]}})
This configures lein build-site as a command that will invoke the
export function in the cjohansen-no.web namespace.
Stasis gives us what we need to build this function:
(def export-dir "dist")
(defn export []
(stasis/empty-directory! export-dir)
(stasis/export-pages (get-pages) export-dir))
While this won't technically fail, it also won't be the whole picture. Had we
not been using Optimus, this would be OK. Since we are using Optimus, we want to
make sure the export is optimized as well. The fix is simple; tell Optimus to
dump assets for us, and add an entry to Stasis' request map extensions so that
Optimus finds the assets. First update the namespace form to require the Optimus
export library:
(ns cjohansen-no.web
(:require [optimus.assets :as assets]
[optimus.export] [optimus.link :as link]
[optimus.optimizations :as optimizations]
[optimus.prime :as optimus]
[optimus.strategies :refer [serve-live-assets]]
[cjohansen-no.highlight :refer [highlight-code-blocks]]
[clojure.java.io :as io]
[clojure.string :as str]
[hiccup.page :refer [html5]]
[me.raynes.cegdown :as md]
[stasis.core :as stasis]))
Then update the export function:
(defn export []
(let [assets (optimizations/all (get-assets) {})]
(stasis/empty-directory! export-dir)
(optimus.export/save-assets assets export-dir)
(stasis/export-pages (get-pages) export-dir {:optimus-assets assets})))
Now, on the command line, run lein build-site. After a short while
you will find your entire site ready to ship in the dist directory.
This can be directly rsynced to your server.
Building sites like we've done in this post opens for various interesting ways
of programatically performing tests and health checks. I will show you two
simple, yet immensely useful tests we can add to a site of this kind. You can of
course also add unit tests for individual functions, and doing so in a system
composed of mostly pure functions is very straight-forward, yet outside the
scope of this post.
One nice test to put in a site like this is an integration test that checks that
every page renders without errors. We will use
Midje for our tests, so let's update
project.clj:
(defproject cjohansen-no "0.1.0-SNAPSHOT"
:description "cjohansen.no source code"
:url "http://cjohansen.no"
:license {:name "BSD 2 Clause"
:url "http://opensource.org/licenses/BSD-2-Clause"}
:dependencies [[org.clojure/clojure "1.5.1"]
[stasis "1.0.0"]
[ring "1.2.1"]
[hiccup "1.0.5"]
[me.raynes/cegdown "0.1.1"]
[enlive "1.1.5"]
[clygments "0.1.1"]
[optimus "0.14.2"]]
:ring {:handler cjohansen-no.web/app}
:aliases {"build-site" ["run" "-m" "cjohansen-no.web/export"]}
:profiles {:dev {:plugins [[lein-ring "0.8.10"]]}
:test {:dependencies [[midje "1.6.0"]] :plugins [[lein-midje "3.1.3"]]}})
We've added a test profile that includes the midje dependencies. Add the
following to test/cjohansen_no/web_test.clj:
(ns cjohansen-no.web-test
(:require [cjohansen-no.web :refer :all]
[midje.sweet :refer :all]))
(fact
"All pages respond with 200 OK"
(doseq [url (keys (get-pages))]
(let [status (:status (app {:uri url}))]
[url status] => [url 200])))
We simply call our get-pages function, loop the resulting map, and
call each page function with a request map consisting only of a URL. The
comparison is made with a vector of the URL and the status. The reason for this
is that the URL will be included in the error message if this fails. This way we
can know which pages fail. To run the tests:
lein with-profile test midje
Doing this will inform us that the generated core_test.clj fails. Just delete
it. Other than that, the test confirms that all is well with our site. To keep
the tests running while working on the site, run autotest:
lein with-profile test midje :autotest
Another useful test to have in place is a link-checker. We will make one that at
least verifies that the internal links between pages in our app are correct, and
that they don't cause any unnecessary redirects (e.g. from /about to /about/).
Enlive is very useful for these things. We will use the select
function to find all links, and then make sure that the href
attribute points to an existing URL if it is a path (not a full URL, which is
treated as an external link). First up is the link-valid? function,
which checks if a single link is valid given a map of pages:
(defn link-valid? [pages link]
(let [href (get-in link [:attrs :href])]
(or
(not (.startsWith href "/"))
(contains? pages href)
(contains? pages (str href "index.html")))))
The link is considered valid if the href attribute either points to a URL that
isn't a path within our app (relative paths are assumed not used) or if it
points to one of the pages in the map. We're lenient enough to allow links to
/about/ when we have /about/index.html. Since we will use enlive to select all
the links, update the test namespace form to this:
(ns cjohansen-no.web-test
(:require [cjohansen-no.web :refer :all]
[midje.sweet :refer :all]
[net.cgrand.enlive-html :as enlive]))
Then add the test itself:
(fact
"All links are valid"
(let [pages (get-pages)]
(doseq [url (keys (get-pages))
link (-> (:body (app {:uri url}))
java.io.StringReader.
enlive/html-resource
(enlive/select [:a]))]
(let [href (get-in link [:attrs :href])]
[url href (link-valid? pages link)] => [url href true]))))
Again, we loop all the pages and get them. For each page, we select all links,
and expect all of them to pass the link checker. Again we make a slightly
strange comparison in the interest of having more than true/false in the output
if one of these fail. If you add an invalid link to the markdown file now,
running the tests will produce this:
FAIL "All links are valid" at (web_test.clj:21)
Expected: ["/my-first-post" "/about/" true]
Actual: ["/my-first-post" "/about/" false]
18 lines of code to verify all links on the site. Pretty nifty! Rather than
having numerous tests that load all the pages, it would probably be a good way
to change the structure of the tests such that we only load each page once, and
instead register various test functions we want to run for each page. This is
left as an exercise for the reader.
I hope this post has shown you the power and flexibility of Stasis and all the
other tools. Perhaps it has even convinced you further of the value of Clojure.
I really do dislike number-of-lines-of-code jerkoffs, but it is worth mentioning
that we were able to build a reasonably feature-complete technical blog in
roughly 100 lines of code using a simple, yet powerful "no batteries included"
library like Stasis (which itself clocks in at just over 100 lines of code). I
hope you will consider Clojure and Stasis for your next semi-static web project.
Big thanks to Magnar Sveen for proof-reading and correcting this post.
