Nico is blogging

My thoughts on software development, clojure, ruby, and trying to deal with complexity in simple ways

More portable (complex) macro musing

TL;DR: Writing macros that target both Clojure & ClojureScript is possible, but it comes with some gotchas. When those macros have to call other macros defined in one namespace in clj but in a different one in cljs, there are a couple more gotchas :)


This is my follow-up to the great Portable Macro Musing by Mike Fikes. Mike does a great job in showing how to write a single macro targeting Clojure, Clojurescript JVM and ClojureScript JS, when the main goal of the macro is to abstract the differences due to host interop: The example macro emits a call to Integer/parseInt on the JVM but to js/parseInt on JS.

Here I’m going to stress test the ability to write portable macros that need to call other macros while targeting both Clojure and ClojureScript. Not only that, the macro will call a certain macro when targeting Clojure, but needs to call a different macro when targeting ClojureScript.

A snippet of code is worth a thousand words, so let’s see our original macro that targets just Clojure:

1
2
3
4
5
6
7
8
9
(ns more-macro-musings.core
  (:require [clojure.test :refer [is]]))

(defmacro given [v & body]
  `(do
     ~@(for [[a b c] (partition 3 body)]
         (case b
           := `(is (= (~a ~v) ~c))
           :!= `(is (not= (~a ~v) ~c))))))

This is a simplified version of juxt.iota/given from juxt/iota (which is healthy macro sugar by the way!).

The idea of given is to allow us to write our test assertions like:

1
2
3
4
(given {:foo "bar" :woo "baz"}
  count := 2
  :foo := "bar"
  :woo :!= "bar")

instead of:

1
2
3
(is (= (count {:woo "baz", :foo "bar"}) 2))
(is (= (:foo {:woo "baz", :foo "bar"}) "bar"))
(is (not= (:woo {:woo "baz", :foo "bar"}) "bar")))

juxt.iota/given supports much more operators apart from := and :!=, you should check the full list in the README.

So, let’s go back to the main goal of this post.

NOTE: you can follow each of the following examples in the more-macro-musings repo, there’s a commit for each step.

1st try: Use reader conditionals

We might be tempted to introduce a reader conditional to port the macro to cljs:

1
2
3
4
5
6
7
8
9
10
(ns more-macro-musings.core
  (:require #?(:clj  [clojure.test :refer [is]]
               :cljs [cljs.test :refer-macros [is]])))

(defmacro given [v & body]
  `(do
     ~@(for [[a b c] (partition 3 body)]
         (case b
           := `(is (= (~a ~v) ~c))
           :!= `(is (not= (~a ~v) ~c))))))

When we compile the cljs code, we’ll see some warnings:

1
2
3
Compiling test/more_macro_musings/core_test.cljc
WARNING: No such namespace: clojure.test, could not locate clojure/test.cljs, clojure/test.cljc, or Closure namespace "" at line 7 test/more_macro_musings/core_test.cljc
WARNING: Use of undeclared Var clojure.test/do-report at line 7 test/more_macro_musings/core_test.cljc

If we try to run the tests we’ll get something like:

1
2
3
4
ERROR in (given-macro-test) (ReferenceError:NaN:NaN)
Uncaught exception, not in assertion.
expected: nil
  actual: #object[ReferenceError ReferenceError: java is not defined]

And if we take a look to the generated js code we’ll find:

1
2
3
4
5
6
7
var result__7612__auto___5695 = cljs.core.apply.call(null,cljs.core._EQ_,values__7611__auto___5694);
if(cljs.core.truth_(result__7612__auto___5695)){
clojure.test.do_report.call(null,new cljs.core.PersistentArrayMap(null, 4, [new cljs.core.Keyword(null,"type","type",1174270348) ...));
} else {
...
}catch (e5689){if((e5689 instanceof java.lang.Throwable)){
...

Well, we won’t see the ...: that was me shortening the output, which is much, much longer.

See the call to clojure.test.do_report and the instanceof java.lang.Throwable? AHA!

As the macro was compiled in Clojure, is was captured as clojure.test/is which in turn emits calls to clojure.test/do-report instead of cljs.test/do-report (and catch Throwable instead of catch :default), which is what we really need in cljs land.

2dn try: Use an alias for the namespace

So we do this:

1
2
3
4
5
6
7
8
9
10
(ns more-macro-musings.core
  (:require #?(:clj  [clojure.test :as test :refer [is]]
               :cljs [cljs.test :as test :refer-macros [is]])))

(defmacro given [v & body]
  `(do
     ~@(for [[a b c] (partition 3 body)]
         (case b
           := `(test/is (= (~a ~v) ~c))
           :!= `(test/is (not= (~a ~v) ~c))))))

The result is the same as in the previous try. That’s because test/is is also captured as clojure.test/is.

BTW, If you want to learn more about variable capture and hygienic macros, check out the excellent Clojure and Hygienic Macros by Marek Kubica

3rd try: Quote-unquote

It must work now!

1
2
3
4
5
6
(defmacro given [v & body]
  `(do
     ~@(for [[a b c] (partition 3 body)]
         (case b
           := `(~'is (= (~a ~v) ~c))
           :!= `(~'is (not= (~a ~v) ~c))))))

It doesn’t. We avoided the wrong variable capture by using the quote-unquote, but we changed it for another wrong variable capture: is is captured now as more-macro-musings.core-test/is which doesn’t exist. We can see this by looking into the generated js code:

1
2
3
more_macro_musings.core_test.given_macro_test.cljs$lang$test = (function (){
more_macro_musings.core_test.is.call(null,cljs.core._EQ_.call(null,new cljs.core.Keyword(null,"foo","foo",1268894036) ...
...

There are no compilation errors or warnings, but the error is not gone, it just changed to Cannot read property 'call' of undefined

4th try: Refer the is macro in the test ns

We change our test ns form from:

1
2
3
4
(ns more-macro-musings.core-test
  (:require #?(:clj  [clojure.test :refer        [deftest]]
               :cljs [cljs.test    :refer-macros [deftest]])
            [more-macro-musings.core #?(:clj :refer :cljs :refer-macros) [given]]))

to:

1
2
3
4
(ns more-macro-musings.core-test
  (:require #?(:clj  [clojure.test :refer        [deftest is]]
               :cljs [cljs.test    :refer-macros [deftest is]])
            [more-macro-musings.core #?(:clj :refer :cljs :refer-macros) [given]]))

It works! But

We don’t want to do that. I mean, we wanted to provide the new given macro as a replacement for the is macro, but we force the users of the given macro to refer the is macro in their ns declaration. And if they forget to do it (and they will), they’ll get the obscure error from our 3rd try. Not nice.

There must be another solution.

5th try: Reader conditionals inside of the macro

We try replacing the is form to something like:

1
:= `(#?(:clj clojure.test/is :cljs cljs.test/is) (= (~a ~v) ~c))

It doesn’t work for the same reasons as in our 1st and 2nd try. Check the full code in the reader-conditional-in-macro tag

6th try: if-cljs FTW!

Let’s introduce a new macro: if-cljs and see the finished code:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
(ns more-macro-musings.core
  (:require #?(:clj  [clojure.test]
               :cljs [cljs.test :include-macros true])))

(defn- cljs-env?
  "Take the &env from a macro, and tell whether we are expanding into cljs."
  [env]
  (boolean (:ns env)))

(defmacro if-cljs
  "Return then if we are generating cljs code and else for Clojure code.
   https://groups.google.com/d/msg/clojurescript/iBY5HaQda4A/w1lAQi9_AwsJ"
  [then else]
  (if (cljs-env? &env) then else))

(defmacro is [& args]
  `(if-cljs
     (cljs.test/is ~@args)
     (clojure.test/is ~@args)))

(defmacro given [v & body]
  `(do
     ~@(for [[a b c] (partition 3 body)]
         (case b
           := `(is (= (~a ~v) ~c))
           :!= `(is (not= (~a ~v) ~c))))))

We used a trick here: (:ns &env) will be nil when compiling with a Clojure target, but will be truthy when compiling for ClojureScript, so we can emit different code depending on that.

Let’s see a snippet from the generated code:

1
2
3
4
5
6
if(cljs.core.truth_(result__6136__auto___6423)){
cljs.test.do_report.call(null,new cljs.core.PersistentArrayMap(null, ...
...
}catch (e6417){var t__6173__auto___6424 = e6417;
cljs.test.do_report.call(null,new cljs.core.PersistentArrayMap(null, ...
...

Nice! We are emitting calls to cljs.test/do-report and we are not catching java.lang.Throwable. We also don’t need to refer is in our test ns, take a look to the test code in the master branch. Run it with lein doo node node-test if you don’t believe me :)

I learned about if-cljs in a conversation in the clojure google group. It’s defined in a couple projects, like Prismatic/schema, ptaoussanis/encore and some others.


Know of a better way to do this? Leave a comment below here, or in the twitter-lands!

Ring-logger 0.7.1 released! Adds option to disable timing

New features

  • Add :timing option to disable logging timing information

Breaking changes

  • The options :pre-logger, :post-logger and :exception-logger were removed

    They were intended as a way to override the way the messages are generated, but we now have multimethods and :printer to do that.


clojars: [ring-logger 0.7.1]

github: nberger/ring-logger

Check it out, and feel free to share any kind of feedback!

I’m way too busy, show me a screenshot, plz!

Ok, here we go:

ring-logger log capture

Converting test.check to .cljc

After working for more than a month on the conversion of test.chuck to .cljc, I was a bit scared when Gary Fredericks suggested to me that I should do the same in test.check. But that feeling lasted for just 1 second and then I started to think that it would be a nice opportunity to do a bigger contribution to the clojure ecosystem, but also an opportunity to learn more about the test.check codebase.


Let’s step back a bit. I’m a big fan of test.check and generative testing in general. I think I was very lucky for being in Clojure/West 2014 when John Hughes gave his incredible talk Testing the Hard Stuff and Staying Sane. As he was talking, everything started to make sense. That feeling that I always had when writing tests, that some piece was missing, that the way we usually write tests was not enough… well John had the answer for that: Generative Testing.

When I came back to Buenos Aires from San Francisco, I started to play a bit with test.check and also to see what similar tools were available in ruby, which was the language I was using during those days. I won’t extend much on this but just as a tl;dr: IMO the best generative testing library in ruby is rantly, but it’s still too far from what test.check offers. I talked about rantly in a Ruby Meetup and also in clojure meetups in Sao Paulo and Buenos Aires, so if you happen to walk close to me there’s a high chance I will be talking about how we should do more property testing and not only testing by example in our codebases.

Reader conditionals (.cljc) in test.check

Today I finished what I think is the complete conversion of the test.check codebase to .cljc. I left the part about random number generators out of the conversion because it’s very different from clj to cljs, so having all that code in a single file doesn’t make much sense. And I’m saying “what I think is the complete conversion” because I just uploaded the patches to TCHECK-79 and now the review process will start and anything might happen, from having to do small changes to having the ticket rejected altogether. If that happens, sorry for all this buzz, I will have enjoyed the journey anyways…

Now, why do we even care about .cljc files and Reader Conditionals??

You might be wondering what .cljc is. Well, .cljc is the extension of the files that want to take advantage of the (relatively) new Reader Conditionals in clojure.

Reader conditionals allow us to share code of multiple clojure dialects (Clojure, ClojureScript, ClojureCLR) in a single file that will be compiled to the corresponding platforms. The main advantage of doing so is that a lot of duplication can be avoided.

Let’s see a small example:

1
2
3
(.getTime
  #?(:clj  (java.util.Date.)
     :cljs (js/Date.)))

This is a very small code sample, but take a look to the commit that converts the generators namespace to .cljc. I know, you won’t click on that link, so here’s a screenshot:

generators.cljc 50 additions and 715 deletions

Yes, that’s 50 lines of code added and 715 deleted!. It’s a lot of duplication that can be avoided thanks to this.

For more info on Reader Conditionals see the design document about reader conditionals in dev.clojure.org, The Reader - Reader Conditionals in clojure.org, Clojure Reader Conditionals by Example by Daniel Compton and Portable Macro Musing by Mike Fikes.

Want to help, or just try it out?

You can help by playing around with the converted code. You can get it by applying the 10 patches from the ticket, or by cloning my fork of test.check and checking out the cljc branch. Cloning the repo will be probably easier than applying the patches :), but YMMV.

Feel free to add comments to the ticket, if you think you have anything to add, that would be of great help!

I’d like to thank Gary Fredericks for all the work he has done in test.chuck and test.check, for being very receptive to contributions on both projects, and also for the great support he gave me while trying to do all this stuff.

Contributing to OSS projects - Part 2

first pull request meme

This is the second part of Contributing to OSS projects where I’m telling my modest journey in being a contributor to OSS projects, as a way to motivate myself to continue doing it but also to show others that it’s not difficult to get started.

In the first part of this series I did a racconto from my first pull-request that got merged in october 2010 up until the last of my pull requests from 2013

So let’s continue…

2014


That’s it for 2014.

I hope this isn’t being too boring. To me, it’s being awesome. It’s liberating in some sense: Even though I’m not proud of all the code I did, I’m proud for having done it and for showing it in the open. A couple years ago I would have been incapable of doing this: I was very afraid of doing a mistake while in the open.

Life is only one. And as (supposedly) Albert Einstein said:

A person who never made a mistake, never tried anything new


See you in the third part, where I’ll cover the current year.

Contributing to OSS projects - Part 1

first pull request meme

I’m relatively new to this thing of “being a contributor”. Even though I got my first pull request accepted into an open source software (OSS) project almost 5 years ago (just checked and it had tests! Pheew!) it was only this year when I started to make frequent and more important contributions.

I wanted to do a recap of this modest journey. I think it might help others who are trying to start with theirs, but also to encourage me to continue the pursuit :)

A 5 years of contributions recap (part 1: Oct/2010 - 2013)

2010

  • formtastic - A small fix for passing options in a select with grouping: Not much to say… I needed this for a project, so I made the change and sent the PR. This is the most usual way to do a contribution: You are using a library and find that you need a small addition to it, so you fork the project, do the change, and send it back. Everyone wins.

2012

  • sidekiq - Added method in testing for running only due jobs (not merged): I think this was a nice addition. It had tests (Pheew again!). I needed it for a project where I was scheduling many jobs with different due times and I wanted to verify the behavior when only some of the jobs were run because the time for them did arrive. Nobody was interested and Mike Perham thought that it was not worth it to add the change, so he closed the PR.

This can be hard when you are starting, but it’s one of the three possible endings for a PR: It will be accepted, rejected, or just kept open to the eternity. Don’t be discouraged by this! Perhaps it’s for the better: The world has too much code already. And remember: the project owner might have a different idea of what’s acceptable or desired for the project, and that’s perfectly fine.

2013

zeus - Add spec for tracking changes to Gemfile.lock in default_bundle process (not merged): Not sure why I wanted to add this test. I should have added a description of the change, at least so I can explain it now :)

rspec_candy - Assume rspec1 if RSpec::Core is not defined: It had tests, a nice description, and it was accepted :)

heroku-resque-workers-scaler - Do nothing if not authorized: I didn’t send it to the upstream repo, and I don’t remember why.

rails - Backport #4956 to 3-2-stable. Postgresql partial indices (not merged): I was using rails a lot by that time. This was the only PR I got to send to the project, and it was rejected because were not accepting new features in the 3-2-stable branch, and that made sense. I just didn’t know about that when I sent the PR. As I commented in the PR, I ended using pg_power which already had this feature.

tictactoe - Add travis.yml / Upgrade to clojure 1.5.1: I was trying to do my first steps with clojure and those PRs were just from trying to run the project, in my local env but also in travis.

tictactoe - Extract board to a separate ns (not merged): More from my first steps in clojure, playing around on this project from my friend @paraseba. I was happy that I was starting to understand what was going on, and I thought it would be better to split the code into new namespaces. @paraseba didn’t agree, and after some time I learned that it’s sometimes better (or at least a common style) to avoid splitting the code into namespaces in clojure, to avoid an “explosion” of namespaces. It’s a tradeoff. I learned something from this PR, so I’m happy I made it.

split - Refactor Experiment#initialize / Refactoring: Extract ExperimentStore / Add redis on travis: Just a bit of refactoring and running tests in travis-ci. I was not using this project nor planning to use it in the short term, but I think I found the code via Code Triage and I thought it needed some serious refactoring, so I started with some. I didn’t continue contributing to the project.


To be continued…

If you got until here, thanks! This was the first part of this post, including my pull requests sent to OSS from October 2010 until the end of 2013. In the next post I’ll finish the 5 years recap until today.

meme credit: http://bukashk0zzz.com/first-pull-request/

Ring-logger 0.7.0 released!

ring logger log capture

I’m pleased to announce the release of ring-logger 0.7.0!

The goal for this release was to refactor the code in preparation for future improvements and to remove the timbre & onelog (which pulls log4j, etc) dependencies. Those were moved to the new companion projects that also have 0.7.0 releases: ring-logger-timbre & ring-logger-onelog

Breaking changes

  • Remove timbre implementation, was moved to ring-logger-timbre
  • Remove OneLog implementation, was moved to ring-logger-onelog
  • Replace keyword arguments with a proper map in wrap-with-logger
  • Replace :logger-impl param to :logger
  • Make Logger protocol more simple: It has just two fns now: log and add-extra-middleware

Try it!

Feedback & contributions are all welcome!

On Forking

Today I’m going to talk about starting a new fork in an open source project. I mean a fork not to just send a patch to the original project but to really go by a different pathway.

There are moments in life when it’s better to split pathways, be it from a job, your own company, a relationship, or an open source project.

If you are going to do that, wait a second and listen:

DON’T DO IT!

I mean: Try hard not to do it! It’s better to join forces together, the best stuff is built as a team, and the world has too much code floating around already.

That’s my first advise on this regard.

But if you truly think you have compelling reasons to change the direction of the project and you can get it started on that direction, then:

DO IT!

You’ll hopefully find others who like the shape the new project is taking (a fork is a new project) and will try and help to move it forward. And not only that: once the forked project is a reality, it might give a new perspective to the original project maintainers, and chances are that you might be able to join forces together in the future. Be it by joining the two projects, or just collaborating in unexpected ways.

Just be open in what you are doing.

This post was motivated by a project I started last week, ring-logger, which is a fork of ring.middleware.logger. The reasons for why I started the fork can be summarized as:

  • Ring.middleware.logger has some transitive dependencies (onelog, log4j) that I don’t want to (or even can) pull when adding a logger to a project.

  • I think the code of r.m.logger would benefit from some refactoring and the extension/customization mechanisms are far from ideal. Doing that could introduce some breaking changes.

  • r.m.logger maintainers team was very clear in that they are fine with keeping those dependencies because that’s what they use, and even though they would accept adding additional implementations, they would not change the default one from onelog. And the way how I was implementing the change, I couldn’t find a clean way to keep onelog as the default implementation while providing how to skip onelog dependency when that’s needed.

In a future post I’ll talk about how I decided to split my pathway from yellowspot, the company I co-founded and where I loved to work for almost 6 years, from which I decided to leave by the end of last year.

But that’s a very different story.

Ship often

When I started with this blog, I made the promise to keep it simple. To hold on that promise I should have blogged more often. I failed on that part, and here I make a new promise to not fail again in the same way, at least not so hard that I didn’t even publish that “keep it simple” post until today.

How is it that “ship often” is related to “keep it simple”?

When you wait too much to publish (ship) a post, most of the times it’s because you think it’s not perfect yet. That you’d better improve it, or people will get a bad impression from you because either you are a bad writer, or you don’t create good content, or you don’t write well in that language that’s not your mother tongue… Or maybe it’s just that…

I don’t know, f*** it! Let’s publish this post now, I need to get started.

Stay tuned!

In the next posts I’ll talk about what I’ve been doing lately: Apart from working on “real” projects, I started to contribute to open source projects more frequently and also took an active role in helping organize the local clojure meetups in Buenos Aires.

Keep it simple

This blog starts here, and I’ll do my best to keep it simple.