Overloading gen-class methods of glorious types with sublime nation of clojure for make great benefit of interop

(Post title shenanigans alert!)

This is a ~~quick~~ note for a few seemingly-simple issues about how clojure interops with java. And how I was surprised by it. Again. I'm also hoping it's useful and/or informative.

With clojure you can create "java" classes using the gen-class macro. With this macro you may construct a class in clojure, specify which interfaces it implements, what state it has and so on. Read more about gen-class at clojuredocs. It's not used often, but it can be necessary in some cases.

I encountered a few problems for which the solutions were not obvious and decided to collect it here in the form of examples.

Creating context; Most basic `gen-class` example

We have one interface, IThing which we will implement in clojure.
We have one java class ThingUser.
ThingUser uses an instance of IThing.
We will instantiate ThingUser from clojure, and give it our implementation of IThing.

To start with; Notice that IThing has two overloads defined on doWithThing1. Each takes a different number of parameters. (I'll use doWithThing2 and doWithThing3 lower down in this article.)

In package test;

public interface IThing {
    void doWithThing1(String a, String b);
    void doWithThing1(String a, String b, String c);

    void doWithThing2(String a, String b);
    void doWithThing2(String a, Integer b);
    void doWithThing2(Integer a, byte[] b);

    default void doWithThing3(String a, String b) {
        doWithThing2(a, b);
    }

}

public class ThingUser {
    public void doThingOneWithTwo(IThing thing) {
        thing.doWithThing1("A", "B");
    }
    public void doThingOneWithThree(IThing thing) {
        thing.doWithThing1("A", "B", "C");
    }
}

In test/thingi.clj:

(ns test.thingi
  (:import
   [test IThing ThingUser]))

(gen-class :name "test.thingi.TestImpl"
           :prefix "impl-"
           :main false
           :implements [test.IThing])

(defn impl-doWithThing1
  ([this a b c]
   (println (str "a: " a 
                 ", b: " b 
                 ", c: " c)))
  ([_this a b]
   (println (str "a: " a 
                 ", b: " b))))

Then in the repl evaluate this: (compile 'test.thingi). At this point everything is ready for the first test.

(let [impl (test.thingi.TestImpl.)
        user (ThingUser.)]
    (.doThingOneWithTwo user impl)
    (.doThingOneWithThree user impl))

a: A, b: B
a: A, b: B, c: C

Observations

The interface method arity is matched to the correct clojure function arity.
We are not "implementing" all of the interface methods on the clojure side yet no errors result.
(compile ...) creates a stub .class by using the bytecodes API. This class is loaded once it is referenced in clojure.
You can change the clojure code without having to restart the JVM. The stub methods use "reflection" to find the correct clojure functions.
By default the class files are only loaded once per JVM. When the class file has to meaningfully change the JVM has to be restarted. There are tools that remove some of this friction.

same arity overloads

In IThing we are now looking at doWithThing2. Add new java class ThingUser2:

public class ThingUser2 {
    public void doThingTwo(IThing thing, String a, String b) {
        thing.doWithThing2(a, b);
    }
    public void doThingTwo(IThing thing, String a, Integer b) {
        thing.doWithThing2(a, b);
    }
    public void doThingTwo(IThing thing, Integer a, byte[] b) {
        thing.doWithThing2(a, b);
    }

In test/thingi.clj add:

(defn impl-doWithThing2
  [_ a b]
  (println (str "a[" (type a) "]: " a "\n"
                "b[" (type b) "]: " b)))

Then eval:

  (let [impl (test.thingi.TestImpl.)
        user (ThingUser2.)]
    (.doThingTwo user impl "a" "b")
    (.doThingTwo user impl "a" (int 66))
    (.doThingTwo user impl (int 65) ^bytes (.getBytes "b")))

I get this result:

a[class java.lang.String]: a
b[class java.lang.String]: b

a[class java.lang.String]: a
b[class java.lang.Integer]: 66

a[class java.lang.Integer]: 65
b[class [B]: [B@3ea69f14

Observations

Regardless of which type arity overload you call, it will find the same function on the clojure side.
Your logic will (probably) require some type inspection on the parameters.
If the interface you implement gets a new overload with parameter types you did not anticipate you might get in trouble. Code defensively for this.

Alternative for overloads differing on parameter types

There is a secret (implementation detail) in the clojure compiler that allows you to create specific functions for gen-class methods, when the overloads differ only in the types of the parameters.

;; My recommendation is not to mix the two styles of implementation
;; (defn impl-doWithThing2
;;   [_ a b]
;;   (println (str "a[" (type a) "]: " a "\n"
;;                 "b[" (type b) "]: " b)))

(defn impl-doWithThing2-String-String
  [_ ^String a ^String b]
  (println "impl-doWithThing2-String-String: a=" a ", b=" b))

(defn impl-doWithThing2-String-Integer
  [_ ^String a ^Integer b]
  (println "impl-doWithThing2-String-Integer a=" a ", b=" b))

(defn impl-doWithThing2-Integer-byte<>
  [_ ^Integer a ^bytes b]
  (println "impl-doWithThing2-Integer-byte<> a=" a ", b=" (String. b)))

With the same test in the repl:

(let [impl (test.thingi.TestImpl.)
        user (ThingUser2.)]
    (.doThingTwo user impl "a" "b")
    (.doThingTwo user impl "a" (int 66))
    (.doThingTwo user impl (int 65) ^bytes (.getBytes "b")))

I get:

impl-doWithThing2-String-String: a= a , b= b
impl-doWithThing2-String-Integer a= a , b= 66
impl-doWithThing2-Integer-byte<> a= 65 , b= b

Observations

The short names of the parameter types are taken, interleaved with -, and appended to the function name to specify the specific overload that is being defined.
The short-name of an array is type[], for example int[], byte[] &c. [ and ] are special characters in clojure and cannot be used for the name of functions. They become <> in this case.
In my opinion mixing the two ways of defining the implementation will most likely lead to bad results so I recommend you choose between inspecting the types at runtime, or specifying multiple functions one for each overload.

Default interface implementations

In java you can add a default implementation to an interface, which allows you to add backwards compatible updates to an interface. However ... (foreboding)

One more java class:

package test;

public class ThingUser3 {
    public void doThingThree(IThing thing, String a, String b) {
    thing.doWithThing3(a, b);
    }
}

Then I run this in the repl:

  (let [impl (test.thingi.TestImpl.)
        user (ThingUser3.)]
    (.doThingThree user impl "a" "b"))

For me, on clojure version 1.11.1 this fails with:

1. Unhandled java.lang.UnsupportedOperationException
   doWithThing3 (test.thingi/impl-doWithThing3 not defined?)

           ThingUser3.java:    5  test.ThingUser3/doThingThree
                      REPL:   55  test.thingi/eval11229
                      REPL:   53  test.thingi/eval11229
...

Whoops! There is a bug in the clojure compiler where it will implement doWithThing3 on the stub implementation and then look for the impl- function and not find it and then give this error.

This is on the clojure team's radar for maybe inclusion in 1.13.

I found this in kafka client code. We implement org.apache.kafka.common.serialization.Deserializer which in the most recent release (3.6) received a new interface method with a default implementation. If you do the same in clojure you will probably find it sooner or later too.

// old interface method
deserialize(String topic, Headers headers, byte[] data)
// new interface method overload
deserialize(String topic, Headers headers, ByteBuffer data)

What was a backwards compatible change in other JVM languages was a breaking change in clojure.

Observations

I suspect the technique of runtime-type-inspecting the arguments is probably the one you are "supposed" to be using.
The mechanism for naming a function after the types of parameters was added for a reason (which can only be guessed at).
For better or for worse people (like myself) now depend on it.
This mechanism is considered an implementation detail and the clojure team does not promise not to break it in the future.
For particularly hairy interop situations, you are encouraged to use Java implementations. (I don't have a source reference for this)

Things.

Just things.