(Post title shenanigans alert!)
This is a quick note for a few seemingly-simple issues about how clojure interops with java. And how I was surprised by it. Again. I'm also hoping it's useful and/or informative.
With clojure you can create "java" classes using the gen-class macro. With this macro you
may construct a class in clojure, specify which interfaces it implements, what state it has
and so on. Read more about gen-class at clojuredocs. It's not used often, but it can be necessary in some cases.
I encountered a few problems for which the solutions were not obvious and decided to collect it here in the form of examples.
Creating context; Most basic gen-class example
- We have one interface,
IThingwhich we will implement in clojure. - We have one java class
ThingUser. -
ThingUseruses an instance ofIThing. - We will instantiate
ThingUserfrom clojure, and give it our implementation ofIThing.
To start with; Notice that IThing has two overloads defined on doWithThing1. Each takes a different number of parameters.
(I'll use doWithThing2 and doWithThing3 lower down in this article.)
In package test;
public interface IThing {
void doWithThing1(String a, String b);
void doWithThing1(String a, String b, String c);
void doWithThing2(String a, String b);
void doWithThing2(String a, Integer b);
void doWithThing2(Integer a, byte[] b);
default void doWithThing3(String a, String b) {
doWithThing2(a, b);
}
}
public class ThingUser {
public void doThingOneWithTwo(IThing thing) {
thing.doWithThing1("A", "B");
}
public void doThingOneWithThree(IThing thing) {
thing.doWithThing1("A", "B", "C");
}
}
In test/thingi.clj:
(ns test.thingi
(:import
[test IThing ThingUser]))
(gen-class :name "test.thingi.TestImpl"
:prefix "impl-"
:main false
:implements [test.IThing])
(defn impl-doWithThing1
([this a b c]
(println (str "a: " a
", b: " b
", c: " c)))
([_this a b]
(println (str "a: " a
", b: " b))))
Then in the repl evaluate this: (compile 'test.thingi). At this point everything is ready for the first test.
(let [impl (test.thingi.TestImpl.)
user (ThingUser.)]
(.doThingOneWithTwo user impl)
(.doThingOneWithThree user impl))
a: A, b: B
a: A, b: B, c: C
Observations
- The interface method arity is matched to the correct clojure function arity.
- We are not "implementing" all of the interface methods on the clojure side yet no errors result.
-
(compile ...)creates a stub.classby using the bytecodes API. This class is loaded once it is referenced in clojure. - You can change the clojure code without having to restart the JVM. The stub methods use "reflection" to find the correct clojure functions.
- By default the class files are only loaded once per JVM. When the class file has to meaningfully change the JVM has to be restarted. There are tools that remove some of this friction.
same arity overloads
In IThing we are now looking at doWithThing2. Add new java class ThingUser2:
public class ThingUser2 {
public void doThingTwo(IThing thing, String a, String b) {
thing.doWithThing2(a, b);
}
public void doThingTwo(IThing thing, String a, Integer b) {
thing.doWithThing2(a, b);
}
public void doThingTwo(IThing thing, Integer a, byte[] b) {
thing.doWithThing2(a, b);
}
In test/thingi.clj add:
(defn impl-doWithThing2
[_ a b]
(println (str "a[" (type a) "]: " a "\n"
"b[" (type b) "]: " b)))
Then eval:
(let [impl (test.thingi.TestImpl.)
user (ThingUser2.)]
(.doThingTwo user impl "a" "b")
(.doThingTwo user impl "a" (int 66))
(.doThingTwo user impl (int 65) ^bytes (.getBytes "b")))
I get this result:
a[class java.lang.String]: a
b[class java.lang.String]: b
a[class java.lang.String]: a
b[class java.lang.Integer]: 66
a[class java.lang.Integer]: 65
b[class [B]: [B@3ea69f14
Observations
- Regardless of which type arity overload you call, it will find the same function on the clojure side.
- Your logic will (probably) require some type inspection on the parameters.
- If the interface you implement gets a new overload with parameter types you did not anticipate you might get in trouble. Code defensively for this.
Alternative for overloads differing on parameter types
There is a secret (implementation detail) in the clojure compiler that allows you to create specific functions for gen-class methods, when the overloads differ only in the types of the parameters.
;; My recommendation is not to mix the two styles of implementation
;; (defn impl-doWithThing2
;; [_ a b]
;; (println (str "a[" (type a) "]: " a "\n"
;; "b[" (type b) "]: " b)))
(defn impl-doWithThing2-String-String
[_ ^String a ^String b]
(println "impl-doWithThing2-String-String: a=" a ", b=" b))
(defn impl-doWithThing2-String-Integer
[_ ^String a ^Integer b]
(println "impl-doWithThing2-String-Integer a=" a ", b=" b))
(defn impl-doWithThing2-Integer-byte<>
[_ ^Integer a ^bytes b]
(println "impl-doWithThing2-Integer-byte<> a=" a ", b=" (String. b)))
With the same test in the repl:
(let [impl (test.thingi.TestImpl.)
user (ThingUser2.)]
(.doThingTwo user impl "a" "b")
(.doThingTwo user impl "a" (int 66))
(.doThingTwo user impl (int 65) ^bytes (.getBytes "b")))
I get:
impl-doWithThing2-String-String: a= a , b= b
impl-doWithThing2-String-Integer a= a , b= 66
impl-doWithThing2-Integer-byte<> a= 65 , b= b
Observations
- The short names of the parameter types are taken, interleaved with
-, and appended to the function name to specify the specific overload that is being defined. - The short-name of an array is
type[], for exampleint[],byte[]&c.[and]are special characters in clojure and cannot be used for the name of functions. They become<>in this case. - In my opinion mixing the two ways of defining the implementation will most likely lead to bad results so I recommend you choose between inspecting the types at runtime, or specifying multiple functions one for each overload.
Default interface implementations
In java you can add a default implementation to an interface, which allows you to add backwards compatible updates to an interface. However ... (foreboding)
One more java class:
package test;
public class ThingUser3 {
public void doThingThree(IThing thing, String a, String b) {
thing.doWithThing3(a, b);
}
}
Then I run this in the repl:
(let [impl (test.thingi.TestImpl.)
user (ThingUser3.)]
(.doThingThree user impl "a" "b"))
For me, on clojure version 1.11.1 this fails with:
1. Unhandled java.lang.UnsupportedOperationException
doWithThing3 (test.thingi/impl-doWithThing3 not defined?)
ThingUser3.java: 5 test.ThingUser3/doThingThree
REPL: 55 test.thingi/eval11229
REPL: 53 test.thingi/eval11229
...
Whoops! There is a bug in the clojure compiler where it will implement doWithThing3 on the stub implementation
and then look for the impl- function and not find it and then give this error.
This is on the clojure team's radar for maybe inclusion in 1.13.
I found this in kafka client code. We implement org.apache.kafka.common.serialization.Deserializer which
in the most recent release (3.6) received a new interface method with a default implementation. If you do the same
in clojure you will probably find it sooner or later too.
// old interface method
deserialize(String topic, Headers headers, byte[] data)
// new interface method overload
deserialize(String topic, Headers headers, ByteBuffer data)
What was a backwards compatible change in other JVM languages was a breaking change in clojure.
Observations
- I suspect the technique of runtime-type-inspecting the arguments is probably the one you are "supposed" to be using.
- The mechanism for naming a function after the types of parameters was added for a reason (which can only be guessed at).
- For better or for worse people (like myself) now depend on it.
- This mechanism is considered an implementation detail and the clojure team does not promise not to break it in the future.
- For particularly hairy interop situations, you are encouraged to use Java implementations. (I don't have a source reference for this)
References
- The original clojure mailing list post in which the secret gen-class overloading mechanism is discussed.
- At least one StackOverflow post in which the post above is referenced and the technique provided as an answer.
- Clojure Jira ticket for default implementations bug.
- Slack thread of Alex Miller predicting when default implementations might get support.
