Overloading gen-class methods of glorious types with sublime nation of clojure for make great benefit of interop

(Post title shenanigans alert!)

This is a quick note for a few seemingly-simple issues about how clojure interops with java. And how I was surprised by it. Again. I'm also hoping it's useful and/or informative.

With clojure you can create "java" classes using the gen-class macro. With this macro you may construct a class in clojure, specify which interfaces it implements, what state it has and so on. Read more about gen-class at clojuredocs. It's not used often, but it can be necessary in some cases.

I encountered a few problems for which the solutions were not obvious and decided to collect it here in the form of examples.

Creating context; Most basic gen-class example

  • We have one interface, IThing which we will implement in clojure.
  • We have one java class ThingUser.
  • ThingUser uses an instance of IThing.
  • We will instantiate ThingUser from clojure, and give it our implementation of IThing.

To start with; Notice that IThing has two overloads defined on doWithThing1. Each takes a different number of parameters. (I'll use doWithThing2 and doWithThing3 lower down in this article.)

In package test;

public interface IThing {
    void doWithThing1(String a, String b);
    void doWithThing1(String a, String b, String c);

    void doWithThing2(String a, String b);
    void doWithThing2(String a, Integer b);
    void doWithThing2(Integer a, byte[] b);

    default void doWithThing3(String a, String b) {
        doWithThing2(a, b);
    }

}

public class ThingUser {
    public void doThingOneWithTwo(IThing thing) {
        thing.doWithThing1("A", "B");
    }
    public void doThingOneWithThree(IThing thing) {
        thing.doWithThing1("A", "B", "C");
    }
}

In test/thingi.clj:

(ns test.thingi
  (:import
   [test IThing ThingUser]))

(gen-class :name "test.thingi.TestImpl"
           :prefix "impl-"
           :main false
           :implements [test.IThing])

(defn impl-doWithThing1
  ([this a b c]
   (println (str "a: " a 
                 ", b: " b 
                 ", c: " c)))
  ([_this a b]
   (println (str "a: " a 
                 ", b: " b))))

Then in the repl evaluate this: (compile 'test.thingi). At this point everything is ready for the first test.

(let [impl (test.thingi.TestImpl.)
        user (ThingUser.)]
    (.doThingOneWithTwo user impl)
    (.doThingOneWithThree user impl))

a: A, b: B
a: A, b: B, c: C

Observations

  • The interface method arity is matched to the correct clojure function arity.
  • We are not "implementing" all of the interface methods on the clojure side yet no errors result.
  • (compile ...) creates a stub .class by using the bytecodes API. This class is loaded once it is referenced in clojure.
  • You can change the clojure code without having to restart the JVM. The stub methods use "reflection" to find the correct clojure functions.
  • By default the class files are only loaded once per JVM. When the class file has to meaningfully change the JVM has to be restarted. There are tools that remove some of this friction.

same arity overloads

In IThing we are now looking at doWithThing2. Add new java class ThingUser2:

public class ThingUser2 {
    public void doThingTwo(IThing thing, String a, String b) {
        thing.doWithThing2(a, b);
    }
    public void doThingTwo(IThing thing, String a, Integer b) {
        thing.doWithThing2(a, b);
    }
    public void doThingTwo(IThing thing, Integer a, byte[] b) {
        thing.doWithThing2(a, b);
    }

In test/thingi.clj add:

(defn impl-doWithThing2
  [_ a b]
  (println (str "a[" (type a) "]: " a "\n"
                "b[" (type b) "]: " b)))

Then eval:

  (let [impl (test.thingi.TestImpl.)
        user (ThingUser2.)]
    (.doThingTwo user impl "a" "b")
    (.doThingTwo user impl "a" (int 66))
    (.doThingTwo user impl (int 65) ^bytes (.getBytes "b")))

I get this result:

a[class java.lang.String]: a
b[class java.lang.String]: b

a[class java.lang.String]: a
b[class java.lang.Integer]: 66

a[class java.lang.Integer]: 65
b[class [B]: [B@3ea69f14

Observations

  • Regardless of which type arity overload you call, it will find the same function on the clojure side.
  • Your logic will (probably) require some type inspection on the parameters.
  • If the interface you implement gets a new overload with parameter types you did not anticipate you might get in trouble. Code defensively for this.

Alternative for overloads differing on parameter types

There is a secret (implementation detail) in the clojure compiler that allows you to create specific functions for gen-class methods, when the overloads differ only in the types of the parameters.

;; My recommendation is not to mix the two styles of implementation
;; (defn impl-doWithThing2
;;   [_ a b]
;;   (println (str "a[" (type a) "]: " a "\n"
;;                 "b[" (type b) "]: " b)))

(defn impl-doWithThing2-String-String
  [_ ^String a ^String b]
  (println "impl-doWithThing2-String-String: a=" a ", b=" b))

(defn impl-doWithThing2-String-Integer
  [_ ^String a ^Integer b]
  (println "impl-doWithThing2-String-Integer a=" a ", b=" b))

(defn impl-doWithThing2-Integer-byte<>
  [_ ^Integer a ^bytes b]
  (println "impl-doWithThing2-Integer-byte<> a=" a ", b=" (String. b)))

With the same test in the repl:

(let [impl (test.thingi.TestImpl.)
        user (ThingUser2.)]
    (.doThingTwo user impl "a" "b")
    (.doThingTwo user impl "a" (int 66))
    (.doThingTwo user impl (int 65) ^bytes (.getBytes "b")))

I get:

impl-doWithThing2-String-String: a= a , b= b
impl-doWithThing2-String-Integer a= a , b= 66
impl-doWithThing2-Integer-byte<> a= 65 , b= b

Observations

  • The short names of the parameter types are taken, interleaved with -, and appended to the function name to specify the specific overload that is being defined.
  • The short-name of an array is type[], for example int[], byte[] &c. [ and ] are special characters in clojure and cannot be used for the name of functions. They become <> in this case.
  • In my opinion mixing the two ways of defining the implementation will most likely lead to bad results so I recommend you choose between inspecting the types at runtime, or specifying multiple functions one for each overload.

Default interface implementations

In java you can add a default implementation to an interface, which allows you to add backwards compatible updates to an interface. However ... (foreboding)

One more java class:

package test;

public class ThingUser3 {
    public void doThingThree(IThing thing, String a, String b) {
    thing.doWithThing3(a, b);
    }
}

Then I run this in the repl:

  (let [impl (test.thingi.TestImpl.)
        user (ThingUser3.)]
    (.doThingThree user impl "a" "b"))

For me, on clojure version 1.11.1 this fails with:

1. Unhandled java.lang.UnsupportedOperationException
   doWithThing3 (test.thingi/impl-doWithThing3 not defined?)

           ThingUser3.java:    5  test.ThingUser3/doThingThree
                      REPL:   55  test.thingi/eval11229
                      REPL:   53  test.thingi/eval11229
...

Whoops! There is a bug in the clojure compiler where it will implement doWithThing3 on the stub implementation and then look for the impl- function and not find it and then give this error.

This is on the clojure team's radar for maybe inclusion in 1.13.

I found this in kafka client code. We implement org.apache.kafka.common.serialization.Deserializer which in the most recent release (3.6) received a new interface method with a default implementation. If you do the same in clojure you will probably find it sooner or later too.

// old interface method
deserialize(String topic, Headers headers, byte[] data)
// new interface method overload
deserialize(String topic, Headers headers, ByteBuffer data)

What was a backwards compatible change in other JVM languages was a breaking change in clojure.

Observations

  • I suspect the technique of runtime-type-inspecting the arguments is probably the one you are "supposed" to be using.
  • The mechanism for naming a function after the types of parameters was added for a reason (which can only be guessed at).
  • For better or for worse people (like myself) now depend on it.
  • This mechanism is considered an implementation detail and the clojure team does not promise not to break it in the future.
  • For particularly hairy interop situations, you are encouraged to use Java implementations. (I don't have a source reference for this)

References

keyboards and web apps, my post/rant for the year

Did you know?

The keyboard as a concept for operating a machine was around for at least 35 years before the mouse came to be. And I'm actually talking nonsense, because it was longer than that. (quick mafs)

And yet.

And yet.

Building a web application to be used by a mouse is easy, reliable and expected. Building a web application so that it can be driven by a keyboard is at best a character building affair.

If you are a web developer in 2023 and your boss/friend/archenemy sweetly asks you to Add Keyboard shortcuts to your company's web app, tell them/they to go f**k themselves do it themselves instead.

Here's a short list of things that will mess with your self-confidence as a developer along the way:

  • Most npm libraries that "deal with" keyboard events use the old and deprecated keyCode API. This API gives you a number to represent the key that was pressed. 38, 38, 40, 40, 37, 39, 37, 39, 66, 65.
  • There is a new API that will tell you what key was pressed by using a set of well-define strings: ["ArrowUp", "ArrowUp", "ArrowDown", "ArrowDown", "ArrowLeft", "ArrowRight", "ArrowLeft", "ArrowRight", "KeyB", "KeyA"].
    • This is cool. I approve of this. 🫶
    • This is (only somewhat) useful to you if you want to create keyboard shortcuts that are based on what motion your hand makes as they type those shortcuts.
  • Part of the new API is the attribute called "key", which lets you know what character was produced by the keyboard, by the operating system, by the keyboard layout in the operating system, and by the browser inside this operating system. To quote from the spec:
    The "Digit2" and "KeyQ" keys are writing system keys that generate "2" and "q" when the US locale is active and "é" and "a" when the French locale is active.
  • Keyboard layouts exist:
    • Keyboard layouts are a software feature provided by your operating system, that makes a different character come out from the software than the one that was pressed by your fingers.
    • People in other countries (not the US) speak languages with characters that cannot be found on a US keyboard.
    • They prefer to use their keyboards to type words that are not in an English dictionary. (French, Greek, Hebrew)
    • These are innocently called International Keyboard Layouts.
    • Some nerds like to use US keyboards, but remap the keys to mean something different than what is printed on them. (Dvorak, Colemak)
    • Some nerds build their own keyboards 😱 and I promise you you have no idea what will happen when they press a key on it.
  • There is a javascript API that will tell you what character will come out of the mess that is a modern software stack when a specific, named key is pressed.
    • Just don't have users who use Firefox ...
    • or Safari.
  • There is NOT an API that will tell you which key must be pressed in order to produce a specific character. (As an implementer of technologies, I understand, but as a user of technologies, this seems silly)
  • Keyboards have things called "Modifier keys" on them: Alt, Shift, Control, Meta and some less well-known others AltGr, Hyper, Super, Fn
    • Those things modify the meaning of the key that is pressed. Typically you hold down one (or more) of these keys while simultaneously pressing another key.
    • Ctrl-S (means save in most places), Ctrl-Z (undo)
    • Shift-a (means A in most places) etc.
    • Most operating systems have opinions about what those modifier keys might be used for.
    • On Windows, you cannot use Alt-F4 as a keyboard shortcut for anything. Meta-l (Windows L) is also usually off-limits.
    • On MacOSX, pressing the Alt key changes the character of the key being pressed: Alt-J -> Δ
    • On linux, with some window managers, Alt is reserved for dragging a window around. On other linuxes (linuxii?) anything else is possible.
  • "Keyboard events" come in flavours: KeyDown, KeyPress & KeyUp
    • If you hold a key down, it is the KeyDown event that is repeated. This is useful for panning, zooming or adding horizontal whitespace. (just kidding, do you think I'm a barbarian?!)
    • The KeyPress event comes between the KeyDown and the KeyUp event. Sometimes.
    • KeyUp is a decent choice for creating keychords with. (A keychord is a sequence of key presses that together triggers some action)
    • On MacOSX, when you hold down the Meta key, KeyUp's sometimes/often don't fire.
  • These things exist:
    • Compose keys; quoting from Wikipedia:
      A compose key (sometimes called multi key) is a key on a computer keyboard that indicates that the following (usually 2 or more) keystrokes trigger the insertion of an alternate character, typically a precomposed character or a symbol. For instance, typing Compose followed by ~ and then n will insert ñ.
    • Dead keys:
      A dead key is a special kind of modifier key on a mechanical typewriter, or computer keyboard, that is typically used to attach a specific diacritic to a base letter.
    • Alt-codes (on Windows, I believe). Example: In Afrikaans words such as these exist: wêreld & sēening. To produce those characters on a Windows PC you hold down Alt and press in sequence 136 or 137, as the case may be.
    • There's more in this category, but these are the ones I myself have used.
  • Languages that have nothing to do with ancient Rome and Latin and of which I know nothing:
    • 한글 (Korean)
    • にほん · 日本 (Japan)
    • 汉字 (China)
    • עִברִית (Israel)
    • हिंदी or தமிழ் or മലയാളം (merely 3 picked at random from Indian subcontinent)

    • Ask yourself how people who speak (and type!) in these languages use computers created by white, English-speaking men.
To get back on point. Keyboards are hard. Mice are easy. Even web applications that get it mostly right, also get it really wrong, sometimes. (❤️❤️ linear.app ❤️❤️)

If you are in this situation, I have two pieces of advice:

  • Don't. Go home. Have a beer. Kiss your favourite person. Become a carpenter, be free, roam the land. Counting is hard.
  • Are you making games?
    • Use physical keyboard codes and use the getLayoutMap() to indicate to your gamers which characters to try to type to go up or down or left or right.
    • Stick to chrome.
  • Are you making productivity software?
    • Use mnemonics for keyboard shortcuts.
    • Use key sequences for keyboard shortcuts.
    • Avoid modifiers, especially Shift, Alt & Meta
    • Ask for a raise.
☮️




























HOWTO set up your programming environment for great fun and glorious profit

I made some notes about how to start an open-source infrastructure-as-code project. Here is the first bit of that in blog form. This post deals with step-0 decisions on how to set up a new programming project, managing the programming language tools, their associated version sprawl and environment variables across different projects or directory trees.

tl;dr -- Introducing the problem for which direnv and asdf-vm is the fix.

What's going on?


Every chicken has an egg problem and every egg has a chicken problem. -- Somebody

The C in IAC stands for Code. It is software and it needs to run or execute. With this interpretation of things something like terraform is an interpreter and it executes *.tf source code files. These files are structured internally in hcl format, which is a type of json. terraform is therefore a programming language that executes json code/data against your cloud fabric.

Every software project targets a specific runtime explicitly and sometimes implicitly. This runtime contains the implementation of the programming language your code is based on, and also the accumulated changes and incompatibilities between released versions. As with any software, updates invalidate assumptions about the program behaviour that nobody knew they held.


Running the code against the wrong version of the tool may result in unknown-unknowable and unexpected problems and error states, so maintaining the execution runtime is important for engineer/operator happiness. Future-you will be happy not to deal with both unexpected upgrade maintenance and regular urgent maintenance at the same time.

This problem exists in programming projects in general. Your specific runtime might be the version of build-essentials that you use, or something else like babashka, terragrunt, ruby, python, ansible, clojure, java etc &c. The point is that as an operator you need to run the software in the git repository against the version(s) of the software it was developed with.

A compounding factor is that you might be working on multiple different codebases at the same time and it's possible and sometimes likely that every repository has its own idea of what version of each tool is required.

We want to stay in the pit of success.

One way to solve this meta-problem is to rely on a tool that will install other tools. This is what a package manager does. Your operating system (ubuntu or arch or something) has a package manager, but usually package managers want to provide you with the version of every package it manages. What you as a software engineer require is a package manager that, 1) doesn't piss off your main package manager and, 2) still allows you to have multiple versions of the multiple tools within easy reach in our work environment. The default system-provided package manager is not good enough, we need something else.

To expand on this a little; Package managers like brew and apt do not understand the concept of having multiple versions of one tool around at the same time. (python and python2 and python3 and python-is-python-2 is dumb...). Yet as a developer you might still have to target both AWS lambda and GCP Cloud functions in one workday. Each of these require a different version of python.

It is to solve this problem that tools like rbenv for ruby and pyenv for python exist. Those tools allow you to have multiple versions of the same tool installed (without requiring sudo...) and make different ones of those available depending on which directory your bash session or process currently is in.

But what about bash or nodejs? What about <insert favourite programming language here>? It is unlikely that every programming community has this custom made tool to solve this problem for its programming environment runtime.

Luckily we can solve the problem by having another standard tool... with plugins for every versioned programming environment you might require.

asdf, that one tool to rule all other tools and their versions


Let's set ourselves up for success.


If you have `asdf` installed, you can use it to install specific versions of programming languages into your programming environment and manage this with a special file called .tool-versions. When your shell, bash or zsh, encounters this file it activates the version of each tool specified in the file.

Example:

$ cat .tool-versions  
terraform 1.2.3
terragrunt 0.38.2
babashka 0.8.157

$ asdf install
babashka 0.8.157 is already installed
terraform 1.2.3 is already installed
terragrunt 0.38.2 is already installed
...

$ terragrunt --version \
&& terraform --version \
&& bb version
terragrunt version v0.38.2
Terraform v1.2.3
on linux_amd64

Your version of Terraform is out of date! The latest version
is 1.2.5. You can update by downloading from https://www.terraform.io/downloads.html
babashka v0.8.157

Oh nice! Look at that friendly warning about another new version of something you need to depend on... /s

So I open nano and change .tool-versions

cat .tool-versions
terraform 1.2.5
terragrunt 0.38.2
babashka 0.8.157

$ asdf install
babashka 0.8.157 is already installed
terraform --version
Terraform v1.2.5
on linux_amd64

Downloading terraform version 1.2.5 from https://releases.hashicorp.com/terraform/1.2.5/terraform_1.2.5_linux_amd64.zip
Verifying signatures and checksums
gpg: keybox '/tmp/asdf_terraform_GLCZwo/pubring.kbx' created
gpg: /tmp/asdf_terraform_GLCZwo/trustdb.gpg: trustdb created
gpg: key 34365D9472D7468F: public key "HashiCorp Security (hashicorp.com/security) <security@hashicorp.com>" imported
gpg: Total number processed: 1
gpg:               imported: 1
gpg: Signature made Wed 13 Jul 2022 12:23:52 SAST
gpg:                using RSA key 374EC75B485913604A831CC7C820C6D5CD27AB87
gpg: Good signature from "HashiCorp Security (hashicorp.com/security) <security@hashicorp.com>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: C874 011F 0AB4 0511 0D02  1055 3436 5D94 72D7 468F
    Subkey fingerprint: 374E C75B 4859 1360 4A83  1CC7 C820 C6D5 CD27 AB87
terraform_1.2.5_linux_amd64.zip: OK
Cleaning terraform previous binaries
Creating terraform bin directory
Extracting terraform archive
terragrunt 0.38.2 is already installed

$ terraform --version
Terraform v1.2.5
on linux_amd64

Problem solved.

(OK - I admit, the gpg warning seems like a problem that requires solving too, but that problem exists on a different plane of problems than the one I'm writing so much about.)

direnv and .envrc


Once more.

You actually have this problem in general. There are tools in the world that engineers and operators use that depend on specific environment variables being set, and being set to some specific values on a per-project basis. Sometimes you want to have the same variable with different values depending on the directory even within the same project. terragrunt is like this.

One example is having the value of AWS_PROFILE=company-prod-account-profile-name in one specific directory tree (eg for .../terragrunt/prod/cdn/terragrunt.hcl) in your terragrunt project. Then in another directory tree in the same project (eg .../terragrunt/dev/cdn/terragrunt.hcl) you can set AWS_PROFILE=company-dev-account-profile-name.

This allows you to deploy to dev environment or prod environment in the same bash session/software-ide-project, WITHOUT needing to maintain secrets in the open. IE You can use your aws configure and set up $HOME/.aws/config and $HOME/.aws/credentials to contain credentials for as many accounts or clients or projects as you require and use specific credentials only in certain directory trees and their subtrees.

$ nano .envrc
...
direnv: error /path/to/iaac/prod/.envrc is blocked. Run `direnv allow` to approve its content                                                                               
$ cat .envrc
export AWS_PROFILE=company-prod-account

$ direnv allow
direnv: loading ~/path/to/iaac/prod/.envrc
direnv: export +AWS_PROFILE

$ env | grep -i aws
AWS_PROFILE=company-prod-account

$ cd `mktemp -d` # other directory somewhere
direnv: unloading

$ echo 'export AWS_PROFILE=company-dev-account' > .envrc
direnv: error /tmp/tmp.ZFPF6NsWAD/.envrc is blocked. Run `direnv allow` to approve its content

$ direnv allow
direnv: loading /tmp/tmp.ZFPF6NsWAD/.envrc
direnv: export +AWS_PROFILE

$ env | grep -i aws
AWS_PROFILE=company-dev-account

Problem solved.

Use the same for any other environment variable that may configure any piece of kit that follows the 12-factor app design.

Other variables I like to set like this: AWS_ACCOUNT_ID, VAULT_TOKEN, VAULT_HTTP_ADDR, SMTP_FROM_ADDRESS &c.

## emacs

In your $HOME/.emacs.d/init.el, using quelpa and use-package together:

(use-package quelpa
  :config
  (setq quelpa-upgrade-interval 7)
  (add-hook #'after-init-hook #'quelpa-upgrade-all-maybe))

(require 'quelpa-use-package)

(use-package direnv
  :config (direnv-mode))

(use-package asdf-vm
  :quelpa (asdf-vm :fetcher github :repo "delonnewman/asdf-vm.el"))
(require 'asdf-vm)
(asdf-vm-init)


vim: direnv & asdf.
vscode:direnv. (I could not find a vscode plugin for asdf-vm)

Conclusion


Now that we know how to pin tool versions and environment variables to specific directories we can actually start to set up a infrastructure as code project. More about this next time.

Unix rules for life

  • Keep it simple: It's cheaper and easier to carry around.
  • Do one thing at a time: Multitasking is a lie.
  • Network: You were born to connect.
  • Say what you mean; nothing is truer than the truth.
  • Hack: Trial and error is the only way we learn anything
  • Be who you are: Even a bent wire can carry a great light.
  • Use leverage; a bigger hammer isn't always the answer.
  • Use what you have: never dig diamonds with a brick of gold.
  • Have faith; all's possible, except maybe skiing through a revolving door.
  • Think ahead, but don't worship your plans; remember today is the first day of the rest of your learning experience.

devopsdays 2018 speaking engagement (now with more prosperity)

I've been invited to speak at devopsdays 2018 Cape Town on the 20th of September 2018.

My talk will be about Hashicorp's nomad, a job schedule. 

Here are my conference slides for future posterity.

EDIT:

I'm adding a link to the source code of the slides on here too.

Seriously. Please give the slides a look so what I'm about to say makes sense, it'll take you 2 seconds. The link takes you to a single-page html/javascript app that I use whenever I have to present ideas to groups of people. I run that code on my laptop and connect an HDMI for the overhead projector...

I was in a serious technical conversation today and I took an argumentative position that clojure/script is a pragmatic implementation language for real software projects by real developers, engineers and devopses, facing real project deadlines. My basic argument is that if you use clojure/script, you need fewer lines of code to achieve a specific result than you'd need if you used some other language. That fewer lines of code means there are fewer things to comprehend or mentalise while you are busy doing maintenance work on that code. Also, fewer lines of code, means fewer logical places where things can go wrong.

The counter argument is that syntax sometimes aids in readability. I don't know about that, to be honest. If you can get by without syntax, then you're left with the meaning behind the code. And if you express the meaning meaningfully, then you get a piece of code that's a delight to work with.

(PS/rant - I hate traditional slide-deck software. I'd rather edit a piece of code to build my slide content than dick around with a mouse and icons I can barely see.)