File watching fails on MacOS Big Sur beta

[chancerussell]

When starting up file watching via wkf/hawk, an exception is thrown. After a little digging, this looks to be caused by changes to how MacOS loads dylibs in 11.0+.

It looks like the JNA folks are already working on a fix, so once that lands, we can update (or fork) BarbaryWatchService, then update wkf/hawk to use it.

Stack trace:

[Figwheel:SEVERE] java.lang.UnsatisfiedLinkError: Unable to load library 'Carbon': dlopen(libCarbon.dylib, 9): image not found
Exception in thread "main" java.lang.NoClassDefFoundError: Could not initialize class com.barbarysoftware.jna.CarbonAPI
        at com.barbarysoftware.jna.CFStringRef.toCFString(CFStringRef.java:10)
        at com.barbarysoftware.watchservice.MacOSXListeningWatchService.register(MacOSXListeningWatchService.java:30)
        at com.barbarysoftware.watchservice.WatchableFile.register(WatchableFile.java:30)
        at com.barbarysoftware.watchservice.WatchableFile.register(WatchableFile.java:39)
        at hawk.watcher$fn__133.invokeStatic(watcher.clj:102)
        at hawk.watcher$fn__133.invoke(watcher.clj:99)
        at hawk.watcher$fn__38$G__29__47.invoke(watcher.clj:24)
        at hawk.core$watch_BANG_.invokeStatic(core.clj:83)
        at hawk.core$watch_BANG_.doInvoke(core.clj:59)
        at clojure.lang.RestFn.invoke(RestFn.java:410)
        at clojure.lang.AFn.applyToHelper(AFn.java:154)
        at clojure.lang.RestFn.applyTo(RestFn.java:132)
        at clojure.core$apply.invokeStatic(core.clj:657)
        at clojure.core$apply.invoke(core.clj:652)
        at figwheel.main.watching$alter_watches.invokeStatic(watching.clj:17)
        at figwheel.main.watching$alter_watches.invoke(watching.clj:11)
        at clojure.lang.Atom.swap(Atom.java:51)
        at clojure.core$swap_BANG_.invokeStatic(core.clj:2345)
        at clojure.core$swap_BANG_.invoke(core.clj:2337)
        at figwheel.main.watching$add_watch_BANG_.invokeStatic(watching.clj:22)
        at figwheel.main.watching$add_watch_BANG_.invoke(watching.clj:21)
        at figwheel.main.css_reload$start_STAR_.invokeStatic(css_reload.cljc:188)
        at figwheel.main.css_reload$start_STAR_.invoke(css_reload.cljc:187)
        at clojure.lang.Var.invoke(Var.java:381)
        at figwheel.main$watch_css.invokeStatic(main.cljc:1188)
        at figwheel.main$watch_css.invoke(main.cljc:1185)
        at figwheel.main$config_watch_css$fn__7373.invoke(main.cljc:1195)
        at figwheel.main$default_compile.invokeStatic(main.cljc:1848)
        at figwheel.main$default_compile.invoke(main.cljc:1812)
        at figwheel.main$build_main_opt.invokeStatic(main.cljc:526)
        at figwheel.main$build_main_opt.invoke(main.cljc:521)
        at cljs.cli$main.invokeStatic(cli.clj:636)
        at cljs.cli$main.doInvoke(cli.clj:625)
        at clojure.lang.RestFn.applyTo(RestFn.java:139)
        at clojure.core$apply.invokeStatic(core.clj:659)
        at clojure.core$apply.invoke(core.clj:652)
        at cljs.main$_main.invokeStatic(main.clj:61)
        at cljs.main$_main.doInvoke(main.clj:52)
        at clojure.lang.RestFn.applyTo(RestFn.java:137)
        at clojure.core$apply.invokeStatic(core.clj:657)
        at clojure.core$apply.invoke(core.clj:652)
        at figwheel.main$_main$fn__7768.invoke(main.cljc:2165)
        at clojure.core$with_redefs_fn.invokeStatic(core.clj:7434)
        at clojure.core$with_redefs_fn.invoke(core.clj:7418)
        at figwheel.main$_main.invokeStatic(main.cljc:2163)
        at figwheel.main$_main.doInvoke(main.cljc:2142)
        at clojure.lang.RestFn.applyTo(RestFn.java:137)
        at clojure.lang.Var.applyTo(Var.java:702)
        at clojure.core$apply.invokeStatic(core.clj:657)
        at clojure.main$main_opt.invokeStatic(main.clj:317)
        at clojure.main$main_opt.invoke(main.clj:313)
        at clojure.main$main.invokeStatic(main.clj:424)
        at clojure.main$main.doInvoke(main.clj:387)
        at clojure.lang.RestFn.applyTo(RestFn.java:137)
        at clojure.lang.Var.applyTo(Var.java:702)
        at clojure.main.main(main.java:37)

[bhauman]

@chancerussell does the following watcher work on Big Sur?

https://github.com/gmethvin/directory-watcher

[bhauman]

Actually forget that.

It would be nice if we could fall back on a simple java based watcher when things like this happen.

[chancerussell]

Yeah, according to their issues they’re bit by the same exact problem.

[rgm]

I found a temporary workaround by falling back to polling, so Figwheel can at least run in a (slightly) degraded fashion. As far as I can tell, the major cost is degraded battery / higher power consumption.

The hawk lib checks to see if it's working on macOS, and uses a multimethod to set up a Barbary watcher. Thus, I can just swap in that multimethod before I start up figwheel. In my user.clj I've added:

(ns user
  (:require [figwheel.main.api :as fig]))

;; Patch hawk for macOS Big Sur
;; hawk can't find the carbon lib any more, so the underlying barbary watcher
;; doesn't work for getting OS-level file change events. We have to fall back to
;; polling by monkey-patching the barbary-vending multimethod for now.
(import [hawk SensitivityWatchEventModifier])
(require '[hawk.watcher])
(defmethod hawk.watcher/new-watcher :barbary [_]
  (prn "WARN - falling back to filesystem polling on macOS")
  (let [sensitivity SensitivityWatchEventModifier/HIGH]
    (hawk.PollingWatchService. sensitivity)))

(fig/start {:mode :serve} "dev")

It doesn't have to be in user.clj, but it does have to run before Figwheel gets going. I start the Figwheel process manually, so I haven't tried it with clj -m figwheel.main, but I would hope this would load and patch OK in that circumstance too.

[rgm]

Of course, now that I've taken enough time to look through the code in preparing a PR, I see that we can explicitly opt-out of using the OS's FS events with https://figwheel.org/config-options.html#hawk-options . I don't think I understood why that would be useful before. This is a lot simpler than my shenanigans above.

[bhauman]

In summary, this problem is a temporary problem that is being worked on and will hopefully be fixed by the time Big Sur is released this fall.

Folks that are using the Big Sur preview need to fall back to a :polling watcher in the :hawk-options Figwheel option.

I.E.

:hawk-options {:watcher :polling}

Big thanks to @rgm for taking the time to track this down.

[rgm]

I'm not sure where this leaves the "real" fix ... is it something like having figwheel rely directly on the patched jna dependency in deps.edn once one's available? Or do we work through getting the upstream deps updated through barbary -> hawk -> figwheel?

[bhauman]

We'll make sure that hawk has this fix.

[dbwiddis]

FYI, JNA 5.6.0 has been released.