How should I deal with packages that require FFI external libraries?

[Max Chernoff]

Hi Christophe,

(Karl and Norbert, a question for you at the bottom)

On Sat, 2023-04-15 at 13:24 +0200, Christophe Jorssen wrote:
> > 
> I think I understand why a TeX distribution didn't need to distribute
> .so or .dll in the past. But with luatex and its "modular" approach
> (everything that can be done outside the engine must be done outside,
> at format level, with lua or even with ffi), maybe that policy could
> change?

(I'm not on the TL team, so this may not be accurate)

There isn't any specific policy against including binary libraries, it's
just (1) TL builds for lots of obscure platforms and you're unlikely to
compile the library for all of them, (2) binaries are less auditable
than interpreted code, and (3) binaries make things more complicated for
downstream distributors. 

If you look at biber and luametatex, they just compile binaries for a
few common platforms then upload them to CTAN/TL. This works pretty
well, and there's no technical reason that you couldn't do the same.
>From a policy level though, I don't think that anyone is too happy about
the biber/luametatex situation and they'd rather not add another similar
package.

> > > 3) Does FFI work in luatex on all architectures supported by TeXlive?
> 
> Ok. I'll have a look. But mac M{1,2} won't work easely. Got it.

If the macOS arm ABI is close enough to the Linux arm ABI, it might just
work. If it doesn't work, you _might_ be able to convince one of the
MacTeX maintainers to add support, but that wouldn't be distributed
until TL2024 at the earliest.

> > 
> I got aware of FFI with a couple of questions on tex.stackexchange.com
> (https://tex.stackexchange.com/questions/tagged/ffi) and an article in
> tugboat written by Henri Menke:
> 
> https://tug.org/TUGboat/tb39-1/tb121menke-ffi.pdf

Interesting. Thanks!
> 

> > You could also write a Lua module in C directly. This would require you
> > to write some C code, but it may end up being easier than using the ffi
> > library since (1) all the CoolProp example code is probably in C, and
> > (2) the compiler would give you proper warnings and errors so you'll
> > always be sure that you're calling the functions correctly.
> 
> That would not solve the initial problem of distributing the compiled C code?

No, it wouldn't really solve any of the problems that you're having.
I've never used the ffi library before, but I've had a pretty good
experience writing Lua modules directly in C. If the ffi library is
working well for you though, then keep doing that. 

> A much simpler way to do all of this would be to use PythonTeX. CoolProp
> distributes binary wheels, so you'd just need to tell your users to run
> 
> > That would be a nice idea but probably not for the average TeX user

Ok, new idea.

If you load your package with no options, then it will try to load
libcoolprop.so from CLUAINPUTS, then from the standard system paths
(/usr/lib, /usr/local/lib, etc.). You can do this with something like

   local path = kpse.find_file("libcoolprop", "clua")
   if path then
       ffi.load(path)
   else
       ffi.load("coolprop")
   end
   
and if it doesn't find the library, it just throws an error and exists.

However, if you load the package with a "download" option, then it will
use luasocket to download "libcoolprop.so" to $TEXMFCACHE, then it will
load the library from there.

This makes it so that non-technical users can easily install the
binaries at runtime, and TL won't need to distribute any binaries.

Karl and Norbert,

Would doing this be okay in a TL package? The library that it's
downloading is open source, the package would only do this if the user
requests it via an option, and the download would be user-local. Doing
this seems fine to me, but I can also understand a policy against
downloading and running external code.

Thanks,
-- Max