How should I deal with packages that require FFI external libraries?

[Christophe Jorssen]

Thank you Norbert, Karl and Max for your detailed answers.

I continue the discussion here as the main issue is related to
distribution of TeX related materials.

Le sam. 15 avr. 2023 à 03:29, Max Chernoff <mseven at telus.net> a écrit :
> >
> > 1) When the external library is loaded, I have something like:
> >
> > lib = ffi.load('<path to the .so file>')
> >
> > For now, the path is hardcoded. Where should I put the .so file so
> > that luatex finds it? What is good practice for CTAN/TeXLive? Does
> > luatex rely on kpsewhich for that kind of thing?
>
> To get the path to the file, the "proper" way is to use
>
>    kpse.find_file("library-name", "clua")
>
> which will look in
>
>    $ kpsewhich --var-value=CLUAINPUTS
>    .:/usr/local/texlive/2023/bin/x86_64-linux/lib/{kpsewhich,unsetengine,}/lua//
>
> by default. If you're distributing your package in a .zip/.tar, that's
> not the easiest path to add things to, so something like
>
>    kpse.find_file("library-name.so") or kpse.find_file("library-name.dll")
>
> will likely be easier to distribute (although not technically
> "correct").
>

Thanks a lot for this suggestion.

> > 2) Can I distribute the .so file (or .dll) with the CTAN/TeXLive
> > package
>
> My full TL installation doesn't have any shared libraries installed
>
>    $ find /usr/local/texlive/2023 -iname '*.so'
>
> so you probably can't distribute it in TL. TLContrib and MiKTeX _might_
> accept it, I have no idea.
>
> There aren't any .so's on CTAN, but there are a few .dll's. All the
> .dll's are over 15 years old though, so I have no idea if they'll accept
> any new ones. You'll have to ask them.
>

I think I understand why a TeX distribution didn't need to distribute
.so or .dll in the past. But with luatex and its "modular" approach
(everything that can be done outside the engine must be done outside,
at format level, with lua or even with ffi), maybe that policy could
change?

I guess there are security reasons or storage reasons or...

As I said earlier, I am not an expert and as a TeXLive user I will
always abide by the rules stated by those who know better (Karl,
Norbert and all the others that are doing an amazing job with the
entire TeX ecosystem).

TLContrib is indeed a good idea (Norbert?) but, in my opinion, not for
the average TeX user I'll talk about below.

> > 3) Does FFI work in luatex on all architectures supported by TeXlive?
>
> The documentation
>
>    https://github.com/TeX-Live/luatex/tree/master/source/texk/web2c/luatexdir/luaffi
>
> says that it only supports Linux and macOS, x86 and x86_64. There is a
> msvc build file and a bunch of files with "win" in the name, so I'm
> pretty sure that it supports Windows too. There are also support files
> with "arm" in the name, but they're 5 years old so probably Linux only.
>

Ok. I'll have a look. But mac M{1,2} won't work easely. Got it.

> > By the way, I cannot find any reference to the ffi library in the
> > luatex manual: I missed something?
>
> You are correct, there is no reference to it at all in the manual. I
> didn't even know that LuaTeX included an FFI library until this email.
>

I got aware of FFI with a couple of questions on tex.stackexchange.com
(https://tex.stackexchange.com/questions/tagged/ffi) and an article in
tugboat written by Henri Menke:

https://tug.org/TUGboat/tb39-1/tb121menke-ffi.pdf

[...]

> > I'm currently writing a generic luatex package that relies on an
> > external library (https://github.com/CoolProp/CoolProp) called via the
> > FFI library in luatex
>
> A much simpler way to do all of this would be to use PythonTeX. CoolProp
> distributes binary wheels, so you'd just need to tell your users to run
>

That would be a nice idea but probably not for the average TeX user I
know. I teach physics and most of my colleagues just want to use LaTeX
out of the box. They often stick with what works for them (even some
really outdated bad practice). If I come to them and say: "Well, there
is a nice new package that can draw advanced thermodynamics diagrams.
All you have to do is update your TeXLive and run luatex with
--shell-escape enabled", maybe they'll give it a try. I know for sure
that if they have to install an entire python ecosystem, use pip
install, understand how pythontex works, mess with their system
python, etc., they just won't use the package.

>
> You could also write a Lua module in C directly. This would require you
> to write some C code, but it may end up being easier than using the ffi
> library since (1) all the CoolProp example code is probably in C, and
> (2) the compiler would give you proper warnings and errors so you'll
> always be sure that you're calling the functions correctly.
>

AFAIU, CoolProp is in C++ but maybe the difference is not relevant
here (I don't know what I'm talking about).

That would not solve the initial problem of distributing the compiled C code?

Thanks again for those detailed answers.

Best regards

-- 
Christophe