[tex-k] File searching very slow

[Johannes Kuester]

Hi Karl,

problem solved … Thank you very much for your help!

In fact, it *was* due to the casefolding stuff.

I wasn't aware that this might have such an impact on performance.

Incidentally, I tested in several directories which have a lot of files
each (mainly "glyph pictures" from font production with metatype1 /
metapost). This way, I didn't notice that the large amount of files
causes the problem, as this never was a noticeable performance issue on
my previous computer (where I still had texlive2017, so without
casefolding).

On my laptop, there are much less files in the respective directories,
as I never did a full font production run there.

After moving / removing most of the files, performance improved a lot.
But still it is much better with "texmf_casefold_search = 0".

So, "time latex …" on my latex run gives the following with

texmf_casefold_search = 1

    latex glyphproof  0.35s user 2.05s system 99% cpu 2.397 total


texmf_casefold_search = 0

    latex glyphproof  0.28s user 0.02s system 99% cpu 0.297 total


("glyphproof" is a 55-page document with about 100 \includegraphics
commands and almost no text; the directory still contains about 2.500
files).


Best,

Johannes



Am 14.04.21 um 23:20 schrieb Karl Berry:
>     np> Something to do with the upper/lower case changes that were
>     introduced inbetween?
>
> I did the casefolding stuff in 2018, so it's present in both JK's (good)
> 2019 and (bad) 2021. It's mentioned (the same way) in both logs.
>
> Nevertheless, Johannes, if you want to disable the casefold stuff for
> testing, change the "texmf_casefold_search = 1" assignment in texmf.cnf
> to 0.
>
> I don't know of any code changes that would have made such a difference.
> And, since no one else has reported such problems (to my recollection),
> it seems it must be something specific to Johannes's system.  I'm not
> smart enough to guess what it is. I know little about Macs.
>
> It does seem like failing to use ls-R is the most likely culprit, but
> that's not what the logs say. I am at a loss.
>
> I can't imagine how zsh vs. bash would make a difference. The shell is
> not called with these filename lookups.
>
> Johannes, here are some other ideas (none particularly good):
>
> 1) run kpsewhich under a debugger on the slow system, and interrupt it
> while it's laboriously searching. Doing that a few times should at least
> show what it's churning away at.
>
> 2) install a minimal (or not so minimal) TL from scratch on
> the new system, and see if it also is slow. See
> https://tug.org/texlive/acquire-netinstall.html.
>
> 3) run some non-TeX operation on both systems to check that speed
> isn't altered for that too. Maybe, I don't know, something like
>   find texmf-dist -name minimal.cls -print.
>   
> 4) install 2019 on the new system, and/or 2021 on the old, for direct
> comparison of the code.
>
> 5) to involve people who know more about Macs, you could write
> mactex at tug.org. I know little about Macs, especially the "new" ones with
> all their "security" features that can do/prevent all kinds of normal
> Unixy behavior. I'm not aware of anything that would cause this
> slowdown, but then, I wouldn't, necessarily.
>
> E.g., I wonder about some kind of system auditing of every disk access
> (maybe at the filesystem level). Maybe check system directories/logs for
> differences?
>
>     normal/expected/correct output from debugging, and what is an error or
>     unusual/suspicious/undesired output. Some examples might be helpful here.
>
> I just can't say what's normal and what's not normal. It depends on
> what's being done.
>
> I systematized the two logs ("2019" vs "2021" and "x86_64-darwin"
> vs. "universal-darwin") and did a diff, and saw nothing.
>
> Evidently it would be helpful to be able to insert elapsed-time values
> onto the log lines (to see where the time is being consumed), but
> unfortunately I'm not going to be able to look into that any time
> soon. Maybe you or someone else here would like to take that up.
> I doubt it would be terribly deep. --good luck, karl.