[tex-k] TL23 test failure, eptexdir/wcfname

Ken Moffat zarniwhoop at ntlworld.com
Sun Mar 26 01:42:28 CET 2023 

On Fri, Mar 24, 2023 at 04:56:19PM -0600, Karl Berry wrote:
> Hi Ken (and Takuji and all),
> 
>      I noticed in the copy that came back to me in the mailing lists
>      that soem of the codepoints looked different (as if random 8-bit
>      ASCII-derivative), 
> 
> Indeed. In my experience, it's not reliable to pass around lots of
> arbitrary UTF-8, let alone non-UTF-8 (SJIS/EUC), in email. I usually
> compress such things in binary attachments if it's important.
> 

Thanks, will use gzip for the attachments here.

At this point I will mention that I was also mistaken about the
problem with 'mv' and non-existent files - they never exist (at least
in the eptex test) and that is not a cause of the failure.

> (Unfortunately the mail archives on tug.org are not UTF-8 friendly
> either, something I hope to remedy in the not-impossibly-distant future,
> but not for a while yet.)
> 
>      but the error appears to be that the files
>      ending in '-tmp.tex' have not been created.
> 
> I don't know much about this test (except that it's complicated and
> fragile :), but I think that the script web2c/tests/fn-generate.perl
> (run from wcfname.test) generates a .tex file.  Then the .tex file is
> run through eptex -ini --shell-escape to generate the -tmp file:
> 
>         $fnameT =~ s/\.tex$/-tmp.tex/;
>     ...
>     \\immediate\\openout1=$fnameT
>     ...
> 
> And there various encoding conversions along the way.
> 
> So I suppose something in there is failing.
> 
> I'm impressed that Takuji was able to write such a complicated test
> (dealing with UTF-8 conversions) to work on just about all systems.
> 
> I surmise that whatever is causing the failure on LFS is going to be
> pretty impossible for anyone not on LFS to debug. Maybe have to run the
> steps of the test by hand on LFS and non-LFS in parallel, to see what
> happens differently and where.
> 
> The good news is that it's unlikely to make any practical difference if
> the test passes or not. The functionality only matters (as I understand
> it) to people using eptex with non-UTF-8 filenames/data. My hunch is
> that number of such people on LFS is zero.
> 
> Best,
> Karl

It was also reported by Johannes (added to Cc), for
Slackware-current so it appears to be something in a recent package,
whether perl or a toolchain package, which causes it.  I know that
Arch is often using newer packages than LFS, but for most packages
they do not seem to run tests - not sure if that applies for
texlive.

I've no idea about a lot of the details in the eptex tests, but I
used to do application support (on IBM mainframe COBOL) and I think
I know enough to instrument bash.

It is definitely not my locale which is a problem, but I do not know
if the eptex tests require the legacy Japanese locales and will skip
if not present ?  In theory, anyone who builds current LFS and runs
tests has euc and sjis locales as well as several UTF-8 including
C.UTF-8.  In practice, our people install as few or as many locales
as they wish.

The eptex tests run two sets of tests, it is the second set which
fail.  I'm attaching: the diff 'wcfname-eptex5.patch' (for
texlive-source) I eventually used, together with gzipped
web2c-debug,log, created-* (what was created by running eptex), and
expected* (what was expected, i.e. the files in the source).

ĸen
-- 
Often there is no more than a little plaque to reveal that, against
all gynaecological probability, someone very famous was born halfway
up a wall.                 -- Equal Rites
-------------- next part --------------
diff -Naur texlive-20230313-source/texk/web2c/eptexdir/wcfname.test texlive-20230313-source.patched/texk/web2c/eptexdir/wcfname.test
--- texlive-20230313-source/texk/web2c/eptexdir/wcfname.test	2023-02-14 13:14:50.000000000 +0000
+++ texlive-20230313-source.patched/texk/web2c/eptexdir/wcfname.test	2023-03-26 00:06:48.590937946 +0000
@@ -24,6 +24,7 @@
   loc=C.UTF-8
   LC_ALL=$loc; LANGUAGE=$loc; export LC_ALL LANGUAGE
 fi
+echo "DEBUG001 loc is $loc"  
 
 perl $srcdir/tests/fn-generate.perl eptests || exit 128
 
@@ -32,10 +33,17 @@
 for ienc in euc sjis; do
 for doc in fn-$fenc fnさざ波-$fenc; do
 
+  echo "DEBUG002 running eptx with fenc $fenc ienc $ienc $doc"  
   echo '>>> Document:'$doc '  File Encoding:'$fenc '  Internal Encoding:'$ienc
   ./eptex -ini -interaction nonstopmode -jobname=$doc-$ienc -kanji=$fenc --kanji-internal=$ienc --shell-escape $doc.tex >eptests/$doc-$ienc-term.log || rc=1
+  echo "DEBUG003 rc after eptex is $rc"  
+  echo "DEBUG004 current fn-prefixed files after eptex:"  
+  ls fn*  
   mv $doc-$ienc.txt $doc-$ienc.log fn*-tmp.tex eptests/
+  # mv always reports error for fn-*.tmp  
+  #echo "DEBUG005 Status from mv was $?"  
   diff eptests/$doc-$ienc.txt $srcdir/tests/fn-$fenc.txt || rc=2
+  echo "DEBUG006 rc is $rc after running diff from $doc-$ienc to fn-$fenc.txt"  
 
 done
 done
@@ -54,10 +62,19 @@
     if [ $fenc != sjis ]; then ienc="euc"; fi
   fi
 
+  echo "DEBUG010 'regacy or legacy encoding' fenc $fenc ienc $ienc $doc" 
   echo '>>> Document:'$doc '  File Encoding:'$fenc '  Internal Encoding:'$ienc
   ./eptex -ini -interaction nonstopmode -jobname=$doc-$ienc -kanji=$fenc --kanji-internal=$ienc --shell-escape $doc.tex >eptests/$doc-$fenc-term.log || rc=3
+  echo "DEBUG011 rc after eptex is $rc"  
+  echo "DEBUG012 current fn-prefixed files after eptex:"  
+  ls fn*  
+  # mv always reports error for fn-*.tmp  
   mv $doc-$ienc.txt $doc-$ienc.log fn*-tmp.tex eptests/
+  #echo "DEBUG013 Status from mv was $?"  
   diff eptests/$doc-$ienc.txt $srcdir/tests/fn-$fenc.txt || rc=4
+  echo "DEBUG014 rc is $rc after running diff from $doc-$ienc.txt to fn-$fenc.txt"  
+  cp -v eptests/$doc-$ienc.txt /tmp/created-$doc-$ienc.txt  
+  cp -v $srcdir/tests/fn-$fenc.txt /tmp/expected-fn-$fenc.txt  
 
 done
 done
-------------- next part --------------
A non-text attachment was scrubbed...
Name: web2c-debug.log.gz
Type: application/x-gunzip
Size: 2368 bytes
Desc: not available
URL: <https://tug.org/pipermail/tex-k/attachments/20230326/b299e5b9/attachment.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: created-fn???-euc-euc.txt.gz
Type: application/x-gunzip
Size: 109 bytes
Desc: not available
URL: <https://tug.org/pipermail/tex-k/attachments/20230326/b299e5b9/attachment-0001.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: created-fn???-sjis-sjis.txt.gz
Type: application/x-gunzip
Size: 117 bytes
Desc: not available
URL: <https://tug.org/pipermail/tex-k/attachments/20230326/b299e5b9/attachment-0002.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: expected-fn-euc.txt.gz
Type: application/x-gunzip
Size: 77 bytes
Desc: not available
URL: <https://tug.org/pipermail/tex-k/attachments/20230326/b299e5b9/attachment-0003.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: expected-fn-sjis.txt.gz
Type: application/x-gunzip
Size: 76 bytes
Desc: not available
URL: <https://tug.org/pipermail/tex-k/attachments/20230326/b299e5b9/attachment-0004.bin>

More information about the tex-k mailing list.