[tex-live] General test suite for TeX-Live

[Karl Berry]

All I have to add is that my previous attempt at automated testing
(years and years ago) is in ./Build/tests.  The directory names will
give the idea:

./Build/tests/
Makefile	 dvi-latex0-small2e/   largefile/
README		 dvi-latex1-sample2e/  pdf-context0-hello/
asytestlibs.asy  dvi-latex2-pdfprim/   pdf-context4-select/
checkdvi.pl*	 dvi-latex5-tugboat/   pdf-latex0-small2e/
checkpdf.pl*	 dvi-tex0-story/       pdf-tex0-story/
common.mak	 dvi-tex5-tugboat/     tryconcat/

My approach for PDF (DVI is easy) was to make images and compare them.
Turned out to be impractical, not surprisingly.

The advent of l3build and "reproducible builds" for PDF should make the
idea much more viable.  -k

P.S.
> https://piratenpad.de/p/TeXLiveTesting

I don't understand why you put ideas in some read-only temporary url.
Here is the text you wrote, for reading/archival on the mailing list
like everything else in the thread.

Thoughts regarding automated TeX LIVE Testing

Requirements

* must run on at least Mac OS, Windows and Linux, more platforms are appreciated
* shall not require manual checks
* shall be self contained
* shall test a majority of user scenarios (letters, reports, books, etc.)
* many small tests may be helpful, bigger test scenarios must not be omitted (I, for example, ran a 600 pages dissertation as part of my testing)
* cannot test each and every scenario users may come across


Design Ideas

* use python (as I know it best)
* make use of its unit test facilities
* run a testcase by calling TeX engine n-times on specific file
* check if the process ended properly
* proper process ending does not mean that file is correct
* Gather statistics on file: Size, number of pages, ...
* Convert generated file to image and run imagemagick to collect statistical data. 
* Compare statistical data to known result. If theshold > level ==> testcase failed.