Tests directory scaling problem.

[cmarcelo]

Gather ideas for our t/ testing and validation suite. t/ is getting to be a maintenance headache. It's not going to scale another 2x, 3x, 4x, whatever. Look at new modern languages (eg, scala, slate, pypy), at big languages (ghc, mono), and elsewhere (where?). Bring us ideas. Skills: as described.

[unobe]

slate

• tests separated by concept (e.g, digraph, dictionary, regex, i18n, unicode)
• benchmark tests
• regression tests

pypy

• unit tests, all which reside in the test/ subdirectory of the module they're testing
  • interpreter-level tests have access to interpreter (implementation) details
  • application-level tests don't have access to interpreter details, so they're standard python

scala

• uses ant
• custom test functions harness,even though there is a JUnit port:
• tests themselves are categorized into separate directories spattered with bug fixes and concept tests:
  • ant (the build system)
  • cli (command line interface related)
  • jvm (JVM 1.4 related)
  • jvm5 (JVM 1.5 related)
  • lib (some .jar's for jvm tests)
  • neg, pos, res (seems that neg doesn't have corresponding check files, pos does, and res I don't know what it does)
  • run (runtime related)
  • script (running script w/arg)
  • shootout (language shootout)
• it seems that xx-uncategorized/ dir is a good place for all the bug reports to be filed in when the reporter isn't sure where it's supposed to go, then someone being more in-the-know can move it to the appropriate place.

ghc

• see http://darcs.haskell.org/testsuite/README
• briefly:
  • uses make:
    • make TEST_HC=another_haskell_compiler_exe
    • make TEST=testname
    • make accept ( update tests to accept whatever is current output. usually run make accept TESTS=testname)
  • top-level directories in ghc-regress are categorized by parts of the compiler (e.g., typechecker), each of which may have at most three subdirectories: should_compile, should_fail, should_run
  • possible files for each test which test harness can use (besides testname.hs): testname.stdout, testname.stdin, testname.stderr
  • test harness described at url above, and is written in python
• I like the ability of automatically updating the correct stdout & stderr for a test with make accept. this looks like a more advanced version of what's happening with the p6regex rx_* stuff.
• the make TEST_HC=compiler looks like something perl6 could use, too (of course, it would probably be TEST_PERL or somesuch). putting this in might help with determining what is and is not specific to the pugs implementation, and help in creating the tiers discussed in ticket #10.