root/t/README

===============================================================================
|    ______                           ______                 __               |
|  /\   __ \                         /\__  _\               _\ \__            |
|  \ \  \/\ \ __  __  ______  ______ \/_/\ \/ ______  _____/\__  _\______     |
|   \ \   __//\ \/\ \/\  __ \/\  ___\   \ \ \/\  ___\/\  ___\_/\ \/\  ___\    |
|    \ \  \/ \ \ \_\ \ \ \/\ \ \___  \   \ \ \ \  ___\ \___  \\ \ \_\___  \   |
|     \ \__\  \ \____/\ \____ \/\_____\   \ \_\ \_____\/\_____\\ \ _\\_____\  |
|      \/__/   \/___/  \/___/\ \/____/     \/_/\/____/ \/____/  \/__//____/   |
|                        /\____/                                              |
|                        \/___/                                               |
|                                                                             |
===============================================================================

                     Welcome to the Pugs Test suite.

Pugs is currently being developed in a highly test-driven manner. Tests are 
written for both implemented and unimplemented features and are based roughly
on the Perl6 Language Synopsis documents. All are welcome and encouraged to
contribute to this test suite. The part under spec/ is considered to be the
offical Perl 6 test suite, and should only test what is specified in the
Synopsis documents. It is also used by other Perl 6 implementations.

-------------------------------------------------------------------------------
Getting Started
-------------------------------------------------------------------------------

Here are some basic guidelines to help you get started writing tests for Pugs. 

- Prerequisites

Please read the Perl 6 Spec first, namely, the Synopses:

  http://perlcabal.org/syn/

or directly from the SVN repos:

  http://svn.perl.org/perl6/doc/trunk/design/syn

A good grasp of the Perl 6 language itself is very important to writing good
tests.

- If you are unsure of something, don't hesitate to ask.

If you have a question about a test, written or unwritten, log on to #perl6 on
irc.freenode.net or send an email to perl6-compiler and ask someone about it.
If you have read the synopses very carefully and are still unsure about a
perl6 language element, we encourage you to ask on #perl6 or email the
perl6-language list and get clarification. Pugs and Perl 6 are group efforts
and asking questions is a good thing.

- What to test

A number of Pugs hackers on #perl6 run regular smoke tests, and you can run 
your own using 'make smoke'. The smoke test produces an HTML graph of what tests
are passing, and what aren't. This can be a good place to start. There's a smoke
server on the web:

  http://m19s28.dyndns.org/cgi-bin/pugs-smokeserv.pl?

There is also a cross linking which is made between the tests and the synopses.
It's generated by running the 'util/smartlinks.pl' script. You
can generate an HTML version of the latest Synopses cross-referenced with test
file snippets from the test suite using the following command:

    $ util/smartlinks.pl t/*/*.t t/*/*/*.t

There's a cross-referenced version of Synopses at

    http://perlcabal.org/syn/

which is automatically updated by the smartlinks.pl script on feather once
an hour.

- Use the Test module.

We have created a basic Test module found in ext/Test/lib/Test.pm. It is 
written in Perl 6 and implements the TAP protocol (and so can be used with 
Test::Harness). The module has its own documentation and I encourage you to 
read it. 

- Pugs tests should have a non-she-bang line of "use v6;".

This line helps both Test::Harness as well as the 'prove6' utility when 
running Pugs tests. 

- Place tests in the appropriate folder.

We have recently undergone a re-organization of the test suite in order to 
make it easier to find what has and has not been tested. It is important as 
the test suite grows that we try to keep this organization. If you have a test
and are unsure of where to put it, ask on #perl6 for help, or put it in the
general/ folder and email perl6-compiler and let us know.

Tests that belong to one synopsis document should go under
t/spec/SXX-SECTION/, where XX is the number of the synopsis and SECTION is the
title (or an important key word of the title) under which the tested behaviour
is described.

Warning: please grep the whole test suite to ensure the tests you want to add
are not already in some file.

- Run the test file with util/prove6 to test your tests.

  util/prove6 t/foo/bar.t

Make sure the outputs are what you *can* expect even if your tests fail.

- If possible, please smart link your tests to the Synopsis

Smart links are explained on the wiki:

  http://rakudo.org/perl6/index.cgi?smart_linking

And in the POD of smartlinks.pl:

  perldoc util/smartlinks.pl

You may also find the following two pugs.blogs.com posts helpful:

  http://pugs.blogs.com/pugs/2006/08/integrating_the.html

  http://pugs.blogs.com/pugs/2006/09/check_smoke_res.html

  http://pugs.blogs.com/pugs/2006/09/the_benefits_of.html

Everytime you have added smartlinks to some .t file, remember to run
util/smartlinks.pl to verify the links' validity:

   $ perl util/smartlinks.pl --check t/some/test.t

By default, the freshness of the synopses will checked every time. Use "--fast"
to skip that step.

- Dealing with parse failures

When developing tests for features that have not been implemented yet,
often you find yourself writing code that doesn't compile. Don't get
stuck on this: wrap the new code with eval and test it anyway. Just make
sure that the test fails as long as the eval does, and until the feature
has been implemented correctly.

Sometimes code is so futuristic, it can even confuse eval. We call this a
"hard parsefail". When this happens, comment out the failing code, but mark
it so it doesn't get forgotten, like so:

    todo :pugs<6.28.0>, v6_pm<0.110>;
    flunk("FIXME parsefail");
    #ok eval('my code here');
    
Or another alternate style is as follows:

    ok eval('# $code.which(%will, @fail)');

    ok eval(q{
        blah blah blah
    });

    is(eval(q{
        my $val;
        # some code here...
        $val;
    }), $expected, 'description');

which essentially comments out your eval, and returns 'undef' to ok().

- When TODO and when not TODO.

All of the functions in the Test module also have a 'todo' function. (See
Test.pm's documentation for more details.)

The general rule about todo tests is that if the feature is not
yet implemented, it is TODO. But if a feature is broken, or a bug is found
then the tests should fail and *not* be TODO.

The only exception to this rule is that we TODO all failing tests before 
each point release. This is so 'make test' will succeed :)

Remember, the failing test *is* your bug report.

# vim: ft=text