From andy at hexten.net Thu Jul 12 15:17:39 2007 From: andy at hexten.net (Andy Armstrong) Date: Thu, 12 Jul 2007 15:17:39 +0100 Subject: [tap-l] Tracking tests over time Message-ID: <46963803.20105@hexten.net> I've just noticed that the MediaWiki team have a test harness that tracks the life of individual assertions over time: * HTML nested bullet list, open tags (bug 5497) [Has never passed] * HTML nested ordered list, open tags (bug 5497) [Has never passed] * Fuzz testing: image with bogus manual thumbnail [Introduced between 08-Apr-2007 07:15:22, 1.10alpha (r21099) and 25-Apr-2007 07:15:46, 1.10alpha Nice eh? With TAP that'd require either adding a persistent identifier to tests, requiring that the test numbering doesn't change over time (new tests added at the end) or that there's some heuristic that attempts to track assertions that have been renumbered. http://testanything.org/wiki/index.php/Assertions_are_named_dynamically A partial solution may lie in the direction of Test Blocks[1]. If it became normal practice for small clusters of related assertions to be grouped in a block then the TAP document would gain structure that would make it easier to automagically work out which assertions had been added or removed. Test Blocks would also localise the impact on test numbering when a new assertion is added; outside of the affected block the numbering wouldn't change. And I still like the Test Blocks proposal for other reasons... [1] http://testanything.org/wiki/index.php/Test_Blocks -- Andy Armstrong, Hexten From mpeters at plusthree.com Thu Jul 12 15:23:38 2007 From: mpeters at plusthree.com (Michael Peters) Date: Thu, 12 Jul 2007 10:23:38 -0400 Subject: [tap-l] [tapx-dev] Tracking tests over time In-Reply-To: <46963803.20105@hexten.net> References: <46963803.20105@hexten.net> Message-ID: <4696396A.90009@plusthree.com> Andy Armstrong wrote: > I've just noticed that the MediaWiki team have a test harness that > tracks the life of individual assertions over time: > > * HTML nested bullet list, open tags (bug 5497) [Has never passed] > * HTML nested ordered list, open tags (bug 5497) [Has never passed] > * Fuzz testing: image with bogus manual thumbnail [Introduced > between 08-Apr-2007 07:15:22, 1.10alpha (r21099) and 25-Apr-2007 > 07:15:46, 1.10alpha > > Nice eh? > > With TAP that'd require either adding a persistent identifier to tests, > requiring that the test numbering doesn't change over time (new tests > added at the end) or that there's some heuristic that attempts to track > assertions that have been renumbered. Why can't you just use test names? ok($is_ok, 'HTML nested bullet list, open tags (bug 5497)'); If tracking a single test over time seems like something people would want, I think it could be added to Smolder without too much difficulty. -- Michael Peters Developer Plus Three, LP From andy at hexten.net Thu Jul 12 16:25:01 2007 From: andy at hexten.net (Andy Armstrong) Date: Thu, 12 Jul 2007 16:25:01 +0100 Subject: [tap-l] [tapx-dev] Tracking tests over time In-Reply-To: <4696396A.90009@plusthree.com> References: <46963803.20105@hexten.net> <4696396A.90009@plusthree.com> Message-ID: <469647CD.3040003@hexten.net> Michael Peters wrote: > Why can't you just use test names? > > ok($is_ok, 'HTML nested bullet list, open tags (bug 5497)'); You certainly could - but I tend to think of them as descriptions rather than names and therefore not entirely ideal as the primary key. I don't really want to be forced not to change a description because I'll break the test system if I do. > If tracking a single test over time seems like something people would want, I > think it could be added to Smolder without too much difficulty. As far as I know the only demand is me thinking "isn't that neat" - but I do think it's neat :) -- Andy Armstrong, Hexten From andy at hexten.net Thu Jul 12 18:24:10 2007 From: andy at hexten.net (Andy Armstrong) Date: Thu, 12 Jul 2007 18:24:10 +0100 Subject: [tap-l] Tracking tests over time In-Reply-To: <46965EC1.6020608@indecorous.com> References: <46963803.20105@hexten.net> <46965EC1.6020608@indecorous.com> Message-ID: <469663BA.4000608@hexten.net> Ian Malpass wrote: > Requiring that test numbering doesn't change seems like an awfully large > burden. It would mean you couldn't add extra values to a loop with a > test in it, for example (unless it happened to be at the end of your > test). It would be difficult to enforce, easy to forget/ignore/not know > about, and failure would be awkward. Yeah, I wasn't advocating that :) > A heuristic would be complicated and fragile, and if (when) it went awry > your data would, again, go screwy. Yeah, possibly. I think it's more or less the same problem diff solves. Again, I'm not actually advocating it though. > As such, I think I'd favour the persistent identifier approach, and add > an extra argument to each test assertion: > > is ( $got, $exptected, $test_name, $id ); > > You could then create and use a subclass of Test::More to care about the > extra argument. Anyone just using plain Test::More wouldn't see any > difference. It wouldn't be particularly taxing when writing tests, and > would only impact those who felt that it would give them some benefit. > > I'm not sure how you'd deal with not using Test::More (or that use > Test::More internally), or using things that do stuff like > all_pod_files_ok(), but those seem like solvable problems. I wouldn't envisage unique IDs being mandatory. We currently have two proposals / work in progress that I think are relevant to this: TAP Diagnostic Syntax http://testanything.org/wiki/index.php/TAP_diagnostic_syntax That gives us somewhere to put a unique test id. TAP Blocks http://testanything.org/wiki/index.php/Test_Blocks Has one of its benefits localising the effects of any renumbering. A block is effectively a nested TAP document with its own plan. Any changes to the numbering inside the block don't propagate outside the block. That'd benefit tests that didn't use a unique ID. -- Andy Armstrong, Hexten From ian at indecorous.com Thu Jul 12 18:02:57 2007 From: ian at indecorous.com (Ian Malpass) Date: Thu, 12 Jul 2007 12:02:57 -0500 Subject: [tap-l] Tracking tests over time In-Reply-To: <46963803.20105@hexten.net> References: <46963803.20105@hexten.net> Message-ID: <46965EC1.6020608@indecorous.com> Andy Armstrong wrote: > With TAP that'd require either adding a persistent identifier to tests, > requiring that the test numbering doesn't change over time (new tests > added at the end) or that there's some heuristic that attempts to track > assertions that have been renumbered. Requiring that test numbering doesn't change seems like an awfully large burden. It would mean you couldn't add extra values to a loop with a test in it, for example (unless it happened to be at the end of your test). It would be difficult to enforce, easy to forget/ignore/not know about, and failure would be awkward. A heuristic would be complicated and fragile, and if (when) it went awry your data would, again, go screwy. As such, I think I'd favour the persistent identifier approach, and add an extra argument to each test assertion: is ( $got, $exptected, $test_name, $id ); You could then create and use a subclass of Test::More to care about the extra argument. Anyone just using plain Test::More wouldn't see any difference. It wouldn't be particularly taxing when writing tests, and would only impact those who felt that it would give them some benefit. I'm not sure how you'd deal with not using Test::More (or that use Test::More internally), or using things that do stuff like all_pod_files_ok(), but those seem like solvable problems. Ian From mpeters at plusthree.com Thu Jul 12 18:50:31 2007 From: mpeters at plusthree.com (Michael Peters) Date: Thu, 12 Jul 2007 13:50:31 -0400 Subject: [tap-l] Tracking tests over time In-Reply-To: <46963803.20105@hexten.net> References: <46963803.20105@hexten.net> Message-ID: <469669E7.4080209@plusthree.com> Andy Armstrong wrote: > I've just noticed that the MediaWiki team have a test harness that > tracks the life of individual assertions over time: > > * HTML nested bullet list, open tags (bug 5497) [Has never passed] > * HTML nested ordered list, open tags (bug 5497) [Has never passed] > * Fuzz testing: image with bogus manual thumbnail [Introduced > between 08-Apr-2007 07:15:22, 1.10alpha (r21099) and 25-Apr-2007 > 07:15:46, 1.10alpha By the way. Do you have a link to show this or is it just something that prints to the console when you run the tests? -- Michael Peters Developer Plus Three, LP From andy at hexten.net Thu Jul 12 21:32:42 2007 From: andy at hexten.net (Andy Armstrong) Date: Thu, 12 Jul 2007 21:32:42 +0100 Subject: [tap-l] Tracking tests over time In-Reply-To: <469669E7.4080209@plusthree.com> References: <46963803.20105@hexten.net> <469669E7.4080209@plusthree.com> Message-ID: <46968FEA.1060909@hexten.net> Michael Peters wrote: > By the way. Do you have a link to show this or is it just something that prints > to the console when you run the tests? They had an accident with the mail server that's obviously supposed to be relaying the tests and about 12 months worth spewed forth onto the list today: For example: http://lists.wikimedia.org/pipermail/wikitech-l/2007-July/032289.html After I'd seen a few hundred of them I decided to read one and was really impressed by the test history thing. -- Andy Armstrong, Hexten From curtis_ovid_poe at yahoo.com Sat Jul 21 11:03:25 2007 From: curtis_ovid_poe at yahoo.com (Ovid) Date: Sat, 21 Jul 2007 03:03:25 -0700 (PDT) Subject: [tap-l] (no subject) Message-ID: <114633.51263.qm@web60816.mail.yahoo.com> Working for my TAP::Parser -> Test::Harness 3.0 presentation for YAPC::EU in Vienna and I think there's a tiny mistake on this page: http://podwiki.hexten.net/podwiki.pl?page=TAP It reads: Any output line that is not a version, a plan, a test line, a diagnostic or a bail out is incorrect. How a harness handles the incorrect line is undefined. I believe that should be something along the lines of: Any output line that is not a version, a plan, a test line, a diagnostic or a bail out is considered an "unknown" line. A TAP parser is required to not consider an unknown line as an error but may optionally choose to capture said line and hand it to the test harness, which may have custom behavior attached. This is to allow for forward compatability. OK, that's awkward, but basically, the behavior is not undefined because currently we discard it but the parser is REQUIRED to not count an incorrect line as an error. Thus, the behavior is not undefined, but a harness may optionally choose to attach additional semantics to it. Is this correct? Cheers, Ovid -- Buy the book - http://www.oreilly.com/catalog/perlhks/ Perl and CGI - http://users.easystreet.com/ovid/cgi_course/ Personal blog - http://publius-ovidius.livejournal.com/ Tech blog - http://use.perl.org/~Ovid/journal/ From andy at hexten.net Sat Jul 21 11:19:33 2007 From: andy at hexten.net (Andy Armstrong) Date: Sat, 21 Jul 2007 11:19:33 +0100 Subject: [tap-l] (no subject) In-Reply-To: <114633.51263.qm@web60816.mail.yahoo.com> References: <114633.51263.qm@web60816.mail.yahoo.com> Message-ID: <42A8D9D9-710B-42AD-A9D8-6460A647A72F@hexten.net> On 21 Jul 2007, at 11:03, Ovid wrote: > Any output line that is not a version, a plan, a test line, a > diagnostic > or a bail out is > considered an "unknown" line. A TAP parser is required to not > consider an unknown > line as an error but may optionally choose to capture said line > and hand it to the > test harness, which may have custom behavior attached. This is > to allow for forward > compatability. I've updated it with that text :) -- Andy Armstrong, hexten.net From publiustemp-tapx at yahoo.com Sat Jul 21 12:21:23 2007 From: publiustemp-tapx at yahoo.com (Ovid) Date: Sat, 21 Jul 2007 04:21:23 -0700 (PDT) Subject: [tap-l] Need comments on TH3 Presenation Message-ID: <144682.70400.qm@web60812.mail.yahoo.com> ----- Original Message ---- From: Andy Armstrong > I've updated it with that text :) Thanks! I still feel that my copy was awkward, so if you have a better way of writing it, that's fine. In other news, still working on my presentation. (http://vienna.yapceurope.org/ye2007/talk/475) Here's my basic plan: * Show version 1 of TAP. * Quickly (one slide per second), whip through the next twelve versions. * Compare the version 1 grammar to the version 13 grammar [1] * TAP is evolving faster than the parsers * Why not refactor Test::Harness? * Test::Harness RT queue * Show things TAP::Parser can do that Test::Harness cannot (or is clumsy at) [2] - TODO and SKIP identification - TODO and SKIP lists (--directives) - Colored test output - GUIs (anyone want to fix the GTK GUI?) - Failures only - Callback actions - Improved diagnostics - Multi-language test suites * To Test::Harness 3.0 and beyond - Improve test coverage - Improve documenation - Parallel tests * TAP::Parser internals I would love feedback on any and all of this. Cheers, Ovid [1] The version 1 grammar looks like this (simplified): tap ::= plan "\n" lines plan ::= '1..' posInteger lines ::= line {line} line ::= 'not ' ? 'ok ' posInteger "\n" posInteger ::= ( digit - '0' ) { digit } The version 13 grammar, of course, is at: http://search.cpan.org/dist/TAP-Parser/lib/TAP/Parser/Grammar.pm#TAP_GRAMMAR http://tinyurl.com/yw7t7e [2] Is there anything that Test::Harness can do that TAP::Parser cannot? -- Buy the book - http://www.oreilly.com/catalog/perlhks/ Perl and CGI - http://users.easystreet.com/ovid/cgi_course/ Personal blog - http://publius-ovidius.livejournal.com/ Tech blog - http://use.perl.org/~Ovid/journal/ From andy at hexten.net Sat Jul 21 12:45:08 2007 From: andy at hexten.net (Andy Armstrong) Date: Sat, 21 Jul 2007 12:45:08 +0100 Subject: [tap-l] Need comments on TH3 Presenation In-Reply-To: <144682.70400.qm@web60812.mail.yahoo.com> References: <144682.70400.qm@web60812.mail.yahoo.com> Message-ID: On 21 Jul 2007, at 12:21, Ovid wrote: > - Improved diagnostics I know that's what we've been calling them - but I wonder if that's not underselling them a bit. We're not limited to diagnostic information there - we can have all sorts of meta about the test run, use them to relate regression tests to specific RT tracker numbers etc, etc. -- Andy Armstrong, hexten.net From mpeters at plusthree.com Sat Jul 21 14:04:59 2007 From: mpeters at plusthree.com (Michael Peters) Date: Sat, 21 Jul 2007 09:04:59 -0400 Subject: [tap-l] Need comments on TH3 Presenation In-Reply-To: <144682.70400.qm@web60812.mail.yahoo.com> References: <144682.70400.qm@web60812.mail.yahoo.com> Message-ID: <46A2047B.2060707@plusthree.com> Ovid wrote: > - GUIs (anyone want to fix the GTK GUI?) You could mention the Smolder (shameless plug). It's sort of a gui. Not to run the tests but to share the results of a run. And the current 1.1 uses TAP::Parser. -- Michael Peters Developer Plus Three, LP From publiustemp-tapx at yahoo.com Mon Jul 23 10:43:14 2007 From: publiustemp-tapx at yahoo.com (Ovid) Date: Mon, 23 Jul 2007 02:43:14 -0700 (PDT) Subject: [tap-l] Help Us With Test::Harness 3.0 Message-ID: <386172.83780.qm@web60818.mail.yahoo.com> Announcement is here: http://www.perlmonks.org/?node_id=628195 Cheers, Ovid -- Buy the book - http://www.oreilly.com/catalog/perlhks/ Perl and CGI - http://users.easystreet.com/ovid/cgi_course/ Personal blog - http://publius-ovidius.livejournal.com/ Tech blog - http://use.perl.org/~Ovid/journal/