From andy at hexten.net Thu Nov 1 03:19:36 2007 From: andy at hexten.net (Andy Armstrong) Date: Thu, 1 Nov 2007 08:19:36 +0000 Subject: [tap-l] Mad TAP proposal In-Reply-To: <200710311711.31585.ewilhelm@cpan.org> References: <638DA795-0F88-4051-8BA3-79A32F3F9508@hexten.net> <200710311711.31585.ewilhelm@cpan.org> Message-ID: On 1 Nov 2007, at 00:11, Eric Wilhelm wrote: >> But what about a more general mechanism? A TAP directive that means >> 'schedule these other tests'. So then you'd have a controller test >> which was the only one directly visible to Test::Harness and that'd >> decide which other tests to run. > > It sounds like it would be re-creating a lot of the same functionality > needed for declarative extra testing and/or Test::Manifest. > > http://scratchcomputing.com/tmp/extra_testing.txt > > I would rather it not be a TAP directive. Yes, it abstracts > the "manifesting" into TAP, but it still requires some code to run to > determine the manifest, and will therefore be less introspectable. Yeah - I slept on it. It should be TSP - Test Steering Protocol. If a test emits TAP it's a test, if it emits TSP it's a controller. TAP should stay pure. -- Andy Armstrong, Hexten From andy at hexten.net Thu Nov 1 03:39:47 2007 From: andy at hexten.net (Andy Armstrong) Date: Thu, 1 Nov 2007 08:39:47 +0000 Subject: [tap-l] Mad TAP proposal In-Reply-To: References: <638DA795-0F88-4051-8BA3-79A32F3F9508@hexten.net> <200710311711.31585.ewilhelm@cpan.org> Message-ID: <874F091D-08C2-40E1-A759-2F7C6A5BC69F@hexten.net> On 1 Nov 2007, at 08:19, Andy Armstrong wrote: > Yeah - I slept on it. It should be TSP - Test Steering Protocol. If a > test emits TAP it's a test, if it emits TSP it's a controller. TAP > should stay pure. http://testanything.org/wiki/index.php/TSP_Test_Steering_Protocol -- Andy Armstrong, Hexten From andy at hexten.net Thu Nov 1 09:26:57 2007 From: andy at hexten.net (Andy Armstrong) Date: Thu, 1 Nov 2007 14:26:57 +0000 Subject: [tap-l] Mad TAP proposal In-Reply-To: <4729DED2.5050408@plusthree.com> References: <638DA795-0F88-4051-8BA3-79A32F3F9508@hexten.net> <200710311711.31585.ewilhelm@cpan.org> <4729DED2.5050408@plusthree.com> Message-ID: <1EDC7601-D18A-4434-84F0-982141307DD8@hexten.net> On 1 Nov 2007, at 14:12, Michael Peters wrote: > Andy Armstrong wrote: > >> Yeah - I slept on it. It should be TSP - Test Steering Protocol. If a >> test emits TAP it's a test, if it emits TSP it's a controller. TAP >> should stay pure. > > Sounds to me like it should just be a custom test harness. If the > user needs to > dynamically figure out which tests to run, then why not create a > custom harness > which will do what they need. This way they can have control over > not only which > files to run, but which routines (if they're using Test::Class), > etc, etc. > Writing your own harness is trivial now. I'd say writing something > to emit a new > protocol is no easier than writing a script that picks the test > files/subs to run. Yeah, but from the user's PoV this is pretty easy: # t/controller.t use Test::Steering; include ('xt/frob') if frob_avail(); include ('xt/slow') if all_the_time_in_the_world(); And it's not too hard to support in the harness. -- Andy Armstrong, Hexten From mpeters at plusthree.com Thu Nov 1 09:32:13 2007 From: mpeters at plusthree.com (Michael Peters) Date: Thu, 01 Nov 2007 10:32:13 -0400 Subject: [tap-l] Mad TAP proposal In-Reply-To: <1EDC7601-D18A-4434-84F0-982141307DD8@hexten.net> References: <638DA795-0F88-4051-8BA3-79A32F3F9508@hexten.net> <200710311711.31585.ewilhelm@cpan.org> <4729DED2.5050408@plusthree.com> <1EDC7601-D18A-4434-84F0-982141307DD8@hexten.net> Message-ID: <4729E36D.9020707@plusthree.com> Andy Armstrong wrote: > Yeah, but from the user's PoV this is pretty easy: > > # t/controller.t > use Test::Steering; > > include ('xt/frob') if frob_avail(); > include ('xt/slow') if all_the_time_in_the_world(); But if they just do their own harness it becomes: push('xt/frob', @tests)if frob_avail(); push('xt/slow) if all_the_time_in_the_world(); Test::Harness::runtests(@tests); -- Michael Peters Developer Plus Three, LP From pagaltzis at gmx.de Thu Nov 1 09:31:32 2007 From: pagaltzis at gmx.de (A. Pagaltzis) Date: Thu, 1 Nov 2007 15:31:32 +0100 Subject: [tap-l] Mad TAP proposal In-Reply-To: <4729DED2.5050408@plusthree.com> References: <638DA795-0F88-4051-8BA3-79A32F3F9508@hexten.net> <200710311711.31585.ewilhelm@cpan.org> <4729DED2.5050408@plusthree.com> Message-ID: <20071101143132.GC2710@klangraum> * Michael Peters [2007-11-01 15:15]: > Sounds to me like it should just be a custom test harness. If > the user needs to dynamically figure out which tests to run, > then why not create a custom harness which will do what they > need. I was going to say the same. Then it occured to me that if you want to make such a harness reusable and you want to make the test plans introspectable, then such a harness needs to parse some sort of config. Which is very nearly what TSP is. I just don?t see the ?protocol? part. I?d rather think of it in terms of the Test Plan Configuration format, and that this be something specific to TAP::Harness rather than documented widely as something common to TAP implementations. It veers into operational questions that I don?t think are applicable to every scenario where TAP is in use. Because? > This way they can have control over not only which files to > run, but which routines (if they're using Test::Class), etc, > etc. ? TAP encompasses scenarios where the notion of files and test routines doesn?t even compute. > Writing your own harness is trivial now. I'd say writing > something to emit a new protocol is no easier than writing a > script that picks the test files/subs to run. Agree. Putting this sort of functionality in the default harness *is* a good idea so people don?t need to reinvent a common wheel, and so you can use a declarative language to define this sort of thing. But just make this a config for TAP::Harness. Regards, -- Aristotle Pagaltzis // From mpeters at plusthree.com Thu Nov 1 09:12:34 2007 From: mpeters at plusthree.com (Michael Peters) Date: Thu, 01 Nov 2007 10:12:34 -0400 Subject: [tap-l] Mad TAP proposal In-Reply-To: References: <638DA795-0F88-4051-8BA3-79A32F3F9508@hexten.net> <200710311711.31585.ewilhelm@cpan.org> Message-ID: <4729DED2.5050408@plusthree.com> Andy Armstrong wrote: > Yeah - I slept on it. It should be TSP - Test Steering Protocol. If a > test emits TAP it's a test, if it emits TSP it's a controller. TAP > should stay pure. Sounds to me like it should just be a custom test harness. If the user needs to dynamically figure out which tests to run, then why not create a custom harness which will do what they need. This way they can have control over not only which files to run, but which routines (if they're using Test::Class), etc, etc. Writing your own harness is trivial now. I'd say writing something to emit a new protocol is no easier than writing a script that picks the test files/subs to run. -- Michael Peters Developer Plus Three, LP From andy at hexten.net Thu Nov 1 09:43:59 2007 From: andy at hexten.net (Andy Armstrong) Date: Thu, 1 Nov 2007 14:43:59 +0000 Subject: [tap-l] Mad TAP proposal In-Reply-To: <20071101143132.GC2710@klangraum> References: <638DA795-0F88-4051-8BA3-79A32F3F9508@hexten.net> <200710311711.31585.ewilhelm@cpan.org> <4729DED2.5050408@plusthree.com> <20071101143132.GC2710@klangraum> Message-ID: On 1 Nov 2007, at 14:31, A. Pagaltzis wrote: >> This way they can have control over not only which files to >> run, but which routines (if they're using Test::Class), etc, >> etc. > > ? TAP encompasses scenarios where the notion of files and test > routines doesn?t even compute. Yes - agree 100%. Hence it shouldn't be part of TAP. >> Writing your own harness is trivial now. I'd say writing >> something to emit a new protocol is no easier than writing a >> script that picks the test files/subs to run. > > Agree. Putting this sort of functionality in the default harness > *is* a good idea so people don?t need to reinvent a common wheel, > and so you can use a declarative language to define this sort of > thing. But just make this a config for TAP::Harness. The config needs to be dynamic at test time - so it might as well be a script that runs and outputs a description of which tests to run, right? So isn't that nearly TSP? :) -- Andy Armstrong, Hexten From andy at hexten.net Thu Nov 1 09:50:53 2007 From: andy at hexten.net (Andy Armstrong) Date: Thu, 1 Nov 2007 14:50:53 +0000 Subject: [tap-l] Mad TAP proposal In-Reply-To: <4729E36D.9020707@plusthree.com> References: <638DA795-0F88-4051-8BA3-79A32F3F9508@hexten.net> <200710311711.31585.ewilhelm@cpan.org> <4729DED2.5050408@plusthree.com> <1EDC7601-D18A-4434-84F0-982141307DD8@hexten.net> <4729E36D.9020707@plusthree.com> Message-ID: <4CC9406A-F2E0-4464-A10F-AFEE56FC93DD@hexten.net> On 1 Nov 2007, at 14:32, Michael Peters wrote: > But if they just do their own harness it becomes: > > push('xt/frob', @tests)if frob_avail(); > push('xt/slow) if all_the_time_in_the_world(); > Test::Harness::runtests(@tests); Yes, they could. But if you write your own harness you presumably lose, e.g., the advantages of all the different ways you can run the tests via prove. And presumably you couldn't use tk-prove or whatever it's called. It's mixing test logic into harness logic. Isn't it better for the tests to be completely self describing and keep the harness generic? Then your tests will work with any Test::Harness based tool and any compatible TAP harness. -- Andy Armstrong, Hexten From mpeters at plusthree.com Thu Nov 1 10:38:06 2007 From: mpeters at plusthree.com (Michael Peters) Date: Thu, 01 Nov 2007 11:38:06 -0400 Subject: [tap-l] Mad TAP proposal In-Reply-To: <4CC9406A-F2E0-4464-A10F-AFEE56FC93DD@hexten.net> References: <638DA795-0F88-4051-8BA3-79A32F3F9508@hexten.net> <200710311711.31585.ewilhelm@cpan.org> <4729DED2.5050408@plusthree.com> <1EDC7601-D18A-4434-84F0-982141307DD8@hexten.net> <4729E36D.9020707@plusthree.com> <4CC9406A-F2E0-4464-A10F-AFEE56FC93DD@hexten.net> Message-ID: <4729F2DE.4020300@plusthree.com> Andy Armstrong wrote: > On 1 Nov 2007, at 14:32, Michael Peters wrote: >> But if they just do their own harness it becomes: >> >> push('xt/frob', @tests)if frob_avail(); >> push('xt/slow) if all_the_time_in_the_world(); >> Test::Harness::runtests(@tests); > > > Yes, they could. But if you write your own harness you presumably lose, > e.g., the advantages of all the different ways you can run the tests > via prove. And presumably you couldn't use tk-prove or whatever it's > called. Ok, that's true. > It's mixing test logic into harness logic. Isn't it better for the tests > to be completely self describing and keep the harness generic? Then your > tests will work with any Test::Harness based tool and any compatible TAP > harness. Isn't this the problem that SKIP was designed to fix? If it needs to dynamically determined if a test should run, isn't the test itself the best place to do that? xt tests shouldn't be run unless explicity done so by the runner of the tests (a developer) who presumably knows that they are doing. Instead of "something else" telling the harness which tests to run, why not have the harness run them all and the individual tests can decide if they don't want to go through with it or not. If "something else" tells the harness which tests to run, we loose the information about which tests didn't run and why. Testing in Perl is really nice and simple and I'm not seeing the benefits or the problems being solved with this complexity that can't already be solved using something else. -- Michael Peters Developer Plus Three, LP From pagaltzis at gmx.de Thu Nov 1 10:44:29 2007 From: pagaltzis at gmx.de (A. Pagaltzis) Date: Thu, 1 Nov 2007 16:44:29 +0100 Subject: [tap-l] Mad TAP proposal In-Reply-To: References: <638DA795-0F88-4051-8BA3-79A32F3F9508@hexten.net> <200710311711.31585.ewilhelm@cpan.org> <4729DED2.5050408@plusthree.com> <20071101143132.GC2710@klangraum> Message-ID: <20071101154429.GD2710@klangraum> * Andy Armstrong [2007-11-01 15:50]: > The config needs to be dynamic at test time - so it might as > well be a script that runs and outputs a description of which > tests to run, right? But it only needs to be dynamic in a minority of cases. So it seems to me it should be the other way around ? rather than run a script to emit the config, have a config that has a directive to run a script and include its output as part of the config. That makes it easier for other tools to analyse the test configs. > So isn't that nearly TSP? :) In some ways. I just don?t think the harness should look at the output of tests to check if it?s TSP rather than TAP. The list of tests to run should be determined up front, and the test scripts should only ever output TAP. Mixing them with scripts that output TSP feels like a confusion of concerns to me. For the configs there are also a number of issues like dealing well with multiple possible configurations and resolution of relative paths (esp. in compound configs). I think this needs a little bit more design. And since YAML is making its way into TAP anyway, maybe it should be YAML-based rather than a completely ad-hoc TAPish format? Regards, -- Aristotle Pagaltzis // From andy at hexten.net Thu Nov 1 11:20:43 2007 From: andy at hexten.net (Andy Armstrong) Date: Thu, 1 Nov 2007 16:20:43 +0000 Subject: [tap-l] Mad TAP proposal In-Reply-To: <4729F2DE.4020300@plusthree.com> References: <638DA795-0F88-4051-8BA3-79A32F3F9508@hexten.net> <200710311711.31585.ewilhelm@cpan.org> <4729DED2.5050408@plusthree.com> <1EDC7601-D18A-4434-84F0-982141307DD8@hexten.net> <4729E36D.9020707@plusthree.com> <4CC9406A-F2E0-4464-A10F-AFEE56FC93DD@hexten.net> <4729F2DE.4020300@plusthree.com> Message-ID: <55D11C32-F98C-4010-A16E-D688B9EEB602@hexten.net> On 1 Nov 2007, at 15:38, Michael Peters wrote: > Isn't this the problem that SKIP was designed to fix? If it needs to > dynamically > determined if a test should run, isn't the test itself the best > place to do > that? xt tests shouldn't be run unless explicity done so by the > runner of the > tests (a developer) who presumably knows that they are doing. The feature request that started all this relates to a large test suite most of which needs to be skipped if a graphical environment isn't available - so presumably it'd involve executing hundreds of tests just to find out that they skipped. > Instead of "something else" telling the harness which tests to run, > why not have > the harness run them all and the individual tests can decide if they > don't want > to go through with it or not. If "something else" tells the harness > which tests > to run, we loose the information about which tests didn't run and why. Having to run each test to find out it didn't want to run doesn't scale well. Having a mechanism to dynamically select subsets of tests can only encourage people to write more tests I'd have thought. If we decide on a set of rules for which xt/* tests are executed under which circumstances it also gives us a language in which to express those rules - so instead of being hard wired into TAP::Harness we'll have a little TSP document that could be used with any TAP parser. I've also seen a discussion of test ordering and the practice of numbering tests to guarantee a ordering recently; ordering and more complex dependencies could also be handled by a t/controller.t that emits TSP. > Testing in Perl is really nice and simple and I'm not seeing the > benefits or the > problems being solved with this complexity that can't already be > solved using > something else. You could presumably have said the same about the idea of TAP based testing. You can after all just have a bunch of programs that pass or fail and run them from the shell. I'm being slightly facetious there :) I agree that parsimony should guide our decisions but I'm not finding it hard to think of use cases for TSP. -- Andy Armstrong, Hexten From andy at hexten.net Thu Nov 1 11:33:58 2007 From: andy at hexten.net (Andy Armstrong) Date: Thu, 1 Nov 2007 16:33:58 +0000 Subject: [tap-l] Mad TAP proposal In-Reply-To: <20071101154429.GD2710@klangraum> References: <638DA795-0F88-4051-8BA3-79A32F3F9508@hexten.net> <200710311711.31585.ewilhelm@cpan.org> <4729DED2.5050408@plusthree.com> <20071101143132.GC2710@klangraum> <20071101154429.GD2710@klangraum> Message-ID: <544AD30E-C612-41F2-8DCA-D6C95F996A9F@hexten.net> On 1 Nov 2007, at 15:44, A. Pagaltzis wrote: > * Andy Armstrong [2007-11-01 15:50]: >> The config needs to be dynamic at test time - so it might as >> well be a script that runs and outputs a description of which >> tests to run, right? > > But it only needs to be dynamic in a minority of cases. So it > seems to me it should be the other way around ? rather than run > a script to emit the config, have a config that has a directive > to run a script and include its output as part of the config. Well if it doesn't need to be dynamic it doesn't need to exist at all - you just have the tests you want to run. > That makes it easier for other tools to analyse the test configs. > >> So isn't that nearly TSP? :) > > In some ways. I just don?t think the harness should look at the > output of tests to check if it?s TSP rather than TAP. The list of > tests to run should be determined up front Well that's desirable certainly - but isn't it also valuable to be able to dynamically include include or exclude large numbers of tests when you need to? And to do so using a well defined protocol? > , and the test scripts > should only ever output TAP. Yeah, I have sympathy with that. I really don't want to complicate things gratuitously. > Mixing them with scripts that output TSP feels like a confusion > of concerns to me. I can see that too. In that sense the existing "Bail Out" directive feels like an anomaly too. > For the configs there are also a number of issues like dealing > well with multiple possible configurations and resolution of > relative paths (esp. in compound configs). Yup - not impossible though. You can use HTML-like semantics - relative paths are resolved relative to the config, abs paths relative to some well defined project root. > I think this needs a little bit more design. And since YAML is > making its way into TAP anyway, maybe it should be YAML-based > rather than a completely ad-hoc TAPish format? Oh sure - that'd be fine. The choice of representation is somewhat orthogonal to whether it's a good idea in the first place though. -- Andy Armstrong, Hexten From geoff at modperlcookbook.org Thu Nov 1 11:53:45 2007 From: geoff at modperlcookbook.org (Geoffrey Young) Date: Thu, 01 Nov 2007 12:53:45 -0400 Subject: [tap-l] Mad TAP proposal In-Reply-To: <55D11C32-F98C-4010-A16E-D688B9EEB602@hexten.net> References: <638DA795-0F88-4051-8BA3-79A32F3F9508@hexten.net> <200710311711.31585.ewilhelm@cpan.org> <4729DED2.5050408@plusthree.com> <1EDC7601-D18A-4434-84F0-982141307DD8@hexten.net> <4729E36D.9020707@plusthree.com> <4CC9406A-F2E0-4464-A10F-AFEE56FC93DD@hexten.net> <4729F2DE.4020300@plusthree.com> <55D11C32-F98C-4010-A16E-D688B9EEB602@hexten.net> Message-ID: <472A0499.5020805@modperlcookbook.org> Andy Armstrong wrote: > On 1 Nov 2007, at 15:38, Michael Peters wrote: >> Isn't this the problem that SKIP was designed to fix? If it needs to >> dynamically >> determined if a test should run, isn't the test itself the best place >> to do >> that? xt tests shouldn't be run unless explicity done so by the runner >> of the >> tests (a developer) who presumably knows that they are doing. > > The feature request that started all this relates to a large test suite > most of which needs to be skipped if a graphical environment isn't > available - so presumably it'd involve executing hundreds of tests just > to find out that they skipped. fwiw, Apache-Test respects an all.t file, which looks something like this: use Apache::Test; # skip all tests unless mod_perl and mod_apreq are available plan tests => 1, need_module('mod_perl.c', 'mod_apreq2.c'); ok 1; the test is silently executed by our harness. if the plan succeeds and the test run, A-T will run all tests in the current directory. if the plan fails, the skip message is printed and all tests in the directory are never executed. it sits next to the tests as t/foo/all.t t/foo/01do.t t/foo/02done.t or whatnot. anyway, its not the best way to handle things necessarily (I didn't write it, and it's over 6 years old now :) but it it does come in handy every once in a while, so the idea in general is probably sound. --Geoff From andy at hexten.net Thu Nov 1 12:09:07 2007 From: andy at hexten.net (Andy Armstrong) Date: Thu, 1 Nov 2007 17:09:07 +0000 Subject: [tap-l] Mad TAP proposal In-Reply-To: <472A0499.5020805@modperlcookbook.org> References: <638DA795-0F88-4051-8BA3-79A32F3F9508@hexten.net> <200710311711.31585.ewilhelm@cpan.org> <4729DED2.5050408@plusthree.com> <1EDC7601-D18A-4434-84F0-982141307DD8@hexten.net> <4729E36D.9020707@plusthree.com> <4CC9406A-F2E0-4464-A10F-AFEE56FC93DD@hexten.net> <4729F2DE.4020300@plusthree.com> <55D11C32-F98C-4010-A16E-D688B9EEB602@hexten.net> <472A0499.5020805@modperlcookbook.org> Message-ID: <7C66B913-171F-45DF-8A0C-52FE1715474A@hexten.net> On 1 Nov 2007, at 16:53, Geoffrey Young wrote: > anyway, its not the best way to handle things necessarily (I didn't > write it, and it's over 6 years old now :) but it it does come in > handy > every once in a while, so the idea in general is probably sound. So that's another use case for TSP right? :) -- Andy Armstrong, Hexten From scratchcomputing at gmail.com Thu Nov 1 12:52:20 2007 From: scratchcomputing at gmail.com (Eric Wilhelm) Date: Thu, 1 Nov 2007 10:52:20 -0700 Subject: [tap-l] dynamic test selection In-Reply-To: <544AD30E-C612-41F2-8DCA-D6C95F996A9F@hexten.net> References: <638DA795-0F88-4051-8BA3-79A32F3F9508@hexten.net> <20071101154429.GD2710@klangraum> <544AD30E-C612-41F2-8DCA-D6C95F996A9F@hexten.net> Message-ID: <200711011052.21011.ewilhelm@cpan.org> # from Andy Armstrong # on Thursday 01 November 2007 09:33: >On 1 Nov 2007, at 15:44, A. Pagaltzis wrote: >> * Andy Armstrong [2007-11-01 15:50]: >>> The config needs to be dynamic at test time - so it might as >>> well be a script... >> But it only needs to be dynamic in a minority of cases. So it >> seems to me it should be the other way around ? rather than run >> a script to emit the config, have a config that has a directive >> to run a script and include its output as part of the config. This would be preferable for introspectability, yes. >Well if it doesn't need to be dynamic it doesn't need to exist at all >- you just have the tests you want to run. But rather than say "not dynamic", let's say "not arbitrary". The config is always dynamic -- e.g. a canned method require_gui() would need to check whatever cross-platform checks and also maybe consult the tester's "allowables" settings. The distinction between arbitrary and dynamic is that arbitrary code hinders introspection while dynamic can mean that it simply refers to a pre-determined check routine (e.g. the Apache Ant 'exec' directive introduces "arbitrary" into what is otherwise a 100% declarative manifest.) >> That makes it easier for other tools to analyse the test configs. Introspection. Not only can we decide whether or not to run the gui tests, but we know which ones require the gui. If you just source `controller.pl` (arbitrary code) and get a valid list for the current environment, you have no way to tell which of the selections are affected by which bits of the environment. To determine the gui tests, you have to turn-off the $DISPLAY (won't work on windows, and possibly mac) run `controller.pl` again and then compare the new output. Note the difficulty of handling "I have a gui, but I don't want the tests messing with it" on platforms like windows. With a declarative scheme, we have a tree of data and we can simply ask which selections require_gui(). Those tests can then be run as their own group, or even simply examined for some other interesting information. And even in the standard use-case, if the tester has opted to not have her gui interrupted by the tests, they don't run. The declarative scheme makes it easy to "do the right thing" because author education is reduced to simply: "set require_gui for the tests that need a gui" (as opposed to e.g. "check $DISPLAY and then check the preferences file, etc.") It's a lot harder to get it wrong if it involves code that you didn't write. >>> So isn't that nearly TSP? :) >> ... don?t think the harness should look at the >> output of tests to check if it?s TSP rather than TAP. The list of >> tests to run should be determined up front > >Well that's desirable certainly - but isn't it also valuable to be >able to dynamically include include or exclude large numbers of tests >when you need to? And to do so using a well defined protocol? Possibly, but it shouldn't be part of the t/*.t files and I think we would do well to make it declarative rather than a protocol emitted by arbitrary code. >> , and the test scripts >> should only ever output TAP. +1 >> ... YAML ... >Oh sure - that'd be fine. The choice of representation is somewhat >orthogonal to whether it's a good idea in the first place though. I think we've decided that "selectable tests" are a good idea WRT "extra testing." Declarative requirements are probably just as useful WRT normal tests. Note though, that Martin mentions "xt/" in the bug report ;-) --Eric -- To a database person, every nail looks like a thumb. --Jamie Zawinski --------------------------------------------------- http://scratchcomputing.com --------------------------------------------------- From mmoore at clusterresources.com Wed Nov 14 15:53:42 2007 From: mmoore at clusterresources.com (Michael Moore) Date: Wed, 14 Nov 2007 13:53:42 -0700 Subject: [tap-l] What's the best way to do logging? Message-ID: <473B6056.4060608@clusterresources.com> Hello, I'm glad to have found a TAP mailing list! If Perl specific questions aren't appropriate for this list, please let me know and forgive me. I'm working on an automated test suite for some of our software using Perl, I had been using Test::Harness and Test::Harness::Straps but saw just yesterday that Test::Harness::Straps won't be developed anymore. Since we want to use these tests for some time, I'd like to move to something that's not discontinued. Could someone here tell me if TAP::Harness and TAP::Parser is what I should be looking at? I need to do fairly extensive logging. Our tests will normally be run as nightly regression tests, and we want to be able to compare results from different nights. On a test by test basis I am currently logging all of the Test::More functions. I log the test comment and the result. I also log the many external programs that are called, the STDOUT, STDERR, and return status. There are several hundred batch tests, which currently use Test::Harness's execute_tests. I log which test was called from the batch test, and what the result for each was. I've added the logging functionality by overriding the functions, grabbing the results and logging them before returning them to the calling function. I've done this by writing a module which I include instead of Test::More, Test::Harness and Test::Harness::Straps. Is TAP::Harness and TAP::Parser what I should be looking at? Or is there an easier way to do logging? Thank you, Michael Moore From andy at hexten.net Wed Nov 14 16:22:02 2007 From: andy at hexten.net (Andy Armstrong) Date: Wed, 14 Nov 2007 21:22:02 +0000 Subject: [tap-l] What's the best way to do logging? In-Reply-To: <473B6056.4060608@clusterresources.com> References: <473B6056.4060608@clusterresources.com> Message-ID: <77531E48-D8CF-4418-B5BF-836B63992659@hexten.net> On 14 Nov 2007, at 20:53, Michael Moore wrote: > I'm glad to have found a TAP mailing list! If Perl specific questions > aren't appropriate for this list, please let me know and forgive me. > > > I'm working on an automated test suite for some of our software using > Perl, I had been using Test::Harness and Test::Harness::Straps but > saw > just yesterday that Test::Harness::Straps won't be developed anymore. > > Since we want to use these tests for some time, I'd like to move to > something that's not discontinued. Could someone here tell me if > TAP::Harness and TAP::Parser is what I should be looking at? Yes. By design TAP::* replaces the existing Straps mechanism. If it won't do what you want it to do we need to log at that design. > I need to do fairly extensive logging. Our tests will normally be > run as > nightly regression tests, and we want to be able to compare results > from > different nights. > > On a test by test basis I am currently logging all of the Test::More > functions. I log the test comment and the result. I also log the many > external programs that are called, the STDOUT, STDERR, and return > status. There are several hundred batch tests, which currently use > Test::Harness's execute_tests. I log which test was called from the > batch test, and what the result for each was. > > I've added the logging functionality by overriding the functions, > grabbing the results and logging them before returning them to the > calling function. > > I've done this by writing a module which I include instead of > Test::More, Test::Harness and Test::Harness::Straps. > > Is TAP::Harness and TAP::Parser what I should be looking at? Or is > there > an easier way to do logging? I think TAP::Harness is the way forwards. Michael Peters wrote TAP::Harness::Archive[1] which is designed to capture everything from a test run. It's used to upload test results to Smolder[2] - which may also be of interest to you. Unfortunately the latest release of Test::Harness has broken T::H::Archive. There's a fixed version in the Test::Harness subversion repo - I guess I need to give Michael a poke. In the mean time you can get the latest version here: http://svn.hexten.net/tapx/TAP-Harness-Archive/trunk/ May I suggest that you start by reading the documentation for TAP::Harness::Archive and see if that might do what you want? [1] http://search.cpan.org/~wonko/TAP-Harness-Archive/ [2] http://sourceforge.net/projects/smolder From mmoore at clusterresources.com Thu Nov 15 12:24:47 2007 From: mmoore at clusterresources.com (Michael Moore) Date: Thu, 15 Nov 2007 10:24:47 -0700 Subject: [tap-l] What's the best way to do logging? In-Reply-To: <77531E48-D8CF-4418-B5BF-836B63992659@hexten.net> References: <473B6056.4060608@clusterresources.com> <77531E48-D8CF-4418-B5BF-836B63992659@hexten.net> Message-ID: <473C80DF.2010409@clusterresources.com> > Andy Armstrong wrote: > On 14 Nov 2007, at 20:53, Michael Moore wrote: >> Since we want to use these tests for some time, I'd like to move to >> something that's not discontinued. Could someone here tell me if >> TAP::Harness and TAP::Parser is what I should be looking at? > > Yes. By design TAP::* replaces the existing Straps mechanism. If it > won't do what you want it to do we need to log at that design. Great. As I've been looking into more, it looks like it'll be even better than Test::Harness. In particular, it looks like it's easier to get the results of tests out of TAP::Parser. >> Is TAP::Harness and TAP::Parser what I should be looking at? Or is >> there >> an easier way to do logging? > > I think TAP::Harness is the way forwards. Michael Peters wrote > TAP::Harness::Archive[1] which is designed to capture everything from > a test run. It's used to upload test results to Smolder[2] - which may > also be of interest to you. > > Unfortunately the latest release of Test::Harness has broken > T::H::Archive. There's a fixed version in the Test::Harness subversion > repo - I guess I need to give Michael a poke. In the mean time you can > get the latest version here: > > http://svn.hexten.net/tapx/TAP-Harness-Archive/trunk/ > > May I suggest that you start by reading the documentation for > TAP::Harness::Archive and see if that might do what you want? > > [1] http://search.cpan.org/~wonko/TAP-Harness-Archive/ > [2] http://sourceforge.net/projects/smolder Thanks I hadn't see TAP::Harness::Archive. It doesn't do exactly what I need, but it was very useful as a reference. With is as an example, I was able to get a sample test working with TAP::Harness. Looks like it's going to do what we need. TAP::Harness here we come. Thanks again, Michael Moore From mmoore at clusterresources.com Thu Nov 15 12:27:10 2007 From: mmoore at clusterresources.com (Michael Moore) Date: Thu, 15 Nov 2007 10:27:10 -0700 Subject: [tap-l] What's the best way to do logging? Message-ID: <473C816E.6000500@clusterresources.com> > Andy Armstrong wrote: > On 14 Nov 2007, at 20:53, Michael Moore wrote: >> Since we want to use these tests for some time, I'd like to move to >> something that's not discontinued. Could someone here tell me if >> TAP::Harness and TAP::Parser is what I should be looking at? > > Yes. By design TAP::* replaces the existing Straps mechanism. If it > won't do what you want it to do we need to log at that design. Great. As I've been looking into more, it looks like it'll be even better than Test::Harness. In particular, it looks like it's easier to get the results of tests out of TAP::Parser. >> Is TAP::Harness and TAP::Parser what I should be looking at? Or is >> there >> an easier way to do logging? > > I think TAP::Harness is the way forwards. Michael Peters wrote > TAP::Harness::Archive[1] which is designed to capture everything from > a test run. It's used to upload test results to Smolder[2] - which may > also be of interest to you. > > Unfortunately the latest release of Test::Harness has broken > T::H::Archive. There's a fixed version in the Test::Harness subversion > repo - I guess I need to give Michael a poke. In the mean time you can > get the latest version here: > > http://svn.hexten.net/tapx/TAP-Harness-Archive/trunk/ > > May I suggest that you start by reading the documentation for > TAP::Harness::Archive and see if that might do what you want? > > [1] http://search.cpan.org/~wonko/TAP-Harness-Archive/ > [2] http://sourceforge.net/projects/smolder Thanks I hadn't see TAP::Harness::Archive. It doesn't do exactly what I need, but it was very useful as a reference. With is as an example, I was able to get a sample test working with TAP::Harness. Looks like it's going to do what we need. TAP::Harness here we come. Thanks again, Michael Moore From andy at hexten.net Thu Nov 15 12:30:36 2007 From: andy at hexten.net (Andy Armstrong) Date: Thu, 15 Nov 2007 17:30:36 +0000 Subject: [tap-l] What's the best way to do logging? In-Reply-To: <473C80DF.2010409@clusterresources.com> References: <473B6056.4060608@clusterresources.com> <77531E48-D8CF-4418-B5BF-836B63992659@hexten.net> <473C80DF.2010409@clusterresources.com> Message-ID: <2F4BE95B-48B0-4A19-9303-C78A07E5D933@hexten.net> On 15 Nov 2007, at 17:24, Michael Moore wrote: > Great. As I've been looking into more, it looks like it'll be even > better than Test::Harness. In particular, it looks like it's easier to > get the results of tests out of TAP::Parser. Should be :) >> [1] http://search.cpan.org/~wonko/TAP-Harness-Archive/ >> [2] http://sourceforge.net/projects/smolder > > Thanks I hadn't see TAP::Harness::Archive. It doesn't do exactly > what I > need, but it was very useful as a reference. With is as an example, I > was able to get a sample test working with TAP::Harness. > > Looks like it's going to do what we need. TAP::Harness here we come. Splendid! :) -- Andy Armstrong, Hexten From mmoore at clusterresources.com Mon Nov 19 16:43:25 2007 From: mmoore at clusterresources.com (Michael Moore) Date: Mon, 19 Nov 2007 14:43:25 -0700 Subject: [tap-l] Nested tests with TAP::Harness and verbose output Message-ID: <4742037D.9050708@clusterresources.com> I've set up our batch tests to use TAP::Harness's runtests to run sub-tests. We have many cases where BatchTestA calls BatchTestB which calls TestC. If I create a TAP::Harness object in both BatchTestA and BatchTestB, it works fine (all good so far!). I wanted to get a little more information out of our tests, so I passed in a hash with the parameter "verbosity => 1" to each of the TAP::Harness objects. I found that If any TAP::Harness object other than the top level test was set to verbose, it would break the TAP output. Is this expected behavior? Here is the resulting output: === Start output === ./LevelA.bat LevelB........ LevelC...... ok 1 - The test in LevelC 1..1 ok 2 All tests successful. Files=1, Tests=1, 0 wallclock secs ( 0.00 usr 0.01 sys + 0.00 cusr 0.02 csys = 0.03 CPU) Result: PASS ok 1 - The test in LevelB 1..1 All 1 subtests passed Test Summary Report ------------------- LevelB.bat (Wstat: 0 Tests: 3 Failed: 2) Failed test number(s): 1-2 Parse errors: Plan (1..1) must be at the beginning or end of the TAP output Tests out of sequence. Found (1) but expected (3) More than one plan found in TAP output Bad plan. You planned 1 tests but ran 3. Files=1, Tests=3, 0 wallclock secs ( 0.01 usr 0.00 sys + 0.00 cusr 0.10 csys = 0.11 CPU) Result: FAIL ok 1 - The test in LevelA 1..1 === End output === I'm not sure if attachments are OK on this list, so I'll just put the three tests I used to narrow down the behavior here inline. If you save these three tests and run LevelA.bat, you should see the described behavior. If you comment out the line "verbosity => 1" in LevelB.bat, the problem will go away. -----LevelA.bat----- #!/usr/bin/perl use strict; use warnings; use TAP::Harness; use Test::More; plan('no_plan'); my %tapHarnessArgs = ( verbosity => 1, ); my $batch = TAP::Harness->new(\%tapHarnessArgs); $batch->runtests("LevelB.bat"); ok(1==1,"The test in LevelA"); -----LevelB.bat----- #!/usr/bin/perl use strict; use warnings; use TAP::Harness; use Test::More; plan('no_plan'); my %tapHarnessArgs = ( verbosity => 1, ); my $batch = TAP::Harness->new(\%tapHarnessArgs); $batch->runtests("LevelC.t"); ok(2==2,"The test in LevelB"); -----LevelC.t----- #!/usr/bin/perl use strict; use warnings; use Test::More; plan('no_plan'); ok(3==3,"The test in LevelC"); Many Thanks, Michael Moore From mmoore at clusterresources.com Mon Nov 19 18:46:19 2007 From: mmoore at clusterresources.com (Michael Moore) Date: Mon, 19 Nov 2007 16:46:19 -0700 Subject: [tap-l] Nested tests with TAP::Harness and verbose output] Message-ID: <4742204B.7090502@clusterresources.com> I tried to send this earlier, but I think it bounced, so here it is again. I've set up our batch tests to use TAP::Harness's runtests to run sub-tests. We have many cases where BatchTestA calls BatchTestB which calls TestC. If I create a TAP::Harness object in both BatchTestA and BatchTestB, it works fine (all good so far!). I wanted to get a little more information out of our tests, so I passed in a hash with the parameter "verbosity => 1" to each of the TAP::Harness objects. I found that If any TAP::Harness object other than the top level test was set to verbose, it would break the TAP output. Is this expected behavior? Here is the resulting output: === Start output === ./LevelA.bat LevelB........ LevelC...... ok 1 - The test in LevelC 1..1 ok 2 All tests successful. Files=1, Tests=1, 0 wallclock secs ( 0.00 usr 0.01 sys + 0.00 cusr 0.02 csys = 0.03 CPU) Result: PASS ok 1 - The test in LevelB 1..1 All 1 subtests passed Test Summary Report ------------------- LevelB.bat (Wstat: 0 Tests: 3 Failed: 2) Failed test number(s): 1-2 Parse errors: Plan (1..1) must be at the beginning or end of the TAP output Tests out of sequence. Found (1) but expected (3) More than one plan found in TAP output Bad plan. You planned 1 tests but ran 3. Files=1, Tests=3, 0 wallclock secs ( 0.01 usr 0.00 sys + 0.00 cusr 0.10 csys = 0.11 CPU) Result: FAIL ok 1 - The test in LevelA 1..1 === End output === I'm not sure if attachments are OK on this list, so I'll just put the three tests I used to narrow down the behavior here inline. If you save these three tests and run LevelA.bat, you should see the described behavior. If you comment out the line "verbosity => 1" in LevelB.bat, the problem will go away. -----LevelA.bat----- #!/usr/bin/perl use strict; use warnings; use TAP::Harness; use Test::More; plan('no_plan'); my %tapHarnessArgs = ( verbosity => 1, ); my $batch = TAP::Harness->new(\%tapHarnessArgs); $batch->runtests("LevelB.bat"); ok(1==1,"The test in LevelA"); -----LevelB.bat----- #!/usr/bin/perl use strict; use warnings; use TAP::Harness; use Test::More; plan('no_plan'); my %tapHarnessArgs = ( verbosity => 1, ); my $batch = TAP::Harness->new(\%tapHarnessArgs); $batch->runtests("LevelC.t"); ok(2==2,"The test in LevelB"); -----LevelC.t----- #!/usr/bin/perl use strict; use warnings; use Test::More; plan('no_plan'); ok(3==3,"The test in LevelC"); Many Thanks, Michael Moore From andy at hexten.net Tue Nov 20 04:26:31 2007 From: andy at hexten.net (Andy Armstrong) Date: Tue, 20 Nov 2007 09:26:31 +0000 Subject: [tap-l] Nested tests with TAP::Harness and verbose output In-Reply-To: <4742037D.9050708@clusterresources.com> References: <4742037D.9050708@clusterresources.com> Message-ID: <0F4D1395-D48C-4446-937E-408F8880789B@hexten.net> On 19 Nov 2007, at 21:43, Michael Moore wrote: > I've set up our batch tests to use TAP::Harness's runtests to run > sub-tests. We have many cases where BatchTestA calls BatchTestB which > calls TestC. > > If I create a TAP::Harness object in both BatchTestA and BatchTestB, > it > works fine (all good so far!). > > I wanted to get a little more information out of our tests, so I > passed > in a hash with the parameter "verbosity => 1" to each of the > TAP::Harness objects. > > I found that If any TAP::Harness object other than the top level test > was set to verbose, it would break the TAP output. Is this expected > behavior? Yeah, I think so. It looks as if you're feeding the console output from the nested harness into the containing harness as TAP. A test that runs under TAP::Harness should output syntactically clean TAP. You could have a look at Test::Steering to discover how to run nested tests. -- Andy Armstrong, Hexten From adrianh at quietstars.com Tue Nov 20 11:41:26 2007 From: adrianh at quietstars.com (Adrian Howard) Date: Tue, 20 Nov 2007 16:41:26 +0000 Subject: [tap-l] Deferred Plans In-Reply-To: <373AE811-37BA-4571-ADB1-BF6D62F76C01@hexten.net> References: <729237.83345.qm@web60811.mail.yahoo.com> <373AE811-37BA-4571-ADB1-BF6D62F76C01@hexten.net> Message-ID: <25E45D81-571E-4408-A345-4F4109FD53AD@quietstars.com> Taken to tap-l from perl-qa... On 19 Nov 2007, at 13:06, Andy Armstrong wrote: [snip] > I think we really need to reach a decision on > > http://testanything.org/wiki/index.php/Test_Groups versus > http://testanything.org/wiki/index.php/Test_Blocks Having just glanced at these again it's occurred that if you change the "end" keyword in the block proposal to an "ok" / "not ok" (based on treating the block of tests as a successful/unsuccessful test run) you get: 1..4 begin 1 Object creation 1..2 ok 1 Object created OK ok 2 Object isa Flunge::Twizzler ok 1 Object creation ok 2 Clone OK begin 3 Methods 1..4 ok 1 has twizzle method ok 2 has burnish method ok 3 has spangle method not ok 4 has frob method not ok 3 Methods ok 4 Resources released which (I think) still reads nicely, and has the advantage of being 100% compatible with older parsers. Makes more work for whatever is generating the TAP of course. Just a thought... Adrian From andy at hexten.net Tue Nov 20 11:54:52 2007 From: andy at hexten.net (Andy Armstrong) Date: Tue, 20 Nov 2007 16:54:52 +0000 Subject: [tap-l] Deferred Plans In-Reply-To: <25E45D81-571E-4408-A345-4F4109FD53AD@quietstars.com> References: <729237.83345.qm@web60811.mail.yahoo.com> <373AE811-37BA-4571-ADB1-BF6D62F76C01@hexten.net> <25E45D81-571E-4408-A345-4F4109FD53AD@quietstars.com> Message-ID: On 20 Nov 2007, at 16:41, Adrian Howard wrote: > Having just glanced at these again it's occurred that if you change > the "end" keyword in the block proposal to an "ok" / "not ok" (based > on treating the block of tests as a successful/unsuccessful test run) > you get: > > 1..4 > begin 1 Object creation > 1..2 > ok 1 Object created OK > ok 2 Object isa Flunge::Twizzler > ok 1 Object creation > ok 2 Clone OK > begin 3 Methods > 1..4 > ok 1 has twizzle method > ok 2 has burnish method > ok 3 has spangle method > not ok 4 has frob method > not ok 3 Methods > ok 4 Resources released > > which (I think) still reads nicely, and has the advantage of being > 100% compatible with older parsers. > > Makes more work for whatever is generating the TAP of course. > > Just a thought... Oooh. I like that very much. It means that if you're combining TAP you have to parse it to generate the summary result and also that the indentation becomes significant - which wasn't my original intention - but I'm OK with both of those things. -- Andy Armstrong, Hexten From andy at hexten.net Tue Nov 20 19:35:01 2007 From: andy at hexten.net (Andy Armstrong) Date: Wed, 21 Nov 2007 00:35:01 +0000 Subject: [tap-l] SKIP_ALL tests should not get hidden In-Reply-To: <4743702C.90308@pobox.com> References: <166662.31668.qm@web60815.mail.yahoo.com> <155AA7B6-809C-4339-B56A-FADB09E21412@hexten.net> <4743702C.90308@pobox.com> Message-ID: On 20 Nov 2007, at 23:39, Michael G Schwern wrote: >> Do we like that? > > Test::Harness 2 put it on it's own line mostly to avoid wrapping off > the right > side of the screen. I still lean in that direction. Hmm. I'm kind of hooked on the new behaviour now. It puts a summary column right where I can find it. >>> However, TAP::Parser doesn't seem to have a &skip_all method >>> (returning boolean or explanation? What if no skip reason given?). >> >> TAP::Parser->skip_all returns a true value if all tests are skipped. >> It either returns the skip reason or '(no reason given)' as above. > > I don't like that it hard wires the default string, that's mixing up > display > with the API. It leaves developers having to write weird things > like "if( > $skip eq '(no reason given)' ) { ...then put in your own > default... }". > > This is a situation where "0 but true" becomes handy. Or in this > case, '' but > true. It might make sense, in the case where there is no reason > given, to > return a boolean and string overloaded object which returns false > for bool and > the empty string as a string. Agreed. I'll take a look at it - unless you want to. -- Andy Armstrong, Hexten From schwern at pobox.com Tue Nov 20 18:39:24 2007 From: schwern at pobox.com (Michael G Schwern) Date: Tue, 20 Nov 2007 15:39:24 -0800 Subject: [tap-l] SKIP_ALL tests should not get hidden In-Reply-To: <155AA7B6-809C-4339-B56A-FADB09E21412@hexten.net> References: <166662.31668.qm@web60815.mail.yahoo.com> <155AA7B6-809C-4339-B56A-FADB09E21412@hexten.net> Message-ID: <4743702C.90308@pobox.com> Andy Armstrong wrote: > I've added logic that produces output like this: > > 13:54] andy $ prove t/sample-tests/skipall t/sample-tests/ > skipall_nomsg t/sample-tests/simple > t/sample-tests/skipall..........skipped: rope > t/sample-tests/skipall_nomsg....skipped: (no reason given) > t/sample-tests/simple...........ok > All tests successful. > Files=3, Tests=5, 0 wallclock secs ( 0.01 usr 0.01 sys + 0.01 cusr > 0.00 csys = 0.03 CPU) > Result: PASS > > Do we like that? Test::Harness 2 put it on it's own line mostly to avoid wrapping off the right side of the screen. I still lean in that direction. >> However, TAP::Parser doesn't seem to have a &skip_all method >> (returning boolean or explanation? What if no skip reason given?). > > TAP::Parser->skip_all returns a true value if all tests are skipped. > It either returns the skip reason or '(no reason given)' as above. I don't like that it hard wires the default string, that's mixing up display with the API. It leaves developers having to write weird things like "if( $skip eq '(no reason given)' ) { ...then put in your own default... }". This is a situation where "0 but true" becomes handy. Or in this case, '' but true. It might make sense, in the case where there is no reason given, to return a boolean and string overloaded object which returns false for bool and the empty string as a string. It occurs to me that an "X but true" CPAN module would be handy. -- Don't try the paranormal until you know what's normal. -- "Lords and Ladies" by Terry Prachett