From publiustemp-perlqa3 at yahoo.com Thu Apr 10 10:42:07 2008 From: publiustemp-perlqa3 at yahoo.com (Ovid) Date: Thu, 10 Apr 2008 02:42:07 -0700 (PDT) Subject: [tap-l] Nested TAP Message-ID: <238318.14601.qm@web65714.mail.ac4.yahoo.com> One issue we covered at the Hackathon was how to do nested TAP. The suggestion that was adopted was not backwards-compatible, but it was felt this was OK because the TAP consumer specifically had to request this. I hated this decision, but was hard-pressed to argue against it because making things backwards-compatible seemed to involve putting a full-blown parser into the TAP producer. (In other words, sort of like embedding Test::Harness into Test::Builder). I *think* I've figured out how to make things backwards-compatible and not require a parser. The background is here: http://use.perl.org/~Ovid/journal/36120 Cheers, Ovid -- Buy the book - http://www.oreilly.com/catalog/perlhks/ Perl and CGI - http://users.easystreet.com/ovid/cgi_course/ Personal blog - http://publius-ovidius.livejournal.com/ Tech blog - http://use.perl.org/~Ovid/journal/ From h.m.brand at xs4all.nl Thu Apr 10 13:58:46 2008 From: h.m.brand at xs4all.nl (H.Merijn Brand) Date: Thu, 10 Apr 2008 14:58:46 +0200 Subject: [tap-l] Parallel testing Message-ID: <20080410145846.56d61b86@pc09.procura.nl> In Oslo Andy and I came up with a way to define how Test::Harness should deal with parallel tests, like this: --8<--- psmoke.conf # never run files in this group side by side, order doesn't matter conflict: folder: ../ext/DB_File/t folder: ../ext/IO_Compress_Zlib/t folder: ../lib/CPANPLUS folder: ../lib/ExtUtils/t # folder: ../lib/Pod/t # first file should be run before file 2, 3, 4, ... depends: depend: [ foo.t bar.t ] -->8--- As Test::Harness `does' TAP, maybe "psmoke.conf" should better be renamed to ".taprc", which brought me to the question if that config file would be called .taprc (or tap.rc on windows/vms) and it would be stackable (~/.taprc, /etc/tap.rc ...) would that be usefull? and then it should be included in the yaml metadata we disdussed? I bet the TAP parsers should know about the tests being run in parallel and what the restrictions were. It maybe also should know what $HARNESS_OPTIONS was set when the run ran. -- H.Merijn Brand Amsterdam Perl Mongers (http://amsterdam.pm.org/) using & porting perl 5.6.2, 5.8.x, 5.10.x on HP-UX 10.20, 11.00, 11.11, & 11.23, SuSE 10.1 & 10.2, AIX 5.2, and Cygwin. http://qa.perl.org http://mirrors.develooper.com/hpux/ http://www.test-smoke.org http://www.goldmark.org/jeff/stupid-disclaimers/ From andy at hexten.net Thu Apr 10 17:05:42 2008 From: andy at hexten.net (Andy Armstrong) Date: Thu, 10 Apr 2008 18:05:42 +0200 Subject: [tap-l] Parallel testing In-Reply-To: <20080410145846.56d61b86@pc09.procura.nl> References: <20080410145846.56d61b86@pc09.procura.nl> Message-ID: <2ED3B530-8971-481D-9F72-9F41CAFFF98D@hexten.net> On 10 Apr 2008, at 14:58, H.Merijn Brand wrote: > if that config file would be called .taprc (or tap.rc on windows/ > vms) and > it would be stackable (~/.taprc, /etc/tap.rc ...) would that be > usefull? I don't know if it should live in ~ or /etc - it's specific to a project. And I'm not sure why you'd want it to be stackable. I'm not saying it's not useful - just that I can't come up with a use case. -- Andy Armstrong, Hexten From h.m.brand at xs4all.nl Thu Apr 10 17:23:53 2008 From: h.m.brand at xs4all.nl (H.Merijn Brand) Date: Thu, 10 Apr 2008 18:23:53 +0200 Subject: [tap-l] Parallel testing In-Reply-To: <2ED3B530-8971-481D-9F72-9F41CAFFF98D@hexten.net> References: <20080410145846.56d61b86@pc09.procura.nl> <2ED3B530-8971-481D-9F72-9F41CAFFF98D@hexten.net> Message-ID: <20080410182353.05e24078@pc09.procura.nl> On Thu, 10 Apr 2008 18:05:42 +0200, Andy Armstrong wrote: > On 10 Apr 2008, at 14:58, H.Merijn Brand wrote: > > if that config file would be called .taprc (or tap.rc on windows/ > > vms) and > > it would be stackable (~/.taprc, /etc/tap.rc ...) would that be > > usefull? > > I don't know if it should live in ~ or /etc - it's specific to a > project. And I'm not sure why you'd want it to be stackable. I'm not > saying it's not useful - just that I can't come up with a use case. Just because we're just having this specific usage for parallel testing does not mean we cannot extend the usage of a .taprc do do more :) Report: Username: Tux Real_Name: withheld Hobbies: [ perl, whiskey, legs^Wbeautiful people ] -- H.Merijn Brand Amsterdam Perl Mongers (http://amsterdam.pm.org/) using & porting perl 5.6.2, 5.8.x, 5.10.x on HP-UX 10.20, 11.00, 11.11, & 11.23, SuSE 10.1 & 10.2, AIX 5.2, and Cygwin. http://qa.perl.org http://mirrors.develooper.com/hpux/ http://www.test-smoke.org http://www.goldmark.org/jeff/stupid-disclaimers/ From mpeters at plusthree.com Thu Apr 10 17:55:39 2008 From: mpeters at plusthree.com (Michael Peters) Date: Thu, 10 Apr 2008 12:55:39 -0400 Subject: [tap-l] Parallel testing In-Reply-To: <20080410145846.56d61b86@pc09.procura.nl> References: <20080410145846.56d61b86@pc09.procura.nl> Message-ID: <47FE468B.8020605@plusthree.com> H.Merijn Brand wrote: > As Test::Harness `does' TAP, maybe "psmoke.conf" should better be > renamed to ".taprc", which brought me to the question Test::Harness does process TAP, but TAP has nothing to do with parallel testing. That's all up to the Harness to handle, process and merge. So call it .harnessrc or something like that. > if that config file would be called .taprc (or tap.rc on windows/vms) and > it would be stackable (~/.taprc, /etc/tap.rc ...) would that be usefull? Just seems kind of YAGNI to me, but maybe I'm missing an important use case. -- Michael Peters Plus Three, LP From scratchcomputing at gmail.com Thu Apr 10 20:34:25 2008 From: scratchcomputing at gmail.com (Eric Wilhelm) Date: Thu, 10 Apr 2008 12:34:25 -0700 Subject: [tap-l] Parallel testing In-Reply-To: <20080410145846.56d61b86@pc09.procura.nl> References: <20080410145846.56d61b86@pc09.procura.nl> Message-ID: <200804101234.25211.ewilhelm@cpan.org> # from H.Merijn Brand # on Thursday 10 April 2008 05:58: >In Oslo Andy and I came up with a way to define how Test::Harness > should deal with parallel tests, like this: I think this discussion belongs on the harness list, as it has little to do with the tap protocol. >conflict: >? ? folder:?????../ext/DB_File/t >? ? folder:?????../ext/IO_Compress_Zlib/t >? ? folder:?????../lib/CPANPLUS >? ? folder:?????../lib/ExtUtils/t ># ? folder:?????../lib/Pod/t > ># first file should be run before file 2, 3, 4, ... >depends: >? ? depend:?????[ foo.t bar.t ] I can understand the desire to parallelize a crufty test suite which is making bad assumptions, but I'm troubled by the idea that this needs to be specified in a config file and I think the 'depends' section is just blessing bad practice. A long time ago, my TAP::Harness::Parallel implemented a very simple partition() method which allowed an option to just automatically partition the process queue by directory. This almost always proved to be sufficient for working-around tests which assumed zero concurrency. It did potentially sacrifice some speed if your directories were too few or imbalanced. But externally configuring it to optimize speed while working around assumptions which cause collisions implies an attempt to polish something with a soft brown surface. As far as situations where automatic directory partitions are incorrect (two files in a directory have a resource conflict), you haven't addressed that unless you add a 'file' keyword. But then you need to e.g. spec a list of files to be run in sequence in one process and suddenly you're writing two copies of your MANIFEST. I saw the next step up from "simple workaround" as "fix your tests", and this proposed config file appears to require half of that work but without getting you halfway to the ideal state of non-broken tests. Further, clustered (multi-node parallelization) testing breaks dependent tests in new and different ways, so whatever is done to line-up the dependencies seems likely to get in the way of clustering. --Eric -- "Time flies like an arrow, but fruit flies like a banana." --Groucho Marx --------------------------------------------------- http://scratchcomputing.com --------------------------------------------------- From schwern at pobox.com Fri Apr 11 11:58:40 2008 From: schwern at pobox.com (Michael G Schwern) Date: Fri, 11 Apr 2008 12:58:40 +0200 Subject: [tap-l] Send TAP discussions to the TAP list. Message-ID: <47FF4460.80309@pobox.com> Note to TAP folks on perl-qa, please post TAP related stuff to the TAP mailing list: tap-l at testanything.org Remember, we're supposed to be encouraging cross-language discussion. It's ok to CC perl-qa but please make this the primary discussion list for TAP. -- I am somewhat preoccupied telling the laws of physics to shut up and sit down. -- Vaarsuvius, "Order of the Stick" http://www.giantitp.com/comics/oots0107.html From schwern at pobox.com Fri Apr 11 11:59:52 2008 From: schwern at pobox.com (Michael G Schwern) Date: Fri, 11 Apr 2008 12:59:52 +0200 Subject: [tap-l] Descriptive vs Proscriptive Message-ID: <47FF44A8.2020301@pobox.com> [For folks who aren't aware, we just had an intense three day hackathon in Oslo during which about a dozen of us tried to hash out new TAP extensions and write some sort of well formed spec. We got a lot done, but didn't have quite the clean resolution I was hoping for. Afterwards I had a number of revelations about the TAP development process which I'd like to share.] If we're all in agreement about how a thing should be used, I get worried. We saw this with the nested TAP syntax in Oslo. 7pm the first day we thought we had it all figured out in 15 minutes. Next morning we found the flaw. It all fell apart and we still haven't put the pieces together. We're a very homogeneous group. We're Unix people. We're Perl programmers. We've all grown up with one particular way of testing and just two primary libraries to do it (Test::More and Test::Harness). If we all agree that something is only going to be used one way you can bet that we haven't thought it through well enough. This is the Test ANYTHING Protocol, but it's being designed by a group with a very narrow experience. We don't have voices from other languages, communities, and testing systems to provide a healthy mix of use cases. So we must step very carefully not on what we allow, but what we disallow. Something I observed about the discussions in the TAP room was that when we had clear, existing use cases to follow we did quite well. We agreed on what needed to be done and it was just a matter of making it work within the confines of the protocol. We were just supporting existing best practice. Paving the cow paths [1]. When we strayed off the cow paths, when we pushed into unknown territory, when there was not yet a clear best practice or burgeoning user need, when we didn't have any clear information about how a feature is going to be used the process fell apart. We argued, but not about technical details but details of use. "The feature will get used like this!" "No no, it'll get used like this!" "People want to read it like this" "No, I never do it that way, I do it like this". They became quite emotional and frustrating. Two points in particular: the test "contexts" [2] which got quite far along but broke down in details because we had never given the idea much thought before. After a lot of arguing that pulled in several other groups at the hackathon we eventually decided that we don't have enough information and haven't given it enough thought so we'll shelve it until we do. The other was user-defined YAML keys. What should we allow? What should we disallow? We should we reserve? Initially it started out with reserving lower case and leaving upper case to users. Then edge cases came up. And people worried what happened if users did crazy things with the keys? What happens if they name a key "#&(!#*("? Or they use an ambiguously cased Hungarian i? Or if they use a font that doesn't show up case vs down case well? Or if their TAP producer has a bug and spits out a lower case key as upper case (or vice versa) and they should be able to spot that! Quite rapidly everyone shifted over to thinking that we should only allow "X-foo" for user keys because it's unambiguous. Then we don't have to worry about characters that don't have an up/down-case concept. And we can eyeball a user vs reserved key slip. And it looks like mail headers and we're all used to reading mail headers. And we can always allow a wider use later. Etc... Seems like a fine solution. Everyone agreed but me. It seemed like I was just being a sore loser, and maybe I am, but I don't often dig in my heels unless I think it's really, really important to get it right. The last time that happened was the business about Test::Harness 3 merging STDOUT and STDERR which took months to resolve. I don't really care so much about doing "Foo" or "X-Foo". It's all an aesthetic choice. What worries me is that we're encoding an aesthetic choice at all. That we're proscribing behaviors because we think it might be ugly or hard to read or harmful or stupid or redundant or difficult to specify. We have too narrow a vision to make that decision. All we can truthfully say about the future is that our predictions will be wrong. If we proscribe what we think might be bad, because we're going to be wrong, we also proscribe what might be good. If we proscribe what is bad now, because things change we also proscribe what might be good later. If we write parsers now which are proscriptive, that complain if they see something we don't like such as a "Foo" key instead of "X-foo", we paint the protocol into a corner. Any relaxing of the protocol later becomes a parser error which is a roadblock to change. We saw it happen several times in the past and in Oslo. If you disallow non-TAP lines you make it impossible to extend the language without upgrading all the parsers in lock-step with the producers. If we make an unrecognized lower-cased "reserved" diagnostic key a parser error then we can't add more keys without another lock-step parser/producer upgrade. If parsers puke when they see a future version of TAP we make it difficult to add new, otherwise backwards compatible features. This is why we should be descriptive instead of proscriptive. Descriptive means to specify only what you need and leave the rest open. It provides a playground for users to fool around in and try things out that we'd never have thought of. It provides the cracks into which really clever people can wedge radical new ideas to advance in wild new directions. It's the flexibility that allows a language to survive for 20 years and all the unpredictable changes that come. Perl survives and grows that way, TAP should too. Yes, it means we allow people to do silly things, but what's silly is often subjective. Yes, it makes parsing a bit more difficult and the spec a bit more complicated, but we've always weighted TAP towards simplicity of human reading and machine writing. Yes, it means we can't lock down the meaning of everything, but that's ok. A little chaos is healthy. A little chaos will allow us to extend the protocol more without breaking everything. TAP is 20 years old and growing because of how little it says so little about how you do your testing. It's worked because it's always been about what people need to do, not about preventing them from doing what we think they shouldn't. That's why I dug in my heels on the user keys. Why I don't just give in when something doesn't feel right but I'm "out voted". I couldn't articulate it properly then but I hope I've made it clear. We were violating a very important TAP design principle. We had strayed off the cow paths. We were proscribing use based upon our own narrow experiences, what we like and what makes sense to us today. Worse, we were walling off users from carving their own cow paths for us to follow. If you disallow anything but "X-foo" you can never learn what people might have done otherwise. We were filling in the cracks that future users may use to extend the protocol past anything we'd ever thought of. [1] Or sleigh tracks in the case of Oslo [2] link to test contexts proposal -- s7ank: i want to be one of those guys that types "s/j&jd//.^$ueu*///djsls/sm." and it's a perl script that turns dog crap into gold. From schwern at pobox.com Fri Apr 11 12:01:19 2008 From: schwern at pobox.com (Michael G Schwern) Date: Fri, 11 Apr 2008 13:01:19 +0200 Subject: [tap-l] User Supplied YAML Diagnostic Keys: Descriptive Version Message-ID: <47FF44FF.6010403@pobox.com> Here's the descriptive way to specify how the diagnostic keys work. 1) We reserve every key which begins with a lower case letter 2) We say nothing about anything else 3) All keys are optional That's it. Remember, the only reason we're saying anything about the keys is because we want to avoid future official keys from clashing with user defined keys. That is the *only* reason we need to nail this down and it allows us to say very little. This makes deciding whether or not something is reserved a simple matter of running islower() on it in POSIX or Unicode or whatever character set you're using. Should it become too difficult to nail down what "lower case" means we can simply reserve [a-z] and say to avoid anything that might be ambiguous. This also means we don't have to get bogged down in edge cases while still allowing almost complete freedom of movement for the users. Remember, the YAML diagnostics are a brand new thing and a huge addition of flexibility to TAP. People are going to want to do all sorts of wild things with it, let them. Let's not wall off the playground because someone might skin their knees. * What about keys with no concept of case? Numbers, asian characters... They're not lower case, we don't reserve them. Have fun with them. * What about ambiguously cased words like that Hungarian i thing that comes out as both upper and lower? We reserve it because it is lower case, and if you want to shoot yourself in the foot with ambiguity, or if you're Hungarian, that's fine by me. * I like using "X-foo" for my custom keys Ok, enjoy! * I like using "Foo" for my custom keys Ok, have fun. * I like writing my custom keys in Klingon! Ka'plah! Just make sure you don't start it with a lower case Klingon character and I'll allow you to die with honor. * What if I want to check that my TAP producer is outputting the right keys? What if I want to force everyone in my group to write their keys in the same style? Write a strict validating parser and use that. We all know it's best to automate tests. * What if I want to write my keys in some funky character set? Go ahead. Nobody else is likely to understand it, but that's ok. They're your custom keys for your use. * It's important to me to differentiate between user defined and reserved keys and I display TAP in a way that doesn't show upper case vs lower case well! What now? Use a prefix that's not ambiguous to your display. Or pick a better font, this one is going to cause you all sorts of grief. How are you seeing the difference between the "have" and "want" value? What about all the other protocols and languages and commands that are case sensitive? -- Stabbing you in the face so you don't have to. From sjn at pvv.org Thu Apr 10 23:33:56 2008 From: sjn at pvv.org (Salve J Nilsen) Date: Fri, 11 Apr 2008 00:33:56 +0200 (CEST) Subject: [tap-l] RFC for tap protocol In-Reply-To: <200704281340.18259.ewilhelm@cpan.org> References: <638975.68413.qm@web60820.mail.yahoo.com> <200704281340.18259.ewilhelm@cpan.org> Message-ID: Hey, guys. Sorry for waking up this old idea. aEric Wilhelm said: > > (psst. We use this list for the protocol stuff, right?) > > # from Andy Armstrong > # on Saturday 28 April 2007 07:16 am: >> On 28 Apr 2007, at 13:16, Ovid wrote: >>> >>> Salve Nilsen of Oslo.pm suggested to me that we go through the IETF >>> process and get an RFC out there to help improve the visibility of >>> TAP. Jos Boumans suggested that I contact him if we're going to do >>> this because he deal with IETF all the time. I think it's a great >>> idea. >>> >>> Thoughts? >> >> Absolutely. Let's do it. > > I'll third that. I think we're even going to have a pretty easy time > meeting the "multiple working reference implementations" criteria. Schwern and I had a few words with Harald T. Alvestrand (former chairman of IETF) about the relevance of putting TAP on a IETF track, and he basically answered "Sure, this sounds like something the application group would want to work on." Anyone want to look into what it takes to make this happen? :) - Salve, who'd volunteer if there was a way to do this offline -- #!/usr/bin/perl sub AUTOLOAD{$AUTOLOAD=~/.*::(\d+)/;seek(DATA,$1,0);print# Salve Joshua Nilsen getc DATA}$"="'};&{'";@_=unpack("C*",unpack("u*",':4@,$'.# '2!--"5-(50P%$PL,!0X354UC-PP%/0\`'."\n"));eval "&{'@_'}"; __END__ is near! :) From david at kineticode.com Fri Apr 11 18:15:48 2008 From: david at kineticode.com (David E. Wheeler) Date: Fri, 11 Apr 2008 10:15:48 -0700 Subject: [tap-l] Descriptive vs Proscriptive In-Reply-To: <47FF44A8.2020301@pobox.com> References: <47FF44A8.2020301@pobox.com> Message-ID: <53D1C256-26FF-48EF-9459-611F67026F5A@kineticode.com> On Apr 11, 2008, at 03:59, Michael G Schwern wrote: > Quite rapidly everyone shifted over to thinking that we should only > allow > "X-foo" for user keys because it's unambiguous. Then we don't have > to worry > about characters that don't have an up/down-case concept. And we > can eyeball > a user vs reserved key slip. And it looks like mail headers and > we're all > used to reading mail headers. And we can always allow a wider use > later. > Etc... What about reserving only lower-case ASCII for internal use, and anything else is fair game? That's unambiguous and pretty easy to enforce, plus is close to your original proposal, since 9x% of the time, people will just use uppercase ASCII anyway. > Seems like a fine solution. Everyone agreed but me. It seemed like > I was > just being a sore loser, and maybe I am, but I don't often dig in my > heels > unless I think it's really, really important to get it right. The > last time > that happened was the business about Test::Harness 3 merging STDOUT > and STDERR > which took months to resolve. It seems unnecessarily limiting to me. > I don't really care so much about doing "Foo" or "X-Foo". It's all an > aesthetic choice. What worries me is that we're encoding an > aesthetic choice > at all. That we're proscribing behaviors because we think it might > be ugly or > hard to read or harmful or stupid or redundant or difficult to > specify. We > have too narrow a vision to make that decision. All we can > truthfully say > about the future is that our predictions will be wrong. If we > proscribe what > we think might be bad, because we're going to be wrong, we also > proscribe what > might be good. If we proscribe what is bad now, because things > change we also > proscribe what might be good later. I agree. I think that the spec should actually limit a very small subset of a universe of options rather than limit the consumers of that spec to a small subset. Make the spec have to think harder about what's allowed. > This is why we should be descriptive instead of proscriptive. > Descriptive > means to specify only what you need and leave the rest open. It > provides a > playground for users to fool around in and try things out that we'd > never have > thought of. It provides the cracks into which really clever people > can wedge > radical new ideas to advance in wild new directions. It's the > flexibility > that allows a language to survive for 20 years and all the > unpredictable > changes that come. Perl survives and grows that way, TAP should too. Amen to that. Nice post, Schwern. Best, David From mpeters at plusthree.com Fri Apr 11 19:43:18 2008 From: mpeters at plusthree.com (Michael Peters) Date: Fri, 11 Apr 2008 14:43:18 -0400 Subject: [tap-l] User Supplied YAML Diagnostic Keys: Descriptive Version In-Reply-To: <87ej9cjksy.fsf@shirts.quite-simply.de> References: <47FF44FF.6010403@pobox.com> <87ej9cjksy.fsf@shirts.quite-simply.de> Message-ID: <47FFB146.8060901@plusthree.com> Steffen Schwigon wrote: > How about reserving a prefix for TAP related keys and allow/ignore > everything else? > > Explained in another way: > > a prefix lowercased "tap-" for TAP > Thumbs down from me. You don't see HTTP headers having to prefix all of their labels with 'HTTP-' or email headers doing a 'Email-' prefix either. -- Michael Peters Plus Three, LP From ss5 at renormalist.net Fri Apr 11 19:54:24 2008 From: ss5 at renormalist.net (Steffen Schwigon) Date: Fri, 11 Apr 2008 20:54:24 +0200 Subject: [tap-l] User Supplied YAML Diagnostic Keys: Descriptive Version In-Reply-To: <47FFB146.8060901@plusthree.com> (Michael Peters's message of "Fri, 11 Apr 2008 14:43:18 -0400") References: <47FF44FF.6010403@pobox.com> <87ej9cjksy.fsf@shirts.quite-simply.de> <47FFB146.8060901@plusthree.com> Message-ID: <873apsjk7z.fsf@shirts.quite-simply.de> Michael Peters writes: > Steffen Schwigon wrote: > >> How about reserving a prefix for TAP related keys and allow/ignore >> everything else? >> >> Explained in another way: >> >> a prefix lowercased "tap-" for TAP >> > > Thumbs down from me. You don't see HTTP headers having to prefix all > of their labels with 'HTTP-' or email headers doing a 'Email-' > prefix either. A see. But TAP isn't SMTP or HTTP and an explicit prefix matches the "be descriptive" of Schwern, doesn't it? And we are talking about the diagnostics part, which is primarily for the user, so the rules are reversed there. Steffen -- Steffen Schwigon Dresden Perl Mongers Deutscher Perl-Workshop From ss5 at renormalist.net Fri Apr 11 19:41:49 2008 From: ss5 at renormalist.net (Steffen Schwigon) Date: Fri, 11 Apr 2008 20:41:49 +0200 Subject: [tap-l] User Supplied YAML Diagnostic Keys: Descriptive Version In-Reply-To: <47FF44FF.6010403@pobox.com> (Michael G. Schwern's message of "Fri, 11 Apr 2008 13:01:19 +0200") References: <47FF44FF.6010403@pobox.com> Message-ID: <87ej9cjksy.fsf@shirts.quite-simply.de> Michael G Schwern writes: > Here's the descriptive way to specify how the diagnostic keys work. > > 1) We reserve every key which begins with a lower case letter > 2) We say nothing about anything else > 3) All keys are optional How about reserving a prefix for TAP related keys and allow/ignore everything else? Explained in another way: a prefix lowercased "tap-" for TAP as practically the *opposite* of the idea with a prefix uppercase "X-" for users. Did I overlook such an idea in your other posts? Steffen -- Steffen Schwigon Dresden Perl Mongers Deutscher Perl-Workshop From publiustemp-perlqa3 at yahoo.com Fri Apr 11 21:05:50 2008 From: publiustemp-perlqa3 at yahoo.com (Ovid) Date: Fri, 11 Apr 2008 13:05:50 -0700 (PDT) Subject: [tap-l] User Supplied YAML Diagnostic Keys: Descriptive Version In-Reply-To: <873apsjk7z.fsf@shirts.quite-simply.de> Message-ID: <902148.18615.qm@web65705.mail.ac4.yahoo.com> --- Steffen Schwigon wrote: > A see. But TAP isn't SMTP or HTTP and an explicit prefix matches the > "be descriptive" of Schwern, doesn't it? > > And we are talking about the diagnostics part, which is primarily for > the user, so the rules are reversed there. There are two goals we want: 1. Make it as human-readable as possible. 2. Maximize flexibility. As for human-readable, consider these: results: tap-have: 4 tap-want: 5 results: have: 4 want: 5 (And remember that there could be plenty of other diagnostics in there). I think the second is more readable, but that's a bit subjective. Still, the idea has merit because it addresses my concern below. Regarding maximum flexibility, Adrian Howard has already pointed out that if we restrict people to 'X-', then we can always be more permissive in the future. If we open it up to just about anything they want and it turns out to be a mistake, we could easily be backed into a corner. Why on earth would we *deliberately* choose an option that we know might be problematic, particularly when someone's non-unicode aware language might get confused about what is and isn't lower-case? It doesn't hurt us to play it safe with such a major change. It *might* hurt us to make a major change when we already KNOW that potential problems are out there. We're not talking about a minor utility that only a handful of people would use. We're talking about playing fast and loose with core infrastructure tool that many people are dependent on. On an implementation detail, parsers will need to differentiate between TAP standard keys an user-supplied keys. Right now it's simple in Perl: if ( $key =~ /^[[:lower:]]/ ) { # internal key } Not all languages make it that easy. Also -- I don't know the specifics here -- are we now going to have to require 'encoding' meta-information in TAP to make sure that the above regex works? Cheers, Ovid -- Buy the book - http://www.oreilly.com/catalog/perlhks/ Perl and CGI - http://users.easystreet.com/ovid/cgi_course/ Personal blog - http://publius-ovidius.livejournal.com/ Tech blog - http://use.perl.org/~Ovid/journal/ From pagaltzis at gmx.de Sun Apr 13 02:45:18 2008 From: pagaltzis at gmx.de (Aristotle Pagaltzis) Date: Sun, 13 Apr 2008 03:45:18 +0200 Subject: [tap-l] Now in your favourite editor Message-ID: <20080413014518.GD11333@klangraum> http://www.vim.org/scripts/script.php?script_id=2213 :-) Regards, -- Aristotle Pagaltzis // From schwern at pobox.com Sun Apr 13 18:41:04 2008 From: schwern at pobox.com (Michael G Schwern) Date: Sun, 13 Apr 2008 18:41:04 +0100 Subject: [tap-l] User Supplied YAML Diagnostic Keys: Descriptive Version In-Reply-To: <902148.18615.qm@web65705.mail.ac4.yahoo.com> References: <902148.18615.qm@web65705.mail.ac4.yahoo.com> Message-ID: <480245B0.6050906@pobox.com> Ovid wrote: > --- Steffen Schwigon wrote: >> And we are talking about the diagnostics part, which is primarily for >> the user, so the rules are reversed there. > > There are two goals we want: > > 1. Make it as human-readable as possible. > 2. Maximize flexibility. > > As for human-readable, consider these: > > results: > tap-have: 4 > tap-want: 5 It's worse than that, Jim. tap-results: tap-have: 4 tap-want: 5 tap tap tap tap tap. Similar things should look similar, so you can quickly and visually group them. Different things should look different so you can quickly and visually separate them. By giving everything the same prefix you make it all look the same. And it's saying that the primary use case is differentiating user vs standard keys. I agree, it's ugly. > Regarding maximum flexibility, Adrian Howard has already pointed out > that if we restrict people to 'X-', then we can always be more > permissive in the future. If we open it up to just about anything they > want and it turns out to be a mistake, we could easily be backed into a > corner. Why on earth would we *deliberately* choose an option that we > know might be problematic, particularly when someone's non-unicode > aware language might get confused about what is and isn't lower-case? Because we pave cow paths. If we restrict users to just one style then they do not have a flexibility to try out new things. We'll never discover what users actually need to do, because they can't do it. No cow paths will form because we've already laid a rail. Furthermore, all the TAP spec has to worry about is the set of reserved keywords. Whatever users do with the non-reserved stuff is their responsibility. We treat users like adults and give them power and responsibility. Remember, the producer and the displayer of the non-reserved keys are both under local user control. They choose the custom keys and they choose what they need and can handle. If Yahoo thinks it's a good idea to write their keys in Klingon then Yahoo will be prepared to handle their keys in Klingon. The user knows best. All TAP needs to do is carve out a wide and clear enough reserved space for future use so we don't bump into the user. We all seem to be basically in agreement about what that set is. Yes, it makes the spec a little fuzzy but it's worth it. Yes, people might do horrible things with it and blow big gaping holes in their feet, but they might also do surprisingly brilliant things with it. TAP is not about having a simple spec or easy parsing, it's about testing anything. > It doesn't hurt us to play it safe with such a major change. It > *might* hurt us to make a major change when we already KNOW that > potential problems are out there. We're not talking about a minor > utility that only a handful of people would use. We're talking about > playing fast and loose with core infrastructure tool that many people > are dependent on. > > On an implementation detail, parsers will need to differentiate between > TAP standard keys an user-supplied keys. TAP parsers should ignore any key they don't explicitly know about, otherwise adding new standard keys becomes difficult. Same reason we ignore non-TAP output. A TAP parser needs to know nothing about the keys, it just passes them through to the displayer. A TAP displayer just has a set of standard and custom keys it knows what to do something about. Anything else it displays in a generic fashion (or ignores). All you need is a simple string equality, something most languages can manage. :) The only time one would need to programmatically separate reserved from user-defined keys is if you're writing a strict-validating parser or you want to explicitly differentiate how standard vs user keys are displayed. And maybe you do, but the consequences of ambiguity in those scenarios is very minor. You still get the right test results and the user sees the diagnostics. > Not all languages make it that easy. Also -- I don't know the > specifics here -- are we now going to have to require 'encoding' > meta-information in TAP to make sure that the above regex works? Two possible solutions: A) Just reserve ASCII [a-z]. This is very easy to check for but I'm worried it's carving out too small a space. B) Reserve "lower case" and leave the spec a little fuzzy around the edges for the moment. I like B. It gives us some room to move. As demonstrated above, determining what's user and what's reserved isn't a big deal. The only reason it exists is so we don't define a standard TAP key that blows over a user one with a different meaning. For this to happen because of ambiguity over what is "lower case" requires a double breakdown: 1) The user has to use an edge case key, which might happen. 2) TAP has to define the same edge case key. Are we really going to define a standard TAP key starting with a Hungarian i? Or musical notes? Are we going to provide standardized keys localized to the user's language? (the displayer can do that) Not likely. We don't have to draw the spec with a fine point pen. It's ok to have some fuzzy bits if we're not sure about it and there's no serious consequences. Character encoding is one. We have very little information about how the YAML diagnostics are going to be used and we know nothing about how they'll interact with alternative character sets. Let's see what happens and when we know more we'll draw the lines a little clearer. -- 100. Claymore mines are not filled with yummy candy, and it is wrong to tell new soldiers that they are. -- The 213 Things Skippy Is No Longer Allowed To Do In The U.S. Army http://skippyslist.com/list/ From david at kineticode.com Sun Apr 13 19:31:36 2008 From: david at kineticode.com (David E. Wheeler) Date: Sun, 13 Apr 2008 11:31:36 -0700 Subject: [tap-l] User Supplied YAML Diagnostic Keys: Descriptive Version In-Reply-To: <480245B0.6050906@pobox.com> References: <902148.18615.qm@web65705.mail.ac4.yahoo.com> <480245B0.6050906@pobox.com> Message-ID: On Apr 13, 2008, at 10:41, Michael G Schwern wrote: > Two possible solutions: > > A) Just reserve ASCII [a-z]. This is very easy to check for but > I'm worried it's carving out too small a space. Why would it be too small? I mean, that's a *lot* of words you can use. > B) Reserve "lower case" and leave the spec a little fuzzy around > the edges for the moment. That seems reasonable to me. > I like B. It gives us some room to move. As demonstrated above, > determining what's user and what's reserved isn't a big deal. The > only reason it exists is so we don't define a standard TAP key that > blows over a user one with a different meaning. For this to happen > because of ambiguity over what is "lower case" requires a double > breakdown: > > 1) The user has to use an edge case key, which might happen. > 2) TAP has to define the same edge case key. > > Are we really going to define a standard TAP key starting with a > Hungarian i? Or musical notes? Are we going to provide > standardized keys localized to the user's language? (the displayer > can do that) Not likely. That's why I suggested lowercase ASCII. Because the use of anything else is incredibly unlikely. But B seems like a reasonable compromise to me, and should anything cause a problem in the future (unlikely, methinks), the spec could always be sharpened by limiting it to lowercase ASCII or Latin-1. > We don't have to draw the spec with a fine point pen. It's ok to > have some fuzzy bits if we're not sure about it and there's no > serious consequences. Character encoding is one. We have very > little information about how the YAML diagnostics are going to be > used and we know nothing about how they'll interact with alternative > character sets. Let's see what happens and when we know more we'll > draw the lines a little clearer. +1 Best, David From schwern at pobox.com Sun Apr 13 19:37:11 2008 From: schwern at pobox.com (Michael G Schwern) Date: Sun, 13 Apr 2008 19:37:11 +0100 Subject: [tap-l] User Supplied YAML Diagnostic Keys: Descriptive Version In-Reply-To: References: <902148.18615.qm@web65705.mail.ac4.yahoo.com> <480245B0.6050906@pobox.com> Message-ID: <480252D7.3070909@pobox.com> David E. Wheeler wrote: > On Apr 13, 2008, at 10:41, Michael G Schwern wrote: > >> Two possible solutions: >> >> A) Just reserve ASCII [a-z]. This is very easy to check for but I'm >> worried it's carving out too small a space. > > Why would it be too small? I mean, that's a *lot* of words you can use. I don't have any particular reason. Just a feeling that "7-bit ASCII should be good enough for anyone" is not such a safe position for anything wanting to look forward. -- 170. Not allowed to ?defect? to OPFOR during training missions. -- The 213 Things Skippy Is No Longer Allowed To Do In The U.S. Army http://skippyslist.com/list/ From david at kineticode.com Sun Apr 13 19:58:18 2008 From: david at kineticode.com (David E. Wheeler) Date: Sun, 13 Apr 2008 11:58:18 -0700 Subject: [tap-l] User Supplied YAML Diagnostic Keys: Descriptive Version In-Reply-To: <480252D7.3070909@pobox.com> References: <902148.18615.qm@web65705.mail.ac4.yahoo.com> <480245B0.6050906@pobox.com> <480252D7.3070909@pobox.com> Message-ID: On Apr 13, 2008, at 11:37, Michael G Schwern wrote: >>> A) Just reserve ASCII [a-z]. This is very easy to check for but >>> I'm worried it's carving out too small a space. >> Why would it be too small? I mean, that's a *lot* of words you can >> use. > > I don't have any particular reason. Just a feeling that "7-bit > ASCII should be good enough for anyone" is not such a safe position > for anything wanting to look forward. Well, it's not for *everyone*. Just for the folks who are adding keys to TAP itself, which is a pretty small set of people, all things told. But anyway, I agree that B seems fine and it can always be reduced to A later, provided, of course, that we only use ASCII or Latin-1 characters anyway, which seems quite likely to me. Best, David From scratchcomputing at gmail.com Sun Apr 13 20:06:49 2008 From: scratchcomputing at gmail.com (Eric Wilhelm) Date: Sun, 13 Apr 2008 12:06:49 -0700 Subject: [tap-l] User Supplied YAML Diagnostic Keys: Descriptive Version In-Reply-To: <480245B0.6050906@pobox.com> References: <902148.18615.qm@web65705.mail.ac4.yahoo.com> <480245B0.6050906@pobox.com> Message-ID: <200804131206.49262.ewilhelm@cpan.org> # from Michael G Schwern # on Sunday 13 April 2008 10:41: >No cow paths will form >because we've already laid a rail. Well, you can always plan to explore the trajectories implied by the craters where multiple people have fallen off of the rail. ;-) >A) Just reserve ASCII [a-z]. This is very easy to check for but I'm >worried it's carving out too small a space. I think the momentum of 105-key keyboards implies that reserving anything else is a waste of effort. Note though that I said to make room to include the numbers and '_' in your builtins. I hope you're just talking about the first character being in [a-z]. --Eric -- "But as to modern architecture, let us drop it and let us take modernistic out and shoot it at sunrise." --F.L. Wright --------------------------------------------------- http://scratchcomputing.com --------------------------------------------------- From schwern at pobox.com Sun Apr 13 21:39:40 2008 From: schwern at pobox.com (Michael G Schwern) Date: Sun, 13 Apr 2008 21:39:40 +0100 Subject: [tap-l] TAP Secretary? Message-ID: <48026F8C.2090702@pobox.com> It occurs to me that it would be handy to have a secretary position for TAP. This is basically the person who makes sure things get out of our collective heads and written down and in some orderly fashion. Responsibilities would include: * Getting proposals written up on the wiki. * Ensuring the proposals stay up to date with current positions and thinking. * Managing formatting standards. * Getting the spec written. * Investigating standards options. This could be done by the secretary, by an array of secretaries or by the secretary kicking other people to do some work. -- 185. My name is not a killing word. -- The 213 Things Skippy Is No Longer Allowed To Do In The U.S. Army http://skippyslist.com/list/ From schwern at pobox.com Sun Apr 13 21:40:52 2008 From: schwern at pobox.com (Michael G Schwern) Date: Sun, 13 Apr 2008 21:40:52 +0100 Subject: [tap-l] User Supplied YAML Diagnostic Keys: Descriptive Version In-Reply-To: <200804131206.49262.ewilhelm@cpan.org> References: <902148.18615.qm@web65705.mail.ac4.yahoo.com> <480245B0.6050906@pobox.com> <200804131206.49262.ewilhelm@cpan.org> Message-ID: <48026FD4.2060209@pobox.com> Eric Wilhelm wrote: > Note though that I said to make room to include the numbers and '_' in > your builtins. I hope you're just talking about the first character > being in [a-z]. Yes, just the first character. Though it might be worthwhile to reserve _ just in case we decide some sort of distinguishing prefix is useful. -- Ahh email, my old friend. Do you know that revenge is a dish that is best served cold? And it is very cold on the Internet! From schwern at pobox.com Sun Apr 13 22:58:33 2008 From: schwern at pobox.com (Michael G Schwern) Date: Sun, 13 Apr 2008 22:58:33 +0100 Subject: [tap-l] User Supplied YAML Diagnostic Keys: Descriptive Version In-Reply-To: <200804131425.40398.chromatic@wgz.org> References: <902148.18615.qm@web65705.mail.ac4.yahoo.com> <480245B0.6050906@pobox.com> <200804131425.40398.chromatic@wgz.org> Message-ID: <48028209.5060809@pobox.com> chromatic wrote: > On Sunday 13 April 2008 10:41:04 Michael G Schwern wrote: > >> Remember, the producer and the displayer of the non-reserved keys are both >> under local user control. They choose the custom keys and they choose what >> they need and can handle. > > That sort of eliminates the upgrading problem, doesn't it? How's that? > Remind me again why TAP needs to reserve a whole range of keywords or invent > disambiguation schemes if there's no upgrading problem? Same reason we have trouble adding new keywords to Perl 5, a user might have defined a key to mean one thing and suddenly we're saying it means something else. Clash. (Apologies for the hokey example, I'm not feeling creative) Let's say Test::Builder spits out "file", "line" and "name" keys automatically. User written producer on top of Test::Builder adds their own "flavor" key meaning ice cream flavor. User extended displayer displays appropriate pictures of ice cream based on the "flavor". TAP decides "flavor" should be all about quarks. Strange, top, bottom, etc... Test::Builder starts emitting "flavor" keys automatically. User written producer's keys and Test::Builder now clash. User written producer overrides (Test::Builder's choice, but maybe in another system it's the other way around) losing the Test::Builder supplied data. TAP displayer's interpretation of "flavor" now clashes with TAP's official meaning. Tests written using other TAP producers use the official "flavor" meaning and confuse the customized TAP displayer. Existing, uncustomized TAP displayers interpret their use of "flavor" incorrectly. User now has to decide between being non-standard and incompatible with other standardized TAP components or renaming all their uses of "flavor" to be "ice_cream_flavor". -- 44. I am not the atheist chaplain. -- The 213 Things Skippy Is No Longer Allowed To Do In The U.S. Army http://skippyslist.com/list/ From schwern at pobox.com Sun Apr 13 23:13:10 2008 From: schwern at pobox.com (Michael G Schwern) Date: Sun, 13 Apr 2008 23:13:10 +0100 Subject: [tap-l] RFC for tap protocol In-Reply-To: References: <638975.68413.qm@web60820.mail.yahoo.com> <200704281340.18259.ewilhelm@cpan.org> Message-ID: <48028576.8040401@pobox.com> Salve J Nilsen wrote: > Schwern and I had a few words with Harald T. Alvestrand (former chairman > of IETF) about the relevance of putting TAP on a IETF track, and he > basically answered "Sure, this sounds like something the application group > would want to work on." When Salve suggested this I burst out laughing because "standards" and "Perl" are like oil and water. In Oslo we were joking about "ISO TAP", how none of us would have been let in the room and the resulting horror that would come out the other end. After I had my laugh, I let Harald set me straight on how the IETF works. The main points I recall are... * The working group is basically an existing mailing list, this one would do. * The IETF doesn't poke their fingers into our business. * Even if we don't go for a full standard, we can still publish as a draft or RFC. * Even though TAP isn't technically an "Internet" protocol it's still within their scope. If nothing else, it would give us some structure to the process of specifying TAP and some legitimacy to the format by publishing with the IETF. It would allow TAP to be seen by a wider community. And hey, haven't you always wanted to publish an RFC? So I think this is worth pursuing. The RFCs about publishing make my eyes glaze over, but hopefully somebody else will plow through them. -- 94. Crucifixes do not ward off officers, and I should not test that. -- The 213 Things Skippy Is No Longer Allowed To Do In The U.S. Army http://skippyslist.com/list/ From pagaltzis at gmx.de Mon Apr 14 03:47:43 2008 From: pagaltzis at gmx.de (Aristotle Pagaltzis) Date: Mon, 14 Apr 2008 04:47:43 +0200 Subject: [tap-l] RFC for tap protocol In-Reply-To: <48028576.8040401@pobox.com> References: <638975.68413.qm@web60820.mail.yahoo.com> <200704281340.18259.ewilhelm@cpan.org> <48028576.8040401@pobox.com> Message-ID: <20080414024743.GG11333@klangraum> * Michael G Schwern [2008-04-14 00:15]: > When Salve suggested this I burst out laughing because > "standards" and "Perl" are like oil and water. In Oslo we were > joking about "ISO TAP", how none of us would have been let in > the room and the resulting horror that would come out the other > end. > > After I had my laugh, I let Harald set me straight on how the > IETF works. If he hadn?t, I would. I?ve gotten my name on two RFCs, so I saw how that sausage got made, and it?s definitely not a stuffy bureaucratic process but rather a kind of standard that would fit the Perl culture. IETF operates on ?rough consensus and working code? by and large, so Perl folks should take to it like a fish takes to water. > If nothing else, it would give us some structure to the process > of specifying TAP and some legitimacy to the format by > publishing with the IETF. It would allow TAP to be seen by a > wider community. I definitely support this move. In fact, I would strongly advocate it. > And hey, haven't you always wanted to publish an RFC? I wouldn?t mind getting my name on another. :-) > So I think this is worth pursuing. The RFCs about publishing > make my eyes glaze over, but hopefully somebody else will plow > through them. Well, there?ll probably be two or so working group chairs and an editor, and it?s them who?ll be doing most of the process work; they, in turn, will get some handholding from the IETF people that get assigned to see the WG through the process. Most contributors will notice almost nothing of the bureaucracy and won?t have to worry about it. What they WILL notice is just how much work it is to shake something down well enough to put the ?specification? stamp on it. Judging by their weblogs, some of the Oslo goers already got a good taste of that? :-) But I think it?s absolutely worth it. Regards, -- Aristotle Pagaltzis // From pagaltzis at gmx.de Mon Apr 14 04:15:27 2008 From: pagaltzis at gmx.de (Aristotle Pagaltzis) Date: Mon, 14 Apr 2008 05:15:27 +0200 Subject: [tap-l] TAP Secretary? In-Reply-To: <48026F8C.2090702@pobox.com> References: <48026F8C.2090702@pobox.com> Message-ID: <20080414031527.GH11333@klangraum> * Michael G Schwern [2008-04-13 22:45]: > It occurs to me that it would be handy to have a secretary > position for TAP. This is basically the person who makes sure > things get out of our collective heads and written down and in > some orderly fashion. > > Responsibilities would include: > > * Getting proposals written up on the wiki. > * Ensuring the proposals stay up to date with current positions > and thinking. > * Managing formatting standards. > * Getting the spec written. > * Investigating standards options. > > This could be done by the secretary, by an array of secretaries > or by the secretary kicking other people to do some work. Suggestion: send mail about wiki edits to the TAP list. As long as the wiki doesn?t have enough traffic to make this too annoying, it?s a good way to incite people to care about things they might not even have known of otherwise. Regards, -- Aristotle Pagaltzis // From pagaltzis at gmx.de Mon Apr 14 04:25:13 2008 From: pagaltzis at gmx.de (Aristotle Pagaltzis) Date: Mon, 14 Apr 2008 05:25:13 +0200 Subject: [tap-l] User Supplied YAML Diagnostic Keys: Descriptive Version In-Reply-To: References: <902148.18615.qm@web65705.mail.ac4.yahoo.com> <480245B0.6050906@pobox.com> <480252D7.3070909@pobox.com> Message-ID: <20080414032513.GI11333@klangraum> * David E. Wheeler [2008-04-13 21:00]: > On Apr 13, 2008, at 11:37, Michael G Schwern wrote: > >>> A) Just reserve ASCII [a-z]. This is very easy to check > >>> for but I'm worried it's carving out too small a space. > >> Why would it be too small? I mean, that's a *lot* of words > >> you can use. > > > > I don't have any particular reason. Just a feeling that > > "7-bit ASCII should be good enough for anyone" is not such > > a safe position for anything wanting to look forward. > > Well, it's not for *everyone*. Just for the folks who are > adding keys to TAP itself, which is a pretty small set of > people, all things told. > > But anyway, I agree that B seems fine and it can always be > reduced to A later, provided, of course, that we only use > ASCII or Latin-1 characters anyway, which seems quite likely > to me. I agree with David on all counts. [a-z] seems perfectly sufficient to me, but saying ?anything for which POSIX `islower` returns true? is acceptably precise. Regards, -- Aristotle Pagaltzis // From publiustemp-perlqa3 at yahoo.com Mon Apr 14 06:30:07 2008 From: publiustemp-perlqa3 at yahoo.com (Ovid) Date: Sun, 13 Apr 2008 22:30:07 -0700 (PDT) Subject: [tap-l] User Supplied YAML Diagnostic Keys: Descriptive Version In-Reply-To: <20080414032513.GI11333@klangraum> Message-ID: <556485.28061.qm@web65711.mail.ac4.yahoo.com> --- Aristotle Pagaltzis wrote: > I agree with David on all counts. > > [a-z] seems perfectly sufficient to me, but saying ???anything for > which POSIX `islower` returns true??? is acceptably precise. I'll go along with this. Can we move forward now? :) Cheers, Ovid -- Buy the book - http://www.oreilly.com/catalog/perlhks/ Perl and CGI - http://users.easystreet.com/ovid/cgi_course/ Personal blog - http://publius-ovidius.livejournal.com/ Tech blog - http://use.perl.org/~Ovid/journal/ From chromatic at wgz.org Sun Apr 13 22:25:40 2008 From: chromatic at wgz.org (chromatic) Date: Sun, 13 Apr 2008 14:25:40 -0700 Subject: [tap-l] User Supplied YAML Diagnostic Keys: Descriptive Version In-Reply-To: <480245B0.6050906@pobox.com> References: <902148.18615.qm@web65705.mail.ac4.yahoo.com> <480245B0.6050906@pobox.com> Message-ID: <200804131425.40398.chromatic@wgz.org> On Sunday 13 April 2008 10:41:04 Michael G Schwern wrote: > Remember, the producer and the displayer of the non-reserved keys are both > under local user control. ?They choose the custom keys and they choose what > they need and can handle. That sort of eliminates the upgrading problem, doesn't it? Remind me again why TAP needs to reserve a whole range of keywords or invent disambiguation schemes if there's no upgrading problem? -- c From chromatic at wgz.org Sun Apr 13 23:58:58 2008 From: chromatic at wgz.org (chromatic) Date: Sun, 13 Apr 2008 15:58:58 -0700 Subject: [tap-l] User Supplied YAML Diagnostic Keys: Descriptive Version In-Reply-To: <48028209.5060809@pobox.com> References: <902148.18615.qm@web65705.mail.ac4.yahoo.com> <200804131425.40398.chromatic@wgz.org> <48028209.5060809@pobox.com> Message-ID: <200804131558.58808.chromatic@wgz.org> On Sunday 13 April 2008 14:58:33 Michael G Schwern wrote: > chromatic wrote: > > On Sunday 13 April 2008 10:41:04 Michael G Schwern wrote: > >> Remember, the producer and the displayer of the non-reserved keys are > >> both under local user control. They choose the custom keys and they > >> choose what they need and can handle. > > That sort of eliminates the upgrading problem, doesn't it? > How's that? Any organization with custom code on both sides of the producer/consumer pipeline has the option of upgrading Test::Builder or TAP::Harness, modifying their local modules, or neither. This is the same choice everyone always has when maintaining local forks. If you don't get your code in upstream, you get to maintain it yourself across releases. If you want to pave cow paths, you're going to have to watch where cows go. Shunting all of the cows into a pasture you'll never see doesn't really help. Put some pressure on downstream to get their customizations in the next revision of TAP (there's a reason TAP has version numbers). The problem with an infinitely expandable protocol that tries to do everything is that it's infinitely expandable and tries to do everything. Might be nice to rein that in a little bit more. Or don't, and instead make it trivial to add ad-hoc keys willy-nilly without planning or communication or consistency, and you'll end up with the protocol equivalent of spaghetti. Anyone care to guess how many X-* headers there are in all of the SMTP clients and servers in the wild? How about HTTP headers? Maybe you don't have to care about them when they're in the spec, but at some point, someone has to write software to produce and consume them. It would be nice if that process didn't completely suck. -- c From ss5 at renormalist.net Mon Apr 14 09:32:09 2008 From: ss5 at renormalist.net (Steffen Schwigon) Date: Mon, 14 Apr 2008 10:32:09 +0200 Subject: [tap-l] User Supplied YAML Diagnostic Keys: Descriptive Version In-Reply-To: <480245B0.6050906@pobox.com> (Michael G. Schwern's message of "Sun, 13 Apr 2008 18:41:04 +0100") References: <902148.18615.qm@web65705.mail.ac4.yahoo.com> <480245B0.6050906@pobox.com> Message-ID: <871w58rg52.fsf@shirts.quite-simply.de> Michael G Schwern writes: > Ovid wrote: >> --- Steffen Schwigon wrote: >>> And we are talking about the diagnostics part, which is primarily for >>> the user, so the rules are reversed there. >> There are two goals we want: >> 1. Make it as human-readable as possible. >> 2. Maximize flexibility. >> As for human-readable, consider these: >> results: >> tap-have: 4 >> tap-want: 5 > > It's worse than that, Jim. > > tap-results: > tap-have: 4 > tap-want: 5 > > tap tap tap tap tap. Then maybe I haven't understood what the standardization of diagnostics is about. I thought it is primarily meant for the *user* of TAP to transport its own diagnostics of its own test runs and test results? I for instance would use it to transport benchmark results inside tests. I hadn't expected much of *TAP* specific stuff there. So the above horror scenario with "tap tap tap everywher" is not really happening. Or are the YAMLis keys of diagnostics expected to transport all what current typical Test::* modules provide? This would include, for instance, all the diagnostics that may result of is ok is_deeply like isa_ok file_exists file_empty file_size file_max_size (min) file_readable (executable) file_mode file_is_symlink owner test_verbose lives_ok dies_ok critic_ok all_critic_ok all_code_files and many more. This is probably not what standardization of keys withing diagnostics is about. Or is it? Can you give me examples what keys you try to standardize as part of TAP inside diagnostics? I interpret the examples of TAP13 with "data", "expect", "got" in http://podwiki.hexten.net/podwiki.pl?page=TAP13 as "user keys", i.e. *not* standardized as TAP. Steffen -- Steffen Schwigon Dresden Perl Mongers Deutscher Perl-Workshop From publiustemp-perlqa3 at yahoo.com Mon Apr 14 09:55:52 2008 From: publiustemp-perlqa3 at yahoo.com (Ovid) Date: Mon, 14 Apr 2008 01:55:52 -0700 (PDT) Subject: [tap-l] User Supplied YAML Diagnostic Keys: Descriptive Version In-Reply-To: <871w58rg52.fsf@shirts.quite-simply.de> Message-ID: <186174.73880.qm@web65702.mail.ac4.yahoo.com> --- Steffen Schwigon wrote: > Then maybe I haven't understood what the standardization of > diagnostics is about. > > I thought it is primarily meant for the *user* of TAP to transport > its own diagnostics of its own test runs and test results? > > I for instance would use it to transport benchmark results inside > tests. And that's fine. You can extend it for any use you like. However, because diagnostics are currently free-form, it's almost impossible to do anything *programmatic* with them. So, for example, consider sample output from Test::Differences: not ok 1 - lang is respected # Failed test 'lang is respected' # at lang.t line 23 # +----+-----+----------+ # | Elt|Got |Expected | # +----+-----+----------+ # * 0|uno |one * # * 1|dos |two * # +----+-----+----------+ Well, that seems fine, but imagine that you have test tool which wants to automatically jump to the file and line number (or auto-navigate you through a stack trace). We have to *guarantee* that this format never changes. Ever worked with a test suite where a bunch of tests failed because someone corrected the spelling in a 'die' message or localization breaks your hard-coded string matching? By offering a well-defined syntax, we can alleviate much of this sort of pain. The above might well translate to this: --- file: lang.t line: 23 results: have: - uno - dos want: - one - two name: lang is respected display: | +----+-----+----------+ | Elt|Got |Expected | +----+-----+----------+ * 0|uno |one * * 1|dos |two * +----+-----+----------+ Then all which really needs to be done is to ensure that your particular TAP consumer knows what to do with it. In fact, it would be trivial to simply transform that back to the unstructured diagnostics we are familiar with. Cheers, Ovid -- Buy the book - http://www.oreilly.com/catalog/perlhks/ Perl and CGI - http://users.easystreet.com/ovid/cgi_course/ Personal blog - http://publius-ovidius.livejournal.com/ Tech blog - http://use.perl.org/~Ovid/journal/ From publiustemp-tapx at yahoo.com Mon Apr 14 12:14:24 2008 From: publiustemp-tapx at yahoo.com (Ovid) Date: Mon, 14 Apr 2008 04:14:24 -0700 (PDT) Subject: [tap-l] RFC for tap protocol In-Reply-To: Message-ID: <131107.20005.qm@web65713.mail.ac4.yahoo.com> --- Salve J Nilsen wrote: > >>> Salve Nilsen of Oslo.pm suggested to me that we go through the > IETF > >>> process and get an RFC out there to help improve the visibility > of > >>> TAP. Jos Boumans suggested that I contact him if we're going to > do > >>> this because he deal with IETF all the time. I think it's a > great > >>> idea. I also like this idea. Heck, anything which offers a stamp of "legitimacy" is a tempting marketing tool. There is a lack of standardization regarding shared test data and it sounds like the IETF process is lightweight enough that we can actually get things done. I guess it goes without saying that I'd like to be part of the IETF group. Cheers, Ovid -- Buy the book - http://www.oreilly.com/catalog/perlhks/ Perl and CGI - http://users.easystreet.com/ovid/cgi_course/ Personal blog - http://publius-ovidius.livejournal.com/ Tech blog - http://use.perl.org/~Ovid/journal/ From pagaltzis at gmx.de Mon Apr 14 12:54:02 2008 From: pagaltzis at gmx.de (Aristotle Pagaltzis) Date: Mon, 14 Apr 2008 13:54:02 +0200 Subject: [tap-l] OT: muttrc (was: User Supplied YAML Diagnostic Keys: Descriptive Version) In-Reply-To: <20080414055926.GM79799@plum.flirble.org> References: <902148.18615.qm@web65705.mail.ac4.yahoo.com> <200804131425.40398.chromatic@wgz.org> <48028209.5060809@pobox.com> <200804131558.58808.chromatic@wgz.org> <20080414055926.GM79799@plum.flirble.org> Message-ID: <20080414115402.GJ11333@klangraum> * Nicholas Clark [2008-04-14 08:00]: > $ grep -c 'ignore X' ~/.muttrc > 100 > > That's the ones I've collected that I don't care about. And > some of those are common prefixes. You know that you can use wildcards to ignore everything (or just big swathes of stuff) by default and then selectively unignore some of non-gunk, right? Regards, -- Aristotle Pagaltzis // From tap at rjbs.manxome.org Mon Apr 14 12:59:34 2008 From: tap at rjbs.manxome.org (Ricardo SIGNES) Date: Mon, 14 Apr 2008 07:59:34 -0400 Subject: [tap-l] Nested TAP In-Reply-To: <238318.14601.qm@web65714.mail.ac4.yahoo.com> References: <238318.14601.qm@web65714.mail.ac4.yahoo.com> Message-ID: <20080414115934.GA6520@knight.local> * Ovid [2008-04-10T05:42:07] > I hated this decision, but was hard-pressed to argue against it because > making things backwards-compatible seemed to involve putting a > full-blown parser into the TAP producer. (In other words, sort of like > embedding Test::Harness into Test::Builder). I *think* I've figured > out how to make things backwards-compatible and not require a parser. > The background is here: > > http://use.perl.org/~Ovid/journal/36120 This looked pretty reasonable to me when I read it. I haven't seen anyone respond. Getting nested TAP onto the table would be a huge win. What are the objections to this approach? -- rjbs From publiustemp-tapx at yahoo.com Mon Apr 14 13:38:06 2008 From: publiustemp-tapx at yahoo.com (Ovid) Date: Mon, 14 Apr 2008 05:38:06 -0700 (PDT) Subject: [tap-l] Nested TAP In-Reply-To: <20080414115934.GA6520@knight.local> Message-ID: <291505.99814.qm@web65707.mail.ac4.yahoo.com> --- Ricardo SIGNES wrote: > > http://use.perl.org/~Ovid/journal/36120 > > This looked pretty reasonable to me when I read it. I haven't seen > anyone > respond. Getting nested TAP onto the table would be a huge win. > What are the > objections to this approach? The objection is that embedding a TAP parser into a TAP generator potentially raises the bar to an unacceptably high level. However, since all we need is *core* tap, I've a 40 line proof of concept: http://use.perl.org/~Ovid/journal/36121 That's how easy this is. So far only Michael Peters has spoken up about this idea (he points out the issue of return codes). I'd like to hear the opinion of others. Cheers, Ovid -- Buy the book - http://www.oreilly.com/catalog/perlhks/ Perl and CGI - http://users.easystreet.com/ovid/cgi_course/ Personal blog - http://publius-ovidius.livejournal.com/ Tech blog - http://use.perl.org/~Ovid/journal/ From andy at hexten.net Mon Apr 14 13:55:13 2008 From: andy at hexten.net (Andy Armstrong) Date: Mon, 14 Apr 2008 13:55:13 +0100 Subject: [tap-l] Nested TAP In-Reply-To: <291505.99814.qm@web65707.mail.ac4.yahoo.com> References: <291505.99814.qm@web65707.mail.ac4.yahoo.com> Message-ID: <31429F66-AB04-4F3F-AFA3-04E2D8430583@hexten.net> On 14 Apr 2008, at 13:38, Ovid wrote: > That's how easy this is. So far only Michael Peters has spoken up > about this idea (he points out the issue of return codes). I'd like > to > hear the opinion of others. I understand the motives - but it's going to mean that /every/ TAP producer from here on is going to carry the burden of being able to generate the summary result. That includes producers written in C, shell script, assembler. I'd rather have a clean definition of nested TAP (no summary redundancy) and be explicit that it's a new feature that, for now, should only be used if you control the whole testing toolchain and can upgrade Test::Harness accordingly. I'm biased - but I can't really get past my original proposal: ok 1 - foo begin 2 - nested 1..2 ok 1 - bar not ok 2 - fribble end 2 - nested not ok 3 - fook That allows TAP to be aggregated just by slapping a begin / end around a complete TAP document. I can use cat to aggregate TAP! I understand the importance of backwards compatibility - but it's just one factor. I think we need to compromise. In ten years time I don't want to be writing a TAP producer and still have to compute the redundant summary result just so that TAP is backwards compatible with code nobody uses any more. -- Andy Armstrong, Hexten From tap at rjbs.manxome.org Mon Apr 14 19:10:51 2008 From: tap at rjbs.manxome.org (Ricardo SIGNES) Date: Mon, 14 Apr 2008 14:10:51 -0400 Subject: [tap-l] Nested TAP In-Reply-To: <31429F66-AB04-4F3F-AFA3-04E2D8430583@hexten.net> References: <291505.99814.qm@web65707.mail.ac4.yahoo.com> <31429F66-AB04-4F3F-AFA3-04E2D8430583@hexten.net> Message-ID: <20080414181051.GA8288@minion169.office.icgroup.com> * Andy Armstrong [2008-04-14T08:55:13] > I'm biased - but I can't really get past my original proposal: > > ok 1 - foo > begin 2 - nested > 1..2 > ok 1 - bar > not ok 2 - fribble > end 2 - nested > not ok 3 - fook > > That allows TAP to be aggregated just by slapping a begin / end around > a complete TAP document. I can use cat to aggregate TAP! What were the objections? Two "ok 1" tests in one stream (if the begin and plan are ignored as apparently-not-TAP) breaks some backcompat? -- rjbs From andy at hexten.net Mon Apr 14 19:21:47 2008 From: andy at hexten.net (Andy Armstrong) Date: Mon, 14 Apr 2008 19:21:47 +0100 Subject: [tap-l] Nested TAP In-Reply-To: <20080414181051.GA8288@minion169.office.icgroup.com> References: <291505.99814.qm@web65707.mail.ac4.yahoo.com> <31429F66-AB04-4F3F-AFA3-04E2D8430583@hexten.net> <20080414181051.GA8288@minion169.office.icgroup.com> Message-ID: <233EDE4C-BC8F-44F6-8116-3A7DD42F725F@hexten.net> On 14 Apr 2008, at 19:10, Ricardo SIGNES wrote: > * Andy Armstrong [2008-04-14T08:55:13] >> I'm biased - but I can't really get past my original proposal: >> >> ok 1 - foo >> begin 2 - nested >> 1..2 >> ok 1 - bar >> not ok 2 - fribble >> end 2 - nested >> not ok 3 - fook >> >> That allows TAP to be aggregated just by slapping a begin / end >> around >> a complete TAP document. I can use cat to aggregate TAP! > > What were the objections? Two "ok 1" tests in one stream (if the > begin and > plan are ignored as apparently-not-TAP) breaks some backcompat? Using it isn't backwards compatible. I /think/ Curtis' idea was to indent the nested block (so it's ignored by current parsers) and provide a summary result for the block - so the whole thing parses correctly with current parsers. -- Andy Armstrong, Hexten From publiustemp-tapx at yahoo.com Mon Apr 14 20:20:52 2008 From: publiustemp-tapx at yahoo.com (Ovid) Date: Mon, 14 Apr 2008 12:20:52 -0700 (PDT) Subject: [tap-l] Nested TAP In-Reply-To: <233EDE4C-BC8F-44F6-8116-3A7DD42F725F@hexten.net> Message-ID: <394614.16379.qm@web65701.mail.ac4.yahoo.com> --- Andy Armstrong wrote: > Using it isn't backwards compatible. I /think/ Curtis' idea was to > indent the nested block (so it's ignored by current parsers) and > provide a summary result for the block - so the whole thing parses > correctly with current parsers. That's correct. The problem with the current ideas for breaking backwards compatibility is there's no way for the TAP producer to downgrade. If the assumption is that it could be difficult for the producer to parse the Core TAP out of a stream and produce a summary. Consider: TAP version 13 ok 1 - foo begin 2 - nested 1..2 ok 1 - bar not ok 2 - fribble end 2 - nested not ok 3 - fook If the TAP consumer (Test::Harness in our case) doesn't say that it understands TAP v13, the producer has *no* way of downgrading the above TAP and would be forced to abort -- unless it had a minimal parser (like my 40 line example which can be made even shorter). Thus, tests which *should* pass will now fail. That violates the entire spirit of TAP. Cheers, Ovid -- Buy the book - http://www.oreilly.com/catalog/perlhks/ Perl and CGI - http://users.easystreet.com/ovid/cgi_course/ Personal blog - http://publius-ovidius.livejournal.com/ Tech blog - http://use.perl.org/~Ovid/journal/ From andy at hexten.net Mon Apr 14 20:56:40 2008 From: andy at hexten.net (Andy Armstrong) Date: Mon, 14 Apr 2008 20:56:40 +0100 Subject: [tap-l] Nested TAP In-Reply-To: <394614.16379.qm@web65701.mail.ac4.yahoo.com> References: <394614.16379.qm@web65701.mail.ac4.yahoo.com> Message-ID: <006FE1D3-FD60-4AD8-90BE-873577E90F3C@hexten.net> On 14 Apr 2008, at 20:20, Ovid wrote: > If the TAP consumer (Test::Harness in our case) doesn't say that it > understands TAP v13, the producer has *no* way of downgrading the > above > TAP and would be forced to abort -- unless it had a minimal parser > (like my 40 line example which can be made even shorter). Thus, tests > which *should* pass will now fail. That violates the entire spirit of > TAP. I'm lost now :) Am I right in thinking that the principal differences between the two approaches can be summarised thusly? Your approach: generated TAP is automatically backwards compatible, producer needs a minimal parser or additional logic to keep track of summary results, producer doesn't need to know the TAP version supported by consumer. My approach: nested TAP isn't b/c, simpleminded producers (no parser), producer can downgrade iff it knows the TAP version supported by consumer. I like the idea of not breaking backwards compatibility /now/ - but I don't like the idea of living with backwards compatibility baggage forever. Both seem like reasonably approaches. I think one's preference would depend on the relative values attached to backwards compatibility and future simplicity. I think future simplicity wins in the long tail :) Another point in favour of your suggestion: there's no need for a producer to contain a parser unless it aggregates tests. If it's just generating nested TAP it can probably generate the summary result based on its own bookkeeping. -- Andy Armstrong, Hexten From mpeters at plusthree.com Mon Apr 14 21:05:17 2008 From: mpeters at plusthree.com (Michael Peters) Date: Mon, 14 Apr 2008 16:05:17 -0400 Subject: [tap-l] Nested TAP In-Reply-To: <006FE1D3-FD60-4AD8-90BE-873577E90F3C@hexten.net> References: <394614.16379.qm@web65701.mail.ac4.yahoo.com> <006FE1D3-FD60-4AD8-90BE-873577E90F3C@hexten.net> Message-ID: <4803B8FD.1020705@plusthree.com> Andy Armstrong wrote: > I like the idea of not breaking backwards compatibility /now/ - but I > don't like the idea of living with backwards compatibility baggage > forever. I'll go one step further and say if we want to have TAP break out into the non-Perl world we should shed as much old baggage as we can now so we can move forward. I agree that always having a lock-step upgrade between the producer and consumer is a bad idea. But doing it this once, this early on in the life of TAP*, to add a radical necessary change to the protocol would be a good use-case IMO. Try explaining to Java folks, or the IETF that we're making some decisions for a language-agnostic protocol base on decisions that Perl made 20 years ago. That would turn me off right then and there. > Both seem like reasonably approaches. I think one's preference would > depend on the relative values attached to backwards compatibility and > future simplicity. I think future simplicity wins in the long tail :) Here here! > Another point in favour of your suggestion: there's no need for a > producer to contain a parser unless it aggregates tests. If it's just > generating nested TAP it can probably generate the summary result > based on its own bookkeeping. Not really. For instance if there are multiple threads producing TAP the parent would have to communicate to the child what test number it is before it can produce the summary line. * - I know it's been around for 20+ years, but not as a language-agnostic protocol it hasn't. -- Michael Peters Plus Three, LP From andy at hexten.net Mon Apr 14 21:13:44 2008 From: andy at hexten.net (Andy Armstrong) Date: Mon, 14 Apr 2008 21:13:44 +0100 Subject: [tap-l] Nested TAP In-Reply-To: <4803B8FD.1020705@plusthree.com> References: <394614.16379.qm@web65701.mail.ac4.yahoo.com> <006FE1D3-FD60-4AD8-90BE-873577E90F3C@hexten.net> <4803B8FD.1020705@plusthree.com> Message-ID: <2F96FA54-F400-4013-9C65-098C0005D758@hexten.net> On 14 Apr 2008, at 21:05, Michael Peters wrote: >> Another point in favour of your suggestion: there's no need for a >> producer to contain a parser unless it aggregates tests. If it's just >> generating nested TAP it can probably generate the summary result >> based on its own bookkeeping. > > Not really. For instance if there are multiple threads producing TAP > the parent > would have to communicate to the child what test number it is before > it can > produce the summary line. Agreed. I could have put that better. What I meant was "there will be some cases in which a useful producer can emit nested TAP without being a parser / aggregator". -- Andy Armstrong, Hexten From publiustemp-tapx at yahoo.com Mon Apr 14 21:23:04 2008 From: publiustemp-tapx at yahoo.com (Ovid) Date: Mon, 14 Apr 2008 13:23:04 -0700 (PDT) Subject: [tap-l] Nested TAP In-Reply-To: <4803B8FD.1020705@plusthree.com> Message-ID: <100071.3634.qm@web65702.mail.ac4.yahoo.com> --- Michael Peters wrote: > Try explaining to Java folks, or the IETF that we're making some > decisions for a > language-agnostic protocol base on decisions that Perl made 20 years > ago. That would turn me off right then and there. That would depend on whether or not those are good or bad decisions. Heck, as were were debating various ideas on Oslo and had tentatively agreed on breaking backwards compatibility, I realized that there wasn't much stopping us from going whole hog and dropping TAP altogether in favor of YAML. By requiring a summary line, we *gain* the simplicity of always having the top level TAP stream parseable without forcing the consumer to introduce complications necessary to parse nested TAP. With my solution, determining if the top level core TAP stream passed or failed is trivial and *that* I think is what we should be focusing on. Someone could then write a minimal producer/consumer and add features as needed rather than going down the more difficult route of requiring them to parse arbitrarily nested TAP. Thus, someone who's parser can only handle basic core TAP could handle the newer versions. I would agree to break backwards compatibility if and only if we can reasonably demonstrate that we're solving more problems than we're introducing. So far I don't see that, but I'm willing to be convinced since I acknowledge that we have issues with nested TAP. Perhaps we should step back and ask ourselves what problems we're trying to solve with nested TAP and ask if there are other approaches? Cheers, Ovid -- Buy the book - http://www.oreilly.com/catalog/perlhks/ Perl and CGI - http://users.easystreet.com/ovid/cgi_course/ Personal blog - http://publius-ovidius.livejournal.com/ Tech blog - http://use.perl.org/~Ovid/journal/ From andy at hexten.net Mon Apr 14 22:10:31 2008 From: andy at hexten.net (Andy Armstrong) Date: Mon, 14 Apr 2008 22:10:31 +0100 Subject: [tap-l] Nested TAP In-Reply-To: <100071.3634.qm@web65702.mail.ac4.yahoo.com> References: <100071.3634.qm@web65702.mail.ac4.yahoo.com> Message-ID: <0E699440-299E-40A3-95FD-24E8C4C6DFB0@hexten.net> On 14 Apr 2008, at 21:23, Ovid wrote: > By requiring a summary line, we *gain* the simplicity of always having > the top level TAP stream parseable without forcing the consumer to > introduce complications necessary to parse nested TAP. With my > solution, determining if the top level core TAP stream passed or > failed > is trivial and *that* I think is what we should be focusing on. OK, that's probably another axis along which we can disagree :) I favour keeping the overall complexity down - but to the extent that we have features that are tricky to implement I'd like that complexity to be in the consumer. In my view the utility of TAP is dependent more on the ease of writing producers than consumers. Users of other languages will derive more benefit from being able to write a producer in their native language. For me one of the crucial advantages of not requiring a summary result is that facilitates assembling TAP streams from multiple producers without any particular co-operation (either shared state or parser/ aggregator) between those producers. In the case of e.g. an embedded system that kind of simplicity seems to me to be a huge win. I can't imagine a use case that has me writing a TAP consumer to operate in such a constrained environment. Requiring people to write a complex TAP producer in, e.g. PIC assembler seems less appealing than requiring them to use a pre- existing TAP consumer written in Perl. In general, in cases where there is a disparity between the platform running the producer and the platform running the consumer, the producer will be operating in a the less salubrious environment. -- Andy Armstrong, Hexten From pagaltzis at gmx.de Tue Apr 15 02:44:06 2008 From: pagaltzis at gmx.de (Aristotle Pagaltzis) Date: Tue, 15 Apr 2008 03:44:06 +0200 Subject: [tap-l] Nested TAP In-Reply-To: <0E699440-299E-40A3-95FD-24E8C4C6DFB0@hexten.net> <4803B8FD.1020705@plusthree.com> <394614.16379.qm@web65701.mail.ac4.yahoo.com> References: <100071.3634.qm@web65702.mail.ac4.yahoo.com> <0E699440-299E-40A3-95FD-24E8C4C6DFB0@hexten.net> <394614.16379.qm@web65701.mail.ac4.yahoo.com> <006FE1D3-FD60-4AD8-90BE-873577E90F3C@hexten.net> <4803B8FD.1020705@plusthree.com> <233EDE4C-BC8F-44F6-8116-3A7DD42F725F@hexten.net> <394614.16379.qm@web65701.mail.ac4.yahoo.com> Message-ID: <20080415014406.GL11333@klangraum> * Ovid [2008-04-14 21:25]: > The problem with the current ideas for breaking backwards > compatibility is there's no way for the TAP producer to > downgrade. That rings alarm bells for me. I don?t think we can *assume* any way for the consumer and producer to communicate at the time of TAP production. The two ends can be completely decoupled in space and time; that is (implict in) the point of TAP. > Consider: > > TAP version 13 > ok 1 - foo > begin 2 - nested > 1..2 > ok 1 - bar > not ok 2 - fribble > end 2 - nested > not ok 3 - fook > > If the TAP consumer (Test::Harness in our case) doesn't say > that it understands TAP v13, the producer has *no* way of > downgrading the above TAP and would be forced to abort -- > unless it had a minimal parser (like my 40 line example which > can be made even shorter). Thus, tests which *should* pass > will now fail. That violates the entire spirit of TAP. Am I correct in thinking that you can translate this to a back- compatible TAP stream merely by inserting an unindented `not ok 2` line before the `begin 2` line? If so, then it is inevitable that someone will have to parse the nested TAP, and it?s merely a question of which end that will be. If that is the case, why are we trying to dictate the side on which this should happen? I would argue that instead of pinning the duty on either end, we should merely describe how TAP v13 can be downgraded to TAP v12 and leave it up to individual users of TAP where they want to shift that complexity. If push comes to shove, they can write an intermediate pipeline step that takes v13 TAP and downgrades it in the manner we have described in the spec before sending it to their v12 consumer. In fact, we should probably make v13 allow both the presence and absence of the back-compat shim line. That way, people can write v13 producers now that work with v12 consumers now, but those who have upgraded their entire infrastructure can shed that crutch. * Michael Peters [2008-04-14 22:10]: > I'll go one step further and say if we want to have TAP break > out into the non-Perl world we should shed as much old baggage > as we can now so we can move forward. ++ * Andy Armstrong [2008-04-14 23:15]: > I favour keeping the overall complexity down - but to the > extent that we have features that are tricky to implement I'd > like that complexity to be in the consumer. In my view the > utility of TAP is dependent more on the ease of writing > producers than consumers. Users of other languages will derive > more benefit from being able to write a producer in their > native language. Fully agreed. One thing we know is that even as we?re working to remove the one-producer-one-consumer situation, we will *always* have more producers than consumers. It is important to avoid upgrade-the-world scenarios, but it is equally important to be able to write completely dumb producers. I think both of these goals need to be kept in mind for every future version of TAP. For nested TAP I think we can achieve this in the way I described above, by allowing consumers to be burdened, but making the burden optional such that it is limited to a transition phase. Those who write producers in PIC assembly can then choose to take the hit of having to upgrade their consumers, which is clearly the cheaper option for them. Regards, -- Aristotle Pagaltzis // From publiustemp-tapx at yahoo.com Tue Apr 15 10:42:38 2008 From: publiustemp-tapx at yahoo.com (Ovid) Date: Tue, 15 Apr 2008 02:42:38 -0700 (PDT) Subject: [tap-l] Nested TAP In-Reply-To: <100071.3634.qm@web65702.mail.ac4.yahoo.com> Message-ID: <28143.58914.qm@web65711.mail.ac4.yahoo.com> Before we all jump on the bandwagon and agree to break backwards compatibility, could we at least *attempt* to address this: > Perhaps we should step back and ask ourselves what problems we're > trying to solve with nested TAP and ask if there are other > approaches? We might be drawing this out for no good reason. I'd rather give up nested TAP than break backwards compatibility, though I realize that, aside from myself, consensus for breaking backwards compatibility seems unanimous. Cheers, Ovid -- Buy the book - http://www.oreilly.com/catalog/perlhks/ Perl and CGI - http://users.easystreet.com/ovid/cgi_course/ Personal blog - http://publius-ovidius.livejournal.com/ Tech blog - http://use.perl.org/~Ovid/journal/ From publiustemp-tapx at yahoo.com Tue Apr 15 11:41:47 2008 From: publiustemp-tapx at yahoo.com (Ovid) Date: Tue, 15 Apr 2008 03:41:47 -0700 (PDT) Subject: [tap-l] Nested TAP In-Reply-To: <006FE1D3-FD60-4AD8-90BE-873577E90F3C@hexten.net> Message-ID: <953246.73686.qm@web65701.mail.ac4.yahoo.com> --- Andy Armstrong wrote: > Your approach: generated TAP is automatically backwards compatible, > producer needs a minimal parser or additional logic to keep track of > summary results, producer doesn't need to know the TAP version > supported by consumer. Yes. > My approach: nested TAP isn't b/c, simpleminded producers (no > parser), > producer can downgrade iff it knows the TAP version supported by > consumer. There are plenty of ways that the producer might not know the consumer (in fact, it's beneficial if it doesn't know since this reduces coupling). For example, what about a consumer reading pregenerated TAP from a database, Web service, file, etc? There's no way the producer can/should know which consumers will read it in the future. > I like the idea of not breaking backwards compatibility /now/ - but I > don't like the idea of living with backwards compatibility baggage > forever. Agreed on both points. I'm willing to break backcompat when it's demonstrated to me that the wins outweigh the losses. Cheers, Ovid -- Buy the book - http://www.oreilly.com/catalog/perlhks/ Perl and CGI - http://users.easystreet.com/ovid/cgi_course/ Personal blog - http://publius-ovidius.livejournal.com/ Tech blog - http://use.perl.org/~Ovid/journal/ From mpeters at plusthree.com Tue Apr 15 16:20:56 2008 From: mpeters at plusthree.com (Michael Peters) Date: Tue, 15 Apr 2008 11:20:56 -0400 Subject: [tap-l] Nested TAP In-Reply-To: <20080415014406.GL11333@klangraum> References: <100071.3634.qm@web65702.mail.ac4.yahoo.com> <0E699440-299E-40A3-95FD-24E8C4C6DFB0@hexten.net> <394614.16379.qm@web65701.mail.ac4.yahoo.com> <006FE1D3-FD60-4AD8-90BE-873577E90F3C@hexten.net> <4803B8FD.1020705@plusthree.com> <233EDE4C-BC8F-44F6-8116-3A7DD42F725F@hexten.net> <394614.16379.qm@web65701.mail.ac4.yahoo.com> <20080415014406.GL11333@klangraum> Message-ID: <4804C7D8.3040005@plusthree.com> Aristotle Pagaltzis wrote: > Am I correct in thinking that you can translate this to a back- > compatible TAP stream merely by inserting an unindented > `not ok 2` line before the `begin 2` line? Yes, or it can appear after as a summary line. > If so, then it is inevitable that someone will have to parse the > nested TAP, and it?s merely a question of which end that will be. > > If that is the case, why are we trying to dictate the side on > which this should happen? > > I would argue that instead of pinning the duty on either end, we > should merely describe how TAP v13 can be downgraded to TAP v12 > and leave it up to individual users of TAP where they want to > shift that complexity. If push comes to shove, they can write an > intermediate pipeline step that takes v13 TAP and downgrades it > in the manner we have described in the spec before sending it to > their v12 consumer. > > In fact, we should probably make v13 allow both the presence and > absence of the back-compat shim line. That way, people can write > v13 producers now that work with v12 consumers now, but those who > have upgraded their entire infrastructure can shed that crutch. I'm starting to warm up to this idea. I'm still concerned with the baggage it adds to the protocol but as long as it's optional it will probably be ignored. Especially if we recommend that it only be used when integrating with earlier parsers. -- Michael Peters Plus Three, LP From ss5 at renormalist.net Tue Apr 15 19:52:38 2008 From: ss5 at renormalist.net (Steffen Schwigon) Date: Tue, 15 Apr 2008 20:52:38 +0200 Subject: [tap-l] Nested TAP In-Reply-To: <0E699440-299E-40A3-95FD-24E8C4C6DFB0@hexten.net> (Andy Armstrong's message of "Mon, 14 Apr 2008 22:10:31 +0100") References: <100071.3634.qm@web65702.mail.ac4.yahoo.com> <0E699440-299E-40A3-95FD-24E8C4C6DFB0@hexten.net> Message-ID: <87fxtndk7d.fsf@shirts.quite-simply.de> Andy Armstrong writes: > On 14 Apr 2008, at 21:23, Ovid wrote: >> By requiring a summary line, we *gain* the simplicity of always having >> the top level TAP stream parseable without forcing the consumer to >> introduce complications necessary to parse nested TAP. With my >> solution, determining if the top level core TAP stream passed or >> failed >> is trivial and *that* I think is what we should be focusing on. > > > OK, that's probably another axis along which we can disagree :) > > I favour keeping the overall complexity down - but to the extent that > we have features that are tricky to implement I'd like that complexity > to be in the consumer. In my view the utility of TAP is dependent more > on the ease of writing producers than consumers. Users of other > languages will derive more benefit from being able to write a producer > in their native language. I fully agree. Beeing so simple to produce is *the* killer feature of TAP. I can convince practically everyone to take part in a to-be-built infrastructure where he only has to produce TAP as simple as it is now from whatever strange language he might use. And convincing people to use something is IMHO a really important aspect. Steffen -- Steffen Schwigon Dresden Perl Mongers Deutscher Perl-Workshop From ss5 at renormalist.net Tue Apr 15 20:16:22 2008 From: ss5 at renormalist.net (Steffen Schwigon) Date: Tue, 15 Apr 2008 21:16:22 +0200 Subject: [tap-l] User Supplied YAML Diagnostic Keys: Descriptive Version In-Reply-To: <186174.73880.qm@web65702.mail.ac4.yahoo.com> (Ovid's message of "Mon, 14 Apr 2008 01:55:52 -0700 (PDT)") References: <186174.73880.qm@web65702.mail.ac4.yahoo.com> Message-ID: <877iezdj3t.fsf@shirts.quite-simply.de> Ovid writes: > --- Steffen Schwigon wrote: > >> Then maybe I haven't understood what the standardization of >> diagnostics is about. >> >> I thought it is primarily meant for the *user* of TAP to transport >> its own diagnostics of its own test runs and test results? >> >> I for instance would use it to transport benchmark results inside >> tests. > > And that's fine. You can extend it for any use you like. However, > because diagnostics are currently free-form, it's almost impossible to > do anything *programmatic* with them. So, for example, consider sample > output from Test::Differences: > > not ok 1 - lang is respected > # Failed test 'lang is respected' > # at lang.t line 23 > # +----+-----+----------+ > # | Elt|Got |Expected | > # +----+-----+----------+ > # * 0|uno |one * > # * 1|dos |two * > # +----+-----+----------+ > > Well, that seems fine, but imagine that you have test tool which wants > to automatically jump to the file and line number (or auto-navigate you > through a stack trace). We have to *guarantee* that this format never > changes. Ever worked with a test suite where a bunch of tests failed > because someone corrected the spelling in a 'die' message or > localization breaks your hard-coded string matching? By offering a > well-defined syntax, we can alleviate much of this sort of pain. The > above might well translate to this: > > --- > file: lang.t > line: 23 > results: > have: > - uno > - dos > want: > - one > - two > name: lang is respected > display: | > +----+-----+----------+ > | Elt|Got |Expected | > +----+-----+----------+ > * 0|uno |one * > * 1|dos |two * > +----+-----+----------+ > > Then all which really needs to be done is to ensure that your > particular TAP consumer knows what to do with it. In fact, it would be > trivial to simply transform that back to the unstructured diagnostics > we are familiar with. Thank you. I believe, I understand the example and the need to structure the diagnostics. Now, what, in your example, are the keys you want to standardize/reserve for tap specific purposes? IMHO all keys (file, line, results, have, want, name, display) are specific to Test::Differences. You probably don't want to standardize them, or do you? I think it's quite rare to have keys that are really specific to TAP (not to TAP producers). That's why I think the horror scenario of "tap-prefixes everywhere" doesn't really apply, therefore it's ok to have a prefix "tap-" as the "namespace" of such fields, instead of a huge overkill "all ascii characters belong to us". Steffen -- Steffen Schwigon Dresden Perl Mongers Deutscher Perl-Workshop From andy at hexten.net Tue Apr 15 20:17:40 2008 From: andy at hexten.net (Andy Armstrong) Date: Tue, 15 Apr 2008 20:17:40 +0100 Subject: [tap-l] Nested TAP In-Reply-To: <20080415014406.GL11333@klangraum> References: <100071.3634.qm@web65702.mail.ac4.yahoo.com> <0E699440-299E-40A3-95FD-24E8C4C6DFB0@hexten.net> <394614.16379.qm@web65701.mail.ac4.yahoo.com> <006FE1D3-FD60-4AD8-90BE-873577E90F3C@hexten.net> <4803B8FD.1020705@plusthree.com> <233EDE4C-BC8F-44F6-8116-3A7DD42F725F@hexten.net> <394614.16379.qm@web65701.mail.ac4.yahoo.com> <20080415014406.GL11333@klangraum> Message-ID: <1AAFCDA1-8B55-432F-8D7A-B6F755BEC22A@hexten.net> On 15 Apr 2008, at 02:44, Aristotle Pagaltzis wrote: > In fact, we should probably make v13 allow both the presence and > absence of the back-compat shim line. That way, people can write > v13 producers now that work with v12 consumers now, but those who > have upgraded their entire infrastructure can shed that crutch. I think that's a good compromise. So we'd have begin and end, tests inside the begin/end block optionally indented and an optional summary result after the end with the same number as the preceding nested block. That allows for 1..3 ok 1 begin 2 1..2 not ok 1 ok 2 end 2 ok 3 and 1..3 ok 1 begin 2 1..2 not ok 1 ok 2 end 2 ok 3 and 1..3 ok 1 begin 2 1..2 not ok 1 ok 2 end 2 not ok 2 ok 3 All are valid TAP v13. The third version is also compatible with v12 and earlier. I think the semantics for the summary assertion should work like this: === After a numbered nested TAP block a TAP producer may optionally emit a "summary result" with the same number as the block. The overall success of the block is (success of nested TAP AND success of summary result). A summary result can therefore cause an otherwise apparently successful nested block to fail. === And in fact that's quite tidy because we can say that a result is either a block, a simple result or both in which case the simple result must follow the block. A small point - that gives us no less than three sites where YAML diagnostics scoped for the entire nested block can be situated 1..3 ok 1 begin 2 TAP version 13 --- yaml: 'here' ... 1..2 not ok 1 ok 2 end 2 --- or: 'here' ... not ok 2 --- or: 'even here' ... ok 3 The first of those three is probably not a problem. I guess the parser will have an API that treats the interior of the block as a self contained TAP document via which the YAML diagnostic can be reached. The other two are resolved either by saying they can both appear or by ruling that only the third location is valid I think. I vote for the latter. -- Andy Armstrong, Hexten From mpeters at plusthree.com Tue Apr 15 23:32:39 2008 From: mpeters at plusthree.com (Michael Peters) Date: Tue, 15 Apr 2008 18:32:39 -0400 Subject: [tap-l] Nested TAP In-Reply-To: <1AAFCDA1-8B55-432F-8D7A-B6F755BEC22A@hexten.net> References: <100071.3634.qm@web65702.mail.ac4.yahoo.com> <0E699440-299E-40A3-95FD-24E8C4C6DFB0@hexten.net> <394614.16379.qm@web65701.mail.ac4.yahoo.com> <006FE1D3-FD60-4AD8-90BE-873577E90F3C@hexten.net> <4803B8FD.1020705@plusthree.com> <233EDE4C-BC8F-44F6-8116-3A7DD42F725F@hexten.net> <394614.16379.qm@web65701.mail.ac4.yahoo.com> <20080415014406.GL11333@klangraum> <1AAFCDA1-8B55-432F-8D7A-B6F755BEC22A@hexten.net> Message-ID: <48052D07.3000103@plusthree.com> Andy Armstrong wrote: > That allows for > > 1..3 > ok 1 > begin 2 > 1..2 > not ok 1 > ok 2 > end 2 > ok 3 > > and > > 1..3 > ok 1 > begin 2 > 1..2 > not ok 1 > ok 2 > end 2 > ok 3 > > and > > 1..3 > ok 1 > begin 2 > 1..2 > not ok 1 > ok 2 > end 2 > not ok 2 > ok 3 > > All are valid TAP v13. The third version is also compatible with v12 > and earlier. ++ > After a numbered nested TAP block a TAP producer may optionally emit a > "summary result" with the same number as the block. The overall > success of the block is (success of nested TAP AND success of summary > result). A summary result can therefore cause an otherwise apparently > successful nested block to fail. ++ > A small point - that gives us no less than three sites where YAML > diagnostics scoped for the entire nested block can be situated > > 1..3 > ok 1 > begin 2 > TAP version 13 > --- > yaml: 'here' > ... > 1..2 > not ok 1 > ok 2 > end 2 > --- > or: 'here' > ... > not ok 2 > --- > or: 'even here' > ... > ok 3 > > The first of those three is probably not a problem. I guess the parser > will have an API that treats the interior of the block as a self > contained TAP document via which the YAML diagnostic can be reached. > The other two are resolved either by saying they can both appear or by > ruling that only the third location is valid I think. I vote for the > latter. I vote for the 2nd. It means the optional summary line comes after everything associated with the nested TAP. So if you don't want to do it you just drop the last line. It's not embedded in the middle of something that's considered to all be part of the same "test". But my feelings on this matter aren't really strong. -- Michael Peters Plus Three, LP From andy at hexten.net Wed Apr 16 00:19:07 2008 From: andy at hexten.net (Andy Armstrong) Date: Wed, 16 Apr 2008 00:19:07 +0100 Subject: [tap-l] Nested TAP In-Reply-To: <48052D07.3000103@plusthree.com> References: <100071.3634.qm@web65702.mail.ac4.yahoo.com> <0E699440-299E-40A3-95FD-24E8C4C6DFB0@hexten.net> <394614.16379.qm@web65701.mail.ac4.yahoo.com> <006FE1D3-FD60-4AD8-90BE-873577E90F3C@hexten.net> <4803B8FD.1020705@plusthree.com> <233EDE4C-BC8F-44F6-8116-3A7DD42F725F@hexten.net> <394614.16379.qm@web65701.mail.ac4.yahoo.com> <20080415014406.GL11333@klangraum> <1AAFCDA1-8B55-432F-8D7A-B6F755BEC22A@hexten.net> <48052D07.3000103@plusthree.com> Message-ID: <2DE793A4-4977-467B-8C9B-64FC8028E747@hexten.net> On 15 Apr 2008, at 23:32, Michael Peters wrote: > I vote for the 2nd. It means the optional summary line comes after > everything > associated with the nested TAP. So if you don't want to do it you > just drop the > last line. It's not embedded in the middle of something that's > considered to all > be part of the same "test". But my feelings on this matter aren't > really strong. Neither are mine. Happy to go with whatever consensus may emerge :) -- Andy Armstrong, Hexten From pagaltzis at gmx.de Wed Apr 16 04:59:40 2008 From: pagaltzis at gmx.de (Aristotle Pagaltzis) Date: Wed, 16 Apr 2008 05:59:40 +0200 Subject: [tap-l] Nested TAP In-Reply-To: <1AAFCDA1-8B55-432F-8D7A-B6F755BEC22A@hexten.net> References: <100071.3634.qm@web65702.mail.ac4.yahoo.com> <0E699440-299E-40A3-95FD-24E8C4C6DFB0@hexten.net> <394614.16379.qm@web65701.mail.ac4.yahoo.com> <006FE1D3-FD60-4AD8-90BE-873577E90F3C@hexten.net> <4803B8FD.1020705@plusthree.com> <233EDE4C-BC8F-44F6-8116-3A7DD42F725F@hexten.net> <394614.16379.qm@web65701.mail.ac4.yahoo.com> <20080415014406.GL11333@klangraum> <1AAFCDA1-8B55-432F-8D7A-B6F755BEC22A@hexten.net> Message-ID: <20080416035940.GA2994@klangraum.plasmasturm.org> * Andy Armstrong [2008-04-15 21:20]: > I think the semantics for the summary assertion should work > like this: > > === > > After a numbered nested TAP block a TAP producer may optionally > emit a "summary result" with the same number as the block. The > overall success of the block is (success of nested TAP AND > success of summary result). A summary result can therefore > cause an otherwise apparently successful nested block to fail. > > === ++ > And in fact that's quite tidy because we can say that a result > is either a block, a simple result or both in which case the > simple result must follow the block. ++ > A small point - that gives us no less than three sites where > YAML diagnostics scoped for the entire nested block can be > situated: [?] The first of those three is probably not a > problem. I guess the parser will have an API that treats the > interior of the block as a self contained TAP document via > which the YAML diagnostic can be reached. The other two are > resolved either by saying they can both appear or by ruling > that only the third location is valid I think. I vote for the > latter. Diagnostics will be useful only in v13 consumers anyway, so I think there is merit at all in diagnostics trailing the optional simple result. However, outright forbidding such diagnostics will only complicate parsers for not much real gain. So I?d say that if, by implementation detail, it is possible for the consumer to make them available in some form, that?s fine. If not, it?s fine to drop them on the floor also. Interoperability for such diagnostics is therefore uncertain, which in RFC?2119 parlance means that such diagnostics SHOULD NOT be emitted. So people are free to put diagnostics there if they want, and if they are certain they have full control over the entire infrastructure then they can rely on those. However, they are strongly encouraged to stick to the one widely accepted location so it will be easy to write code to utilise their data. As for diagnostics trailing the block vs those that are part of the interior TAP stream, I agree that the trailing diagnostics should be made available as the test?s diagnostics in the outer TAP scope. That way, flipping a test between a single assertion and a nested block does not cause a sudden change in behaviour. Regards, -- Aristotle Pagaltzis // From publiustemp-perlqa3 at yahoo.com Wed Apr 16 09:05:13 2008 From: publiustemp-perlqa3 at yahoo.com (Ovid) Date: Wed, 16 Apr 2008 01:05:13 -0700 (PDT) Subject: [tap-l] User Supplied YAML Diagnostic Keys: Descriptive Version In-Reply-To: <877iezdj3t.fsf@shirts.quite-simply.de> Message-ID: <286979.34568.qm@web65707.mail.ac4.yahoo.com> --- Steffen Schwigon wrote: > Now, what, in your example, are the keys you want to > standardize/reserve for tap specific purposes? IMHO all keys (file, > line, results, have, want, name, display) are specific to > Test::Differences. You probably don't want to standardize them, or do > you? OK, I think I may have misunderstood something. None of those keys are specific to Test::Differences. Consider: is 3, 2, "say it ain't so"; That outputs something like: not ok 1 - say it ain't so # Failed test 'say it ain't so' # at t/foo.t line 23. # got: '3' # expected: '2' As you can see, we still have file, line, results, have, want, name and display, but in an unstructured form (display is almost useless here, but it can be quite useful in many other contexts). > I think it's quite rare to have keys that are really specific to TAP > (not to TAP producers). The specific keys mentioned above will likely be very common. It's already fairly straightforward to capture the file and line, but the rest is waiting on Test::Builder 2. Cheers, Ovid -- Buy the book - http://www.oreilly.com/catalog/perlhks/ Perl and CGI - http://users.easystreet.com/ovid/cgi_course/ Personal blog - http://publius-ovidius.livejournal.com/ Tech blog - http://use.perl.org/~Ovid/journal/ From andy at hexten.net Wed Apr 16 22:18:02 2008 From: andy at hexten.net (Andy Armstrong) Date: Wed, 16 Apr 2008 22:18:02 +0100 Subject: [tap-l] Pragmas Message-ID: <04CF9EFA-1DBC-4663-9A36-87792DFC1E41@hexten.net> In Oslo I completely forgot that I'd added pragmas to the version of TAP Test::Harness supports. Sorry about that. Right now the only supported pragma is strict. It works like this: TAP Version 13 1..2 ok 1 foo ok 2 is OK, but TAP Version 13 1..2 pragma +strict ok 1 foo ok 2 fails - because there's a TAP syntax error. Are we OK with this interface? Currently the only supported pragma is 'strict' but I've just committed a change to T::H that adds an ignore_exit option at various places (at the request of the Parrot folks) and it seems that it'd be useful to support pragma +ignore_exit so that a script that knew it couldn't control its exit status could tell the parser to ignore it. Do we like that? -- Andy Armstrong, Hexten From david at kineticode.com Thu Apr 17 06:57:21 2008 From: david at kineticode.com (David E. Wheeler) Date: Wed, 16 Apr 2008 22:57:21 -0700 Subject: [tap-l] User Supplied YAML Diagnostic Keys: Descriptive Version In-Reply-To: <200804131558.58808.chromatic@wgz.org> References: <902148.18615.qm@web65705.mail.ac4.yahoo.com> <200804131425.40398.chromatic@wgz.org> <48028209.5060809@pobox.com> <200804131558.58808.chromatic@wgz.org> Message-ID: <916F1164-26F0-499D-AB47-CF388A660F88@kineticode.com> On Apr 13, 2008, at 15:58, chromatic wrote: > The problem with an infinitely expandable protocol that tries to do > everything > is that it's infinitely expandable and tries to do everything. > Might be nice > to rein that in a little bit more. Or don't, and instead make it > trivial to > add ad-hoc keys willy-nilly without planning or communication or > consistency, > and you'll end up with the protocol equivalent of spaghetti. Anyone > care to > guess how many X-* headers there are in all of the SMTP clients and > servers > in the wild? How about HTTP headers? Maybe you don't have to care > about > them when they're in the spec, but at some point, someone has to write > software to produce and consume them. It would be nice if that > process > didn't completely suck. Hey chromatic, In principal I completely agree with you, chromatic (that is, I agree with the principal you espouse here; my agreement is not purely theoretical ;-)). But how does that work in practice? Specifically with regard to YAML diagnostic keys in TAP? What do you suggest? Maybe just reserving the keys we know we need now, and then adding more later as we need them, even though doing so might conflict with other folks using such keys? I just want to know how we might put what you've said into practice. Thanks, David From publiustemp-tapx at yahoo.com Thu Apr 17 09:24:48 2008 From: publiustemp-tapx at yahoo.com (Ovid) Date: Thu, 17 Apr 2008 01:24:48 -0700 (PDT) Subject: [tap-l] Pragmas In-Reply-To: <04CF9EFA-1DBC-4663-9A36-87792DFC1E41@hexten.net> Message-ID: <195262.56627.qm@web65706.mail.ac4.yahoo.com> --- Andy Armstrong wrote: > Are we OK with this interface? Currently the only supported pragma is > 'strict' but I've just committed a change to T::H that adds an > ignore_exit option at various places (at the request of the Parrot > folks) and it seems that it'd be useful to support > > pragma +ignore_exit > > so that a script that knew it couldn't control its exit status could > tell the parser to ignore it. > > Do we like that? I do like the syntax, but can you give a use case here? I could imagine the harness wanting to ignore exit codes if it cannot reliably determine them (reading TAP from an archive), but I'm unclear about the producer directing this. However, even the slightest use case is tempting. That being said, I'm not entirely sure I like that test success/failure is dependent on exit codes as that's "out of band" information. Cheers, Ovid -- Buy the book - http://www.oreilly.com/catalog/perlhks/ Perl and CGI - http://users.easystreet.com/ovid/cgi_course/ Personal blog - http://publius-ovidius.livejournal.com/ Tech blog - http://use.perl.org/~Ovid/journal/ From andy at hexten.net Thu Apr 17 11:10:49 2008 From: andy at hexten.net (Andy Armstrong) Date: Thu, 17 Apr 2008 11:10:49 +0100 Subject: [tap-l] Pragmas In-Reply-To: <195262.56627.qm@web65706.mail.ac4.yahoo.com> References: <195262.56627.qm@web65706.mail.ac4.yahoo.com> Message-ID: <715E7006-FED8-40A8-9A75-6E3B15CA9CE0@hexten.net> On 17 Apr 2008, at 09:24, Ovid wrote: > I do like the syntax, but can you give a use case here? I could > imagine the harness wanting to ignore exit codes if it cannot reliably > determine them (reading TAP from an archive), but I'm unclear about > the > producer directing this. However, even the slightest use case is > tempting. The specific use case was that prompted this was a set of tests in parrot/languages/perl6 that are munged through a pre-processor ('fudge') that Larry wrote that alters the test to return a true value to indicate that it's been munged. More generally one can conceive of programs that could be modified to emit TAP when they see a --test switch but whose exit value already has some other meaning which can't conveniently be overloaded. The advantage of the pragma is that such programs can still be self- contained testable entities; no need for a wrapper to discard the exit status. > That being said, I'm not entirely sure I like that test success/ > failure > is dependent on exit codes as that's "out of band" information. Yeah, agree. But that's probably a convention we're stuck with, no? -- Andy Armstrong, Hexten From chromatic at wgz.org Thu Apr 17 17:42:19 2008 From: chromatic at wgz.org (chromatic) Date: Thu, 17 Apr 2008 09:42:19 -0700 Subject: [tap-l] User Supplied YAML Diagnostic Keys: Descriptive Version In-Reply-To: <916F1164-26F0-499D-AB47-CF388A660F88@kineticode.com> References: <902148.18615.qm@web65705.mail.ac4.yahoo.com> <200804131558.58808.chromatic@wgz.org> <916F1164-26F0-499D-AB47-CF388A660F88@kineticode.com> Message-ID: <200804170942.19430.chromatic@wgz.org> On Wednesday 16 April 2008 22:57:21 David E. Wheeler wrote: > In principal I completely agree with you, chromatic (that is, I agree > with the principal you espouse here; my agreement is not purely > theoretical ;-)). But how does that work in practice? Specifically > with regard to YAML diagnostic keys in TAP? What do you suggest? Maybe > just reserving the keys we know we need now, and then adding more > later as we need them, even though doing so might conflict with other > folks using such keys? > > I just want to know how we might put what you've said into practice. That's my suggestion. Figure out the minimal set of keys that we expect to use in the near future and reserve those. Document them. Suggest that we might add more keys later, if there's a rough consensus and working code. Don't forbid people from adding their own keys, but recommend that they bring them up for public discussion. It's a combination of the once, twice, refactor rule and the IETF's "no standards without at least two implementations, and one of them public" rule. -- c From pagaltzis at gmx.de Thu Apr 17 18:05:09 2008 From: pagaltzis at gmx.de (Aristotle Pagaltzis) Date: Thu, 17 Apr 2008 19:05:09 +0200 Subject: [tap-l] User Supplied YAML Diagnostic Keys: Descriptive Version In-Reply-To: <200804170942.19430.chromatic@wgz.org> References: <902148.18615.qm@web65705.mail.ac4.yahoo.com> <200804131558.58808.chromatic@wgz.org> <916F1164-26F0-499D-AB47-CF388A660F88@kineticode.com> <200804170942.19430.chromatic@wgz.org> Message-ID: <20080417170509.GJ17290@klangraum.plasmasturm.org> * chromatic [2008-04-17 18:45]: > IETF's "no standards without at least two implementations, and > one of them public" rule That?s the W3C, actually. Regards, -- Aristotle Pagaltzis // From schwern at pobox.com Thu Apr 17 18:59:32 2008 From: schwern at pobox.com (Michael G Schwern) Date: Thu, 17 Apr 2008 10:59:32 -0700 Subject: [tap-l] Pragmas In-Reply-To: <195262.56627.qm@web65706.mail.ac4.yahoo.com> References: <195262.56627.qm@web65706.mail.ac4.yahoo.com> Message-ID: <48079004.5060804@pobox.com> Ovid wrote: >> 'strict' but I've just committed a change to T::H that adds an >> ignore_exit option at various places (at the request of the Parrot >> folks) and it seems that it'd be useful to support >> >> pragma +ignore_exit >> >> so that a script that knew it couldn't control its exit status could >> tell the parser to ignore it. >> >> Do we like that? > > I do like the syntax, but can you give a use case here? I could > imagine the harness wanting to ignore exit codes if it cannot reliably > determine them (reading TAP from an archive), but I'm unclear about the > producer directing this. However, even the slightest use case is > tempting. I'm sorta neutral on the whole idea, I haven't really thought it through or if it's a good idea or what the pragmas or syntax should be. strict TAP and ignoring the exit code do seem tempting. I do agree that the producer should be able to control pragmas as the test author knows best whether or not their test can produce strict TAP or if the exit code should be ignored. The user running the parser could also tell the parser to turn on pragmas, but that would be the user making assumptions about the tests. One thought... rather than just ignoring the exit code entirely, might it be better to instead redefine what is an ok exit? I'm just thinking that if, say, your test program passes with an exit of 1 that it might be handy to still guard against segfaults. This is particularly handy for segfaults on process cleanup, after all test results have been printed. > That being said, I'm not entirely sure I like that test success/failure > is dependent on exit codes as that's "out of band" information. Remember, we're working on bringing that into band. -- Defender of Lexical Encapsulation From schwern at pobox.com Thu Apr 17 23:16:53 2008 From: schwern at pobox.com (Michael G Schwern) Date: Thu, 17 Apr 2008 15:16:53 -0700 Subject: [tap-l] User Supplied YAML Diagnostic Keys: Descriptive Version In-Reply-To: <200804170942.19430.chromatic@wgz.org> References: <902148.18615.qm@web65705.mail.ac4.yahoo.com> <200804131558.58808.chromatic@wgz.org> <916F1164-26F0-499D-AB47-CF388A660F88@kineticode.com> <200804170942.19430.chromatic@wgz.org> Message-ID: <4807CC55.4070103@pobox.com> chromatic wrote: > On Wednesday 16 April 2008 22:57:21 David E. Wheeler wrote: > >> In principal I completely agree with you, chromatic (that is, I agree >> with the principal you espouse here; my agreement is not purely >> theoretical ;-)). But how does that work in practice? Specifically >> with regard to YAML diagnostic keys in TAP? What do you suggest? Maybe >> just reserving the keys we know we need now, and then adding more >> later as we need them, even though doing so might conflict with other >> folks using such keys? >> >> I just want to know how we might put what you've said into practice. > > That's my suggestion. Figure out the minimal set of keys that we expect to > use in the near future and reserve those. Document them. Suggest that we > might add more keys later, if there's a rough consensus and working code. > Don't forbid people from adding their own keys, but recommend that they bring > them up for public discussion. That's the plan. Official keys will be formalizations of what users do out in the real world. In most cases hopefully just a down-casing. We'd like folks to be able to add their own keys as they need without first wondering whether it might be useful for others or worrying if we might add a key of the same name, but different functionality, later. Thus the separation of local from official keys. > It's a combination of the once, twice, refactor rule and the IETF's "no > standards without at least two implementations, and one of them public" rule. I think we're in violent agreement. To make one thing clear, there's already a few dozen keys under consideration for official status. This issue covers not just test diagnostics but also test meta data such as host environment (hostname, ip, cpu, platform, versions...), access information (user, group, etc...), exit status, date/time, benchmarking information, etc... See http://testanything.org/wiki/index.php/TAP_meta_information for more info. Rather than rushing in and defining a bunch of keys that might turn out to be poorly thought out and unportable (for example, access information and environment), we can go ahead and define the dead obvious ones now, see how the rest play out and make the good ones official. -- Life is like a sewer - what you get out of it depends on what you put into it. - Tom Lehrer From chromatic at wgz.org Thu Apr 17 23:44:36 2008 From: chromatic at wgz.org (chromatic) Date: Thu, 17 Apr 2008 15:44:36 -0700 Subject: [tap-l] User Supplied YAML Diagnostic Keys: Descriptive Version In-Reply-To: <4807CC55.4070103@pobox.com> References: <902148.18615.qm@web65705.mail.ac4.yahoo.com> <200804170942.19430.chromatic@wgz.org> <4807CC55.4070103@pobox.com> Message-ID: <200804171544.36103.chromatic@wgz.org> On Thursday 17 April 2008 15:16:53 Michael G Schwern wrote: > chromatic wrote: > > That's my suggestion. Figure out the minimal set of keys that we expect > > to use in the near future and reserve those. Document them. Suggest > > that we might add more keys later, if there's a rough consensus and > > working code. Don't forbid people from adding their own keys, but > > recommend that they bring them up for public discussion. > That's the plan. Official keys will be formalizations of what users do out > in the real world. In most cases hopefully just a down-casing. That part I don't mind. > We'd like folks to be able to add their own keys as they need without first > wondering whether it might be useful for others or worrying if we might add > a key of the same name, but different functionality, later. Thus the > separation of local from official keys. This part I think will not work. You can't pave the cow paths if you never see the pasture, and you can't not break the DarkPAN because the only way to know if the DarkPAN even exists is to see if light bends in its vicinity. Worse, if you make the argument that name collisions are bad, because you can't determine programmatically the semantics of a key from the surrounding TAP document (and I don't mind this argument; it's a designing principle of Roles in Perl 6 for example) as the reason why TAP keys need a separate namespace from user-defined keys, you've just moved the problem somewhere else. To wit: what if I use multiple producers/consumers which produce false cognate keys? Solution one: don't try, at least for the general case. There's TAP keys and there's everything else, and if everything else doesn't play together nicely, them's the breaks. You get both pieces. Solution two: tag every key with an organizational prefix. This includes TAP keys. Any non-prefixed key is invalid. Can we enforce solution two? Not entirely; only as far as producers and consumers agree roughly on a version of the TAP spec. Further, we're pushing the namespacing problem in a different direction. Organizational names may conflict. However, they're less likely to have the false cognate problem. -- c From schwern at pobox.com Fri Apr 18 00:17:48 2008 From: schwern at pobox.com (Michael G Schwern) Date: Thu, 17 Apr 2008 16:17:48 -0700 Subject: [tap-l] User Supplied YAML Diagnostic Keys: Descriptive Version In-Reply-To: <200804171544.36103.chromatic@wgz.org> References: <902148.18615.qm@web65705.mail.ac4.yahoo.com> <200804170942.19430.chromatic@wgz.org> <4807CC55.4070103@pobox.com> <200804171544.36103.chromatic@wgz.org> Message-ID: <4807DA9C.9050802@pobox.com> chromatic wrote: >> We'd like folks to be able to add their own keys as they need without first >> wondering whether it might be useful for others or worrying if we might add >> a key of the same name, but different functionality, later. Thus the >> separation of local from official keys. > > This part I think will not work. You can't pave the cow paths if you never > see the pasture, and you can't not break the DarkPAN because the only way to > know if the DarkPAN even exists is to see if light bends in its vicinity. I don't understand, what will we break on the DarkPAN? The whole point of separating user from official keys is so we don't break the DarkPAN. > Worse, if you make the argument that name collisions are bad, because you > can't determine programmatically the semantics of a key from the surrounding > TAP document (and I don't mind this argument; it's a designing principle of > Roles in Perl 6 for example) as the reason why TAP keys need a separate > namespace from user-defined keys, you've just moved the problem somewhere > else. > > To wit: what if I use multiple producers/consumers which produce false cognate > keys? Again, I don't understand. > Solution one: don't try, at least for the general case. There's TAP keys and > there's everything else, and if everything else doesn't play together nicely, > them's the breaks. You get both pieces. > > Solution two: tag every key with an organizational prefix. This includes TAP > keys. Any non-prefixed key is invalid. > > Can we enforce solution two? Not entirely; only as far as producers and > consumers agree roughly on a version of the TAP spec. Further, we're pushing > the namespacing problem in a different direction. Organizational names may > conflict. However, they're less likely to have the false cognate problem. Now I really don't understand. Are you advocating key prefixes per organization like "tap.foo" and "xerox.foo" and, dare I delve into Java land, "com.google.foo"? -- 184. When operating a military vehicle I may *not* attempt something ?I saw in a cartoon?. -- The 213 Things Skippy Is No Longer Allowed To Do In The U.S. Army http://skippyslist.com/list/ From chromatic at wgz.org Fri Apr 18 00:29:19 2008 From: chromatic at wgz.org (chromatic) Date: Thu, 17 Apr 2008 16:29:19 -0700 Subject: [tap-l] User Supplied YAML Diagnostic Keys: Descriptive Version In-Reply-To: <4807DA9C.9050802@pobox.com> References: <902148.18615.qm@web65705.mail.ac4.yahoo.com> <200804171544.36103.chromatic@wgz.org> <4807DA9C.9050802@pobox.com> Message-ID: <200804171629.19717.chromatic@wgz.org> On Thursday 17 April 2008 16:17:48 Michael G Schwern wrote: > chromatic wrote: > >> We'd like folks to be able to add their own keys as they need without > >> first wondering whether it might be useful for others or worrying if we > >> might add a key of the same name, but different functionality, later. > >> Thus the separation of local from official keys. > > > > This part I think will not work. You can't pave the cow paths if you > > never see the pasture, and you can't not break the DarkPAN because the > > only way to know if the DarkPAN even exists is to see if light bends in > > its vicinity. > I don't understand, what will we break on the DarkPAN? The whole point of > separating user from official keys is so we don't break the DarkPAN. I've never thought there was an upgrade problem, but everyone else keeps saying that there's an upgrade problem and that there absolutely must be a separation between TAP keys and everything else because there's an upgrade problem and we don't want to break the DarkPAN. If we agree that that's not actually the real problem, good. > > Worse, if you make the argument that name collisions are bad, because you > > can't determine programmatically the semantics of a key from the > > surrounding TAP document (and I don't mind this argument; it's a > > designing principle of Roles in Perl 6 for example) as the reason why TAP > > keys need a separate namespace from user-defined keys, you've just moved > > the problem somewhere else. > > To wit: what if I use multiple producers/consumers which produce false > > cognate keys? > Again, I don't understand. Official Blessed TAP module: I use the keys TAPONLYFILENAME, TAPONLYLINENUMBER, TAPONLYEXPECTED, TAPONLYRECEIVED. Bob's unblessed TAP module: I use the key "time", which means the time of the test run. "Starttime" is just too long, and who wants to process an ISO-compliant datecode? Chuck's unblessed TAP module: I use the key "time", which means the duration of the test run. "Duration" is just too long. See the problem yet? chromatic: Gee, there's a collision between Bob's module and Chuck's module, which are both very very useful, especially when combined in the same program. If only TAP::ESP could disambiguate between the two. The problem is not upgrading. The problem has never been upgrading. The problem has always been name collisions. > > Solution one: don't try, at least for the general case. There's TAP keys > > and there's everything else, and if everything else doesn't play together > > nicely, them's the breaks. You get both pieces. > > > > Solution two: tag every key with an organizational prefix. This includes > > TAP keys. Any non-prefixed key is invalid. > > > > Can we enforce solution two? Not entirely; only as far as producers and > > consumers agree roughly on a version of the TAP spec. Further, we're > > pushing the namespacing problem in a different direction. Organizational > > names may conflict. However, they're less likely to have the false > > cognate problem. > > Now I really don't understand. Are you advocating key prefixes per > organization like "tap.foo" and "xerox.foo" and, dare I delve into Java > land, "com.google.foo"? If you remove the "Are you seriously suggesting we eat babies?" stink of the word "Java" there, yes I am -- if you want to solve the real problem of allowing arbitrary keys. -- c From schwern at pobox.com Fri Apr 18 01:56:25 2008 From: schwern at pobox.com (Michael G Schwern) Date: Thu, 17 Apr 2008 17:56:25 -0700 Subject: [tap-l] User Supplied YAML Diagnostic Keys: Descriptive Version In-Reply-To: <200804171629.19717.chromatic@wgz.org> References: <902148.18615.qm@web65705.mail.ac4.yahoo.com> <200804171544.36103.chromatic@wgz.org> <4807DA9C.9050802@pobox.com> <200804171629.19717.chromatic@wgz.org> Message-ID: <4807F1B9.1030909@pobox.com> chromatic wrote: > On Thursday 17 April 2008 16:17:48 Michael G Schwern wrote: > >> chromatic wrote: > >>>> We'd like folks to be able to add their own keys as they need without >>>> first wondering whether it might be useful for others or worrying if we >>>> might add a key of the same name, but different functionality, later. >>>> Thus the separation of local from official keys. >>> This part I think will not work. You can't pave the cow paths if you >>> never see the pasture, and you can't not break the DarkPAN because the >>> only way to know if the DarkPAN even exists is to see if light bends in >>> its vicinity. >> I don't understand, what will we break on the DarkPAN? The whole point of >> separating user from official keys is so we don't break the DarkPAN. > > I've never thought there was an upgrade problem, but everyone else keeps > saying that there's an upgrade problem and that there absolutely must be a > separation between TAP keys and everything else because there's an upgrade > problem and we don't want to break the DarkPAN. > > If we agree that that's not actually the real problem, good. I think I get it now, but I want to take it from the top anyway. We don't want a new, official TAP key to clash with something being used in the wild. Why? We're working around the same issue Perl 5 is having adding new keywords. In Perl 5, since keywords and user-defined subroutines share the same space, we can't add a new keyword without risking clashing with a user-defined subroutine and giving it new meaning. Thus the "use feature" shame. By reserving all the lower case keys for official TAP keys and leaving everything else for users, TAP is free to add new keys. The issue of name collisions is only an issue between official TAP keys and user-defined keys. Collisions between user-defined keys is actually beneficial. Lemme 'splain. Let's pretend that we make everyone prefix their user keys to avoid collision. Test::Stuff's "time" key might be "Test.Stuff-time" and Test::Thing's "time" key might be "Test.Thing-time". They mean slightly different things, maybe one is in Unix time and one is an ISO datetime, but they serve the same purpose. With the prefix, they can go on being slightly different. This is not what we want. We want convergence. If user defined keys are to have meaning for more than just the user or group that defined them, they must converge on a common meaning. User key collisions encourage that. Users and test producer authors can spot the collision and work it out. Lists of common keys and meanings can be published. TAP displayers can make use of this and start to do things with common, user defined keys. As the wide-spread utility of a key becomes apparent, they can be lower cased and made official. It's much like tagging, with most of the chaotic problems and benefits, except that there will be a growing set of well-defined, official keys to provide some stability. However, if collision is your main concern and what you really want is unique keys, you're free to prefix yours. Prefixing is undesirable for other reasons... Human readability of the raw diagnostics is important. If you prefix every key it makes all the keys output by a given producer look the same. This emphasizes who output the key and deemphasizes the key itself. The important thing the reader of the diagnostics wants to know is why their test failed, not what output the diagnostics. It's also annoying to write. Though that's not nearly as important as diagnostics should usually be generated, not hand-written, for each test. Also, we'd have to define what a prefix should look like to avoid collisions amongst prefixes. Perl namespaces would seem obvious, but this is the Test Anything Protocol, and other languages have their own ideas about namespaces and might collide. So now it's "Perl.Test.Stuff-time" or maybe by author instead but what if two authors have the same name? The issue of avoiding conflicts amongst user keys continues and I'd rather just avoid it entirely. With readability problems, specification complications and inhibiting convergence, prefixing adds more problems than it solves. -- 29. The Irish MPs are not after ?Me frosted lucky charms?. -- The 213 Things Skippy Is No Longer Allowed To Do In The U.S. Army http://skippyslist.com/list/ From chromatic at wgz.org Fri Apr 18 02:16:57 2008 From: chromatic at wgz.org (chromatic) Date: Thu, 17 Apr 2008 18:16:57 -0700 Subject: [tap-l] User Supplied YAML Diagnostic Keys: Descriptive Version In-Reply-To: <4807F1B9.1030909@pobox.com> References: <902148.18615.qm@web65705.mail.ac4.yahoo.com> <200804171629.19717.chromatic@wgz.org> <4807F1B9.1030909@pobox.com> Message-ID: <200804171816.57320.chromatic@wgz.org> On Thursday 17 April 2008 17:56:25 Michael G Schwern wrote: > We're working around the same issue Perl 5 is having adding new keywords. > In Perl 5, since keywords and user-defined subroutines share the same > space, we can't add a new keyword without risking clashing with a > user-defined subroutine and giving it new meaning. Thus the "use feature" > shame. By reserving all the lower case keys for official TAP keys and > leaving everything else for users, TAP is free to add new keys. Perl 5 also solves the problems of Bob's user-defined subroutines having the same name as Chuck's user-defined subroutines. C doesn't. C++ does. PHP didn't until (I believe PHP 5.2). I'm sure you can guess which language feature this is. > We want convergence. If user defined keys are to have meaning for more > than just the user or group that defined them, they must converge on a > common meaning. User key collisions encourage that. Users and test > producer authors can spot the collision and work it out. Lists of common > keys and meanings can be published. TAP displayers can make use of this > and start to do things with common, user defined keys. > As the wide-spread utility of a key becomes apparent, they can be lower > cased and made official. > It's much like tagging, with most of the chaotic problems and benefits, > except that there will be a growing set of well-defined, official keys to > provide some stability. It didn't work for SMTP headers. It didn't work for HTTP headers. Given that SMTP and HTTP are somewhat more popular and widespread than TAP, what makes you think it will possibly work for TAP? > Human readability of the raw diagnostics is important. I don't find these big blobs of YAML particularly readable when compared to the freeform diagnostics we have now. That said, here's a refinement: TAP keys (presumably the most common keys) don't have prefixes. Other keys do. 80% solution to readability, and a kind of pressure to TAP producers to standardize their keys. > With readability problems, specification complications and inhibiting > convergence, prefixing adds more problems than it solves. In the absence of prefixing, what solves the problem of key collision between unrelated producers? Expecting users to debug these problems and convince two or more producers to present their case for convergence to the TAP working group in a timely fashion is only one thinly-veiled metaphor about banana plantations short of the canonical 20th century Latin American novel. lo real maravilloso, -- c From schwern at pobox.com Fri Apr 18 03:10:21 2008 From: schwern at pobox.com (Michael G Schwern) Date: Thu, 17 Apr 2008 19:10:21 -0700 Subject: [tap-l] User Supplied YAML Diagnostic Keys: Descriptive Version In-Reply-To: <200804171816.57320.chromatic@wgz.org> References: <902148.18615.qm@web65705.mail.ac4.yahoo.com> <200804171629.19717.chromatic@wgz.org> <4807F1B9.1030909@pobox.com> <200804171816.57320.chromatic@wgz.org> Message-ID: <4808030D.20900@pobox.com> Executive summary: User key collision is not a show stopper. chromatic wrote: > On Thursday 17 April 2008 17:56:25 Michael G Schwern wrote: > >> We're working around the same issue Perl 5 is having adding new keywords. >> In Perl 5, since keywords and user-defined subroutines share the same >> space, we can't add a new keyword without risking clashing with a >> user-defined subroutine and giving it new meaning. Thus the "use feature" >> shame. By reserving all the lower case keys for official TAP keys and >> leaving everything else for users, TAP is free to add new keys. > > Perl 5 also solves the problems of Bob's user-defined subroutines having the > same name as Chuck's user-defined subroutines. C doesn't. C++ does. PHP > didn't until (I believe PHP 5.2). I'm sure you can guess which language > feature this is. You're right, it's sensible for a programming language to separate user functions and variables written by different users because programming languages require very precise meaning. >> We want convergence. If user defined keys are to have meaning for more >> than just the user or group that defined them, they must converge on a >> common meaning. User key collisions encourage that. Users and test >> producer authors can spot the collision and work it out. Lists of common >> keys and meanings can be published. TAP displayers can make use of this >> and start to do things with common, user defined keys. > >> As the wide-spread utility of a key becomes apparent, they can be lower >> cased and made official. > >> It's much like tagging, with most of the chaotic problems and benefits, >> except that there will be a growing set of well-defined, official keys to >> provide some stability. > > It didn't work for SMTP headers. It didn't work for HTTP headers. Given that > SMTP and HTTP are somewhat more popular and widespread than TAP, what makes > you think it will possibly work for TAP? Sorry, I'm not terribly familiar with the history of SMTP and HTTP and I don't know what you're referring to. AFAIK HTTP has a well defined set of headers and I'm not aware of their defining any mechanism for user defined headers. I don't know what the analogy is. As for why it'll work with TAP, with a few exceptions (exit_status, or whatever we decide to call it, is currently the only one), diagnostic keys do not effect test parsing. It's not a show stopper. At worst, a displayer that has been customized to do something special with a user defined key might show some odd output. The tests still pass and fail as before. Prefixing brings an additional problem, how do you tell a TAP displayer to do anything with all those prefixed keys? Let's say Test::Stuff has Perl.Test.Stuff-time. So you program your displayer to do something with Perl.Test.Stuff-time. Now the Test::Thing author thinks that's a neat idea and adds Perl.Test.Thing-time. So you add in Perl.Test.Thing-time to your displayer to do the same thing as Perl.Test.Stuff-time. Test::Foo picks up on it, and you have to add another special case. Test::Bar adds it in, another special case in your displayer. Another module, add another to the list. You're either going to wind up with a big list of prefixes and keys, which is annoying work, or you're going to break down and match on /-time$/ and defeat the point of prefixing. >> Human readability of the raw diagnostics is important. > > I don't find these big blobs of YAML particularly readable when compared to > the freeform diagnostics we have now. The choice of YAML is a compromise between machine parsability, extensibility and human readability. Could be worse, could be XML. Fortunately the TAP displayer can make it look all rainbows and lollipops if you like, but it's important to ensure it remain eyeballable. > That said, here's a refinement: TAP keys (presumably the most common keys) > don't have prefixes. Other keys do. 80% solution to readability, and a kind > of pressure to TAP producers to standardize their keys. I'm unhappy with the readability consequences, still emphasizing the wrong thing: official vs not. It's also a pressure in the wrong direction. The pressure should be convergence, not standardization. The standard can pluck keys out of the wild on it's own. Also this still requires defining how to make a globally unique prefix that doesn't read like a UPC code. Finally it falls afoul of the multiplicity of prefixes making TAP displayer behavior customization involve a bunch of accounting mentioned above. >> With readability problems, specification complications and inhibiting >> convergence, prefixing adds more problems than it solves. > > In the absence of prefixing, what solves the problem of key collision between > unrelated producers? Expecting users to debug these problems and convince > two or more producers to present their case for convergence to the TAP > working group in a timely fashion is only one thinly-veiled metaphor about > banana plantations short of the canonical 20th century Latin American novel. User key collision is not a show stopper for the user. As mentioned above, at worst a customized TAP displayer shows some confused output for that key. So it's not urgent that it be worked out. The authors of the two colliding producers simply hash it out amongst themselves. There's no need to involve anyone else or get any sort of official blessing. Getting the key blessed is an orthogonal problem. -- 100. Claymore mines are not filled with yummy candy, and it is wrong to tell new soldiers that they are. -- The 213 Things Skippy Is No Longer Allowed To Do In The U.S. Army http://skippyslist.com/list/ From chromatic at wgz.org Fri Apr 18 03:34:44 2008 From: chromatic at wgz.org (chromatic) Date: Thu, 17 Apr 2008 19:34:44 -0700 Subject: [tap-l] User Supplied YAML Diagnostic Keys: Descriptive Version In-Reply-To: <4808030D.20900@pobox.com> References: <902148.18615.qm@web65705.mail.ac4.yahoo.com> <200804171816.57320.chromatic@wgz.org> <4808030D.20900@pobox.com> Message-ID: <200804171934.44381.chromatic@wgz.org> On Thursday 17 April 2008 19:10:21 Michael G Schwern wrote: > As for why it'll work with TAP, with a few exceptions (exit_status, or > whatever we decide to call it, is currently the only one), diagnostic keys > do not effect test parsing. It's not a show stopper. At worst, a > displayer that has been customized to do something special with a user > defined key might show some odd output. The tests still pass and fail as > before. Then why is there a problem with reserved keys at all? If TAP v15 adds a new reserved key, anyone who deliberately upgrades may need to modify both the producer and consumer to deal with the collision, if that person even cares. > Prefixing brings an additional problem, how do you tell a TAP displayer to > do anything with all those prefixed keys? Let's say Test::Stuff has > Perl.Test.Stuff-time. So you program your displayer to do something with > Perl.Test.Stuff-time. Now the Test::Thing author thinks that's a neat idea > and adds Perl.Test.Thing-time. So you add in Perl.Test.Thing-time to your > displayer to do the same thing as Perl.Test.Stuff-time. Test::Foo picks up > on it, and you have to add another special case. Test::Bar adds it in, > another special case in your displayer. Another module, add another to the > list. > > You're either going to wind up with a big list of prefixes and keys, which > is annoying work, or you're going to break down and match on /-time$/ and > defeat the point of prefixing. Every part of customizable diagnostics has this problem. > User key collision is not a show stopper for the user. As mentioned above, > at worst a customized TAP displayer shows some confused output for that > key. So it's not urgent that it be worked out. Then why is it urgent to bless a namespace or character set or subset of a character set? You're arguing that collisions don't matter; someone can just write code to fix it, if it matters to them. > The authors of the two colliding producers simply hash it out amongst > themselves. Or they don't, which they won't. Non-executive summary: customizable diagnostic keys aren't that useful, no one's using them, and collisions aren't a problem, so why even bother standardizing them? Here's the question I should have asked much earlier in this thread, before people who want these diagnostic keys started arguing both sides of the question: can anyone provide a real, actually encountered use case where collisions are a real, actually encountered problem, or is this just an attempt to predict the future badly based on what people think might be a problem? (Given that even unstructured diagnostics have never actually appeared in TAP documents before, my guess is "No, everyone's arguing out of ignorance", which seems to me to be a poor way to write a specification, and probably at least half of the reason I keep getting different answers when I ask questions like "Is this useful" and "Is this actually a problem".) -- c From schwern at pobox.com Fri Apr 18 04:21:52 2008 From: schwern at pobox.com (Michael G Schwern) Date: Thu, 17 Apr 2008 20:21:52 -0700 Subject: [tap-l] User Supplied YAML Diagnostic Keys: Descriptive Version In-Reply-To: <200804171934.44381.chromatic@wgz.org> References: <902148.18615.qm@web65705.mail.ac4.yahoo.com> <200804171816.57320.chromatic@wgz.org> <4808030D.20900@pobox.com> <200804171934.44381.chromatic@wgz.org> Message-ID: <480813D0.5080602@pobox.com> chromatic wrote: > On Thursday 17 April 2008 19:10:21 Michael G Schwern wrote: > >> As for why it'll work with TAP, with a few exceptions (exit_status, or >> whatever we decide to call it, is currently the only one), diagnostic keys >> do not effect test parsing. It's not a show stopper. At worst, a >> displayer that has been customized to do something special with a user >> defined key might show some odd output. The tests still pass and fail as >> before. > > Then why is there a problem with reserved keys at all? If TAP v15 adds a new > reserved key, anyone who deliberately upgrades may need to modify both the > producer and consumer to deal with the collision, if that person even cares. I don't understand. There can be no collision. Official TAP keys all start with a lower case letter. User defined keys don't. A new, official key cannot collide with anything. Can you provide an example of your scenario? >> Prefixing brings an additional problem, how do you tell a TAP displayer to >> do anything with all those prefixed keys? Let's say Test::Stuff has >> Perl.Test.Stuff-time. So you program your displayer to do something with >> Perl.Test.Stuff-time. Now the Test::Thing author thinks that's a neat idea >> and adds Perl.Test.Thing-time. So you add in Perl.Test.Thing-time to your >> displayer to do the same thing as Perl.Test.Stuff-time. Test::Foo picks up >> on it, and you have to add another special case. Test::Bar adds it in, >> another special case in your displayer. Another module, add another to the >> list. >> >> You're either going to wind up with a big list of prefixes and keys, which >> is annoying work, or you're going to break down and match on /-time$/ and >> defeat the point of prefixing. > > Every part of customizable diagnostics has this problem. Sorry, I don't understand. Can you provide an example? >> User key collision is not a show stopper for the user. As mentioned above, >> at worst a customized TAP displayer shows some confused output for that >> key. So it's not urgent that it be worked out. > > Then why is it urgent to bless a namespace or character set or subset of a > character set? You're arguing that collisions don't matter; someone can just > write code to fix it, if it matters to them. There is a line being walked here. Your concerns about collision are valid. It's a question of how big the collision problem is vs how much the solution fucks up readability and extensibility. Separating official vs user keys solves a good chunk of the collision problem, it provides some stability and allows us to add new keys without worry. Upper vs lower case doesn't effect readability much and users still get a big wilderness of prime real estate to homestead. Big benefit, minimal drawback. Prefixing solves a much smaller problem, and adds problems, while screwing up readability. So it's not worth it. >> The authors of the two colliding producers simply hash it out amongst >> themselves. > > Or they don't, which they won't. > > Non-executive summary: customizable diagnostic keys aren't that useful, no > one's using them, and collisions aren't a problem, so why even bother > standardizing them? Nobody's using them because nothing supports them yet. Just because they're not a show stopper doesn't mean they're not useful. These diagnostics will be extremely useful, probably the biggest thing to happen to TAP since "ok". We have a decent pile of use cases already. Realize this solves the oft cried for problem of parsable failure diagnostics. There's a huge need for this. Remember how many times I've chastised people on perl-qa for trying to parse out Test::More diagnostics? > Here's the question I should have asked much earlier in this thread, before > people who want these diagnostic keys started arguing both sides of the > question: can anyone provide a real, actually encountered use case where > collisions are a real, actually encountered problem, or is this just an > attempt to predict the future badly based on what people think might be a > problem? Are you talking about collisions between official and user-defined keys? Or are you talking about collisions between different authors' user-defined keys? For the former, you're right. We have no information about this and we're trying to predict the future. However, it's a pretty safe assumption. We've seen the problem in Perl 5. We have some pretty clear use cases and some stability is desirable. The impact of the "lower case reserved" solution on readability and expression is minimal. The implementation and spec is simple. And if it turns out to be wrong, we can lift the restriction. As to the latter, we weren't arguing about that. > (Given that even unstructured diagnostics have never actually appeared in TAP > documents before, my guess is "No, everyone's arguing out of ignorance", > which seems to me to be a poor way to write a specification, and probably at > least half of the reason I keep getting different answers when I ask > questions like "Is this useful" and "Is this actually a problem".) I believe you misinterpreted an argument that was carried over from Oslo. We were still arguing using X- vs /^[a-z]/ to denote official keys which has been mostly settled. It was all clear to the folks who were in Oslo but probably not so much to everyone else. -- package Outer::Space; use Test::More tests => 9; From chromatic at wgz.org Fri Apr 18 06:44:21 2008 From: chromatic at wgz.org (chromatic) Date: Thu, 17 Apr 2008 22:44:21 -0700 Subject: [tap-l] User Supplied YAML Diagnostic Keys: Descriptive Version In-Reply-To: <480813D0.5080602@pobox.com> References: <902148.18615.qm@web65705.mail.ac4.yahoo.com> <200804171934.44381.chromatic@wgz.org> <480813D0.5080602@pobox.com> Message-ID: <200804172244.21961.chromatic@wgz.org> On Thursday 17 April 2008 20:21:52 Michael G Schwern wrote: > chromatic wrote: > > If TAP v15 adds a > > new reserved key, anyone who deliberately upgrades may need to modify > > both the producer and consumer to deal with the collision, if that person > > even cares. > I don't understand. There can be no collision. Official TAP keys all > start with a lower case letter. User defined keys don't. A new, official > key cannot collide with anything. > > Can you provide an example of your scenario? 1) Suppose that TAP discards this silly idea of namespacing or prefixing or reserving a character set subset for reserved keys 2) Suppose that a DarkPAN TAP producer/consumer pair uses a diagnostic key not reserved by TAP 3) Suppose that the next version of TAP uses that diagnostic key to mean something else Is this a problem? No. As you yourself point out, producer/consumer pairs have to upgrade in tandem to process diagnostic keys in a way that's semantically correct. A) If the user does not upgrade to other tools which produce or consume the new version of TAP, there's no problem. B) If the user performs the upgrade and upgrades the DarkPAN producer/consumer pair, there's no problem. C) If the user performs the upgrade and does not upgrade the producer/consumer pair, the only problem is that some diagnostic information may display incorrectly. Tests do not fail. Planes do not fall out of the sky. You are willing to live with this; you said as much in your previous message. Now replace #1. 1) Suppose that TAP retains the silly idea of namespacing/prefixing/character set reserving. A) If the user does not upgrade, there's no problem. B) If the user performs the upgrade and upgrades the DarkPAN producer/consumer pair, there's no problem. C) If the user performs the upgrade but does not upgrade the producer/consumer pair, the only problem is that some diagnostic information may display incorrectly (that is, not at all). Tests do not fail. Planes do not fall out of the sky. Thus the silly idea of namespacing/prefixing/character set reserving only changes the type of the trivial upgrading problem. Why? Because custom diagnostic keys produced by a custom producer need to be consumed by a custom consumer. There is a mutual dependency. > >> You're either going to wind up with a big list of prefixes and keys, > >> which is annoying work, or you're going to break down and match on > >> /-time$/ and defeat the point of prefixing. > > > > Every part of customizable diagnostics has this problem. > Sorry, I don't understand. Can you provide an example? If the default behavior of a consumer is to ignore unknown keys, then the consumer needs to know which keys it knows. You're either going to wind up with a big list of keys, which is annoying work, or you're going to... well okay, you're going to wind up with a big list of keys. The silly idea of namespacing/prefixing/character set reserving has no bearing on this. > Separating official vs user keys solves a good chunk of the collision > problem, It may solve the problem where a collision can occur between official keys and a single producer/consumer pair, but that's not actually a problem, and it does nothing to solve the problem of collisions between unofficial keys in multiple producers, which is much likelier to happen. Compare the number of TAP producing modules on the CPAN to the number of versions of TAP. There's an order of magnitude difference. Now imagine that half or even a quarter of them decide to use diagnostic keys. If you really need to solve a collision problem in TAP diagnostics, solve that one. The official/non-official key collision problem is a red herring. I don't know how to put this any more clearly, so I'm content to let this thread die here and watch TAP v15 careen off into crazy town. (Alternately, I could be the one careening off into crazy town, but at the risk of making an argument from authority, I *have* written three TAP producers still in use today.) -- c From schwern at pobox.com Fri Apr 18 07:48:07 2008 From: schwern at pobox.com (Michael G Schwern) Date: Thu, 17 Apr 2008 23:48:07 -0700 Subject: [tap-l] User Supplied YAML Diagnostic Keys: Descriptive Version In-Reply-To: <200804172244.21961.chromatic@wgz.org> References: <902148.18615.qm@web65705.mail.ac4.yahoo.com> <200804171934.44381.chromatic@wgz.org> <480813D0.5080602@pobox.com> <200804172244.21961.chromatic@wgz.org> Message-ID: <48084427.4080608@pobox.com> chromatic wrote: >>> If TAP v15 adds a >>> new reserved key, anyone who deliberately upgrades may need to modify >>> both the producer and consumer to deal with the collision, if that person >>> even cares. > >> I don't understand. There can be no collision. Official TAP keys all >> start with a lower case letter. User defined keys don't. A new, official >> key cannot collide with anything. >> >> Can you provide an example of your scenario? > > 1) Suppose that TAP discards this silly idea of namespacing or prefixing or > reserving a character set subset for reserved keys > > 2) Suppose that a DarkPAN TAP producer/consumer pair uses a diagnostic key not > reserved by TAP > > 3) Suppose that the next version of TAP uses that diagnostic key to mean > something else > > Is this a problem? No. As you yourself point out, producer/consumer pairs > have to upgrade in tandem to process diagnostic keys in a way that's > semantically correct. > > A) If the user does not upgrade to other tools which produce or consume the > new version of TAP, there's no problem. > > B) If the user performs the upgrade and upgrades the DarkPAN producer/consumer > pair, there's no problem. > > C) If the user performs the upgrade and does not upgrade the producer/consumer > pair, the only problem is that some diagnostic information may display > incorrectly. Tests do not fail. Planes do not fall out of the sky. You are > willing to live with this; you said as much in your previous message. The stability of user-defined keys is a concern. The incorrect/misleading display of diagnostic information is a problem. We want people to use them without having to worry about being blown over by an official key later on. We also do not want to require the producer and consumer to have to upgrade in lock-step. The question is how much readability and user extendability are we willing to sacrifice to solve it? This is a question of magnitude and balance, not a binary either/or. I am willing to sacrifice a small amount of readability and extensibility to solve 90% of the problem. Thus the official vs user distinction. Solving the user vs user clash sacrifices too much for too little (or no) gain. > Now replace #1. > > 1) Suppose that TAP retains the silly idea of namespacing/prefixing/character > set reserving. > > A) If the user does not upgrade, there's no problem. > > B) If the user performs the upgrade and upgrades the DarkPAN producer/consumer > pair, there's no problem. > > C) If the user performs the upgrade but does not upgrade the producer/consumer > pair, the only problem is that some diagnostic information may display > incorrectly (that is, not at all). Tests do not fail. Planes do not fall > out of the sky. > > Thus the silly idea of namespacing/prefixing/character set reserving only > changes the type of the trivial upgrading problem. > > Why? > > Because custom diagnostic keys produced by a custom producer need to be > consumed by a custom consumer. There is a mutual dependency. Ahh, there's the misunderstanding. We recommend that the displayer just show any unrecognized key/values rather than totally ignore them. So the user still sees the new information, just in a raw form. Sorry if this wasn't made clear. >>>> You're either going to wind up with a big list of prefixes and keys, >>>> which is annoying work, or you're going to break down and match on >>>> /-time$/ and defeat the point of prefixing. >>> Every part of customizable diagnostics has this problem. >> Sorry, I don't understand. Can you provide an example? > > If the default behavior of a consumer is to ignore unknown keys, then the > consumer needs to know which keys it knows. You're either going to wind up > with a big list of keys, which is annoying work, or you're going to... well > okay, you're going to wind up with a big list of keys. > > The silly idea of namespacing/prefixing/character set reserving has no bearing > on this. Since all unknown key/values should be displayed by default, there's no need to specify a key in the displayer unless you're doing something special with it. The important distinction is that in the prefixed scheme, if you have a behavior for a user-defined key you have to list out each prefixed-key which triggers the same behavior. 'Perl.Test.Stuff-time', 'Perl.Test.Things-time', 'Perl.Test.Wibble-time', and so on... would all have to be listed to trigger the same custom behavior and added to each time a new producer starts using that key. With the unprefixed scheme, you just have a one-to-one map of a behavior for each key. Once you determine what "Time" should do you're done. >> Separating official vs user keys solves a good chunk of the collision >> problem, > > It may solve the problem where a collision can occur between official keys and > a single producer/consumer pair, but that's not actually a problem, and it > does nothing to solve the problem of collisions between unofficial keys in > multiple producers, which is much likelier to happen. > > Compare the number of TAP producing modules on the CPAN to the number of > versions of TAP. There's an order of magnitude difference. Now imagine that > half or even a quarter of them decide to use diagnostic keys. > > If you really need to solve a collision problem in TAP diagnostics, solve that > one. The official/non-official key collision problem is a red herring. > > I don't know how to put this any more clearly, so I'm content to let this > thread die here and watch TAP v15 careen off into crazy town. (Alternately, > I could be the one careening off into crazy town, but at the risk of making > an argument from authority, I *have* written three TAP producers still in use > today.) We'll see what happens. I think encouraging user key convergence is worth the risk. Fortunately, if user-vs-user clashes do become the problem users can begin prefixing on their own. We're not walling that possibility off. -- 3. Not allowed to threaten anyone with black magic. -- The 213 Things Skippy Is No Longer Allowed To Do In The U.S. Army http://skippyslist.com/list/ From publiustemp-tapx at yahoo.com Fri Apr 18 11:18:33 2008 From: publiustemp-tapx at yahoo.com (Ovid) Date: Fri, 18 Apr 2008 03:18:33 -0700 (PDT) Subject: [tap-l] User Supplied Ontologies In-Reply-To: <200804171816.57320.chromatic@wgz.org> Message-ID: <741026.20498.qm@web65706.mail.ac4.yahoo.com> First off, I am pretty much in agreement with chromatic here. What follows is why. --- chromatic wrote: > > Human readability of the raw diagnostics is important. > > I don't find these big blobs of YAML particularly readable when > compared to the freeform diagnostics we have now. They can be trivially rewritten to any form you want. Because they're YAML, you can do that. > That said, here's a refinement: TAP keys (presumably the most > common keys) don't have prefixes. Other keys do. 80% solution to > readability, and a kind of pressure to TAP producers to standardize > their keys. That satisfies my desire for an 'X-' prefix and limits the problem space to a manageable level, though from chromatic's arguments, I see that this is doesn't solve the 'global namespace' problem. We're effectively using YAML to define an unstructured ontology (and thus, not really an ontology) and without namespaces, meaning is not only ambiguous, but subject to serious bugs. Consider the words 'preservative' and 'preservatif'. They sound similar, but one is English and the other is French. Their meaning is radically different, as discovered by a friend of mine in a horribly embarrassing situation. Java attempted to deal with this with the 'com.foo.bar' syntax. It works reasonably well, though it's limited. Perl 6 has tried a different, more correct, yet heavyweight approach. RDF uses namespaces to unambiguously describe their ontologies. Prolog suffers due to ambiguity inherent in its freeform structure and that's what we're suggesting here? This is a well-known, well-researched problem and trying to be captain of the USS Make Stuff Up means throwing away a lot of research into a fairly well-understood area. I would suggest, like chromatic, that TAP standard keys have no prefix, but that we define a well-understood syntax for unambiguously describing prefixes for user-supplied keys. Sure, we can scrap this and try to keep things simple. Prolog is simple. We see how successful that's been. > In the absence of prefixing, what solves the problem of key collision > between unrelated producers? Nothing. Cheers, Ovid -- Buy the book - http://www.oreilly.com/catalog/perlhks/ Perl and CGI - http://users.easystreet.com/ovid/cgi_course/ Personal blog - http://publius-ovidius.livejournal.com/ Tech blog - http://use.perl.org/~Ovid/journal/ From publiustemp-perlqa3 at yahoo.com Fri Apr 18 11:42:19 2008 From: publiustemp-perlqa3 at yahoo.com (Ovid) Date: Fri, 18 Apr 2008 03:42:19 -0700 (PDT) Subject: [tap-l] User Supplied Ontologies In-Reply-To: <741026.20498.qm@web65706.mail.ac4.yahoo.com> Message-ID: <345136.2235.qm@web65701.mail.ac4.yahoo.com> --- Ovid wrote: > Java attempted to deal with this with the 'com.foo.bar' syntax. It > works reasonably well, though it's limited. Perl 6 has tried a > different, more correct, yet heavyweight approach. RDF uses > namespaces > to unambiguously describe their ontologies. Prolog suffers due to > ambiguity inherent in its freeform structure and that's what we're > suggesting here? Thinking about this more, consider this ugly compromise: --- file: t/resource.t line: 23 results: have: 3 want: { "foo":3 } tags: - api - database user: com.foo.bar: have-type: xml want-type: json com.example.com: have-type: html have-type: JSON ... If you want unambiguous user keys, they need to specify a namespace with an authority. Since an authority is far outside the scope of what we want, we still have the 'time as duration' and 'time as start time' ambiguity. Usually this is unlikely to be an issue since most test suites will be in-house, but as previously mentioned, we're talking about an ontology (even if we don't realize it) and the basic definition, taken from Wikipedia: An ontology is a representation of a set of concepts within a domain and the relationships between those concepts. It is used to reason about the properties of that domain, and may be used to define the domain. This is a well-researched area dealing the the *semantics* of things and since we're arguing over the semantics, it's worth consulting research in this field. Note that I'm not arguing *for* a particular approach here, so long as we acknowledge that our ad-hoc semantic specification completely ignores known problem areas and it's important to not sweep this under the rug as 'too hard'. Sometimes hard problems are hard. Cheers, Ovid -- Buy the book - http://www.oreilly.com/catalog/perlhks/ Perl and CGI - http://users.easystreet.com/ovid/cgi_course/ Personal blog - http://publius-ovidius.livejournal.com/ Tech blog - http://use.perl.org/~Ovid/journal/ From publiustemp-tapx at yahoo.com Fri Apr 18 12:15:21 2008 From: publiustemp-tapx at yahoo.com (Ovid) Date: Fri, 18 Apr 2008 04:15:21 -0700 (PDT) Subject: [tap-l] User Supplied YAML Diagnostic Keys: Descriptive Version In-Reply-To: <4808030D.20900@pobox.com> Message-ID: <361605.86614.qm@web65714.mail.ac4.yahoo.com> --- Michael G Schwern wrote: > As for why it'll work with TAP, with a few exceptions (exit_status, > or > whatever we decide to call it, is currently the only one), diagnostic > keys do > not effect test parsing. It's not a show stopper. At worst, a > displayer that > has been customized to do something special with a user defined key > might show > some odd output. The tests still pass and fail as before. Except that you're telling people they can build rich clients on keys they can't depend on. > You're either going to wind up with a big list of prefixes and keys, > which is > annoying work, or you're going to break down and match on /-time$/ > and defeat > the point of prefixing. 1. Having no prefixes whatsoever allows this problem. 2. Having ambiguous prefixes which nevertheless have a recommended structure mitigates this problem. 3. Having an authority for each prefix eliminates this problem. I recommend #2 as being the most bang for the buck. #3 is far beyond the scope of what we can usefully determine at this time. #1 is merely plugging your ears and singing 'la-la-la' over and over again :) Cheers, Ovid -- Buy the book - http://www.oreilly.com/catalog/perlhks/ Perl and CGI - http://users.easystreet.com/ovid/cgi_course/ Personal blog - http://publius-ovidius.livejournal.com/ Tech blog - http://use.perl.org/~Ovid/journal/ From publiustemp-tapx at yahoo.com Fri Apr 18 12:38:25 2008 From: publiustemp-tapx at yahoo.com (Ovid) Date: Fri, 18 Apr 2008 04:38:25 -0700 (PDT) Subject: [tap-l] User-Supplied Diagnostics Are *Important* In-Reply-To: <200804171934.44381.chromatic@wgz.org> Message-ID: <196589.49130.qm@web65716.mail.ac4.yahoo.com> --- chromatic wrote: > (Given that even unstructured diagnostics have never actually > appeared in TAP > documents before, my guess is "No, everyone's arguing out of > ignorance", This is not true at all. If you rephrase this slightly: Given that even unstructured diagnostics have never actually appeared in test documents before, my guess is "No, everyone's arguing out of ignorance", There's a lot of prior art in this area, just not in TAP. Perl's needs have been relatively primitive because Perl's tools have been relatively primitive. We've become comfortable eating gruel for breakfast because that's all we have. Take a look at TestNG, the ant XML test output or any other well-used non-TAP test information format and you can see that people can and do use this diagnostic information all the time. We really have no ability to offer a robust solution if we're offering Notepad and someone needs to output RTF. Further, we can learn from their mistakes. Consider this snippet of TestNG output (from http://testng.org/doc/documentation-main.html): java.lang.AssertionError ... Removed 22 stack frames By reading through TestNG, we can see that it has a heavy bias towards OO programming, something that doesn't necessarily fit what TAP is trying to accomplish. Further, look at their started-at, finished-at and duration-ms attributes. Change that duration-ms attribute to have the value 4483. Clearly this conflicts with the start and end times. What the hell does that mean? This duplication of information is something we can learn from and avoid. That being said, there's a lot of stuff in there that someone who uses TestNG would want, but won't necessarily be for all types of testing. User supplied diagnostics gives people the ability to have the information they need (and heck, even translate TAP into TestNG XML format). Lots of folks at the 2006 Google Test Automation Conference in London kept talking about the need for a way of exchanging test information between different systems, but current implementations were inadequate in large part because of the assumptions they make. TAP makes too few assumptions and TestNG appears to make too many. We can provide a common ground of making common assumptions and allowing users to provide the bits they need. Cheers, Ovid -- Buy the book - http://www.oreilly.com/catalog/perlhks/ Perl and CGI - http://users.easystreet.com/ovid/cgi_course/ Personal blog - http://publius-ovidius.livejournal.com/ Tech blog - http://use.perl.org/~Ovid/journal/ From scratchcomputing at gmail.com Fri Apr 18 16:51:07 2008 From: scratchcomputing at gmail.com (Eric Wilhelm) Date: Fri, 18 Apr 2008 08:51:07 -0700 Subject: [tap-l] User Supplied Ontologies In-Reply-To: <741026.20498.qm@web65706.mail.ac4.yahoo.com> References: <741026.20498.qm@web65706.mail.ac4.yahoo.com> Message-ID: <200804180851.08031.ewilhelm@cpan.org> # from Ovid # on Friday 18 April 2008 03:18: >> That said, here's a refinement: TAP keys (presumably the most >> common keys) don't have prefixes. ?Other keys do. ?80% solution to >> readability, and a kind of pressure to TAP producers to standardize >> their keys. > >That satisfies my desire for an 'X-' prefix and limits the problem >space to a manageable level, though from chromatic's arguments, I see >that this is doesn't solve the 'global namespace' problem. What if none of the official tap keys ever contained a '.'? Could we live with that? Users can then prefix their keys with whatever they want, including 'foo.' for project foo. Yes, a trailing '.' would work in that case (m/\./), but I don't think TAP needs to dictate readability -- perhaps it is enough to recommend a leading '.' in the case of "too lazy/don't care to think of a prefix". --Eric -- Introducing change is like pulling off a bandage: the pain is a memory almost as soon as you feel it. --Paul Graham --------------------------------------------------- http://scratchcomputing.com --------------------------------------------------- From publiustemp-tapx at yahoo.com Fri Apr 18 17:35:11 2008 From: publiustemp-tapx at yahoo.com (Ovid) Date: Fri, 18 Apr 2008 09:35:11 -0700 (PDT) Subject: [tap-l] User Supplied Ontologies In-Reply-To: <200804180851.08031.ewilhelm@cpan.org> Message-ID: <112923.53027.qm@web65707.mail.ac4.yahoo.com> --- Eric Wilhelm wrote: > # from Ovid > # on Friday 18 April 2008 03:18: > > >> That said, here's a refinement: TAP keys (presumably the most > >> common keys) don't have prefixes. Other keys do. 80% solution to > >> readability, and a kind of pressure to TAP producers to > standardize > >> their keys. > > > >That satisfies my desire for an 'X-' prefix and limits the problem > >space to a manageable level, though from chromatic's arguments, I > see > >that this is doesn't solve the 'global namespace' problem. > > What if none of the official tap keys ever contained a '.'? Could we > live with that? Users can then prefix their keys with whatever they > want, including 'foo.' for project foo. Yes, a trailing '.' would > work > in that case (m/\./), but I don't think TAP needs to dictate > readability -- perhaps it is enough to recommend a leading '.' in the > case of "too lazy/don't care to think of a prefix". I can live with this. It's easy to read (I think), and we can have a 'recommends' for namespaces/prefixes and outline the reasoning, but allow people to leave it off to keep Schwern happy ;) This can allow people to have their namespaces, not dictate a structure, and perhaps let things evolve to something which works. Any issues with this approach? Cheers, Ovid -- Buy the book - http://www.oreilly.com/catalog/perlhks/ Perl and CGI - http://users.easystreet.com/ovid/cgi_course/ Personal blog - http://publius-ovidius.livejournal.com/ Tech blog - http://use.perl.org/~Ovid/journal/ From david at kineticode.com Fri Apr 18 18:34:02 2008 From: david at kineticode.com (David E. Wheeler) Date: Fri, 18 Apr 2008 10:34:02 -0700 Subject: [tap-l] User Supplied YAML Diagnostic Keys: Descriptive Version In-Reply-To: <200804172244.21961.chromatic@wgz.org> References: <902148.18615.qm@web65705.mail.ac4.yahoo.com> <200804171934.44381.chromatic@wgz.org> <480813D0.5080602@pobox.com> <200804172244.21961.chromatic@wgz.org> Message-ID: <93905BEF-D807-4068-B420-93DCFC20E559@kineticode.com> On Apr 17, 2008, at 22:44, chromatic wrote: > I don't know how to put this any more clearly, so I'm content to let > this > thread die here and watch TAP v15 careen off into crazy town. > (Alternately, > I could be the one careening off into crazy town, but at the risk of > making > an argument from authority, I *have* written three TAP producers > still in use > today.) You've convinced me: there should be nothing to distinguish official from unofficial keys at all, until or unless it actually becomes an issue. Funny how this tends to be the opposite of the conclusion that Ovid draws from your argument. But there you go. ;-) Best, David From chromatic at wgz.org Fri Apr 18 18:50:28 2008 From: chromatic at wgz.org (chromatic) Date: Fri, 18 Apr 2008 10:50:28 -0700 Subject: [tap-l] User Supplied YAML Diagnostic Keys: Descriptive Version In-Reply-To: <93905BEF-D807-4068-B420-93DCFC20E559@kineticode.com> References: <902148.18615.qm@web65705.mail.ac4.yahoo.com> <200804172244.21961.chromatic@wgz.org> <93905BEF-D807-4068-B420-93DCFC20E559@kineticode.com> Message-ID: <200804181050.28203.chromatic@wgz.org> On Friday 18 April 2008 10:34:02 David E. Wheeler wrote: > You've convinced me: there should be nothing to distinguish official > from unofficial keys at all, until or unless it actually becomes an > issue. > > Funny how this tends to be the opposite of the conclusion that Ovid > draws from your argument. But there you go. ;-) My argument was complex: solve the real problem or don't solve it. The in between position is silly and won't make anyone happy. (However, the first person to suggest RDF triples gets a lecture from *all* parties.) Maybe whichever side you prefer depends on how much Gabriel Garc?a M?rquez you've read. Injecting equal parts culture and surly into the discussion since 1999, -- c From david at kineticode.com Fri Apr 18 19:15:24 2008 From: david at kineticode.com (David E. Wheeler) Date: Fri, 18 Apr 2008 11:15:24 -0700 Subject: [tap-l] User Supplied YAML Diagnostic Keys: Descriptive Version In-Reply-To: <200804181050.28203.chromatic@wgz.org> References: <902148.18615.qm@web65705.mail.ac4.yahoo.com> <200804172244.21961.chromatic@wgz.org> <93905BEF-D807-4068-B420-93DCFC20E559@kineticode.com> <200804181050.28203.chromatic@wgz.org> Message-ID: On Apr 18, 2008, at 10:50, chromatic wrote: > My argument was complex: solve the real problem or don't solve it. > The in > between position is silly and won't make anyone happy. (However, > the first > person to suggest RDF triples gets a lecture from *all* parties.) Yes. The choices, as I see them, are: 1. Do we start out conservative, and loosen things up as the cow paths indicate (Ovid's position)? 2. Do we start out liberal, allowing anything, and formalize whatever turns up in the cow paths (what I'm now suggesting)? 3. Or, do we go the middle road, have a relatively mild conservative position (reserve lowercase ASCII), and tight or loosen up as the cow paths indicate (Schwern's position)? I get that you're against 3. I'm against 1 (no chance for cow paths at all, in my view) but can live with 2 or 3. Best, David From chromatic at wgz.org Sat Apr 19 04:24:40 2008 From: chromatic at wgz.org (chromatic) Date: Fri, 18 Apr 2008 20:24:40 -0700 Subject: [tap-l] User Supplied Ontologies In-Reply-To: <0DA5B5B5-B240-40AA-9F82-E8A10AF7692E@chrisdolan.net> References: <345136.2235.qm@web65701.mail.ac4.yahoo.com> <0DA5B5B5-B240-40AA-9F82-E8A10AF7692E@chrisdolan.net> Message-ID: <200804182024.40236.chromatic@wgz.org> On Friday 18 April 2008 20:18:40 Chris Dolan wrote: > How can the above example occur? How do two different user tags get > applied to a single test result? In the Test::Exceptions vs. > Test::Deep examples mentioned earlier (IIRC) I can see how a single > TAP *stream* can have conflicting tags, but I can't see how different > tags get applied to a single test result. Nothing precludes it, but you're right -- it's unlikely to happen with the structure of the Perl 5, Perl 6, and Parrot TAP producers as it exists now. -- c From schwern at pobox.com Sat Apr 19 16:15:40 2008 From: schwern at pobox.com (Michael G Schwern) Date: Sat, 19 Apr 2008 08:15:40 -0700 Subject: [tap-l] User Supplied YAML Diagnostic Keys: Descriptive Version In-Reply-To: References: <902148.18615.qm@web65705.mail.ac4.yahoo.com> <200804172244.21961.chromatic@wgz.org> <93905BEF-D807-4068-B420-93DCFC20E559@kineticode.com> <200804181050.28203.chromatic@wgz.org> Message-ID: <480A0C9C.3020606@pobox.com> David E. Wheeler wrote: > On Apr 18, 2008, at 10:50, chromatic wrote: > >> My argument was complex: solve the real problem or don't solve it. >> The in >> between position is silly and won't make anyone happy. (However, the >> first >> person to suggest RDF triples gets a lecture from *all* parties.) > > Yes. The choices, as I see them, are: > > 1. Do we start out conservative, and loosen things up as the cow paths > indicate (Ovid's position)? > > 2. Do we start out liberal, allowing anything, and formalize whatever > turns up in the cow paths (what I'm now suggesting)? > > 3. Or, do we go the middle road, have a relatively mild conservative > position (reserve lowercase ASCII), and tight or loosen up as the cow > paths indicate (Schwern's position)? > > I get that you're against 3. I'm against 1 (no chance for cow paths at > all, in my view) but can live with 2 or 3. Good sum up. Before I give my own sum up I'd like to propose an alternative way to look at the problem. We all seem to mostly agree that "user vs user" is at least potentially a problem. I'm balking at the solution: prefixing. It wrecks readability, convergence and extensibility. I hate solving a problem by causing another one. The prefixing solution sucks, but it's all we have... and that's a bad place to be. Rather than arguing about a sucky solution, does anyone have another solution to offer? If a better solution can be found then the arguments go away. We now return you to your regularly scheduled frustrating argument. I agree about #1, too restrictive too early, no cow paths can form. #2 is problematic as it offers no stability but for the official keys, a set which will grow slowly. I agree with Ovid that stability is important. It potentially places us in the same position as Perl 5 is right now. Somebody at some point is going to argue that if we add a new key somebody in the wild might already be using it to do something else. At that point I will laugh and laugh and laugh and then cry. #3 is just #2 following an existing cow path. In short, we have a good idea that official vs user is going to be a problem. Is anyone arguing it won't? We have a simple, elegant solution to it that doesn't cause another problem. The cost/benefit ratio is excellent. So why wait? chromatic's argument is that unless we solve both A (official vs user) and B (user vs user) we shouldn't bother solving A at all. It's not very pragmatic. If A and B were coupled, if you couldn't solve A without solving B, then I'd understand. But that hasn't been demonstrated. If solving A blocked solving B, I'd understand that. But the proposed solution to B (prefixing) remains available if needed and it's not like this is the final version of TAP. In short, what is the harm of solving the understood "official vs user" problem while leaving the fuzzier "user vs user" for later when we have more information and experience? (Note, this is distinct from "user vs user will not be a problem"). -- 3. Not allowed to threaten anyone with black magic. -- The 213 Things Skippy Is No Longer Allowed To Do In The U.S. Army http://skippyslist.com/list/ From schwern at pobox.com Sat Apr 19 16:21:29 2008 From: schwern at pobox.com (Michael G Schwern) Date: Sat, 19 Apr 2008 08:21:29 -0700 Subject: [tap-l] User Supplied Ontologies In-Reply-To: <0DA5B5B5-B240-40AA-9F82-E8A10AF7692E@chrisdolan.net> References: <345136.2235.qm@web65701.mail.ac4.yahoo.com> <0DA5B5B5-B240-40AA-9F82-E8A10AF7692E@chrisdolan.net> Message-ID: <480A0DF9.2090207@pobox.com> Chris Dolan wrote: > I'm not on the tap-l list (why is this cross-posted to perl-qa???) We're trying to move discussion of TAP to a broader, non-perl audience, thus the non-perl TAP mailing list. Since most TAP discussion has been on perl-qa, and since many of the people interested in TAP are subscribed to perl-qa and not TAP, we're cross posting for the moment until people adjust their subscriptions. Otherwise folks on perl-qa would not be aware of these discussions at all. -- If at first you don't succeed--you fail. -- "Portal" demo From chris at chrisdolan.net Sat Apr 19 04:18:40 2008 From: chris at chrisdolan.net (Chris Dolan) Date: Fri, 18 Apr 2008 22:18:40 -0500 Subject: [tap-l] User Supplied Ontologies In-Reply-To: <345136.2235.qm@web65701.mail.ac4.yahoo.com> References: <345136.2235.qm@web65701.mail.ac4.yahoo.com> Message-ID: <0DA5B5B5-B240-40AA-9F82-E8A10AF7692E@chrisdolan.net> On Apr 18, 2008, at 5:42 AM, Ovid wrote: > Thinking about this more, consider this ugly compromise: > > --- > file: t/resource.t > line: 23 > results: > have: 3 > want: { "foo":3 } > tags: > - api > - database > user: > com.foo.bar: > have-type: xml > want-type: json > com.example.com: > have-type: html > have-type: JSON I'm not on the tap-l list (why is this cross-posted to perl-qa???) so I haven't been following this discussion too closely. But this one caught my eye. How can the above example occur? How do two different user tags get applied to a single test result? In the Test::Exceptions vs. Test::Deep examples mentioned earlier (IIRC) I can see how a single TAP *stream* can have conflicting tags, but I can't see how different tags get applied to a single test result. Even in TestNG (which I use extensively at work) where you can mix TestNG and JUnit assertions in the same test method, each single assertion is controlled by just one assertion class. Furthermore, in TestNG it's the harness that creates the stream. The individual test cases just succeed or throw AssertionError, so it's a much simpler case than the TAP approach where every case is documented. Chris From chris at chrisdolan.net Sat Apr 19 04:45:21 2008 From: chris at chrisdolan.net (Chris Dolan) Date: Fri, 18 Apr 2008 22:45:21 -0500 Subject: [tap-l] User Supplied Ontologies In-Reply-To: <200804182024.40236.chromatic@wgz.org> References: <345136.2235.qm@web65701.mail.ac4.yahoo.com> <0DA5B5B5-B240-40AA-9F82-E8A10AF7692E@chrisdolan.net> <200804182024.40236.chromatic@wgz.org> Message-ID: On Apr 18, 2008, at 10:24 PM, chromatic wrote: > On Friday 18 April 2008 20:18:40 Chris Dolan wrote: > >> How can the above example occur? How do two different user tags get >> applied to a single test result? In the Test::Exceptions vs. >> Test::Deep examples mentioned earlier (IIRC) I can see how a single >> TAP *stream* can have conflicting tags, but I can't see how different >> tags get applied to a single test result. > > Nothing precludes it, but you're right -- it's unlikely to happen > with the > structure of the Perl 5, Perl 6, and Parrot TAP producers as it > exists now. > > -- c Hmm, good point, I was thinking just in terms of Perl (this *was* posted to perl-qa after all). The cases I can think of are 1) a TAP transformer that edits a stream, adding its own tags and 2) a TAP aggregator that merges upstream results into single results (say, a de-nester). Chris From scratchcomputing at gmail.com Sat Apr 19 17:07:47 2008 From: scratchcomputing at gmail.com (Eric Wilhelm) Date: Sat, 19 Apr 2008 09:07:47 -0700 Subject: [tap-l] User Supplied YAML Diagnostic Keys: Descriptive Version In-Reply-To: <480A0C9C.3020606@pobox.com> References: <902148.18615.qm@web65705.mail.ac4.yahoo.com> <480A0C9C.3020606@pobox.com> Message-ID: <200804190907.47390.ewilhelm@cpan.org> # from Michael G Schwern # on Saturday 19 April 2008 08:15: >The prefixing solution sucks, but it's all we have... and that's a bad > place to be. ?Rather than arguing about a sucky solution, does anyone > have another solution to offer? I'm not sure what you mean by "prefixing"[1], or what sucks about m/\./ being a straightforward identifier for user keys. sloppy. .better_but_lazy .org.vectorsection.officially_registered .File-Fu.perl_domain That is, the recommendation is that user keys start with a '.', but the condition which guarantees that it will never clash is simply that $key =~ m/\./. The further suggestion is that to avoid conflicts, the keys start with a prefix which is registered or claimed in some way. Of course, I would want strict key checking to be off by default and enabled only by the 'strict' pragma. But conveniently: the pragma is declared by the tap stream (i.e. emitter.) [1] If prefixing means "one prefix to rule them all", or "every builtin starts with 'tap_'", or "X-" -- then yes, that sucks. I think alternatives which move the clash-prevention information out of the key text suck more though (then you get into some sort of namespace declaration, context, etc.) --Eric -- Hot dogs: just another condiment. --Heart-attack Man --------------------------------------------------- http://scratchcomputing.com --------------------------------------------------- From scratchcomputing at gmail.com Sat Apr 19 17:10:01 2008 From: scratchcomputing at gmail.com (Eric Wilhelm) Date: Sat, 19 Apr 2008 09:10:01 -0700 Subject: [tap-l] User Supplied YAML Diagnostic Keys: Descriptive Version In-Reply-To: <200804190907.47390.ewilhelm@cpan.org> References: <902148.18615.qm@web65705.mail.ac4.yahoo.com> <480A0C9C.3020606@pobox.com> <200804190907.47390.ewilhelm@cpan.org> Message-ID: <200804190910.01784.ewilhelm@cpan.org> # from Eric Wilhelm # on Saturday 19 April 2008 09:07: >Of course, I would want strict key checking to be off by default and >enabled only by the 'strict' pragma. ?But conveniently: ?the pragma is >declared by the tap stream (i.e. emitter.) And further: strictness must be automatically disabled when $tap_version > $parser_supports_version. This means unrecognized builtins (present in a later version of TAP) don't choke the parser. --Eric -- But you can never get 3n from n, ever, and if you think you can, please email me the stock ticker of your company so I can short it. --Joel Spolsky --------------------------------------------------- http://scratchcomputing.com --------------------------------------------------- From david at kineticode.com Sat Apr 19 17:17:57 2008 From: david at kineticode.com (David E. Wheeler) Date: Sat, 19 Apr 2008 09:17:57 -0700 Subject: [tap-l] User Supplied YAML Diagnostic Keys: Descriptive Version In-Reply-To: <480A0C9C.3020606@pobox.com> References: <902148.18615.qm@web65705.mail.ac4.yahoo.com> <200804172244.21961.chromatic@wgz.org> <93905BEF-D807-4068-B420-93DCFC20E559@kineticode.com> <200804181050.28203.chromatic@wgz.org> <480A0C9C.3020606@pobox.com> Message-ID: <2F4EA81A-B397-4BA1-B22A-4809DE56382B@kineticode.com> On Apr 19, 2008, at 08:15, Michael G Schwern wrote: > #3 is just #2 following an existing cow path. In short, we have a > good idea that official vs user is going to be a problem. Is anyone > arguing it won't? We have a simple, elegant solution to it that > doesn't cause another problem. The cost/benefit ratio is excellent. > So why wait So do that. Consensus? David From Smylers at stripey.com Sat Apr 19 17:24:46 2008 From: Smylers at stripey.com (Smylers) Date: Sat, 19 Apr 2008 17:24:46 +0100 Subject: [tap-l] User Supplied YAML Diagnostic Keys: Descriptive Version In-Reply-To: <480A0C9C.3020606@pobox.com> References: <902148.18615.qm@web65705.mail.ac4.yahoo.com> <200804172244.21961.chromatic@wgz.org> <93905BEF-D807-4068-B420-93DCFC20E559@kineticode.com> <200804181050.28203.chromatic@wgz.org> <480A0C9C.3020606@pobox.com> Message-ID: <20080419162446.GQ20100@stripey.com> Michael G Schwern writes: > David E. Wheeler wrote: > > > On Apr 18, 2008, at 10:50, chromatic wrote: > > > > > My argument was complex: solve the real problem or don't solve it. > > > The in between position is silly and won't make anyone happy. > > > > Yes. The choices, as I see them, are: > > > > 1. Do we start out conservative, and loosen things up as the cow > > paths indicate (Ovid's position)? An example of this would be saying: We promise that all future 'official' keys will start with a lower-case letter. It is prohibited for you to start an in-house key you create with a lower-case letter. > > 2. Do we start out liberal, allowing anything, and formalize > > whatever turns up in the cow paths (what I'm now suggesting)? An example of this could be saying: All current 'official' keys are lower-case (and start with a letter) and we expect future ones will be as well. If you choose to create an in-house key which starts with a lower-case letter then you risk a clash in future should we happen to introduce a key with that name. In practice, what's the difference between those two? The first is apparently stricter, but the prohibition can't actually be inforced: if a user chooses to ignore it then we can't send him to prison or anything; it's just that there's a (small) chance of his system later failing. Similarly if the future Tap maintainers bizarrely do create a key named in some other way, whether they are breaking a promise or a mere expectation doesn't make any difference: it isn't like users can sue over a broken promise. So since choice 1 can't be enforced, it seems to behave exactly like choice 2 anyway. > The prefixing solution sucks, but it's all we have... and that's a bad > place to be. Rather than arguing about a sucky solution, does anyone > have another solution to offer? In practice users can name keys however they want. Namespace clashes will be a bigger issue for some users than others. Issue advice on how to mitigate the problem and let users choose whether to follow it: It's possible that a key you pick will also be used by somebody else for a different purpose, which could cause problems if your systems later get used together. You can reduce the risk of name clashes by starting all your keys with a standard prefix or your organization or project name (or something else likely to be unique). Smylers From pagaltzis at gmx.de Sat Apr 26 21:40:08 2008 From: pagaltzis at gmx.de (Aristotle Pagaltzis) Date: Sat, 26 Apr 2008 22:40:08 +0200 Subject: [tap-l] Tap tap Message-ID: <20080426204008.GA16035@klangraum.plasmasturm.org> Is this thing on? (No pun intended.) Regards, -- Aristotle Pagaltzis // From andy at hexten.net Sat Apr 26 21:41:28 2008 From: andy at hexten.net (Andy Armstrong) Date: Sat, 26 Apr 2008 21:41:28 +0100 Subject: [tap-l] Tap tap In-Reply-To: <20080426204008.GA16035@klangraum.plasmasturm.org> References: <20080426204008.GA16035@klangraum.plasmasturm.org> Message-ID: On 26 Apr 2008, at 21:40, Aristotle Pagaltzis wrote: > Is this thing on? > > (No pun intended.) Yes. Don't faucet. -- Andy Armstrong, Hexten