Subject: Re: Changes to the way we manage RFPs

From: Brian E Carpenter <[email-masked]>
Date: Wed, 26 Feb 2020 16:56:02 +1300

Hi Jay,

I fully understand the arguments for this change. However, I think it would be good to continue to send a *summary* of the CFP messages to ietf-announce, for community awareness. Ditto for the "Award of contract" messages.

If I see "RFP for IETF Cookie services" or "Award of contract for IETF Cookie services" and I care, then I can go and look for more.

Regards
   Brian

On 26-Feb-20 16:15, IETF Executive Director wrote:
> The IETF Administration LLC is changing the way we manage RFPs.  These changes are part of a larger overhaul that will see further changes come in future.  The changes being made today are:
> 
> 1. As of next week, we will be using a new RFP Announcement Mailing List [1] [email-masked] for all announcements relating to RFPs and stop using IETF-Announce.  This change is intended to help us reach a wider network of potential vendors, both directly through a lower volume, more focused list and indirectly through automated redistribution of issued RFPs.  
> 
> 2. We have a new formal process for managing RFPs, published on the RFPs page [2].
> 
> 3. The RFPs page [2] now includes links to the all of the contracts for RFPs issued by the LLC.  These have the financial informaton redacted and where possible, signatures also redacted.
> 
> 
> Planned changes include:
> 
> * Publishing vendor Conflict of Interest (COI) forms, once reviewed by the LLC Board.
> 
> * Publishing contract templates and non-negotiables (e.g. contract transparency) to allow vendors to study them before bidding.
> 
> * Engaging the community on the principles and processes we use for managing RFPs and what is and isn't appropriate for us to expect in our contracts.
> 
> * We will also investigate implementing some elements of the Open Contracting Data Standard [3]
> 
> 
> Please feel free to contact me directly if you have any questions or feedback.
> 
> [1] https://www.ietf.org/mailman/listinfo/rfp-announce
> [2] https://www.ietf.org/about/administration/rfps/
> [3] https://www.open-contracting.org/data-standard/
>

---

From: Bob Hinden <[email-masked]>
Date: Tue, 25 Feb 2020 21:01:08 -0800

Jay,

> On Feb 25, 2020, at 7:15 PM, IETF Executive Director <[email-masked]> wrote:
> 

This sounds fine to me, I agree it isn’t reasonable to ask potential bidders to watch the ietf-announce list for RFPs.   That said, I think there is value to also cc: the ietf-announce list, doing that would improve transparency.    For example, I have no plans to subscribe to an rfp-announce list, but I do read the ietf-announce list.

Bob

> --
> Jay Daley
> IETF Executive Director
> [email-masked]
> _______________________________________________
> IETF-Announce mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ietf-announce

---

From: Jay Daley <[email-masked]>
Date: Wed, 26 Feb 2020 18:22:39 +1300

Brian 

(sent from my

> On 26/02/2020, at 4:56 PM, Brian E Carpenter <[email-masked]> wrote:
> 
> Hi Jay,
> 

Those are two of the three types of RFP message sent to IETF-announce and coincidentally two of the three types planned for rfp-announce (the third one being “Here’s the Q&A for this RFP”).  In other words, I don’t want rfp-announce to grow on volume. 

It would useful for me to understand why subscribing to rfp-announce is not an option?  In the context of the IETF’s high usage of mailing lists, this seems like a particularly low barrier to entry. 

Jay

-- 
Jay Daley
IETF Executive Director
+64 21 678840

---

From: Jay Daley <[email-masked]>
Date: Wed, 26 Feb 2020 19:09:50 +1300

Bob

> On 26/02/2020, at 6:01 PM, Bob Hinden <[email-masked]> wrote:
> 
> Jay,
> 

As I said to Brian, it would be useful for me to understand why subscribing to rfp-announce - did I just need to make it clearer that this is a zero sum change as far as email volume goes?

My concern btw is the lack of a clear principle that decides what gets copied to ietf-announce and why (assuming you don’t mean everything that goes to rfp-announce). 

Jay

---

From: Jay Daley <[email-masked]>
Date: Wed, 26 Feb 2020 19:17:10 +1300

I’m sorry, using mobile devices accurately is defeating me today  - I meant “useful for me to understand why subscribing to rfp-announce is not an option”

-- 
Jay Daley
IETF Executive Director

> On 26/02/2020, at 7:10 PM, Jay Daley <[email-masked]> wrote:
> 
> Bob
> 
>

---

From: John C Klensin <[email-masked]>
Date: Wed, 26 Feb 2020 02:18:49 -0500

Jay,

--On Wednesday, February 26, 2020 19:09 +1300 Jay Daley
<[email-masked]> wrote:

> As I said to Brian, it would be useful for me to understand
> why subscribing to rfp-announce - did I just need to make it
> clearer that this is a zero sum change as far as email volume
> goes?
> 
> My concern btw is the lack of a clear principle that decides
> what gets copied to ietf-announce and why (assuming you
> don't mean everything that goes to rfp-announce). 

My answer may differ from Bob's and Brian's, but have an
observation and possible reason that generalizes far beyond this
change.  There are at least two kinds of openness and
transparency.  One is to have information available to those who
know where to find it and take the time to go looking for it.
The other involving "pushing" the information and its
availability with the expectation that people may stumble across
information of interest, especially information about topics on
with they are not only interested but expert but would not have
thought to go looking for.   The most important example of this
applies to our standards process and one of the big advantages
of f2f meetings over remote participation: if someone is at a
f2f meeting and doesn't have a WG in which they are particularly
involved in a given slot, they may find something in that slot
about which they are mildly curious, wander in, and actually
learn something and/or contribute.  People participating from
their own desks are much more likely to find something else to
do.

It may seem like a stretch to get from there to announcements
about RFPs, but understanding what RFPs are out there and, more
important, how you and the IETF Administration LLC are breaking
things up into pieces may be more important to the community
than we generally realize.  It may be particularly important for
the community to monitor the tendencies of all bureaucracies to
divide work up into more pieces in ways that justify more
contracts, contractors, and administration and therefore more
staff and other resources.   I am _not_ suggesting that we have
that problem or that we would ever have it under your
administration, but it is a long-term, known risk.  Probably
more important in the near term, an unexpected notice of an RFP
might allow some member of the community with particular
expertise, one who normally has no interest in RPFs, to speak up
and say "hey, I know something about that and, while I have no
interest in bidding, I think you are structuring it all wrong
for the following reasons".  It would be unfortunate to lose
that type of expertise and opportunity for input even if it
comes infrequently.  It would also be unfortunate to lose the
possibility of someone spotting a particular RFP topic for which
they know someone with expertise who might not otherwise hear
about an opportunity and alerting that person even though they,
themselves, have no interest in ever bidding on an RFP. 

I worried about the same thing when we split the Last Call list
out: as IETF's work gets more specialized, if someone comes to
the IETF specifically to participate in a specific WG or two,
they may not feel a need to subscribe to the Last Call list
because notifications for Last Calls on that WG's work will be
clear from the WG list and discussions.  But, if they do not
subscribe to the Last Call list, we lose out on whatever topics
they might serendipitously be alerted to for which they have
expertise -- in other WGs, in other areas, and in individual
submissions.  
 
Those are really not arguments to avoid splitting the lists.
Instead, they suggest that, if you are going to do such a thing,
that you be aware of possible unintended side-effects and figure
out a way to mitigate them or even improve things because of
them.  Would we benefit from a monthly summary report from you,
one that summarizes or details outstanding RFPs, not just
plenary reports?  Do we need more explanatory material about why
those who subscribe to the IETF-announce list might want to
subscribe to the other lists too?  Should subscribing to
IETF-Announce automatically put one on the other lists on an
opt-out basis rather than requiring people to find them one at a
time and subscribe (that would protect people who only want to
see the RFP list from being bothered by the irrelevant-to-them
traffic on the IETF-Announce but would keep the information for
subscribers to the latter constant)?  As we continue to break
things out (I definitely see a trend) should we think of
IETF-Announce as a list of lists to which people can subscribe
(or opt out) selectively if they so choose but whose default is
"all announcements"?

And so on.  I don't know the answers to any of those questions,
but would hope that you (and we) would think about them.

best,
   john

---

From: Jay Daley <[email-masked]>
Date: Wed, 26 Feb 2020 20:54:01 +1300

John

> On 26/02/2020, at 8:19 PM, John C Klensin <[email-masked]> wrote:
> 
...

> (or opt out) selectively if they so choose but whose default is

Yes, this is the important issue to consider and a useful set of questions to help frame that. 

It’s also a wider problem than announcements/reports - I only realised last week that there was an active mtgvenue list, one of many “admin” lists, and also a tools-discuss list, one of many “devops(?)” lists. 

> but would hope that you (and we) would think about them.

Will do. I suspect the way forward requires a dose of creativity. 

Jay

-- 
Jay Daley
IETF Executive Director
+64 21 678840

---

From: tom petch <[email-masked]>
Date: Wed, 26 Feb 2020 10:04:48 +0000

> It would useful for me to understand why subscribing to rfp-announce is not an option?  In the context of the IETF’s high usage of mailing lists, this seems like a particularly low barrier to entry.

[tongue in cheek] Because it is impossible to subscribe to any IETF 
lists.  I send a message to subscribe, or unsubscribe or ... and the 
mailer generates a response which I never get.  My ESP - one of the 
larger in the world - has a policy of silently discarding what it 
regards as spam, the rules for which are company confidential. 
Empirically, I see that any short message, such as
'This is a test'
or
'Confirm'
is silently discarded whereas
'This has been padded with a load of semantic rubblis blss ball blaa to 
circumvent teh uneanted intrucion ao an errant ESP etc etc'
gets through.

Unless and until the IETF includes such information - how about a copy 
of Note Well? - in the confirmation message, then confirmation messages 
never arrive and so it is impossible to complete the three way handshake 
and to subscribe to any IETF mailing list.  Of course, when I say ESP, 
it just could be another organisation such as Cloudflare, that is making 
this impossible.  All I know  is that the confirmation message never 
arrives.

Tom Petch

---

From: Michael Richardson <[email-masked]>
Date: Wed, 26 Feb 2020 07:49:50 -0500

> 2. We have a new formal process for managing RFPs, published on the

Are you, or do you plan to, use any kind of workflow process for RFPs?
(Something that should be outsourced, there are many systems out there)
Will you engage in a digital contract signature system?

--
]               Never tell me the odds!                 | ipv6 mesh networks [
]   Michael Richardson, Sandelman Software Works        |    IoT architect   [
]     [email-masked]  http://www.sandelman.ca/        |   ruby on rails    [

--
Michael Richardson <[email-masked]>, Sandelman Software Works
 -= IPv6 IoT consulting =-

---

From: Michael Richardson <[email-masked]>
Date: Wed, 26 Feb 2020 07:58:56 -0500

Jay Daley <[email-masked]> wrote:
    > It would useful for me to understand why subscribing to rfp-announce is
    > not an option?  In the context of the IETF’s high usage of mailing
    > lists, this seems like a particularly low barrier to entry.

I think that Brian would like to see the awarding as part of a bigger IETF
LLC community oversight process.  Something that many people do in an
informal way, and while there are a half-dozen usual suspects, there are
probably another could dozen observers who aren't quite aware that they are
interested.   Some people just want to look out the kitchen window to see
that the kids are still in the yard.

The ietf-lastcall split has been good for ietf-announce volume, but I also
wonder if [email-masked] is the right place for followup and/or for community
engagement with the ietf-llc.   maybe there is another list needed. (maybe
there is already one I know not of)
I'm not sure here exactly: the IETF-LLC would post it's agenda and minutes to
such a list, but it would be open.

---

From: Michael Richardson <[email-masked]>
Date: Wed, 26 Feb 2020 08:08:13 -0500

Jay Daley <[email-masked]> wrote:
    > It’s also a wider problem than announcements/reports - I only realised
    > last week that there was an active mtgvenue list, one of many “admin”
    > lists, and also a tools-discuss list, one of many “devops(?)” lists.

Yes, so it might be that for the set of non-WG related lists (mtgvenue is a
WG though) that we should have a policy of periodically (1-2/year) of posting
something (even if it is just the terms of reference for the list) to ietf@
or ietf-announce@ with appropriate Reply-To:

---

From: Bob Hinden <[email-masked]>
Date: Wed, 26 Feb 2020 07:11:21 -0800

Jay,

> On Feb 25, 2020, at 10:09 PM, Jay Daley <[email-masked]> wrote:
> 
> Bob
> 
> 

I was assuming that the people who would subscribe to the rfp-announce list were people who might want to bid on IETF work.   I also assume the reason to create a separate list was for the people who were interested in bidding, but who didn’t care about other  things like what RFCs were published.

I am not planning to bid on any IETF RFPs.   I would still like to follow what is being put out to bid and what contracts were awarded.   I think copying the larger group would make the bidding process visible to a larger group of IETFers.

> My concern btw is the lack of a clear principle that decides what gets copied to ietf-announce and why (assuming you don’t mean everything that goes to rfp-announce).

My answer to that is things that are of general interest to the IETF should be sent to the IETF-announce list.   This includes RFP and contract awards.

Is there a reason to not copy these emails to ietf-announce?   I note it might result in more bids as they would reach a larger audience.

Hope this helps.

Bob

---

From: Jay Daley <[email-masked]>
Date: Thu, 27 Feb 2020 07:34:09 +1300

Michael

> On 27/02/2020, at 1:49 AM, Michael Richardson <[email-masked]> wrote:
> 
> 

I am looking at that but our process is sufficiently simple and our volume so low that we can comfortably manage it ourselves.  I’ve managed and participated in far more complex systems that had internally managed workflow.

Jay

-- 
Jay Daley
IETF Executive Director
[email-masked]
+64 21 678840

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Wed, 26 Feb 2020 12:54:25 -0600

Hi, Jay,

Greetings to another IETF mailing list explorer. May you live long and
prosper. :-)

Inline ...

On Wed, Feb 26, 2020 at 1:54 AM Jay Daley <[email-masked]> wrote:

> ....

I pointed out when the discussion about COVID-19 popped up on THIS very
mailing list, that the IESG had created a design team to think about a
massively-remote IETF meeting that became massively remote on short notice,
soon after a similar set of posts on this mailing list when we had the IETF
95 Buenos Aires meeting in 2016, amidst concerns about the Zika virus
(remember the Zika virus?). I believe Stephen Farrell suggested the name,
https://www.ietf.org/mailman/listinfo/manycouches, with this description:
"Manycouches -- List is a design team list to identify issues that would
arise should an IETF meeting ever be held with O(1000) 'remote'
participants."

That discussion (with smart people) rain until IETF 98, and I'd thought it
had converged, so wasn't subscribed when Wes and Ted restarted discussion
about a year ago, and that's been pretty active, including discussions
about COVID-19, ever since, but I didn't know it, and many of the points
being made on that mailing list were also being made here.

So, my point is that there are mailing lists that are having discussions of
interest to the community, that it's not easy to for the community to find
out about. And now, I might have told you about one more!

It happens that I'm subscribing to mailing lists at a couple of other SDOs.
One has a "here's a list of all the mailing lists you can subscribe to"
page, and the other one ... doesn't.

> Will do. I suspect the way forward requires a dose of creativity.

And communication! Darn it ...

Do the right thing, of course.

Spencer

Jay

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 27 Feb 2020 09:26:03 +1300

Jay,

My understanding has always been that every active IETF participant needs to subscribe
to ietf-announce because that (and only that) conveys what every participant needs
to know, and doesn't convey anything that is optional to know. Things like
- Last Calls
- WG charters for comment
- IESG decisions
- RFC publications
- Meeting announcements
- NomCom announcements
- "other announcements of interest to the IETF community"

All other mailing lists are optional.

I think that significant actions by IETF LLC belong there too. People need to
know about these even if it makes them yawn. John Klensin gave some reasons
why; my reason is one word: transparency.

Clearly, people who are potential bidders will be happy to join the dedicated list,
so the message to the ietf-announce list could be minimal.

Subject: RFP for IETF Cookie services
Body: IETF-LLC has published a Request for Proposals. Please see https://mailarchive.ietf.org/arch/browse/rfp-announce/ for details.

Regards
   Brian

---

From: John C Klensin <[email-masked]>
Date: Wed, 26 Feb 2020 15:34:08 -0500

--On Wednesday, February 26, 2020 12:54 -0600 Spencer Dawkins at
IETF <[email-masked]> wrote:

>...
> So, my point is that there are mailing lists that are having
> discussions of interest to the community, that it's not easy
> to for the community to find out about. And now, I might have
> told you about one more!
> 
> It happens that I'm subscribing to mailing lists at a couple
> of other SDOs. One has a "here's a list of all the mailing
> lists you can subscribe to" page, and the other one ...
> doesn't.

Noting that we have two lists that, combined, are all the
mailing lists one can subscribe to, it is clear (at least to me)
that, as we split historically-combined lists, we need to do
more than that.  Mixing, as
https://datatracker.ietf.org/list/nonwg does, an RFP
announcement list in with the "non-WG lists" collection (a
mixture of lists from closed WGs, lists proposing work,
meeting-specific lists, assorted IAB and IRTF lists, now
assorted "ag" lists, several lists that probably belonged to the
IAOC and that don't appear to have been moved over, some lists
that were created because someone (possibly a frequent
disrupter) asked for them and some AD gave in, a few lists that
are horribly outdated, and so on.  If one happens to know the
name of a particular list, that list may provide pointers to
information.  Or it may not: there are many lists for which the
"Description" is missing from that page, the page does not
distinguish among categories of lists (not even the ex-WG ones
are consistently identified in descriptions), and a large
fraction of the lists have "About" sections on their web pages
that are essentially information-free.  Adding another
complication, some are open, some aren't and there is rarely any
way to distinguish from that page.  I also found several dead
links as I sampled my way through that page today.

So, Jay, let me suggest a project for you or someone you
recruit.  Go through that "non-WG mailing lists" page and see if
it is possible to index it in compact and accessible form,
perhaps following the model of https://www.ietf.org/links/.  In
the process, see if you can identify lists, including the
several meeting planning ones, that should "belong to" the IETF
Administrative LLC and/or you as ExecDir, think about doing a
little winnowing if appropriate and clean up the descriptions
for the rest.  Think about putting an "IETF Administration LLC"
entry on that links page and listing LLC-related lists,
including those meeting ones, the RFP one, etc., under it.  In
the process, you might want to split "hopelessly outdated and of
interest only to historians" lists off from the main body of the
non-WG list and put them into a nice place of their own (for
example, I doubt anyone is likely, in their normal course of
IETF work, to consult "101all", which is probably good because
the other list of lists (https://www.ietf.org/mailman/listinfo )
claims there is no such list).

At least that should help clarify the problem of "maybe there is
a list for that, now how do I find it").   Our historical
tendency to use names for lists and WGs that are stronger on
cuteness than information obviously does not help.

>> > And so on.  I don't know the answers to any of those
>> > questions, but would hope that you (and we) would think
>> > about them.
>> 

Indeed.... to both communication and creativity.

> Do the right thing, of course.

Of course.

best,
   john

---

From: Michael Richardson <[email-masked]>
Date: Wed, 26 Feb 2020 15:39:26 -0500

Jay Daley <[email-masked]> wrote:
    >> (Something that should be outsourced, there are many systems out
    >> there) Will you engage in a digital contract signature system?

    > I am looking at that but our process is sufficiently simple and our
    > volume so low that we can comfortably manage it ourselves.  I’ve
    > managed and participated in far more complex systems that had
    > internally managed workflow.

If our volume is so low, then I'm not convinced we need the rfp-announce list :-)

---

From: "Salz, Rich" <[email-masked]>
Date: Wed, 26 Feb 2020 20:44:40 +0000

>    If our volume is so low, then I'm not convinced we need the rfp-announce list :-)
  
WE don't need it.  Potential bidders might find it more useful than wading through other stuff.

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 27 Feb 2020 09:44:52 +1300

It's *easy* to find the non-WG mailing lists:
https://datatracker.ietf.org/list/nonwg
There are only 518 of them, what's your problem?

Regards
   Brian Carpenter

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Wed, 26 Feb 2020 15:08:09 -0600

> There are only 518 of them, what's your problem?

Nothing that introducing a level of hierarchy for scaling wouldn't fix.
Like basically everything else in networking :-)

Spencer

---

From: Barry Leiba <[email-masked]>
Date: Wed, 26 Feb 2020 13:58:06 -0800

On Wed, Feb 26, 2020 at 4:59 AM Michael Richardson
<[email-masked]> wrote:
>
> The ietf-lastcall split has been good for ietf-announce volume, but

But here's the thing: That split should have not affected the volume
on ietf-announce *at all*.  It should have reduced volume on
<[email-masked]>, which is a very different thing.  This isn't the
message thread to discuss that further, but I mention it here to lead
into agreement with some other comments:

I believe that having a new "rfp-announce" list is a fine thing *if*
RFC announcements are posted to *both* ietf-announce and rfc-announce.
That should satisfy all the concerns here: those who subscribe to
ietf-announce need to change nothing and will still get RFP
announcements, and those who only want RFC announcements and not the
other stuff can subscribe only to rfp-announce -- it's an individual
choice.

Barry

---

From: Jay Daley <[email-masked]>
Date: Thu, 27 Feb 2020 11:02:32 +1300

Thanks Barry

> On 27/02/2020, at 10:58 AM, Barry Leiba <[email-masked]> wrote:
> 

In order to close this thread I will cc: ietf@ietf on all posts to rfp-announce, which is trivial to do.

Jay

-- 
Jay Daley
IETF Executive Director
[email-masked]
+64 21 678840

---

From: Barry Leiba <[email-masked]>
Date: Wed, 26 Feb 2020 14:09:12 -0800

My suggestion isn't to copy them to [email-masked], but, rather, to
copy them to [email-masked].

b

---

From: Jay Daley <[email-masked]>
Date: Thu, 27 Feb 2020 11:14:02 +1300

> On 27/02/2020, at 11:09 AM, Barry Leiba <[email-masked]> wrote:
> 

Sorry, that’s what I meant.

Jay

-- 
Jay Daley
IETF Executive Director
[email-masked]
+64 21 678840

---

From: Michael Richardson <[email-masked]>
Date: Wed, 26 Feb 2020 17:53:25 -0500

> But here's the thing: That split should have not affected the volume on
    > ietf-announce *at all*.  It should have reduced volume on

Yes, sorry, I mis-typed that.
The split has helped me manage the threads and volume better.

    > other stuff can subscribe only to rfp-announce -- it's an individual

I can live with this.