Subject: Content at notes.ietf.org is not archival

From: Robert Sparks <[email-masked]>
Date: Wed, 15 Sep 2021 13:21:47 -0500

Be aware that any content at notes.ietf.org is intended to be ephemeral.

We expect the content to remain available as long as we are running the 
service, but there is no guarantee it will be preserved. At some point 
in the future, it may be replaced with another service, and the content 
will not be automatically extracted.

Anything that should be archived needs to be stored at an archival 
service. For instance, minutes should be captured in the datatracker as 
minutes documents. If it isn't obvious where to keep any given content, 
contact a relevant member of the leadership.

Please keep in mind that the content at notes.ietf.org may not be there 
in the future as you choose share URLs to it in other places.

Robert Sparks
Tools team chair

---

From: Eric Rescorla <[email-masked]>
Date: Wed, 15 Sep 2021 12:21:40 -0700

Hi Robert,

This is the kind of PSA that is often made and ignored and then people come
to rely on the existing state of affairs, making it hard to change, ISTM
that if you want to actually have it be easy to deprecate the data then you
probably want to make it actively unreliable now.

-Ekr

On Wed, Sep 15, 2021 at 11:22 AM Robert Sparks <[email-masked]> wrote:

> ___________________________________________________________
> Tools-discuss mailing list - [email-masked]
> This list is for discussion, not for action requests or bug reports.
> * Report datatracker and mailarchive bugs to: [email-masked]
> * Report tools.ietf.org bugs to: [email-masked]
> * Report all other bugs or issues to: [email-masked]
> List info (including how to Unsubscribe):
> https://www.ietf.org/mailman/listinfo/tools-discuss
>

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 16 Sep 2021 09:39:55 +0200

On 15. Sep 2021, at 21:21, Eric Rescorla <[email-masked]> wrote:
> 
> Hi Robert,
> 
> This is the kind of PSA that is often made and ignored and then people come to rely on the existing state of affairs, making it hard to change, ISTM that if you want to actually have it be easy to deprecate the data then you probably want to make it actively unreliable now.

I’m not sure there isn’t an emoticon missing from this message…

The message reminds me a bit of Windows UAC and the idea that users are responsible for the reliability of their operating systems.
(One main reason that this never can work is that users have work to do and are not in a mode to interrupt their work for arcane unrelated activities.)

Here, users are effectively asked to make manual backups of work that has been done in notes.ietf.org.
Many people already occasionally make such backups, stash them away in some unreliable way; others won’t and will start making queries of who has a backup and whether that is the most recent one.  (This already has happened.  Don’t ask.)

Instead of burdening everyone with their own unstructured backup mechanisms, maybe it does make sense to keep something going at the IETF side.

(Being able to underlay a git repository would be the best technical solution that still keeps most responsibility on the user side.
But I don’t want to talk mechanism here, just that it is not enough to issue another RFC 6919 "MUST (BUT WE KNOW YOU WON’T)”.)

Grüße, Carsten

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 16 Sep 2021 17:46:00 +1000

Or, just automatically delete notes 30 days after creation.

> On 16 Sep 2021, at 5:39 pm, Carsten Bormann <[email-masked]> wrote:
> 

--
Mark Nottingham   https://www.mnot.net/

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 16 Sep 2021 10:07:57 +0200

> On 16. Sep 2021, at 09:46, Mark Nottingham <[email-masked]> wrote:
> 

(Again, I don’t know whether this was a serious proposal.
I’ll treat is as such for the purpose of this message.)

Or, don’t offer the service at all?
(Probably better than deleting it 30 days after creation (!).)

Most of the people who have done work in the IETF know about the value of accessing random traces of that work from the past.
After a couple of decades, we mostly got rid of the 6-month ideology around Internet-Drafts; I wouldn’t want to repeat that painful experience.

Collecting notes in our own service (as opposed to hackmd.io or Google docs) gives us more control.
Another benefit of having a common service is that, after a while, everybody knows how to use it and we can integrate it into other tools.

Crippling such a service because we can’t figure out how to keep up some ideology strikes me as an expression of contempt for the work of the IETF’s contributors.
You don’t have to feel the same way, but it should be well understood that this is the signal that is being sent.

Grüße, Carsten

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 16 Sep 2021 18:31:27 +1000

It was a very serious proposal.

We already have a large number of places that people need to look for data. Adding another is not optimal, especially when the purpose of the tool is collaborative editing, NOT long-term reference.

> On 16 Sep 2021, at 6:07 pm, Carsten Bormann <[email-masked]> wrote:
> 
>

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 16 Sep 2021 10:34:02 +0200

> We already have a large number of places that people need to look for data. Adding another is not optimal, especially when the purpose of the tool is collaborative editing, NOT long-term reference.

Sure.  I just don’t know how crippling notes.ietf.org will help with Google docs, hackmd.io, and all these other places.  Actually, it will drive people back to those.

Grüße, Carsten

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 16 Sep 2021 18:53:41 +1000

> On 16 Sep 2021, at 6:34 pm, Carsten Bormann <[email-masked]> wrote:
> 

I have so many questions :)

Why is it an issue that people use those other services for scratch documents? 

If temporary documents aren't appropriate, what guarantees do you need regarding archival storage, reliability, availability, etc?

Why does the IETF have to run this service, instead of relying on a third party?

How do these now long-lived documents interact with RFCs, Internet-Drafts, Datatracker and meeting proceedings (the various sources of truth we currently have)? In particular, when someone references a document there and presumes it's IETF-sanctioned?

Who determines that it's worth the cost of providing such a long-lived service?

Cheers,

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 16 Sep 2021 11:02:39 +0200

> I have so many questions :)

… and that is completely OK, because different WGs have different ways to do work, so this all may not be obvious.

> Why is it an issue that people use those other services for scratch documents? 

It is not an issue; it is a missed opportunity.

> If temporary documents aren't appropriate, what guarantees do you need regarding archival storage, reliability, availability, etc?

Reasonable care.
The point is not that this needs to be protected against nuclear attacks, but that we keep some basic level of service that is not intentionally throwing away other people’s work.

> Why does the IETF have to run this service, instead of relying on a third party?

See above.
It doesn’t “have” to run this service; it just historically has done so and it turned out to be useful (much more useful after the switch to CodiMD/Hedgedoc).

> How do these now long-lived documents interact with RFCs, Internet-Drafts, Datatracker and meeting proceedings (the various sources of truth we currently have)? In particular, when someone referenc…

Then they are wrong.  But they already can do that with Internet-Drafts.
If that helps, add an obnoxious disclaimer at the top.

> Who determines that it's worth the cost of providing such a long-lived service?

I don’t know the cost, but it seemed to be limited.
(Etherpad was available for a long time, but was having availability problems.
Kudos to Robert for “creating” this problem in the first place by switching to a more reliable software base.)

Who determines the value?  The users.  I think we have a clear result there (maybe not in your quarters of the IETF).

Grüße, Carsten

---

From: Michael Richardson <[email-masked]>
Date: Thu, 16 Sep 2021 15:22:31 -0400

> (Again, I don’t know whether this was a serious proposal.  I’ll treat
    > is as such for the purpose of this message.)

Well, I'm serious.
I think 30 days might be way too long.  

WG and design team leaders should know to download minutes at the end of the
session.  It's not hard.
But, in order to train that, the contents needs to disappear within the day,
and there will be some crying.   Some WG session minutes will go away.  

    > Collecting notes in our own service (as opposed to hackmd.io or Google
    > docs) gives us more control.  Another benefit of having a common
    > service is that, after a while, everybody knows how to use it and we
    > can integrate it into other tools.

Agreed.

-- 
Michael Richardson <[email-masked]>, Sandelman Software Works
 -= IPv6 IoT consulting =-

---

From: Christian Huitema <[email-masked]>
Date: Thu, 16 Sep 2021 12:39:00 -0700

>      > can integrate it into other tools.

The IETF already archives mail messages and Internet drafts. How hard 
would it be to archive notes and minutes? After all, this texts are part 
of the standard making process, covered by the Note Well, etc.

-- Christian Huitema

---

From: Michael Richardson <[email-masked]>
Date: Thu, 16 Sep 2021 15:50:01 -0400

Christian Huitema <[email-masked]> wrote:
    > would it be to archive notes and minutes? After all, this texts are
    > part of the standard making process, covered by the Note Well, etc.

Sure, just upload them as minutes to the DT, and it gets archived already.

For now, WG chairs need to do that, which also means that they have an
opportunity to review the conent.

A significant annoyance is that we don't have a standard markdown across all our
content archives.

-- 
]               Never tell me the odds!                 | ipv6 mesh networks [ 
]   Michael Richardson, Sandelman Software Works        | network architect  [ 
]     [email-masked]  http://www.sandelman.ca/        |   ruby on rails    [

---

From: Larry Masinter <[email-masked]>
Date: Thu, 16 Sep 2021 12:58:33 -0700

> A significant annoyance is that we don't have a standard markdown across
all
> our content archives.

I'm not sure there's a standard way of signifying which flavor of markdown
was used, for that matter.

---

From: "Andrew G. Malis" <[email-masked]>
Date: Thu, 16 Sep 2021 16:02:33 -0400

Christian,

Completed minutes are archived in the Datatracker. These are draft
minutes that we're talking about here.

Cheers,
Andy

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 17 Sep 2021 09:09:26 +1200

>     > (Again, I don’t know whether this was a serious proposal. 
 I’ll treat
> and there will be some crying.   Some WG session minutes will go away.

One day is too short; back in the good old days, when most of us suffered
jet lag after each meeting, a week to capture the minutes and copy-edit
them was hardly enough. But I do agree, this is a major duty of WG chairs
or secretaries, and needs to be done promptly. 

I wouldn't call it notes.ietf.org. I'd call it temp.ietf.org. Or at least,
put everything in notes.ietf.org/temp/ and move it to notes.ietf.org/recycle/
after 30 days and then to /dev/null after another 30 days. Something like
that, anyway, depending on how it would integrate with HedgeDoc.

   Brian

---

From: Henk Birkholz <[email-masked]>
Date: Thu, 16 Sep 2021 23:14:34 +0200

Minutes are not the only notes taken at notes.ietf.org. I would stop 
using it, if things would simply disappear, obviously. How is this a 
good idea?

---

From: Tero Kivinen <[email-masked]>
Date: Fri, 17 Sep 2021 00:17:26 +0300

> I’ll treat is as such for the purpose of this message.)

I also think this automatic delete after 30 days would be good
improvement. Or actually I would say 30 days after last edit (and if
the last editor has datatracker account we could send email notifying
that the note will expire in 7 days or similar). 

In addition that we do need an easy way to transfer data from the
notes.ietf.org to the datatracker. I.e., a way I can go to the
datatracker for my WG, and say this note in the notes.ietf.org should
be imported to the datatracker with this title (i.e., interm meeting
notes, or IETF xxx minutes or whatever). 

> Most of the people who have done work in the IETF know about the
> value of accessing random traces of that work from the past. After a
> couple of decades, we mostly got rid of the 6-month ideology around
> Internet-Drafts; I wouldn’t want to repeat that painful experience.
> 
> Collecting notes in our own service (as opposed to hackmd.io or
> Google docs) gives us more control. Another benefit of having a
> common service is that, after a while, everybody knows how to use it
> and we can integrate it into other tools.
> 
> Crippling such a service because we can’t figure out how to keep up
> some ideology strikes me as an expression of contempt for the work
> of the IETF’s contributors. You don’t have to feel the same way, but
> it should be well understood that this is the signal that is being
> sent.

The problem with notes.ietf.org is not that we can't keep the data,
itis that we can't find the data. I have no idea what was the etherpad
url for my session 4 years ago. I can most likely find it from the
chair slides if I happened to copy it there, but otherwise I would
need to go and find it from the somewhere else.

If we force people to move pages out from notes.ietf.org after the
session finishes, they are archived in the location where we can
actually find them. They can be stored to the datatracker as minutes
etc, or they can be moved to the working group wiki or similar.

Storing data is useless if people can't find it, and
etherpad/notes.ietf.org are both in that category.
-- 
[email-masked]

---

From: Christian Huitema <[email-masked]>
Date: Thu, 16 Sep 2021 14:26:48 -0700

> need to go and find it from the somewhere else.

Of course, that creates a special problem for archival, standard 
processes, etc. What if someone writes something they should not, gets 
the point across in the WG, and then edits the note to remove the 
offensive text? What is supposed to be archived? Just the note, or the 
whole editing history?

-- Christian Huitema

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 17 Sep 2021 10:16:18 +1200

> using it, if things would simply disappear, obviously. How is this a 

Henk, I may well be old-fashioned, but when I want something to be kept,
I keep it: save it on my disk, and make sure it's in a directory that
gets backed up routinely.

Anybody who used geocities.com knows why.

(I'm biased on this question. In about 1993 I tried to persuade Tim
Berners-Lee to make an expiry date obligatory for every HTML page.
I still think he was wrong to refuse.)

It's also important to specify this from the start. We probably don't
want rough notes to become part of the public record of standards
activities, and subject to subpoena.

    Brian

---

From: "Joel M. Halpern" <[email-masked]>
Date: Thu, 16 Sep 2021 19:40:13 -0400

To amplify Michael's point and respond to a couple of others...

1) These are draft notes.  They are likely to be wrong.  If we archive 
them, we may be archiving information that is false to fact.  That is 
why chairs / WG Secretaries review minutes before posting, and then ask 
the WG to review them as well.   We should not expect the leadership to 
maintain two sets of accurate minutes.  So getting rid of the draft is good.

2) If folks want to use the tools for something else, great.  But that 
does not create an obligation on the IETF to make the tool behave 
differently.  If there is some other use case that is needed, either use 
some other tool, or convince the IETF that we need a tool to support the 
use case you have.

Yours,
Joel

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 17 Sep 2021 04:15:11 +0200

On 2021-09-17, at 01:40, Joel M. Halpern <[email-masked]> wrote:
> 
> 2) If folks want to use the tools for something else, great.  But that does not create an obligation on the IETF to make the tool behave differently.  If there is some other use case that is needed, either use some other tool, or convince the IETF that we need a tool to support the use case you have.

Joel,

You are falling prey to an availability bias here.

You have only ever been using that tool for minutes, so you think the tool needs to be changed to more closely follow some random thinking about minutes.
I don’t care that much whether the IETF destroys its draft minutes, because I do know to back up those drafts that I’ll likely need.

But leading this discussion in complete disregard for the other uses of this tool is exactly the kind of “I don’t understand this, so let’s break it” thinking that has recently become so characteristic of IETF process discussions.

Grüße, Carsten

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 17 Sep 2021 12:32:24 +1000

Carsten,

It has been an informal tool for some time. I (along with others, apparently) am concerned about it becoming a formal dependency by stealth, consuming resources and causing confusion.

If you have use cases for it, that's great -- bring them up and we can have a discussion about whether they should be supported (with the corresponding dedication of resources). You haven't done that, though - you seem to be assuming that because you've used an informal tool in a particular way in the past, that will automatically be supported on an ongoing basis.

In doing so, you've now also characterised others' positions in a disparaging way and expressed cynicism about the IETF process. I understand you may be frustrated, but I don't think doing so is constructive.

Cheers,

> On 17 Sep 2021, at 12:15 pm, Carsten Bormann <[email-masked]> wrote:
>

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 17 Sep 2021 04:47:56 +0200

> In doing so, you've now also characterised others' positions in a disparaging way and expressed cynicism about the IETF process. I understand you may be frustrated, but I don't think doing so is co…

Mark,

sometimes it is absolutely necessary to express how a discourse makes one feel.

Yes, this discussion is frustrating and unlikely to lead anywhere good, and I have to choose my battles.

I made my points, and I will shut up now.

Grüße, Carsten

---

From: tom petch <[email-masked]>
Date: Fri, 17 Sep 2021 09:47:38 +0100

> It has been an informal tool for some time. I (along with others, apparently) am concerned about it becoming a formal dependency by stealth, consuming resources and causing confusion.

Mark

That is the way the IETF works; I do not think that it is a good way but 
it is the way.  If I want to know what resources are available, as in 
supported, maintained, told when they will be withdrawn and such like, I 
know of nowhere to go.

I think this is the reason why a survey earlier this year showed great 
variation in how I-D are produced; there is nowhere to go to find out. 
I have been pointed at, or stumbled across, ways of getting work done in 
the IETF and get upset when they disappear with or without warning.

I had never heard of notes.ietf.org prior to this thread.  If I had, I 
might have found it useful and become dependent on it:-(  Go look at 
tools.ietf.org with the eyes of someone who has been told the way to do 
it in the IETF it to write an I-D.  First thing you see is RFC 
Dependency checker followed by Bibtex Citation Converter ...  We do a 
really bad job of encouraging people to write I-D.

Tom Petch

> .

---

From: Robert Sparks <[email-masked]>
Date: Fri, 17 Sep 2021 08:28:32 -0500

(Setting Reply-To once again - lets please reduce the crossposting and 
move the whole conversation onto tools-discuss@)

I'm going to take up part of Carsten's argument now, attempting to wear 
no hats.

I personally don't want a "delete n days after creation" type of policy. 
I could accept (and probably support) a "delete after n days of 
inactivity" policy where n was large - at least two meeting-cycles 
large, if not larger.

This tool is being used for other things than working group minuting.

There are teams that I am on that have used notes as places to develop 
things like tools implementation plans - the notes served a longer than 
a few (even 30) days purpose until the effort finished. I can see the 
same being useful for design teams, where the real artifacts that come 
out are emails to the list and drafts. The expectation, in my opinion, 
is that these should be expected to be there for awhile, but not _forever_.

What I don't think is ok is carrying the need to redirect these urls a 
few years from now when the technology under them has moved on. URLs 
into this service shouldn't be placed in RFCs, giving us the burden to 
preserve the content and its location forever.

Putting my tools team hat back on - hedgedoc does not (to my knowledge) 
support this autodelete concept at this time. We could make a feature 
request whether or not it ends up being something we decide to use. But 
should we decide to have such a delete policy, it would be some time 
before it is implemented.

RjS

---

From: John C Klensin <[email-masked]>
Date: Fri, 17 Sep 2021 09:51:06 -0400

Robert,

(Still copying ietf@ because I think these issues are important
to the whole community, not just those with the time and energy
to track tools-discuss)

I don't know the notes facility well enough to ask this question
precisely --I qualify as one of those who had barely heard of it
before these threads started -- but it seems to me that you've
made a strong case for per-thread timeouts/ delete limits.  Some
things (like minutes) should disappear quickly for reasons
others have given; others (like design team efforts) may need
"after inactivity" or other much longer timeouts.   So it sounds
as if any feature requests should allow for per-thread/
per-activity timeouts, not just a global setting or that we
should start distinguishing, RSN, among different types of notes.

FWIW, I note that, while I-Ds supposedly expire after six
months, things have evolved to the point that we still say that
but old ones are essentially treated as archival with no special
effort need to get to them.   That is an extreme example of the
bottom of the slippery slope I think others have been concerned
about.

best,
  john

--On Friday, September 17, 2021 08:28 -0500 Robert Sparks
<[email-masked]> wrote:

> (Setting Reply-To once again - lets please reduce the
> crossposting and move the whole conversation onto
> tools-discuss@)
> 
> I'm going to take up part of Carsten's argument now,
> attempting to wear no hats.
> 
> I personally don't want a "delete n days after creation" type
> of policy. I could accept (and probably support) a "delete
> after n days of inactivity" policy where n was large - at
> least two meeting-cycles large, if not larger.
> 
> There are teams that I am on that have used notes as places to
> develop things like tools implementation plans - the notes
> served a longer than a few (even 30) days purpose until the
> effort finished. I can see the same being useful for design
> teams, where the real artifacts that come out are emails to
> the list and drafts. The expectation, in my opinion, is that
> these should be expected to be there for awhile, but not
> _forever_.
> 
> What I don't think is ok is carrying the need to redirect
> these urls a few years from now when the technology under them
> has moved on. URLs into this service shouldn't be placed in
> RFCs, giving us the burden to preserve the content and its
> location forever.
> 
> Putting my tools team hat back on - hedgedoc does not (to my
> knowledge) support this autodelete concept at this time. We
> could make a feature request whether or not it ends up being
> something we decide to use. But should we decide to have such
> a delete policy, it would be some time before it is
> implemented.
> 
> RjS
> 
> 
>>> In doing so, you've now also characterised others' positions
>>> in a disparaging way and expressed cynicism about the IETF
>>> process. I understand you may be frustrated, but I don't
>>> think doing so is constructive.
>> Mark,
>> 
>> Yes, this discussion is frustrating and unlikely to lead
>> anywhere good, and I have to choose my battles.
>>

---

From: Lloyd W <[email-masked]>
Date: Sat, 18 Sep 2021 19:54:31 +1000

> On 16 Sep 2021, at 17:41, Carsten Bormann <[email-masked]> wrote:
> 

That's as crazy as expecting drivers to be responsible for the reliability of the cars they drive. It will never catch on.

https://en.wikipedia.org/wiki/European_Computer_Driving_Licence

---

From: Lloyd W <[email-masked]>
Date: Sun, 19 Sep 2021 15:31:12 +1000

> On 16 Sep 2021, at 18:32, Mark Nottingham <[email-masked]> wrote:
> 
> the purpose of the tool is collaborative editing, NOT long-term reference

I thought github was the tool that the IETF was supposed to use to do collaborative editing? See RFC8874.

Lloyd Wood
[email-masked]

wondering what the IETF mandated editor is.

---

From: David Noveck <[email-masked]>
Date: Sun, 19 Sep 2021 05:17:20 -0400

RFC 8874 is an informational document discussing wg policies for using
GitHub.  It does not say GitHub "is supposed to be" used for collaborative
editing, let alone "IS SUPPOSED TO BE". There is no suggestion that it is
the only way to do that.

Actually, the process described there is, while more collaborative than
previous approaches, not fully collaborative in that there is an editor who
is assumed to be the editor of the corresponding working group document.

---

From: Carsten Bormann <[email-masked]>
Date: Sun, 19 Sep 2021 17:03:55 +0200

Asynchronous collaboration: git. Usually in combination with a website like gitlab or GitHub. (This is what the GitHub discussion in the IETF was about.) 
Asynchronous means not everyone had to be present at the same time. 

Synchronous collaboration: more variety. Today often hackmd/hedgedoc. Traces of Google docs and such. 
Synchronous means the focus is on groups that meet and edit at the same time, but of course hedgedoc can be “abused” for asynchronous collaboration. Since sessions end and synchronous and asynchronous phases may alternate, a good git interface would be a highly desirable element of a tool for synchronous collaboration. 

The more successful of either of these employ markdown as a storage/interchange format for textual information (text is code). 

Oh, I’m talking about how my more successful students work, but the pattern may be transferable to work in the IETF (and has been).

Sent from mobile, sorry for terse

> On 19. Sep 2021, at 07:42, Lloyd W <[email-masked]> wrote:
> 
>