Subject: /.well-known (RFC5785) vs versioned APIs [and EST/RFC7030]

From: Michael Richardson <[email-masked]>
Date: Sun, 24 Feb 2019 12:54:42 -0500

I'm not sure where this query goes, so [email-masked].

Outside of the IETF there seems to be a good tradition of versioning RESTful
APIs using URLs like /api/v1.0.0 or such.

The OpenAPI specification is promoted by Swagger.IO.
(It does not appear to have been endorsed by any SDO-like entity)
These APIs are typically versioned like:
      /vendor/version/endpoint

The question is, how would one combine this into a /.well-known endpoint?
It seems that originally /.well-known was intended to be for things that
might be *combined* with other operator-controlled content.
Really good example is robots.txt, webfinger, acme-challenge.
/.well-known/core is about discoverying other things.

RFC7030 established /.well-known/est, and it is being extended by the ANIMA
WG in a number of documents.  It's not clear to me that this ever belonged
in /.well-known, as the service in question is very much dedicated to the
being a Registrar.

In particular, it seems a bit awkward to build versioned APIs that live
in /.well-known.   Not impossible, it just feels wrong.

Some references/readings:
  https://www.predic8.de/rest-api-versioning.htm
  https://www.baeldung.com/rest-versioning
  https://restfulapi.net/versioning/
  https://swagger.io/docs/specification/about/

--
Michael Richardson <[email-masked]>, Sandelman Software Works
 -= IPv6 IoT consulting =-

---

From: "Martin Thomson" <[email-masked]>
Date: Sun, 24 Feb 2019 15:12:19 -0500

Michael, 

My view on this, which- in the interest of transparency - is not uniformly held, is that versioning in URLs is generally not advisable for the same reason x- isn't. See rfc 6648.

I see /api/v1 all the time, but don't see the point. What that does is bake semantics into the path, which is akin to an rfc 7320 violation.

The goal is to ensure that incompatible versions are properly distinguished. For that purpose, all you need to do is ensure that the URL for v1 is not the same as the URL for v2. To that end, /api and /apiv2 work amply.  Or you could name the protocol.

For est,  I tend to think that /.well-known/ is overused in general, because configuring a URL is usually adequate.  But I can't hold a strong position on a protocol that I barely know. Adding /est2 might be an answer if backwards compatibility would be problematic.   It is also possible to use other ways to extend the protocol that are less violent. 

On Mon, Feb 25, 2019, at 04:55, Michael Richardson wrote:
> Attachments:
> * signature.asc

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Sun, 24 Feb 2019 23:29:10 -0500

> APIs using URLs like /api/v1.0.0 or such.

I think that the IETF already has the answer to the problem you are
raising. But it is spread out so that while the existing standards
essentially constrain the space, there is no RFC that puts them together
and so I wrote one myself:

https://tools.ietf.org/html/draft-hallambaker-web-service-discovery-00

This has expired but will be replaced in a few hours time. I am in fact
updating the whole document set right now.

The problem as I see it is how does a client:

* Discover an Internet service.
* Select between the Hosts offering that service.
* Obtain credentials for the host it is connecting to.

SRV is not quite enough. You also need to have information that describes
the services you are connecting to. This part is described in RFC6763.

https://tools.ietf.org/html/rfc6763

RFC6763 is the way more protocols negotiate connections than anything else
but when it was proposed, some folk thought that it was the wrong approach
so it got pushed into a corner and largely ignored in IETF.

There are a few issues I have with RFC6763, the main one being that it is
written as a set of options for protocol designers. I don't want there to
be a choice of discovery mechanisms. I want this to be an IETF standard
specifying EXACTLY ONE way to do things. That way we can have a DNS
resolution protocol of the form 'find a XYZ service on example.com which
supports versions 1.3-3.0 with ASN.2 encoding' and it can respond.

The other issue I have is a consequence of the first. I don't want every
protocol to specify a different set of TXT attributes describing the
service and a different way to manage them, I want these to be an IETF
standard common to as many protocols as possible. That allows me to use one
set of management tools to administer all my services on a machine.

Standards are good.

So lets take a DNS configuration for my Mathematical Mesh service:

   _mmm._tcp.example.com SRV host1.example.com 0 10 80 host1.example.com
   _mmm._tcp.example.com SRV host2.example.com 0 40 80 host2.example.com
   _mmm._tcp.example.com TXT "version=1.0-2.0"
   mmm.example.com       CNAME host3.example.com
   host1.example.com     A 10.0.1.1
   host2.example.com     A 10.0.1.2
   _mmm._tcp.host2.example.com TXT "path=/service"
   host3.example.com     A 10.0.1.1
   host3.example.com     A 10.0.1.2

Here we have two hosts that offer the service but with slightly different
configurations. Both hosts support versions 1.0 to 2.0 inclusive. Host1
offers the service on the default .well-known/srv/mmm/ path while host2
specifies a different path.

Note that this configuration has fallback for the case where the SRV and
TXT records can't be resolved. This is essential because in the real world,
100% compatibility with legacy infrastructure is an absolute requirement.
So if the client cant get an SRV record, it tries mmm.example.com because
that can be made to work on the most antique setups.

So my response to Michael is that you don't want your version numbers in
the .well-known path, you want them in the DNS records because then they
can be used to select the host to connect to according to capabilities.

Lets say we have 10 hosts and we are upgrading them to a new version of the
spec over a two year period (licenses sometimes cost money, people always
do). So we might upgrade 2 hosts to start with and advertise the fact with
host specific TXT records.

We can even do a multi-tiered scheme in which we have multiple mail
delivery services advertised and the client selects the one it prefers.

This scheme provides an obvious means of advertising host specific
information relevant to TLS. We could specify a host specific key in a TXT
record that could be used to encrypt the initial TLS handshake. We could
specify the fingerprint of a certificate that must be in the Certificate
path for the service certificate. We can specify a minimum TLS version.

Of course, to actually make use of a scheme like this in actual deployment,
the DNS records have to be 100% reliable. Which is not going to happen if
they are being maintained by hand. So an adjunct specification that would
be required is a means of automating the management of the DNS records.
When the service comes up and every n minutes thereafter, it should ping
the outbound DNS to tell it that it is still alive and what current
configuration information it needs to publish.

The ability to secure attributes of this type gives DNSSEC the potential to
improve application security. We can now specify that the service MUST be
encrypted.

---

From: Mark Nottingham <[email-masked]>
Date: Mon, 25 Feb 2019 17:42:54 +1100

Agreed with Martin, to a large extent.

You could do /.well-known/foo/v1, but of course you could also just call it something else, e.g., /.well-known/bar. The point being that if there's backwards-incompatible change in the semantics, it really is another thing.

And of course ../v1.0 doesn't make any kind of sense, because minor versions are supposed to signify backwards-compatible changes, but by changing the URL, they become incompatible. More about this at <https://www.mnot.net/blog/2012/12/04/api-evolution>.

One of the things on my list is to write something explaining how to use OpenAPI for standards; it's designed more for single-deployment APIs (see <https://www.mnot.net/blog/2012/06/25/http_api_complexity_model>), not ones that have multiple, uncoordinated deployments, so there are some foot guns there.

Cheers,

> On 25 Feb 2019, at 7:12 am, Martin Thomson <[email-masked]> wrote:
> 
> Michael, 
> 

--
Mark Nottingham   https://www.mnot.net/

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 25 Feb 2019 07:57:15 +0100

Right, versions are massively overrated.  (Many people think they can transfer the knowledge they have gained about software versioning to protocol versions.  Totally wrong.)

The reason I’m replying is that, for EST-CoAPS, the situation is even more complicated, because CoAP has a standard discovery mechanism (which is incidentally fired by /.well-known/core), and EST-CoAPS also defines everything that is needed to make that work for them.  So there already is an established way to obtain the server-defined entry point URIs to the various services.  The /.well-known/est stuff mainly seems to be there  because there is a belief that implementers in this space want to nail down everything and won’t do the discovery.

Grüße, Carsten

---

From: Michael Richardson <[email-masked]>
Date: Mon, 25 Feb 2019 10:51:55 -0500

> Agreed with Martin, to a large extent.

Good to know.

    > You could do /.well-known/foo/v1, but of course you could also just
    > call it something else, e.g., /.well-known/bar. The point being that if
    > there's backwards-incompatible change in the semantics, it really is
    > another thing.

Registering /.well-known/bar requires an IANA action, and pollutes that
namespace, so it seems like a bad idea.

    > And of course ../v1.0 doesn't make any kind of sense, because minor
    > versions are supposed to signify backwards-compatible changes, but by
    > changing the URL, they become incompatible. More about this at
    > <https://www.mnot.net/blog/2012/12/04/api-evolution>.

I agree that semantic versioning in the URL seems dumb, because only the
first digit should matter as far being compatible.

I suppose using /api/1.0.1 and getting a 404 is a pretty clear statement that a
minor change is not supported, but I don't really get it.  So we agree on /v1
vs /v2 makes more sense than 1.0.0 and 2.0.0

    > One of the things on my list is to write something explaining how to
    > use OpenAPI for standards; it's designed more for single-deployment
    > APIs (see
    > <https://www.mnot.net/blog/2012/06/25/http_api_complexity_model>), not
    > ones that have multiple, uncoordinated deployments, so there are some
    > foot guns there.

I would welcome a discussion about OpenAPI in IETF specifications.
Mechanically, how to do it (throw the YAML in as a an appendix? Inline?
Split up the YAML?  Which text is normative...), and pieces of OpenAPI to
avoid.

My EST extension protocol is a bit complex from an OpenAPI point of
view. First Alice talks to Bob, and get something, and gives it to Frank, and
Frank then speaks to Joe, and returns something to Alice, which returns to
Bob with it. OAUTH2 has a similar level of complexity.  I don't see an
obvious way in OpenAPI to express the coupling between API calls.

I found your blog posting made a good point.  My experience is that people
have the experience of starting "One Client, One Server", and then they are
suddendly in Many Client,Many Server, either because the company was
restructured, or because they went "public".  The Besos legend includes him
declaring that all APIs could be opened at any time, and people better be
ready.  (Many have lamented that Google doesn't have the same approach)

--
]               Never tell me the odds!                 | ipv6 mesh networks [
]   Michael Richardson, Sandelman Software Works        |    IoT architect   [
]     [email-masked]  http://www.sandelman.ca/        |   ruby on rails    [

---

From: Michael Richardson <[email-masked]>
Date: Mon, 25 Feb 2019 10:57:32 -0500

Carsten Bormann <[email-masked]> wrote:
    > Right, versions are massively overrated.  (Many people think they can
    > transfer the knowledge they have gained about software versioning to
    > protocol versions.  Totally wrong.)

I agree.
Further, we don't do enough advance work in "version 1" to say how we will
use minor protocol numbers.  They are almost always useless in my opinion,
but it's taken a long time to get to that view.  Minor extensions must always
be accomodated without protocol version changes.

    > The reason I’m replying is that, for EST-CoAPS, the situation is even
    > more complicated, because CoAP has a standard discovery mechanism
    > (which is incidentally fired by /.well-known/core), and EST-CoAPS also
    > defines everything that is needed to make that work for them.  So there
    > already is an established way to obtain the server-defined entry point
    > URIs to the various services.  The /.well-known/est stuff mainly seems
    > to be there  because there is a belief that implementers in this space
    > want to nail down everything and won’t do the discovery.

Yes, I've pushed to not sit on the fence, here, and lost :-)

Either we always do discovery, in which case, we have no need for
/.well-known/est, or we can operate on never doing discovery, in which case,
we don't need discovery.   It's clearly a code space vs network round trips
argument.

I think that once we have a standardized CBOR reply for CoAP discovery, so no
parsing of ASCII strings, that discovery will become more popular.

WOuld you say that in a CoAP space that protocol enhancements are simply
indicated by the discovery returning new end points.  We can name
them /2/blah if we want, but /rfc1234/blah would be equally valid/useful.

---

From: Michael Richardson <[email-masked]>
Date: Mon, 25 Feb 2019 11:04:31 -0500

Phillip Hallam-Baker <[email-masked]> wrote:
    > I think that the IETF already has the answer to the problem you are raising.
    > But it is spread out so that while the existing standards essentially
    > constrain the space, there is no RFC that puts them together and so I wrote
    > one myself:

    > updating the whole document set right now.

okay, I see it now, and I'll read it.

    > So my response to Michael is that you don't want your version numbers in the .
    > well-known path, you want them in the DNS records because then they can be
    > used to select the host to connect to according to capabilities.

Interesting. I think that I agree.

    > host specific TXT records.

It might also be that operating both versions is more than one wants to do
with a single host, so one wants to upgrade only as the clients upgrade in
order to match traffic loads.

I like that our document includes load balancing as a built-in goal.

    > they are being maintained by hand. So an adjunct specification that would be
    > required is a means of automating the management of the DNS records. When the
    > service comes up and every n minutes thereafter, it should ping the outbound
    > DNS to tell it that it is still alive and what current configuration
    > information it needs to publish.

    > improve application security. We can now specify that the service MUST be

+1

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 25 Feb 2019 18:00:51 +0100

> Interesting. I think that I agree.

For CoAP, we don’t want to become dependent on DNS.  But we can do similar things with discovery and “rt” target attributes.

Grüße, Carsten

---

From: Ted Lemon <[email-masked]>
Date: Mon, 25 Feb 2019 14:34:23 -0500

On Feb 25, 2019, at 12:00 PM, Carsten Bormann <[email-masked]> wrote:
> For CoAP, we don’t want to become dependent on DNS.

Why not?

---

From: Kent Watsen <[email-masked]>
Date: Mon, 25 Feb 2019 19:49:23 +0000

I think putting the version into the URL path is wrong, though that is what is done 99.5% of the time.

Many years ago, I designed a versioned REST API whereby the versioning was implemented via
media-types.  Everything was an object, and objects had versions.  Backwards compatibility was
supported by the server maintaining support for a few versions back of each kind of object.  Each
time the server interacted with the client, it would up/down-convert to/from "current" as needed.
Through HATEOAS, clients passing an Accept type for newer version objects, would automatically
get all the appropriate rel links for it.  We shipped a product for a few years using this approach.
There were many debates, but it seemed popular with the team, especially as releases only had
to support a few new object-versions, as opposed to a whole new API version root.

FWIW.

Kent

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 25 Feb 2019 23:09:29 +0100

>> For CoAP, we don’t want to become dependent on DNS.

It’s a long story, and I’m not sure this is the right place to repeat the discussion.

First of all, there is nothing wrong with using CoAP and DNS together.  We just don’t want to form an addiction.

DNS is this great internet-scale naming system.  For most IoT deployments, I have no idea what that would have to do with it.

In many organizations, DNS is highly loaded with bizarreness (often originally motivated with genuine security considerations).  Staying clear of that may be extremely useful.

DNS provides a limited form of REST (I know you have been working on retrofitting more of it), but it is not the real thing.  Why would I use DNS if I can use CoAP?

Grüße, Carsten

---

From: Ted Lemon <[email-masked]>
Date: Mon, 25 Feb 2019 17:20:57 -0500

> DNS provides a limited form of REST (I know you have been working on retrofitting more of it), but it is not the real thing.  Why would I use DNS if I can use CoAP?

Because there’s a large infrastructure that supports it, and because services are discovered by consulting a single source that is updated as needed by services, rather than by surveying the set of all providers of service every time a service is sought.   And because it is something that nearly every non-IoT device on the network knows how to search for services.

As you say, this probably isn’t the place to discuss it, but at the same time I’ve noticed that it tends never to be discussed outside of constrained device/network circles, and I think that’s probably not good, so when you said what you said, I took the opportunity to question it in a larger venue. :)

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Tue, 26 Feb 2019 13:57:03 -0500

On Mon, Feb 25, 2019 at 12:01 PM Carsten Bormann <[email-masked]> wrote:

> For CoAP, we don’t want to become dependent on DNS.  But we can do similar
> things with discovery and “rt” target attributes.
>

But what do you mean by DNS?

There is the DNS name registry, the DNS data model and the DNS resolution
protocol. All three can be called DNS.

We discussed this at the naming workshop. I see two different types of
name, universal so that we all interpret them the same way and local. If I
am talking to an Internet of Thing in my house 'open the garage door' is
unambiguous. If I am on the Internet, it is not.

We can certainly have a personal name resolution scheme and this should be
a standard. But we should not introduce a new data model unless we really
need to.

The DNS client-resolver resolution protocol is awful. Lets stop trying to
put lipstick on that pig and sidegrade to an alternative resolution
protocol that preserves the same data model.

One of the reasons I want to establish the convention of resolving the
service named 'foo' using the convention it maps to /.well-known/srv/foo/
is precisely so that in protocols such as coap, we can make the service
name a transport connection attribute and dispense with .well-known
entirely.

So putting it together, lets say Carsten has two devices in his house, that
support the not-on-fire protocol that is offered over HTTP, CoAP and in one
room, Splunge:

_not-on-fire._tcp.carsten.house  TXT "trans=coap trans=http"
_not-on-fire._tcp.carsten.house  SRV 0 10 80 garage.carsten.house
_not-on-fire._tcp.carsten.house  SRV 0 10 80 kitchen.carsten.house
_not-on-fire._tcp. kitchen.carsten.house TXT "trans=splunge"

[http://mathmesh.com/Documents/draft-hallambaker-web-service-discovery.html]

The development tools I use allow me to offer XML, ASN.1, JSON, etc
encodings of the same service very easily. It is just a switch on the
compiler right now. Next generation it will be just a different library.

On the naming side, I have a scheme for that as well. But that takes a
rather more holistic view of the problem. My basic idea is that if I buy a
device, I should be in control of it and it should only interact with me
and other devices I specify for it to interact with. It should only use the
name services that I nominate and no others.