Small cleanups.new working demo driver, lots of docs (not complete)

docs/0.8.3-notes.md

@@ -0,0 +1,68 @@
+# Sewer 0.8.3 Release Notes
+
+This will attempt to list all the changes that affect users of the
+`sewer-cli` program, including even cosmetic changes.  If you use sewer as a
+library you may find internal changes not called out here.
+
+**New `sewer-cli` features are usually just mentioned, see
+[sewer-cli.md](sewer-cli) for more complete documentation.**
+
+## What's New
+
+- added many words in the /docs directory.  A lot of it is internals
+  documentation; a lot of it was written to help me understand exactly how
+  some things worked and decide how they should be improved - design essays,
+  as it were.  Much of it is sure to still be in some intermediate state.
+
+- new documentation of the [`sewer-cli` user command](sewer-cli).  Tries to
+  be _as-is in 0.8.3_ while warning about things that are sure to change
+  later.
+
+- `--acme_timeout` has been added (revised from menduo's PR #154)
+
+- `alias` parameter (in DNSProviderBase) added; available through `--p_opt
+  alias=...` for `sewer-cli`.  **NB: legacy drivers do NOT implement
+  aliasing yet - code changes req'd**
+
+- `prop_delay`, `prop_timeout` and `prop_sleep_times` (in DNSProviderBase)
+  Available through the `--p_opt` option in `sewer-cli`, though no legacy
+  provider has support for `prop_timeout` or `prop_sleep_times`.  One
+  advantage to the `--p_opt` approach is that these parameters *will* be
+  available to any provider that supports them (and likewise, other future
+  additions) without any change to `sewer-cli`.
+
+**NB: `--alias_domain` and the planned `--prop_*` options were only added
+during PRE-0.8.3, so they will just be dropped in the release.**
+
+## What's Changed
+
+Mostly I've tried to avoid changes that were likely to break things.  More
+so for `sewer-cli` than those who use the inner workings of Client, of
+course.
+
+- the default log level for providers changed from INFO to WARNING,
+  so by default they don't natter so much.  Makes `sewer-cli` a little quieter
+  when there are no problems.
+
+- all the legacy DNS providers have been minimally revised to accept some
+  new options and pass them up to their parent classes.
+
+- `sewer-cli` interface to providers has been augmented to pass some new options
+  (--acme_timeout, parameters from --p_opts)
+
+- `sewer-cli` has had long option abbreviations disabled in argparse.  This
+  was never documented in sewer, and is an attractive nuisance since the
+  addition of another option can break an abbreviation.  Everyone wants more
+  options, right?  <wink>
+
+- `unbound_ssh` driver, a working demonstration of using aliasing support in
+  a legacy DNS driver.  Needs a rather specific environment to work, but I
+  just renewed a handful of certificates using it the other day.
+
+## Breakage
+
+- removed all the imports in __init__.py (both sewer & dns_providers).  This
+  *will* affect you if you've just done `import sewer` and access especially
+  the provider classes as eg.  `sewer.ThatDNSDns`.  Using proper imports,
+  eg., `import sewer.dns_providers.thatdns.ThatDNSDns` is the current
+  workaround, sorry.  **This does NOT affect `sewer-cli` users.**

docs/ACME.md

@@ -0,0 +1,48 @@
+## ACME, RFCs, and confusion, oh my!
+
+ACME grew out of early, ad-hoc procedures designed to let CAs issue large
+numbers of certificates with low overhead.  As described in RFC855, these
+would go something like this:
+
+> * Generate a PKCS#10 [RFC2986] Certificate Signing Request (CSR).
+> * Cut and paste the CSR into a CA's web page.
+> * Prove ownership of the domain(s) in the CSR by one of the following methods:
+>     + Put a CA-provided challenge at a specific place on the web server
+>     + Put a CA-provided challenge in a DNS record corresponding to the target domain
+>     + Receive a CA-provided challenge at (hopefully) an administrator-controlled email 
+>       address corresponding to the domain, and then respond to it on the CA's web page
+> * Download the issued certificate and install it on the user's web server
+>
+> With the exception of the CSR itself and the certificates that are
+> issued, these are all completely ad hoc procedures and are
+> accomplished by getting the human user to follow interactive natural
+> language instructions from the CA rather than by machine-implemented
+> published protocols.
+
+HTTP validation was the first mechanism, matching the first method of
+proving ownership in the above.  The rest of what
+[Let's Encrypt](https://letsencrypt.org)
+added was automating the process (and rearranging it a bit, having the proof
+of control happen before the CSR, etc.).  Years later, the IETF standardized
+the ACME protocol, and there are other variants that have been (or will be)
+standardized.
+
+## RFC8555
+
+The [IETF](https://www.ietf.org/) accepted(?) and published
+[RFC8555](https://tools.ietf.org/html/rfc8555) defining the ACME protocol
+for http-01 and dns-01 validations of dns-name authorizations.  These are
+the sort of ACME authorizations that we usually think of, and which sewer
+works with.  The RFC was published in the spring of 2019, but it wasn't
+until near the end of that year that Let's Encrypt adopted the full v.2 on
+only their *staging* server.  There's some elaborate and, from what I can
+make out, often-shifting schedule for various partial transitions, but I'm
+not going to try to make sense of them.  As of the beginning of 2020, the
+only immediate effect on sewer was that one could no longer run it against
+the *staging* server.  The next big change is when that same restriction is
+rolled out on LE's *production* server late in the year.  Since sewer
+v0.8.2, which implemented the final RFC8555 protocol at least well enough to
+work with LE's server implementation, our tl;dr is just this:
+
+> If you get a failure running an older version of sewer, get v0.8.2 or
+> later.  This is a known problem: v0.8.2 is the fix.

docs/Aliasing.md

@@ -0,0 +1,91 @@
+# Aliasing for ACME Validation
+
+The idea is presented (for dns-01 authorizations) in [an article at
+letsencrypt.org](https://letsencrypt.org/2019/10/09/onboarding-your-customers-with-lets-encrypt-and-acme.html)
+which shows an example of DNS aliasing and describes what was likely the
+original motivation - a hosting provider running the certificate process on
+behalf of his customers.  Like all DNS aliasing, it uses a CNAME at the
+canonical name `_acme-challenge.domain.tld` to redirect the ACME server to a
+different fqdn that is more convenient for provisioning of the validation
+responses.  I'm not going to try to convince you that you should use
+aliasing, because if you need it you probably already know that, or at least
+know that the process isn't working smoothly as-is.
+
+The `alias` option in sewer is available to drivers that derive from
+`DNSProviderBase`.
+
+>Added in 0.8.3: `--p_opts alias=...`, but legacy drivers don't take
+advantage of the aliasing support in their parent classes yet.
+
+## Isn't aliasing just for DNS?
+
+No.  HTTP has had, as a side effect of common web server and client behavior,
+a kind of aliasing since the very beginning of ACME.  Usually it's
+convenient enough to provision the validation at the canonical
+`/.well-known/acme-challenge/<token>` location.  But if it isn't, either
+`acme-challenge` or `.well-known` can be mapped (by server configuration or
+externally by symlink, usually) to some other location.  If it's desired to
+serve the validation file from some other domain or server altogether, an
+HTTP redirect can often be used (and that's also a third way to place the
+file elsewhere within the web server's accesible filesystem).
+
+The RFC says that (for http-01 challenges) the ACME server "SHOULD follow
+redirects", which would allow for an analogous aliasing.  Lets' Encrypt's
+servers [do follow redirects and
+CNAMES](https://letsencrypt.org/docs/challenge-types/).
+
+So aliasing can be used with HTTP validations, though it's probably less
+often needed since the privilege needed to directly configure the canonical
+response file is likely to be the same (or even less) than that needed to
+setup the new certificate.  And it's possible that you've already used it
+without thinking of it as _aliasing_ because it uses such basic HTTP
+behavior.
+
+## Preparing for DNS aliasing
+
+The first thing which you must have is a way to manage DNS TXT records.  In
+fact, you need to be able to control both the real domain's records (in
+order to setup the CNAME entries, but that's something that needs be done
+only once) as well as managing the alias domain records through the
+service-specific driver.  Generally, expect to need a new-model driver
+rather than an existing legacy driver, on the assumption that it's not much
+more work to migrate the legacy driver to the new interface while adding the
+alias support.  I have my fingers crossed, at any rate...
+
+With alias-capable driver in hand, you then setup CNAME records for every
+DNS name that you wish to use with the alias domain.  In traditional zone
+file form that might look something like this [excerpt]:
+
+    ; existing record for your web or other server
+    name.example.com.                  IN  A     111.222.333.444
+
+    ; then add the CNAME at the ACME-prefixed name
+    _acme-challenge.name.example.com.  IN  CNAME name.example.com.alias.org
+
+In online domain editors, the names are usually given without the full
+domain suffix that's shown here (example.com).  The A record (or it could be
+a CNAME) for `name.example.com` that directs to your server is shown as an
+example here.
+The added CNAME record is the redirect from the conventional ACME challenge
+DNS name, pointing to the TXT record in the alias domain.  When it sees that
+CNAME, the ACME server will proceed to look for the challenge's TXT record
+at `name.example.com.alias.org`.  Since the alias-aware driver will have
+setup that TXT record, the server will retrieve it and validate your right
+to issue for `name.example.com`.
+
+Note that the alias domain can be ANY valid domain that you can manage.  In
+particular, it can be in a different tld (as shown here) or a different
+domain in the same tld, or even in a sub-domain (eg. 
+`validation.example.com`) of the target's domain that has been delgated to
+that more convenient DNS server.  And you can setup aliased TXT challenge
+records for names from any number of _real_ domains as long as the CNAME
+redirects can be provisioned.
+
+## Using DNS aliases in sewer
+
+This is really pretty short & sweet.
+All that's needed, once the setup is done, is to pass `alias=alias.org` to
+the alias-supporting driver when it's created.
+For users of the command line tool, you would add an option `--alias_domain
+alias.org` as well as specifying a DNS driver that supports the alias
+method, when those become available.

docs/DNS-Propagation.md

@@ -0,0 +1,104 @@
+## Waiting for Mr. DNS or Someone Like Him
+
+Q: How long does it take after you've setup the challenge response TXT records
+until they're actually accessible to the ACME server?
+
+A: Good Question!
+
+According to [Let's Encrypt](https://letsencrypt.org/docs/challenge-types/#dns-01-challenge)
+it can take up to an hour.  Depends on the DNS service.  Some provide a way
+to check that your changes are fully propagated to all their servers.  With
+many, however, you just have to wait.
+
+Sewer provides a flexible _delay until actually published_ mechanism through
+three optional driver parameters, `prop_delay`, `prop_timeout`,
+`prop_sleep_times`, and the [`unpropagated` method](unpropagated).
+Let's see how they're used in various circumstances.
+
+### No API support, no reliable way to check: just delay
+
+If you can't check that the TXT records are fully published, then all you
+can do is delay for a while.  Perhaps the DNS service will suggest a safe
+time.  If not, you'll have to start with a guess and adjust from there based
+on your experience.  Choosing the right number - long enough but not
+excessively long - can be hard, but applying it is easy.  Just add
+`prop_delay=SIMPLE_DELAY_TIME` to the driver's initialization parameters,
+and sewer's engine will add that many seconds of delay after the challenge
+setup returns before it signals the ACME server to validate those
+challenges.
+
+**CLI option --p_opt prop_delay=... is available for all providers as of 0.8.3**
+
+### API support or can check: use a timeout
+
+If the DNS service gives you a way to check that the propagation is
+complete, or if there are not too many authoritative servers (viz., not an
+anycast system), you can use that actual check (implemented in the driver's
+`unpropagated` method) and the engine will run that check until it succeeds
+or until a timeout you specify is exceeded.  However the check is being
+done, you setup the timeout by adding `prop_timeout=MAX_WAIT_TIME` to the
+driver parameters.  If you know that it takes at least some minimum time to
+propagate, you may also pass `prop_delay` to make the engine delay that long
+before it starts checking.  And there's a delay between checks that has a
+hopefully sensible default, but which you can adjust if necessary through
+the `prop_sleep_times` parameter.
+
+**legacy providers do not implement `unpropagated` as of 0.8.3**
+
+### You probably don't need to change `prop_sleep_times`
+
+Unless you do, but if it's not obvious, just leave it.
+
+This parameter defines the lengths of sleeps the engine will add following a
+call to `unpropagated` that reports not ready.  As an optional parameter
+passed to the driver, `prop_sleep_times` can be an integer number of seconds
+or a list or tuple of such delays which will be used in order.  The final
+value in the sequence will be reused indefinitely.
+
+Example: the default value is (1, 2, 4, 8) which provides an exponential
+backoff up to an 8 second delay, then sticks there.  _[the values could
+change - it's just what seemed reasonable to me]_  So if there's no delay,
+and the check call takes no measurable time (and reports not ready each
+time), it will look something like this with `prop_timeout=20`:
+
+| time | action |
+| ---: | --- |
+| 0 | call unpropagated, sleep(1) |
+| 1 | call unproagagted, sleep(2) |
+| 2 | call unpropagated, sleep(4) |
+| 6 | call unpropagated, sleep(8) |
+| 14 | call unpropagated, sleep(8) |
+| 22 | call unpropagated, timeout! |
+
+This shows both the last value repeating and the way the timeout and sleeps
+interact.  The check for timeout is done only AFTER a call to unpropagated
+AND the chance to exit with success if it finally reports the challenges are
+ready.  So the timeout isn't a hard maximum time, but it's bounded to be no
+more than one sleep interval (plus actual time to run `unpropagated`, of
+course) over `prop_timeout`.
+
+### Other Notes and Advanced Use
+
+These values are setup through the Provider on the reasonable assumption
+that they will vary most directly with the choice of service provider, so
+the individual drivers are best suited to provide sensible defaults where
+appropriate (and possible!).  Sewer's engine implements the delay and check
+loop (with timeout) because the mechanism is the same for all providers (and
+may be useful for other than the DNS-based challenges for which it has been
+implemented).
+
+If you are using sewer as a library and find that you can make a better
+estimate of the propagation after the driver is setup (perhaps using a
+service-specific method to access part of the service's API or run some
+tests), you could adjust those parameters through the same-named attributes
+on the Provider instance.  This is solidly in the categories of don't do it
+unless you're sure you need to, and be prepared to own both the pieces if
+you break it!
+
+### Could this be used with non-DNS Providers?
+
+Yes!  I have no experience with http-01 in any setting where such a delay
+might be needed, but the mechanism is implemented in sewer's engine, and all
+that needs be done is to setup the parameters (and implement unpropagated in
+the driver if using more than just `prop_delay`) as described above and
+there you are!

docs/LegacyDNS.md

@@ -2,38 +2,38 @@
 
 ### `BaseDns` shim class
 
-A child of `ProviderBase` that acts as an adapter between the Provider
-interface and the Legacy DNS providers.
+A child of `DNSProviderBase` that acts as an adapter between the new
+Provider interface and the legacy DNS provider interface.
 
 #### `__init__(self, **kwargs: Any) -> None`
 
 Accepts no arguments itself; doesn't expect any to be passed by Legacy code.
-Injects chal_types=["dns-01"].
+Injects `chal_types=["dns-01"]`.
 
 #### `setup(self, challenges: Sequence[Dict[str, str]]) -> Sequence[Dict[str, str]]`
 
-Iterates over the challenges, extracting the values needed for dns-01 from
-each challenge in the list, and passing them to create_dns_record.
-Always returns an empty list since there is no error return from
-create_dns_record other than raising an exception.
+Iterates over the challenges, extracting the values needed for the legacy
+DNS interface from each challenge in the list, and passing them to
+`create_dns_record`.  Always returns an empty list since there is no error
+return from `create_dns_record` other than raising an exception.
 
 #### `unpropagated(self, challenges: Sequence[Dict[str, str]]) -> Sequence[Dict[str, str]]`
 
 Always returns an empty list, signalling "all ready as far as I know".
-A DNS provider wishing to do something useful here must migrate to the new
-API.
+A legacy DNS driver wishing to do something useful here MAY implement
+`unpropagated` without updating the rest of its interface.
 
 #### `clear(self, challenges: Sequence[Dict[str, str]]) -> Sequence[Dict[str, str]]`
 
-Same as setup except it calls the legacy delete_dns_record, of course.
+Same as setup except it calls the legacy `delete_dns_record`, of course.
 
 ### Legacy DNS class
 
-#### `__init__(self, ...)`
+#### `__init__(self, ..., **kwargs)`
 
-Args are explicitly named per provider; no provision for passing any to
-`super().__init__` - which makes sense, since there used not to be any the
-parent was prepared to receive.
+Args handled by the driver should be explicitly named, with defaults where
+that makes sense.  Starting in 0.8.3, the `**kwargs` bucket has been added
+to provide pass-through to the base class.
 
 #### `def create_dns_record(self, domain_name, domain_dns_value)`
 
@@ -45,4 +45,12 @@ All very provider-dependent.
 
 In theory it should undo the effects of setup.
 In practice, at least one of the services is unable to do that
-(according to the comment).
+(according to the author's comment).
+
+### Legacy DNS vs Aliasing
+
+Legacy DNS drivers MAY change to use the [aliasing](DNS-ALiasing) methods
+inherited from `DNSProviderBase`, though this will require a potentially
+fragile faking of the new-model challenge dict in the driver.  See the
+`unbound_ssh` example driver, and bear in mind that a change to the data
+type of the challenge items IS anticipated, perhaps in 0.9.