Document the dogfood deployment of this code

[exarkun]

Since dogfood is the preferred testing methodology, make it obvious to everyone how to do that.

Point to the deployment, describe how it gets updated (sounds like every push to any branch gets deployed?), maybe mention some interactions that are supposed to work (and ... continue to work?).

[glyph]

I've bounced off this ticket a few times while pondering the question of what the right format is for this documentation so why don't I stream of consciousness here in a comment and see where we can go from there.

The deployment is here:

https://github.com/twisted-infra/lists.twistedmatrix.com/blob/3a918dfc08ae648cadf007b5d20f8e21980abf8d/deploy

triggered from here:

lists.twistedmatrix.com/.travis.yml

Line 2 in 3a918df

- "./deploy"

The idea behind this extremely limited deployment idiom is that master ought to go live to production. Secrets are injected via the travis encrypt system; you can see which are expected by looking for os.environ in the code.

However, there's a bug right now where every branch goes straight to production as long as it's a branch on this repo.

You can run a local instance to test against by using the ./develop script, which sets up certain things locally, but is honestly a little specific to my environment at this point. For example it has a hook for "airplane mode" development, which expects that you're running a local devpi server and translates 127.0.0.1 to the hard-coded IP 10.200.10.1

lists.twistedmatrix.com/buildandrun

Line 15 in 3a918df

echo "-e=PIP_TRUSTED_HOST=10.200.10.1 -e=PIP_INDEX_URL=$(echo $PIP_INDEX_URL | sed s/127.0.0.1/10.200.10.1/g)"; # as per 'docker for mac' networking guide alias

because that's the recommended thing to hack for docker for mac https://docs.docker.com/docker-for-mac/networking/#i-want-to-connect-from-a-container-to-a-service-on-the-host .

More to come.

[glyph]

The local development script also expects that your machine has a globally valid certificate for its hostname, which is maybe a little weird. This is actually reasonably straightforward to do with current versions of txsni and txacme; here's a script https://gist.github.com/glyph/bce4c45095c9f1513adbf12f294c6bc1 that will do this against Rackspace Cloud DNS (authenticatedly is just a little wrapper around Keyring https://pypi.org/project/keyring/ that grabs a secret asynchronously; it's on my to-do list to open source this as a public thing).

[glyph]

This code is also dogfooding the largely untested "form-hacks" branch on my fork of Klein https://github.com/glyph/klein/tree/form-hacks which needs to be broken up into a series of smaller PRs against Klein proper (and better tested). However, using this as a real-life test bed is supposed to help validate some of the design ideas. Unfortunately this interacts poorly with pip caching and docker building which means "airplane mode" development is not quite a reality.

[glyph]

@exarkun Those are some initial thoughts on how this is set up presently, do you have any more specific questions I could answer? (I am not proposing that these comments should result in a resolution of the ticket, they should still go in a real document somewhere, but hopefully having this info here even in an unstructured blob might enable someone other than me personally to make progress on such a document.)

[glyph]

Oh also an important idea behind this as a dogfooding / demo thing is that while we (i.e. Twisted Matrix Labs at large) have been pretty good at continuous integration, our lack of any "fun" applications to run means we've not been great at continuous deployment. The idea behind this app is that it is focused foremost on ensuring the latest version is live in production immediately with zero steps to manually deploy it; ideally we can retrofit / front-load this with proper unit testing as well, but this should serve as a testbed which will allow us to run a fully integrated up-to-the-minute Twisted app that will let us spot real-life issues in an integration environment, potentially illustrating holes in our unit-test coverage or compatibility process.

[exarkun]

That's certainly all good information. Thanks for sharing it. I don't think it's sufficient for me to do much significant contribution, though. It doesn't do much into what I had in mind when I filed this issue, which was that I don't know much about application level behavior. So now I have some hints about how to test and deploy changes, but still not much idea about what to test for. I could imagine up my own specification and goals for the project and work towards those and maybe that would even work out given our shared email experiences... but I doubt it would scale beyond that and I'm not even that sure it would work for just the two of us.

I have some good feelings about certain parts of BDD. Do you? Do you have another comparable specification strategy that could be applied here?

Alternatively, I could avoid feature work entirely and just fix the narrowest, most obvious of bugs (maybe #2 and #3, the only other issues currently open, even qualify for this). That doesn't quite square with your ideas about a "fun" application but if it would be helpful then meh whatever.

[glyph]

It doesn't do much into what I had in mind when I filed this issue, which was that I don't know much about application level behavior

"deployment" suggested that you were really just asking about how the code gets to the server, which is really just that one script :).

However, this is also a reasonable question, so, a little more stream-of-consciousness:

Mailgun operates an SMTP server which will happily deliver to "discussion"-style lists (i.e. any registered subscriber can post). However, it doesn't have self-service "subscribe" or "unsubscribe" APIs, and it won't store any message for more than 3 days.

The goal of this service is to replicate the behavior of Mailman, but in a very simple way; we don't use 99% of the features of Mailman, so it's not like we'd be reimplementing the whole thing.

This code's objective, then, is to:

• Allow for automated subscription and unsubscription to discussion lists. Mainly this is twisted-python, with twisted-web a close second.
• Carefully rate-limit confirmation messages when subscribing and unsubscribing people, since that's the thing that really got us into this mess in the first place.
• Store and render message archives, in such a way that historical links from the pipermail archives can easily be directed to them. This includes the two subtasks of:
  • Ingest the existing pipermail archives, maintaining the existing mailman message counter, and
  • Respond to the Mailgun webhook that is issued when a message is delivered to a list address. This is currently configured manually per list, but some kind of administrative "create list" / "remove list" functionality would be a nice-to-have for this software.

That's pretty much it. If we can do these three things with a sensible backup strategy then we could probably turn off mailman… and thereby turn off inbound SMTP on dornkirk entirely. (Separately, I can give out SMTP credentials to folks with twistedmatrix.com addresses, and maybe we want a different system for self-service changing where your mail forwards to, but that's really entirely unrelated...)

[glyph]

I have some good feelings about certain parts of BDD. Do you? Do you have another comparable specification strategy that could be applied here?

I don't have feelings one way or another, and I'd be very interested to hear what you come up with here.

[glyph]

Hi. It's been a while. I should probably just note a few things:

• front-line email (i.e.: inbound & outbound SMTP, DKIM, SPF, etc) for twistedmatrix.com is now entirely handled by Mailgun; this has, as expected, massively reduced the administrative load associated with dornkirk. the only annoyance is that occasionally co-located bad actors will get us onto an IP-based RBL. (we would, however, find ourselves on RBLs on our own server too; while it was possible to do more about it there, it also just much harder to notice, and practically speaking we never did.)
  • real talk: https://www.spamcop.net is terrible
    • everyone should stop using it
    • literally everything is colocated these days
    • the real entity you're trusting (or not) is the domain and not the IP address
    • there's public-key crypto in those DNS records, use it
• mailing lists, however, are still handled by mailman; dornkirk is now mailman.twistedmatrix.com; it receives email through forwarding rules and sends email as a smarthost, via mailgun, using an SMTP password
• the perpetual tire-fire of mailman's misconfiguration has been quelled somewhat:
  • mailman has now been configured to use a CSRF token (SUBSCRIBE_FORM_SECRET)
  • the HTTPS URLs on the listinfo & admin pages are now consistent, thanks to fix_url.py; no more mixed content or bad form posts
  • the image URLs are fixed
• I don't work at Rackspace any more (and besides, Rackspace spun out Mailgun before I left) so my professional motivation to work on this code as an API demo is mostly gone. still interested in it as a community testbed.
• speaking of rackspace, carina doesn't exist any more, and the deployment is now to hyper.sh.

so work here has slowed somewhat.

however, it's still a nice little deployment test bed, and it has all the secrets set up to do stuff with our mailgun APIs, so I'm happy if its mission morphs a little bit.

[glyph]

I would, also, long-term, prefer to have something radically simpler than mailman managing our lists; for the time being, however, it definitely isn't a crisis any more.