rr2.dev: Cloudflare migration, part 2

TL;DR: Two days before rr2.dev's TLS certificate was due to expire, Netlify reported that it couldn't renew it. This was a leftover from the GoDaddy → Cloudflare migration. Rahul deleted Netlify's old DNS zone and renewed the certificate. It was fixed in a few minutes, with no downtime.

What broke

Netlify's renewal failed with NXDOMAIN looking up TXT for _acme-challenge.rr2.dev.

Before the migration, Netlify ran rr2.dev's DNS, so it could add the proof record itself and get a wildcard certificate. The migration moved DNS to Cloudflare, but Netlify's old DNS zone stayed behind. Netlify still thought it was in charge, so at renewal time it wrote the proof record into that old zone. Let's Encrypt only checks Cloudflare, so it never found the record. Certificates last 90 days, which is why this only showed up at the first renewal after the move.

The fix

1. Deleted the leftover Netlify DNS zone for rr2.dev. Nothing was served from it, so nothing went down.
2. Renewed the certificate in Netlify's HTTPS settings. Without the zone, Netlify gets a separate certificate for each hostname and checks them through the website, which works with Cloudflare as the DNS host.
3. Checked that Cloudflare's records still point to Netlify and that there's no CAA record blocking Let's Encrypt.

This works on Netlify's free plan.

How a TXT record proves you own a domain

Before Let's Encrypt gives out a certificate, it must check that you control the domain.

• Who controls a domain: The registrar points rr2.dev to a set of nameservers. Only the person who can edit those nameservers controls the domain.
• The test: Let's Encrypt gives a random code. You must add it as a TXT record at _acme-challenge.rr2.dev. A TXT record is a short piece of text stored in DNS.
• Why wildcards need this: A normal certificate is for one name, like rr2.dev. Let's Encrypt can check it by visiting that website. A wildcard certificate (*.rr2.dev) is for every name under rr2.dev, including names you haven't created yet. Let's Encrypt can't visit websites that don't exist. So it checks the one place that controls all those names: DNS. This is how the check works:
  1. Let's Encrypt asks the registrar which nameservers are in charge of rr2.dev. Today, those are Cloudflare's.
  2. It asks those nameservers for the TXT record at _acme-challenge.rr2.dev.
  3. If the record holds the same random code it gave out, the check passes. Only someone who can edit rr2.dev's DNS could have added that code, and that person controls every name under rr2.dev.
  4. Let's Encrypt runs this check from several places on the internet at the same time, so a single faked answer can't trick it.

In this case, Netlify added the code to its old DNS zone. The internet now reads DNS from Cloudflare, so the code was never found. NXDOMAIN means "this name does not exist."

Copying the old _acme-challenge record to Cloudflare during the migration wouldn't have helped either. A new random code is created for each renewal, so an old one never works. After the fix, no TXT record is needed at all, because Netlify now checks each hostname through the website.

Next steps

• Add two steps to the DNS migration checklist: delete the old provider's DNS zone, then force a certificate renewal and confirm it works.
• Add independent certificate expiry monitoring as a backup to Netlify's email (for example UptimeRobot or Cloudflare's certificate alerts).