I Fixed the Certificate — and Next Morning the Office Lost the Site Entirely

This is a sequel to the story about a certificate that was valid for everyone except the office. Back then I swapped the expired Plesk certificate for a valid certbot one, the office came back to life, and my colleague wrote “yep, all working.” Nice ending.

The next morning: “White screen, and it says ‘can’t reach the site’.”

This article is about how I spent half a day looking for the problem in the wrong place, even though the answer had been on my screen within the first two hours — and about why that happened.

The symptom is not the one it looks like

The first thing I did right was read the error text instead of the browser’s logo.

ERR_TIMED_OUT
The host example.com took too long to respond.

This is not a certificate error. Yesterday it was NET::ERR_CERT_DATE_INVALID — a warning page you can click through. That means the connection was established, the browser reached the server, and the only complaint was about a date in the certificate.

Now: a timeout. The connection isn’t established at all. That’s a different class of problem, and no new certificate will fix it.

Lesson #1: an error code is diagnostic information, not just red text. CERT_* means “we arrived but don’t trust you.” TIMED_OUT means “packets vanished into silence” — classic firewall DROP behavior. CONNECTION_RESET means “we were actively cut off.” CONNECTION_REFUSED means “the port is closed, nobody’s listening.” Four different causes, four different directions to search.

Second input: it failed on both her phone and her computer, but the same phone on mobile data opened the site fine. Later it turned out colleagues in her department couldn’t reach it either.

So: the binding isn’t to a device or a person. It’s to a network.

First, prove the server is alive

The temptation was to roll back yesterday’s change — maybe I broke something. I didn’t give in, and that was right: first you have to establish what is actually broken.

The server sits behind NAT (an OpenStack floating IP), so start by checking what nginx is even listening on:

ss -tlnp | grep ':443'

LISTEN 0 511 192.168.2.137:443 0.0.0.0:* users:(("nginx",pid=1711554),...)

Important detail: nginx listens on a specific internal address, not on 0.0.0.0. Which is why my first test was useless:

curl -vk https://127.0.0.1/ -H "Host: example.com"
# Connection refused

That’s not a fault — there’s simply nobody listening there. You have to test the address the service actually sits on:

curl -vk https://192.168.2.137/ -H "Host: example.com" -o /dev/null

100  641k    0  641k    0     0  16.2M      0 --:--:-- --:--:-- --:--:-- 16.4M
* Connection #0 to host 192.168.2.137 left intact

And the certificate actually being served:

echo | openssl s_client -connect 192.168.2.137:443 -servername example.com 2>/dev/null \
  | openssl x509 -noout -issuer -subject -dates

issuer=C = US, O = Let's Encrypt, CN = YR2
subject=CN = example.com
notAfter=Sep 18 02:34:20 2026 GMT

The server is healthy, yesterday’s work is correct, no rollback needed. That’s an important intermediate result: it eliminated a whole layer of hypotheses and killed the temptation to “just reboot the server” — which, incidentally, the hosting support had already tried during the previous incident, to no effect.

A find I wasn’t looking for: we’re banning our own CDN

Next I went to look at the firewall. There were no explicit DROP rules — only fail2ban chains on ports 80, 443, 7080, 7081.

fail2ban-client status

Number of jail: 13
Jail list: plesk-apache, plesk-apache-badbot, plesk-dovecot, plesk-modsecurity,
plesk-one-week-ban, plesk-panel, plesk-permanent-ban, plesk-postfix,
plesk-proftpd, plesk-roundcube, plesk-wordpress, recidive, ssh

And here something surfaced that had nothing to do with the office, but turned out to be more serious:

fail2ban-client status plesk-modsecurity

Banned IP list: 198.51.100.42 172.70.208.104

fail2ban-client status recidive

Banned IP list: 203.0.113.9 162.158.48.132 162.158.94.47 198.51.100.15 203.0.113.77

172.70.208.104, 162.158.48.132, 162.158.94.47 are Cloudflare addresses (ranges 172.64.0.0/13 and 162.158.0.0/15).

So the server was banning its own CDN. Every such ban silently cuts off whatever slice of the real audience Cloudflare happens to route through that particular edge node. With no error in monitoring anywhere: the site “works,” just not for everyone.

The cause: the origin doesn’t restore visitors’ real IPs from the CF-Connecting-IP header. In the logs every guest looks like Cloudflare, fail2ban counts all their violations together, and bans the edge instead of the offender. One bot through that node — and normal people go down with it.

Lesson #2: CDN + fail2ban without mod_remoteip (Apache) or real_ip (nginx) is a landmine. Sooner or later you’ll ban yourself, and you’ll find out not from monitoring but from a random user.

And now — the mistake that cost me half a day

I was scanning the ban lists for the office IP. I didn’t know the office IP.

Instead of asking for it, I deduced it. In the ModSecurity logs I found an address hitting port 7081 — that’s the Plesk panel:

[21/Jul/2026:11:55:38] ... 198.51.100.42 38802 127.0.0.1 7081
X-Real-IP: 198.51.100.42

The logic looked airtight: only insiders use the Plesk panel, insiders sit in the office, therefore 198.51.100.42 is the office. I lifted the ban:

fail2ban-client set plesk-modsecurity unbanip 198.51.100.42

And the browser error changed: ERR_TIMED_OUT → ERR_CONNECTION_RESET.

That’s where it got me. The symptom changed immediately after my action — so the action must have been right, there’s just another layer underneath. That’s how I explained it to myself.

In reality it was a coincidence. But I now had “confirmation” of my theory, and spent the next few hours building more theories on top of it: Split DNS in the office, a stale local DNS record, half-dead IPv6 on the corporate network, SSL inspection on the router. Each one plausible. Each one requiring action from people who “aren’t technical.” None of them the cause.

Lesson #3: a symptom changing after your action is not proof the action was right. Time correlation during active debugging happens constantly, because you’re making many changes in a row. Proof is when you can break it and fix it back.

There was a second layer to the same mistake. When I finally added a whitelist, I put 198.51.100.42 in it — the address I never actually confirmed. I made the “final fix” based on a guess.

The resolution: one click I should have asked for hours earlier

Eventually I asked a colleague to open https://ifconfig.me and send me the number. It requires zero technical skill — it’s just a link.

203.0.113.77

I went back to the recidive output I’d gotten two hours earlier:

Banned IP list: 203.0.113.9 162.158.48.132 162.158.94.47 198.51.100.15 203.0.113.77

Last address in the list. It had been on my screen the entire time. Looking at that very output, I had written “the office IP isn’t here” — because I was comparing it against my guess, not against a fact.

Unbanning is one command, and it clears the ban across all jails at once:

fail2ban-client unban 203.0.113.77

The office worked immediately.

Why this happened at all

The mechanism turned out to be a closed loop:

The office sits behind a single NAT — to the server, every employee is one address.
Someone opens the Plesk panel on port 7081. The unlicensed panel returns errors, and ModSecurity writes them to modsec_audit.log.
The plesk-modsecurity jail reads that log and bans the address.
The recidive jail reads fail2ban.log itself and bans whoever has been banned before — but for a week.
Along with the “offender,” the entire office loses access to the site.

And because recidive bans for so long, the problem didn’t resolve itself.

Worth noting separately why this never showed up before: office traffic goes to the origin directly via Split DNS, bypassing Cloudflare. External visitors arrive from Cloudflare addresses, so a ban on the office IP doesn’t touch them. The same Split DNS that exposed the expired certificate last time made the office the only victim of the ban this time.

The fix

Unban:

fail2ban-client unban 203.0.113.77
fail2ban-client unban 172.70.208.104
fail2ban-client unban 162.158.48.132
fail2ban-client unban 162.158.94.47

Whitelist the office and all Cloudflare networks in /etc/fail2ban/jail.local, under [DEFAULT]:

ignoreip = 127.0.0.1/8 ::1 <server_internal_IP> <office_IP>
  173.245.48.0/20 103.21.244.0/22 103.22.200.0/22 103.31.4.0/22
  141.101.64.0/18 108.162.192.0/18 190.93.240.0/20 188.114.96.0/20
  197.234.240.0/22 198.41.128.0/17 162.158.0.0/15 104.16.0.0/13
  104.24.0.0/14 172.64.0.0/13 131.0.72.0/22
  2400:cb00::/32 2606:4700::/32 2803:f800::/32 2405:b500::/32
  2405:8100::/32 2a06:98c0::/29 2c0f:f248::/32

(All on one line. Current ranges live at cloudflare.com/ips.)

systemctl reload fail2ban
fail2ban-client get recidive ignoreip

And the main thing still left to do: enable real-IP restoration (mod_remoteip / real_ip with Cloudflare’s trusted networks). The whitelist covers the symptom; without real IPs, fail2ban will keep counting every visitor as one of a handful of CDN addresses.

What I took from it

Ask for a fact instead of deriving it. One click on ifconfig.me would have cost 30 seconds. Deducing it from Plesk logs cost half a day and led into three false theories. “They won’t figure it out” is often an excuse — anyone can open a link.
A changed symptom is not a confirmed hypothesis. Especially when you’re making ten changes in a row.
Re-read your own data after a new fact arrives. The answer was in output I had already seen. I missed it because I was looking for a different number.
NAT turns an individual violation into collective punishment. One office = one IP = one ban for everyone. Networks like that belong in ignoreip by default.
A CDN in front of the server changes the rules for everything that works with IPs. fail2ban, rate limiting, geo-blocking, analytics — all of it sees the CDN, not people, until you explicitly configure otherwise.
Exonerate the server first. Twenty minutes of curl to the origin and a certificate check removed the rollback question and saved me from “let’s just reboot” — an action with zero upside and real risk.

The expensive part of this story wasn’t finding the problem. The expensive part was letting go of the first plausible theory — after it had already received a false confirmation.