BosniaAir gets its data from the WAQI API. When I moved the site's API into Next.js route handlers on Vercel (more on that move in another post), the new code could not reach WAQI. The requests failed with ETIMEDOUT.
The code was a plain fetch with a token and a URL. There was nothing in it to find, which is what makes this kind of bug annoying. You keep reading code that is correct.
What was actually going on
The WAQI API, api.waqi.info, is hosted in Tokyo. My server was in Europe. That is a long way for a TCP connection to go, but long is not the same as broken. A connect from Europe to Japan takes a noticeable amount of time and still finishes.
The problem sat one layer below fetch. When a hostname resolves to more than one address, for example one IPv4 and one IPv6, Node uses an algorithm called happy eyeballs (the option is named autoSelectFamily). It tries the addresses in turn, and if an attempt takes too long it moves on to the next one instead of waiting. The "too long" is a timeout per attempt, and in the Node version I was running it defaulted to 250 ms.
Two things combined:
- 250 ms is shorter than the time a connection from Europe to Tokyo needs to complete.
- The IPv6 address was often unreachable from where the code ran, so that attempt could not succeed at all.
So Node tried an address, gave up after 250 ms before the handshake could finish, tried the next, gave up again, and reported a timeout. Nothing was wrong with the server on the other end. The client was giving up faster than the distance allowed.
The fix
One call at startup, in the module that makes the requests:
import net from 'node:net'
// api.waqi.info is hosted in Tokyo. Node's default happy-eyeballs timeout (250 ms per
// address) is shorter than the TCP connect time from Europe, and its IPv6 address is
// often unreachable, so fetch() fails with ETIMEDOUT. Give each attempt more time.
if (typeof net.setDefaultAutoSelectFamilyAttemptTimeout === 'function') {
net.setDefaultAutoSelectFamilyAttemptTimeout(2500)
}Each attempt now gets 2.5 seconds. The typeof check is there because the function only exists in newer Node versions, and an older runtime should not crash on a line that is only a tuning knob.
On the request itself the code also sets an overall deadline of 15 seconds with AbortSignal.timeout(15000). That is a separate limit. The first one decides how long a single connection attempt may take, and the second decides how long the whole call may hang. You want both, because a generous attempt timeout without a deadline is how you end up with a page that spins forever.
Why it was hard to see
The message says ETIMEDOUT, which reads as "the server did not answer". That points your attention at the network, the API, the token and any firewall. All of those were fine. The actual cause was a default inside the runtime, and it is only wrong for some routes between some places.
It also does not show up in tests that stub fetch, and it does not show up when the API is close to you. It appears after you deploy, which is the moment you least want a mystery.
What I would check next time
If a request to a far-away host times out and the code looks fine, I would go in this order:
- Call the same URL with
curlfrom a machine in the same place and see how long the connect takes. - Check whether the host resolves to both IPv4 and IPv6, and whether the machine can use IPv6 at all.
- Look at the client's connect timeout, not only its request timeout.
The third one is the one I had not known to look for.
The lesson
A timeout error is a statement about your client's patience as much as about the other side. When both sides look healthy, read the client's defaults. I had treated them as part of the furniture, and they were chosen with a different geography in mind than mine.
The explanation stays in a comment above the line. The line itself looks arbitrary, and without the comment the next person to read it, me included, will delete it as noise.