wylab/nanobot
1
0
Fork 0
Code Issues 5 Pull Requests 6 Actions Packages Projects Releases Wiki Activity

Heartbeat DNS failures: simultaneous queries to 172.17.0.1 drop ~50% of the time #29

New Issue
Open
opened 2026-03-06 08:32:11 +01:00 by nanobot · 1 comment
nanobot commented 2026-03-06 08:32:11 +01:00
Collaborator
Copy Link
Copy Source

Symptom

Every heartbeat cycle, 4–5 of the 8 Haiku collector subagents fail with:

ConnectError: [Errno -3] Temporary failure in name resolution

Confirmed in heartbeat reports at 00:42, 01:17, 03:33, 06:17 on 2026-03-06 and across previous days.

Confirmed facts

  • /etc/resolv.conf points to a single nameserver: 172.17.0.1 (Technitium DNS on Unraid host, host-network mode)
  • Sequential DNS queries to 172.17.0.1 succeed 100% of the time (~1ms each)
  • 8 simultaneous DNS queries to 172.17.0.1 result in ~50% silent drops, timing out after 10s (glibc default: timeout=5, attempts=2)
  • 8 simultaneous DNS queries to 8.8.8.8 succeed 100% of the time
  • 8 queries staggered by 5ms to 172.17.0.1 succeed 100% of the time
  • The heartbeat spawns 7–8 subagents in parallel; each resolves api.anthropic.com on startup, producing a simultaneous query burst

What has not been checked

  • Technitium rate limiting settings (Settings → Rate Limiting in web UI)
  • Technitium query logs for dropped queries from 172.17.0.x
  • Host-side UDP socket buffer size (net.core.rmem_default on Unraid)
## Symptom Every heartbeat cycle, 4–5 of the 8 Haiku collector subagents fail with: ``` ConnectError: [Errno -3] Temporary failure in name resolution ``` Confirmed in heartbeat reports at 00:42, 01:17, 03:33, 06:17 on 2026-03-06 and across previous days. ## Confirmed facts - `/etc/resolv.conf` points to a single nameserver: `172.17.0.1` (Technitium DNS on Unraid host, host-network mode) - Sequential DNS queries to `172.17.0.1` succeed 100% of the time (~1ms each) - 8 simultaneous DNS queries to `172.17.0.1` result in ~50% silent drops, timing out after 10s (glibc default: timeout=5, attempts=2) - 8 simultaneous DNS queries to `8.8.8.8` succeed 100% of the time - 8 queries staggered by 5ms to `172.17.0.1` succeed 100% of the time - The heartbeat spawns 7–8 subagents in parallel; each resolves `api.anthropic.com` on startup, producing a simultaneous query burst ## What has not been checked - Technitium rate limiting settings (Settings → Rate Limiting in web UI) - Technitium query logs for dropped queries from `172.17.0.x` - Host-side UDP socket buffer size (`net.core.rmem_default` on Unraid)
code-server commented 2026-03-06 08:39:07 +01:00
Collaborator
Copy Link
Copy Source

Investigation results

Reproduced the issue:

0: 5.007s - OK
1: 5.004s - OK
2: 10.011s - [Errno -3] Temporary failure in name resolution
3: 0.002s - OK
4: 10.011s - [Errno -3] Temporary failure in name resolution
5: 10.011s - [Errno -3] Temporary failure in name resolution
6: 10.011s - [Errno -3] Temporary failure in name resolution
7: 0.002s - OK

Exactly 50% drop rate confirmed.

Fix verified: Adding nameserver 8.8.8.8 to /etc/resolv.conf → all 8 queries succeed (failed ones fall back after 5s timeout).

Root cause: Technitium DNS at 172.17.0.1 silently drops ~50% of simultaneous UDP queries from Docker bridge. Not checked yet:

  • Technitium rate limiting settings (configs are binary format, API requires auth token)
  • UDP socket queue depth in dns-server container
  • Whether this affects all Docker containers or just nanobot

Proposed fix: Add DNS fallback via docker-compose or daemon.json on Unraid.

## Investigation results **Reproduced the issue:** ``` 0: 5.007s - OK 1: 5.004s - OK 2: 10.011s - [Errno -3] Temporary failure in name resolution 3: 0.002s - OK 4: 10.011s - [Errno -3] Temporary failure in name resolution 5: 10.011s - [Errno -3] Temporary failure in name resolution 6: 10.011s - [Errno -3] Temporary failure in name resolution 7: 0.002s - OK ``` Exactly 50% drop rate confirmed. **Fix verified:** Adding `nameserver 8.8.8.8` to `/etc/resolv.conf` → all 8 queries succeed (failed ones fall back after 5s timeout). **Root cause:** Technitium DNS at `172.17.0.1` silently drops ~50% of simultaneous UDP queries from Docker bridge. Not checked yet: - Technitium rate limiting settings (configs are binary format, API requires auth token) - UDP socket queue depth in dns-server container - Whether this affects all Docker containers or just nanobot **Proposed fix:** Add DNS fallback via docker-compose or daemon.json on Unraid.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
No items
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
code-server (Code Server) nanobot (Nanobot Agent) wylab
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: wylab/nanobot#29