ara/ss14-cicd/logic/experiments.md

Experiments

E01: Architecture-tagged cache key A/B comparison

• Verifies: C01, C05
• Setup:
  • Model: wylab-station-14 CI workflow (Gitea Actions, .gitea/workflows/build.yaml)
  • Hardware: Two runners — arm64 macOS (OrbStack) + x86-64 Unraid host
  • Dataset: Multiple commits to wylab-station-14 triggering builds
  • System: Gitea at git.wylab.me, shared cache backend (local file cache on shared NFS mount or act-cache-server)
• Procedure:
  1. Configure workflow with architecture-agnostic cache key (baseline): key: dotnet-${{ hashFiles('**/*.csproj') }}
  2. Trigger 5 builds alternating between arm64 and x86-64 runners
  3. Inspect produced Docker image: verify target architecture with docker inspect --format '{{.Architecture}}'
  4. Switch to architecture-tagged key: key: dotnet-${{ runner.arch }}-${{ hashFiles('**/*.csproj') }}
  5. Repeat 5 builds alternating runners
  6. Inspect produced Docker images again
• Metrics: Binary artifact architecture correctness (amd64 vs arm64 output); cache hit rate per architecture; build success rate
• Expected outcome:
  • Baseline (agnostic key): some builds produce wrong-architecture artifacts despite green CI status
  • Tagged key: all builds produce correct x86-64 artifacts; cache hit rate maintained; no cross-arch pollution
• Baselines: Agnostic-key multi-arch setup (documented failure state from 2025-12-18 sessions)
• Dependencies: none

E02: Runner label pinning vs. architecture-tagged keys — effectiveness comparison

• Verifies: C01, C05
• Setup:
  • Model: wylab-station-14 CI workflow
  • Hardware: Same two-runner setup (arm64 Mac + x86-64 Unraid)
  • Dataset: 10 consecutive commits
  • System: Gitea Actions with label routing
• Procedure:
  1. Add runs-on: unraid to all build jobs in workflow YAML
  2. Ensure Unraid runner config.yml has labels: [unraid, self-hosted, linux]
  3. Trigger 10 builds and verify all execute on Unraid runner only
  4. Inspect Docker image architecture for all 10 builds
  5. Measure cache hit rate and build time compared to E01 tagged-key baseline
• Metrics: Fraction of jobs routed to Unraid runner; artifact architecture correctness; build time distribution
• Expected outcome:
  • All 10 builds execute on x86-64 Unraid runner
  • All 10 artifacts are amd64 architecture
  • Build times are consistent (no cross-arch cache misses)
  • Cache hit rate on Unraid runner is higher than in mixed setup
• Baselines: Mixed-runner setup without label pinning (documented as failure state)
• Dependencies: E01

E03: Local file cache vs. act-cache-server reliability comparison

• Verifies: C02, C04
• Setup:
  • Model: wylab-station-14 CI workflow with .NET cache step
  • Hardware: Single runner (either Unraid x86-64 or external VPS 45.137.68.83)
  • Dataset: 10 consecutive builds
  • System: Gitea Actions; two cache configurations tested sequentially
• Procedure:
  1. Configure runner to use native act-cache-server (port 39913 enabled in runner config)
  2. Run 10 builds; record cache step duration per build
  3. Reconfigure runner to use local file cache (volume-mounted directory, no HTTP protocol)
  4. Run 10 builds; record cache step duration per build
  5. Compare: cache step latency distribution, cache hit/miss count, error rate
• Metrics: Cache step duration (seconds); cache hit rate; ETIMEDOUT error rate; total build duration
• Expected outcome:
  • act-cache-server: some fraction of builds show 5-minute cache step (ETIMEDOUT → miss); inconsistent
  • Local file cache: cache step consistently fast; zero timeout errors; higher hit rate
• Baselines: Uncached builds (no cache step) as lower bound reference
• Dependencies: none

E04: DNS configuration strategies for runner job containers

• Verifies: C03
• Setup:
  • Model: wylab-station-14 CI workflow with a git clone git.wylab.me/... step
  • Hardware: Unraid host running act-runner in Docker container (bridge network)
  • Dataset: Single test commit triggering build
  • System: act-runner config.yml, Docker daemon.json on Unraid
• Procedure:
  1. Baseline: default bridge network, no custom DNS — record failure (git.wylab.me unresolvable)
  2. Test A: Add dns: ["1.1.1.1"] to runner process container — record result
  3. Test B: Use container.network: host in runner config.yml — record result (1/6 success expected)
  4. Test C: Configure container.dns: ["172.17.0.1"] in runner config.yml (Docker bridge gateway, which forwards to Technitium) — record result
  5. Test D: Configure container.dns: ["192.168.1.50"] directly (Technitium IP, only works if reachable from bridge) — record result
• Metrics: DNS resolution success rate (0-6 job containers resolving git.wylab.me); build trigger rate; pipeline jobs succeeding end-to-end
• Expected outcome:
  • Test A (1.1.1.1): fails — cannot resolve private internal hostname
  • Test B (host network): partially works — host DNS available, but security boundary removed
  • Test C (bridge gateway 172.17.0.1): expected to work — bridge gateway forwards to Technitium which resolves internal names
  • Test D (192.168.1.50 direct): may work if Technitium is reachable from the bridge subnet
• Baselines: Default bridge network (O2 documented failure state); host networking (O2 1/6 partial success)
• Dependencies: none