Full build log of a 10-VM carrier telecom infrastructure — from hypervisor network isolation to SIP proxy clustering, real-time billing, database high availability, centralized security monitoring, and a planned BGP routing fabric. Built on isolated virtual networks with MTU 9000 jumbo frames. Every value sanitized for community learning.
Phase 7 extends the stack across sites using GRE tunnels at the SIP proxy layer. Each Kamailio node is a GRE leaf endpoint. SIP/RTP rides the GRE tunnel (low overhead, no encryption penalty on the data plane). WireGuard handles the control and management plane separately with full encryption.
Phase 9 introduces a BGP routing fabric using CSR1000v virtual routers as spine nodes. Each SIP proxy cluster becomes a BGP leaf — advertising its SIP prefix routes. ECMP across spine nodes provides carrier-grade load balancing and sub-second route convergence on failures.
| VM | Role | Networks | Phase 2 |
|---|---|---|---|
kamailio-01 |
SIP Proxy — primary (LCR + fraud) | ExtVoiceSync | Monitoring |
kamailio-02 |
SIP Proxy — secondary (failover) | ExtVoiceSync | ✓ Agents |
asterisk-01 |
PBX — primary | ExtVoiceData | ✓ Agents |
asterisk-02 |
PBX — secondary | ExtVoiceData | ✓ Agents |
asterisk-03 |
PBX — tertiary | ExtVoiceData | ✓ Agents |
a2billing-01 |
Billing — primary (real-time auth) | ExtData | ✓ Agents |
a2billing-02 |
Billing — secondary | ExtData | Recovering |
mariadb-01 |
Database — primary | DataSync | No ext NIC |
mariadb-02 |
Database — replica | DataSync | No ext NIC |
monitor-01 |
OSSEC · Zabbix · OpenSearch | ExtVoiceData | Installing |
| Network | Physical Uplink | MTU | IP Range | Purpose |
|---|---|---|---|---|
| Management | Physical NIC | 1500 | <mgmt-net> | Hypervisor management — invisible to VMs |
| External | Physical NIC | 1500 | <ext-net> | SIP trunks, internet, customer portals |
| Voice | None (isolated) | 9000 | 10.10.10.0/24 | Kamailio ↔ Asterisk internal SIP/RTP |
| Data | None (isolated) | 9000 | 10.20.20.0/24 | MariaDB ↔ A2Billing ↔ Asterisk CDR |
| Sync | None (isolated) | 9000 | 10.30.30.0/24 | DB replication + Kamailio cluster heartbeat |
Server on monitor-01, agents on all 9 VMs. Real-time file integrity monitoring of /etc, /bin, /sbin, /usr/bin/sbin. Active response auto-blocks attacking IPs within seconds via local firewall rules. Alerts on Level 7+ events via email.
Zabbix Agent2 on all VMs. ISP connectivity monitoring (ping loss to external resolver). SIP health via OPTIONS ping to each Kamailio node. Database connection checks. CPU/RAM/disk thresholds with pre-emptive alerts before service impact.
Single-node OpenSearch on monitor-01 aggregates logs from all VMs via Filebeat — OSSEC alerts, Kamailio SIP logs, Asterisk CDR, Apache access logs, auth.log. OpenSearch Dashboards provides full-text search and visual analysis across the stack.
Zone-based firewall between External, Voice, Data, and Sync segments. VMs can't communicate across zones without explicit permit rules. Complements the hypervisor air gap — software firewalling above the vSwitch layer for defense in depth.
Kamailio evaluates carrier rate tables on every outbound call and routes to the cheapest carrier for the destination E.164 prefix. Daily automated CSV import refreshes rates. Automatic failover to next cheapest if the primary carrier is unreachable or returns 503.
Every call triggers an AGI (Asterisk Gateway Interface) script that queries A2Billing before connecting. Sub-second authorization decision over the isolated Data network. Insufficient balance results in immediate rejection before ringing — no CDR post-processing lag.
Voice, Data, and Sync have no physical NIC attached. This is a physical impossibility, not a firewall rule. No wire connects MariaDB to the outside world. Even a misconfigured iptables or a kernel exploit cannot reach the database from the internet.
Standard 1500-byte MTU was designed for shared Ethernet in 1982. On an isolated virtual switch, there is no physical constraint. Jumbo frames deliver 6× payload per packet, fewer interrupts per MB, measurable reduction in CPU cycles on MariaDB replication streams.
vmkfstools clones at metadata level — a 30GB VM clones in seconds, not minutes, on NVMe storage. Pages are copy-on-write. Critical: systemd uses /etc/machine-id as DHCP client ID. All clones must reset it via systemd-machine-id-setup to avoid identity collisions.
A managed switch VLAN SVI can occupy a VM's intended IP, winning the ARP race. The VM becomes unreachable from outside but is still reachable from a peer VM on the same L2 segment. SSH ProxyCommand tunnels through the peer to access and reconfigure the conflicted VM.
| Codec | Standard | Where Used | Why |
|---|---|---|---|
| G.711 μ-law | NANP PSTN (North America) | All carrier termination in North America | Mandatory for US/CA PSTN interconnect; no transcoding on external hops |
| G.711 A-law | ETSI PSTN (Europe/Intl) | International carrier termination | Standard codec for non-NANP carriers; passes through without modification |
| G.722 | HD Voice — 16 kHz wideband | Internal VoIP phones on LAN | Zero PSTN interaction — HD voice is only relevant inside the LAN |