Building a Production CLEC Stack: Asterisk 22 + Kamailio 5 + A2Billing on Docker
This article documents the full integration of a production-grade Competitive Local Exchange Carrier (CLEC) platform using Asterisk 22, Kamailio 5, and A2Billing 2.x deployed as a containerized Docker Compose stack. Particular focus is given to twelve integration bugs encountered during the build — most of which are entirely undocumented in official resources — along with their root causes and confirmed fixes.
Asterisk 22 completely removedchan_sip. A2Billing's dashboard, AMI integration, and AGI pipeline all assumedchan_sipwas present. None of this is documented in A2Billing's official resources. Every failure described here required original root-cause analysis.
Architecture
INTERNET / PSTN CARRIERSSIP port 15060Kamailio SBCSIP Proxy · Pike Rate Limiter · Dispatcher · Load Balancerport 15060 UDP/TCP · sip-external networksip-plane · internal Docker networkAsterisk PBX — Node 1wts-pbx-mac-01PJSIP · Extensions · Dialplan · AGIport 5060 · 105 extensionsAsterisk PBX — Node 2wts-pbx-mac-02PJSIP · Extensions · Dialplan · AGIport 5060 · 105 extensionsdata-plane · internal Docker networkMySQL 8.4wts-db-mac-01 · CDR + A2Billing database · data-planeA2Billingwts-billing-mac-01 · Billing portal · AMI · AGI · port 8081Grafana + Prometheus — Monitoring · observability networkDocker networks:
sip-plane— SIP signalling between Kamailio and Asterisk (internal only)data-plane— Database access between Asterisk, MySQL, A2Billing (internal only)sip-external— Internet-facing SIP traffic (Kamailio only)observability— Metrics collection across all containers
Issue 1 — Kamailio Pike Rate Limiter Blocking Load Tests
Pike module throttled all SIPp traffic from 127.0.0.1
SymptomLoad test at 50+ cps showed only ~22 cps effective throughput. Kamailio logs: FLOOD from 127.0.0.1 blocked. Approximately 50% of INVITE requests returned 503.
All SIPp load-test processes originate from 127.0.0.1. Pike applies rate limiting per source IP without distinguishing localhost from external traffic. Multiple SIPp processes hit the threshold in the first 100ms and were blocked for the remainder of the sampling window.
request_route {
if ($si != "127.0.0.1") {
if (!pike_check_req()) {
sl_send_reply("503","Service Unavailable");
exit;
}
}
}Issue 2 — RTP Port Range Mismatch: Silent Calls
Asterisk assigned RTP ports outside Docker's published range — no audio
SymptomSIP signalling completed successfully (INVITE → 200 OK → ACK) but all calls were silent in both directions.
Root CauseAsterisk's default rtp.conf uses ports 10000–20000. Docker Compose only published 10000–10099. Asterisk assigned RTP sessions above port 10099, which Docker silently dropped. No error was logged anywhere.
# rtp.conf — cap range to match Docker port mapping exactly
[general]
rtpstart=10000
rtpend=10099
strictrtp=yes
icesupport=no# docker-compose.yml
volumes:
- ./asterisk/rtp.conf:/etc/asterisk/rtp.conf:ro
ports:
- "10000-10099:10000-10099/udp"Issue 3 — Asterisk Does Not Expand Shell Variables in Config Files
password=${PASS1001} treated as a literal string
SymptomExtension 1001 always returned 401 Unauthorized regardless of the password entered in the phone.
Root CauseAsterisk does not expand shell environment variables in configuration files. The literal string ${PASS1001} was set as the password.
Generate passwords at provisioning time and write them as literal values. Example auto-provisioning script:
PASS=$(openssl rand -base64 16 | tr -d '+/=' | head -c 16)
cat >> /etc/asterisk/pjsip.conf << EOF
[auth${EXT}]
type=auth
auth_type=userpass
username=${EXT}
password=${PASS}
EOFIssue 4 — PJSIP Identify match=0.0.0.0/0 Blocks Phone Registration
All SIP traffic matched to trunk endpoint — phones could never register
SymptomPhone REGISTER requests returned 401 or were silently rejected regardless of credentials.
Root CauseThe PJSIP identify block matched every source IP to the Kamailio trunk endpoint. Individual phone extension endpoints were never evaluated because the catch-all matched first.
[kamailio]
type=identify
endpoint=kamailio
match=0.0.0.0/0 ; ← This matches phones tooFixRestrict the identify match to Kamailio's actual internal IP on the sip-plane network:
[kamailio]
type=identify
endpoint=kamailio
match=<kamailio_internal_ip>Expose Asterisk's SIP port (5060) directly for phone registration, keeping the SBC port (15060) for trunk traffic only.
Issue 5 — Missing [from-phones] Dialplan Context
All calls from registered phones rejected immediately
SymptomPhones registered successfully but dialing any number caused immediate rejection. Asterisk log: No such context 'from-phones'.
The PJSIP phone template sets context=from-phones, but that context was never added to extensions.conf.
[from-phones]
exten => _1[0-9][0-9][0-9],1,Dial(PJSIP/${EXTEN},30,rT)
exten => _1[0-9][0-9][0-9],n,VoiceMail(${EXTEN}@default,u)
exten => _1[0-9][0-9][0-9],n,Hangup()
exten => _NXXNXXXXXX,1,Dial(PJSIP/+1${EXTEN}@kamailio,60,rT)
exten => _NXXNXXXXXX,n,Hangup()
exten => *43,1,Echo() ; Echo test
exten => *98,1,VoiceMailMain() ; VoicemailIssue 6 — A2Billing Config Written to Wrong Path
Error: A2Billing configuration file is missing! on every page
SymptomA2Billing showed the "configuration file is missing" error despite the config existing in the container.
Root CauseA2Billing's PHP source hardcodes the config path to /etc/a2billing.conf:
define('A2B_CONFIG_DIR', '/etc/');
define('DEFAULT_A2BILLING_CONFIG', A2B_CONFIG_DIR . 'a2billing.conf');The Dockerfile was writing the config to /var/www/html/a2billing.conf — the location of the A2Billing source template. The actual read path /etc/a2billing.conf did not exist.
Use a Docker entrypoint script to write the config to the correct path at container startup, reading the database password from a Docker secret at runtime:
#!/bin/bash
DB_PASS=$(cat /run/secrets/mysql_a2billing_password | tr -d '\n')
cat > /etc/a2billing.conf << EOF
[database]
hostname = mysql
user = a2billing
password = ${DB_PASS}
dbname = a2billing
dbtype = mysql
EOF
exec apache2-foregroundIssue 7 — A2Billing Wrong Config Key Names
"Connection failed" — dbuser/dbpassword vs user/password
Symptom"Connection failed" on every page after fixing the config path. ADOdb built the DSN as mysqli://:@mysql/a2billing — empty credentials.
A2Billing's DbConnect() reads $this->config['database']['user'] and $this->config['database']['password']. Using keys dbuser and dbpassword causes both to resolve to null.
; WRONG CORRECT
dbuser = ... → user = ...
dbpassword = ... → password = ...Verify with ADOdb directly inside the container:
php -r "require 'vendor/autoload.php';
\$db = NewADOConnection('mysqli://user:pass@mysql/a2billing');
echo \$db ? 'OK' : 'FAIL';"Issue 8 — MySQL 8.4 Removed mysql_native_password
Attempting to re-enable native_password crashed MySQL entirely
SymptomMySQL 8.4 refused to start after adding mysql_native_password=ON to the config:
ERROR [MY-000067] unknown variable 'default_authentication_plugin=mysql_native_password'Root CauseMySQL 8.4 completely removed mysql_native_password and the default_authentication_plugin system variable. Both were deprecated in 8.0 and fully removed in 8.4.
PHP 7.4's mysqli extension supports caching_sha2_password (MySQL 8.4 default) natively. No plugin change is required. The actual problem was wrong config key names (Issue 7). Revert any mysql_native_password configuration and fix the application config instead.
Issue 9 — Asterisk AMI Disabled by Default
A2Billing had no AMI connection — call origination non-functional
SymptomA2Billing web UI loaded but could not originate calls, run callbacks, or display live call data.
Root CauseAsterisk ships with AMI disabled (enabled = no). No manager.conf was mounted in Docker Compose. A2Billing's database still held default placeholder credentials (manager_host=localhost, manager_username=myasterisk).
# manager.conf
[general]
enabled = yes
port = 5038
bindaddr = 0.0.0.0
[a2billing]
secret = <strong_random_password>
permit = 172.16.0.0/255.240.0.0
read = system,call,log,agent,reporting,cdr,dialplan
write = system,call,agent,originateFor clustered deployments, register all nodes in cc_server_manager:
INSERT INTO cc_server_manager (id_group, manager_host, manager_username, manager_secret)
VALUES (1, 'asterisk-node-1', 'a2billing', '<password>'),
(1, 'asterisk-node-2', 'a2billing', '<password>');Issue 10 — A2Billing AGI Scripts Not Accessible to Asterisk
AGI Script a2billing.php not found — no call billing
SymptomCalls routed to [a2billing] context failed. No call detail records were created in the A2Billing database.
A2Billing's AGI scripts live at /var/www/html/AGI/ inside the A2Billing container. Asterisk looks for AGI scripts in /var/lib/asterisk/agi-bin/ inside its own container. In a containerized deployment these are separate filesystems — the scripts are invisible to Asterisk.
Share the Asterisk data volume with A2Billing in docker-compose.yml, then copy scripts at container startup:
# docker-compose.yml
a2billing:
volumes:
- asterisk-data:/var/lib/asterisk # shared with asterisk service# entrypoint.sh
cp /var/www/html/AGI/a2billing.php /var/lib/asterisk/agi-bin/
cp -r /var/www/html/AGI/lib /var/lib/asterisk/agi-bin/
chmod 755 /var/lib/asterisk/agi-bin/a2billing.php# extensions.conf
[a2billing]
exten => _X.,1,AGI(a2billing.php,1)
exten => _X.,n,Hangup()
[a2billing-did]
exten => _X.,1,AGI(a2billing.php,1,did)
exten => _X.,n,Hangup()Issue 11 — A2Billing Dashboard Uses Removed chan_sip Commands
Asterisk Summary shows SIP Registrations: -2, blank peer lists
SymptomA2Billing's Asterisk Summary page showed SIP Registrations: -2 and empty channel/peer counts on Asterisk 22.
A2Billing's dashboard (A2B_asteriskinfo.php) was written for Asterisk 1.4–1.8. It issues AMI commands that were removed in Asterisk 18+:
"sip show channels" // removed — chan_sip no longer exists
"sip show registry" // removed
"sip show peers" // removed
"iax2 show peers" // chan_iax2 not loadedWhen these return nothing, the parser computes array_count - 3, producing -2.
Replace all chan_sip and iax2 commands with PJSIP equivalents:
"pjsip show endpoints" // replaces sip show peers
"pjsip show registrations" // replaces sip show registry
"pjsip show channels" // replaces sip show channels
"core show channels" // active call count
"core show version"
"core show uptime"Issue 12 — Asterisk 22 AMI Response Format Changed: phpagi Captures No Data
All PJSIP command results showed 0 — phpagi discarded all output lines
SymptomAfter replacing commands with PJSIP equivalents, all counts still showed 0. phpagi's send_request() returned an empty data field.
Asterisk 22 AMI changed the Command action response format. Older Asterisk sent a multi-line block terminated with --END COMMAND--. Asterisk 22 sends each output line as a separate Output: header:
Response: Success
Message: Command output follows
Output: <line 1>
Output: <line 2>
Output: <line 3>
<— blank line terminates packetThe phpagi library stores key-value pairs in a PHP array. Since PHP arrays cannot have duplicate keys, only the last Output: line is retained. For pjsip show endpoints returning 400+ lines, 399 were silently discarded. Additionally, phpagi waited for --END COMMAND-- which never arrives — causing a timeout on every call.
Bypass phpagi for Command actions. Use a raw socket that reads all Output: lines until the blank-line packet terminator. Use a single persistent connection per page load to avoid repeated login overhead (which caused ~15 second page loads):
class AMISession {
private $s = null;
public function connect() {
$this->s = fsockopen(MANAGER_HOST, 5038, $e, $m, 2);
stream_set_timeout($this->s, 2);
fread($this->s, 256); // banner
fputs($this->s, "Action: Login\r\nUsername: " . MANAGER_USERNAME .
"\r\nSecret: " . MANAGER_SECRET . "\r\n\r\n");
$buf = '';
while (!feof($this->s) && strpos($buf, 'Authentication accepted') === false)
$buf .= fread($this->s, 512);
return strpos($buf, 'Authentication accepted') !== false;
}
public function command($cmd) {
fputs($this->s, "Action: Command\r\nCommand: {$cmd}\r\n\r\n");
$data = ''; $t = 0;
while (!feof($this->s) && $t++ < 30) {
$chunk = fread($this->s, 8192);
$data .= $chunk;
// Packet ends with \r\n\r\n in Asterisk 22
if (strlen($chunk) === 0 || substr($data, -4) === "\r\n\r\n") break;
usleep(100000);
}
// Extract all Output: lines
$lines = explode("\r\n", $data);
$out = [];
foreach ($lines as $l)
if (strpos($l, 'Output:') === 0) $out[] = ltrim(substr($l, 7));
return implode("\n", $out);
}
}This reduced the Asterisk Summary page load time from ~15 seconds to under 1 second.
Performance Reference
| Platform | Realistic CPS | Concurrent Calls | Notes |
|---|---|---|---|
| Mac Mini M4 Pro (Docker, ARM) | 50–100 | ~150 | ARM emulation overhead, shared resources |
| x86 bare metal (single node) | 400–500 | 2,000+ | Dedicated cores, no virtualization overhead |
| x86 bare metal (3-node cluster) | 1,200+ | 5,000+ | Kamailio dispatcher across 3 Asterisk nodes |
Load testing with SIPp: the Mac Mini Docker environment correctly validated cluster logic, failover behavior, and the full call flow — but reached a hardware ceiling around 50 cps due to ARM emulation. Bare-metal x86 ESXi is required for production load targets.
Key Takeaways
- A2Billing requires patching for Asterisk 18+. It was written for Asterisk 1.4–1.8 and no official migration guide exists for PJSIP. Every interface — dashboard, AMI, AGI — needs updating.
- Asterisk 22 completely removed
chan_sip. Any tooling referencingsip show *commands will silently fail with no meaningful error. - Asterisk 22 AMI changed the
Commandresponse format. Code waiting for--END COMMAND--will hang. Code reading thedatafield from phpagi will get empty results. ReadOutput:headers directly via raw socket. - MySQL 8.4 removed
mysql_native_passwordentirely. Do not attempt to re-enable it — fix the application's authentication approach instead. - In containerized deployments, no two services share a filesystem. AGI scripts, configuration files, and TLS certificates must be explicitly shared via Docker volumes or copied at startup.
- Docker port ranges have a hard boundary. Asterisk's RTP range must exactly match what Docker publishes — a mismatch produces silent one-way audio with no error messages anywhere in the stack.
- A single AMI connection per page load is critical. Opening a new TCP connection and authenticating for each AMI command produces ~3 second overhead per command. Five commands per page = 15 second page loads. Reuse one session object.
References
- Asterisk PJSIP Configuration: wiki.asterisk.org
- Kamailio Dispatcher Module: kamailio.org
- A2Billing Source: github.com/Star2Billing/a2billing
- Asterisk AMI v2 Specification: wiki.asterisk.org
Ioannis (John) Alexander Konstas — Founder & Managing Principal,
Cybersecurity · Enterprise Infrastructure · Telecommunications Engineering
·