Clustering
An aer cluster is a set of machines that run aer cluster agent. Two commands can use the agents:
aer test --clustersends test classes to the agents, so a large test suite runs on the CPUs of several machines.aer server --cluster, on a PostgreSQL database, forwards API requests and async Apex to the agents, which run them against the same database.
An agent runs one test job and one server job at a time, so the same agents can serve a server and run tests.
Set up a cluster
Create a key, once per cluster:
aer cluster keygenMake the key available on every machine in the
AER_CLUSTER_KEYenvironment variable. You can also pass it with--keytoaer clustercommands and--cluster-keytoaer testandaer server, or set either flag in an.aerrcfile. A key given on the command line is visible to other users of the machine in the process list, so prefer the environment variable or an.aerrcfile only you can read.On each machine that should help, start an agent with the same aer release as the machine that starts runs:
aer cluster agentCheck which agents a machine can see:
aer cluster membersThe list shows each agent’s CPU count, whether it is running tests or serving a server, and its aer release.
Machines on one network segment find each other with mDNS. Where multicast is not available, as on most cloud and CI networks and across routers, give the address of any one cluster node instead:
aer cluster agent --join 10.0.0.5 # on each further agent
aer test force-app --cluster-join 10.0.0.5 # on the machine starting a run--cluster-join implies --cluster.
Run tests on several machines
aer test force-app --clusterThe machine that starts the run, the coordinator, loads the source as usual: it parses and type checks the Apex, builds the schema, and prepares the test database. It then sends each agent the loaded program. Agents do not need the source and do not parse or type check anything. Workers on the agents and on the coordinator take test classes from one queue, one class at a time per worker, and report each method’s result as it finishes.
The coordinator prints the results and writes the --json, --junit, and --coverage output, as in a local run. Exit codes are the same as for a local run.
- Failures. A node that cannot be reached, rejects the job, or fails to start is reported on stderr, and the run continues without it. If a node disconnects or stops during the run, the methods it had not reported run on another node. When no agent is usable, every test runs locally.
- Transfers. An agent keeps the files of earlier jobs and asks only for the ones it lacks. The first job for a project includes the test database template, which can be several hundred megabytes for a large project. A later run after an Apex change sends only the program.
- Flags. Flags that change how tests execute, such as
--timezone,--runs,--timeout,--read-only,--heap-limit,--feature, and--default-namespace, apply on every node. Source paths,--path,--package,--package-dir,--bootstrap-db, and the certificate flags are applied on the coordinator, and the agents receive the result..aerrcsettings are read on the coordinator;.aerrcfiles on an agent’s machine do not affect a job. - Time zone. Workers run with the coordinator’s
TZenvironment variable. - Incompatible flags.
--watch,--debug,--db,--profile,--trace, and--integration-testscannot be combined with--cluster.
A run uses the agents that are in the cluster when it starts; an agent that joins later is used by the next run. Agents start working after the coordinator has loaded the source and sent the job, so a short run can finish before the agents are ready.
License
--cluster requires a valid license on the coordinator. The license is sent with the job and covers it on every node, so agents need no license. A license’s test limit applies to the run as a whole.
Serve requests from several machines
aer server --cluster requires a PostgreSQL database that the agents can connect to:
aer cluster agent # on each helper machine
aer server force-app --db postgres://aer:secret@db.internal/aer --clusterClients keep using the server’s address. The server loads the source, migrates and seeds the database, and serves requests as it does without --cluster. It also sends each agent its loaded program and schema, and the agent starts a worker that connects to the same database without migrating or seeding it.
Which requests run on the agents
The server authenticates each request, then runs it itself or forwards it to the worker with the fewest running requests for its share of CPUs. These requests can run on a worker:
- the REST API’s
sobjects,query,ui-api, andactionsresources, andtooling/executeAnonymous - Apex REST (
/services/apexrest/) - Partner and Enterprise SOAP API calls, other than
login,logout, andgetUserInfo
Logins, Metadata API deploys and retrieves, the other Tooling API resources, Bulk API requests, streaming subscriptions (CometD and Pub/Sub API), and the development and Lightning pages run on the server.
When a worker cannot be reached, the server runs the request itself. If a worker stops after a request was sent to it, a GET or HEAD request is run again on the server, and any other request fails with SERVER_UNAVAILABLE, since the worker may already have committed it.
Platform events published on a worker are delivered through the server to its CometD, Pub/Sub API, and empApi subscribers.
Async Apex and Bulk API batches
Every node runs queued queueable, batch, and future jobs, and each job runs once. A batch job’s records are divided into BatchApexWorker jobs of up to 10,000 records each, as in Salesforce, and any node can take one. Each worker job runs its records in execute() scopes of the batch size, and each scope commits on its own. The worker jobs of a batch that is not Database.Stateful run at the same time; those of a Database.Stateful batch run one after another, so each scope sees the state the previous one left. If a node stops while running a worker job, another node resumes it at its next scope. At most five batch jobs are queued or running at once, and more wait in the flex queue, as in Salesforce.
Bulk API requests that create jobs and upload data run on the server, and the batches they queue run on every node. A Bulk API 2.0 ingest job’s data is divided into batches of up to 10,000 records. A batch whose node stops leaves none of its records behind and is run again from the start.
The Apex Jobs page (/lightning/setup/AsyncApexJobs/home) and the Bulk Data Load Jobs page (/lightning/setup/AsyncApiJobStatus/home) show which node ran each job, scope, and batch.
Deploys and reloads
A Metadata API deploy, a package upgrade, or a --watch reload pauses the cluster: the server stops forwarding requests, waits for the forwarded requests to finish, and has each worker finish its running requests and jobs. A worker that does not finish within two minutes is ended. After the change, the server sends each worker the new program, and workers serve requests again once they run it.
Requirements and flags
--dbmust be apostgres://database.postgresqltest://is rejected, because its database is reachable only from the server’s machine.- When the
--dbhost islocalhost, a loopback address, or a Unix socket, only agents on the server’s own machine are used. - Settings that requests run with, such as
--timezone,--locale,--currency,--feature,--assign-perms,--allow-callouts,--allow-email, and--enforce-governor-limits, are sent to the workers. The server alone uses its address, login credentials,--bootstrap-db,--sandbox-post-copy, and--watch. --clusteror--cluster-joinin an.aerrcfile applies toaer serveronly when its--dbis apostgres://database; otherwise the server prints a warning and runs without a cluster. Given on the command line, they make the server fail unless--dbis apostgres://database.- Start an agent with
--no-server-jobsto keep it out of server jobs, for example when it cannot reach the database or should only run tests.
Cluster status
Setup Home (/lightning/setup/SetupOneHome/home) on the server has a Server Cluster section listing every member with its role, membership status, and CPUs. For each agent running the server’s worker, it shows the worker’s state, its running and completed requests, and whether it runs the server’s current program. For an agent running no worker, it gives the reason, such as a different aer release, --no-server-jobs, serving another server, or a database host it cannot reach.
Several runs on one machine
Without a cluster, aer test starts one worker per CPU. When several runs execute on one machine at the same time, they divide the CPUs among themselves instead of each starting a worker per CPU:
- Runs that can use more than an equal share each get an equal share.
- A run that needs fewer CPUs than an equal share gets what it needs, and the rest are divided among the others.
- Every run keeps at least one worker.
With a PostgreSQL --db, each worker counts as two CPUs, since its database process runs alongside it. Two PostgreSQL runs on a 28-CPU machine run 7 workers each; a PostgreSQL run and an in-memory run get 7 and 14.
A run adjusts its worker count once a second, and when a run ends the others use its CPUs within a second. An agent’s jobs take part, with the agent’s --cpus as their maximum. With -vv, a run prints each change of its worker count to stderr. Set AER_HOST_SHARE=off to make a run use all of its workers regardless of other runs.
Agent options
| Flag | Meaning |
|---|---|
--cpus N | CPUs a job may use (default: all) |
--name NAME | Node name, unique in the cluster (default: the host name) |
--bind ADDR | Address to listen on (default: all interfaces) |
--advertise ADDR | Address other nodes use to reach this agent, when it differs from the bind address |
--join ADDR | Address of another cluster node (repeatable) |
--no-mdns | Do not use mDNS |
--data-dir DIR | Where job files are kept (default: aer-cluster in the user cache directory) |
--data-limit MB | Megabytes of job files to keep between jobs; the least recently used are removed first (default: 8192) |
--work-port N | TCP port for jobs and forwarded requests (default: 7951; 0 picks a free port) |
--gossip-port N | TCP and UDP port for cluster membership (default: 7950) |
--no-server-jobs | Refuse server jobs and run test jobs only |
--allow-upgrade | Run a job from a coordinator on another aer release with that release |
Ports
An agent listens on TCP and UDP 7950 for cluster membership and on TCP 7951 for jobs and forwarded requests. The coordinator connects to the agents, so only agents need to accept connections on the work port. Membership traffic goes in both directions; the coordinator binds a free port for it.
Releases
An agent takes jobs only from a coordinator running the same aer release, since two releases can run a test differently. An agent started with --allow-upgrade also takes a job from a coordinator on another release: it downloads that release for its own operating system and architecture from the aer releases on GitHub, as aer upgrade does, and runs the job with it. It keeps each downloaded release in the builds directory of its data directory for later jobs and never removes them; delete the directory to reclaim the space.
Security
- Membership traffic is encrypted with AES-256-GCM under a key derived from the cluster key. A machine without the key cannot join or read the member list.
- Jobs and forwarded requests travel over TLS 1.3, and each side must prove it holds a key derived from the cluster key before any data is sent. A job contains the program, the schema, the test database, the license key, certificate keys, and, for a server job, the database connection string with its password.
- mDNS is plain multicast and is not authenticated. Anyone on the network segment can see which hosts run agents. Use
--no-mdnswith--join, and--cluster-joinfor runs, where that matters. - One key for the whole cluster. There is no per-node identity: anyone with the key can act as an agent and receive jobs, or as a coordinator and run work on every agent, including making an agent connect to a database of their choosing. Removing a machine from the cluster means changing the key on the others.
- Job files are stored unencrypted in the agent’s data directory, readable only by the user the agent runs as. Run the agent as a user you would trust with the source.