Các broker realtimeRealtime brokers

• Each broker is its own docker-compose stack joined to the external fxconsole_default network to reach fxconsole-redis and fxconsole-web. Always bring the fxconsole stack up first.
• Dockerfile node:20-alpine + openssl; start command node gen-cert.js "${CERT_HOST:-localhost}" && node broker.js. First boot writes a self-signed certs/server.key + certs/server.cert, RSA 2048, valid 10 years (3650 days), CN = CERT_HOST. Existing certs are kept. The ./certs volume must be writable.
• Changing the CN: docker compose down, delete certs/, docker compose up -d.
• REDIS_PREFIX (default laravel-database-) must match Laravel's redis.options.prefix. TickBroker writes unprefixed keys.
• Brokers that read settings use MySQL settings → Redis ${REDIS_PREFIX}app_config:<key> → env, 30s per-key cache, DB failures retried after 5s, NULL/blank = not set (since commit cb272d5).
• Running node broker.js directly does not read .env — set variables in PowerShell, e.g. $env:REDIS_URL="redis://127.0.0.1:6379"; node broker.js.
• MT5: SocketConnect() only reaches hosts listed in Tools → Options → Expert Advisors → Allow WebRequest for listed URL.

# Whole system (fxconsole + 4 brokers)
bash deploy.sh

# A single broker
cd monitor/fxconsole && docker compose up -d     # creates fxconsole_default
cd ../<Broker>
cp .env.example .env                            # fill secrets + DB_*
docker compose up -d --build
docker compose logs -f

deploy.sh calls compose_up for AccountStatusBroker, TickBroker, VpsStreamBroker, EaLinkBroker. Before each up, ensure_env: (1) no .env → creates it from .env.example with every change_me… value left empty (never boot with a guessable secret), chmod 600; (2) if .env.example has DB_HOST → fills DB_* from monitor/fxconsole/.env (DB_DATABASE→DB_NAME, DB_USERNAME→DB_USER); an existing file only gets empty keys filled, never overwritten. The script ends by listing fields to fill by hand.

Receives realtime ticks, symbol lists and the MT5 economic calendar from the TickStreamerPub EA; writes Redis Streams for the Charts page and fans ticks out over WebSocket. Source: monitor/TickBroker/broker.js.

First line: HELLO|tickpub|<account_id>|<shared_secret>|<broker_name>. Wrong secret or format → the broker writes ERR|auth and closes. The broker name comes from HELLO and is pinned for the connection (max 64 chars; | and newlines become spaces). Then JSON Lines:

• Each calendar event has id (MT5 value id, stable per release), eid (event id, stable per indicator), cur, ctry, code (ISO 3166 alpha-2), imp (0 none / 1 low / 2 moderate / 3 high), name, time (unix seconds), actual, forecast, prev, unit, mult. At most 4000 events per snapshot.
• seq/total are optional: TickStreamerPub ≥ 1.02 splits a multi-day window across frames because one line is capped by the 64 KiB buffer. The broker reassembles per connection and writes Redis only when every part has arrived; a new seq 0 discards the buffered parts; a missing middle part skips that cycle. Version 1.01 sends one frame without these fields → treated as a one-part snapshot.
• tzoff (≥ 1.03): seconds the trade server's clock runs ahead of GMT. time is an MQL5 datetime on the server clock, not UTC, so it is off by exactly this much. Not sent → null; consumers cannot correct and must say so.
• mode (≥ 1.04): full replaces the snapshot; delta (every ~30s, only a few hours around now) is merged into the snapshot by id (or time|cur|name without an id). Frames without mode are always full. Deltas are logged only with BROKER_DEBUG=1.

After subscribing, the client immediately gets the last tick (XREVRANGE … COUNT 1). The WS has no auth. The server pings every 30s and terminates sockets that don't pong.

# TLS reachable from outside
openssl s_client -connect fxconsole.gamelau.uk:9600 -servername fxconsole.gamelau.uk

# Logs (expect: "TickBroker pub TCP listening on 0.0.0.0:9600 (TLS)")
cd monitor/TickBroker && docker compose logs -f tick-broker

# Last tick and watchlist for one broker
docker exec -it fxconsole-redis redis-cli XREVRANGE "ticks:<broker>:XAUUSD" + - COUNT 1
docker exec -it fxconsole-redis redis-cli HGETALL "broker_latest:<broker>"

Receives account status from the AccountStatusUpdater EA over TLS-TCP (TCP_SOCKET / HYBRID transports), writes Redis with exactly the schema of App\Services\AccountStatusStore::upsert so the dashboard's polling endpoints stay fresh, and fans out over WebSocket to dashboards in "Stream (socket)" mode. It is a parallel path, not a replacement: lot_history inserts and EquityAlertService still need the HTTP control plane (HYBRID). Source: monitor/AccountStatusBroker/broker.js.

• composite_id = slug(ACCOUNT_COMPANY)::slug(ACCOUNT_SERVER)::login, regex ^[a-z0-9-]+::[a-z0-9-]+::[A-Za-z0-9_-]+$, max 160 chars. account_id = the last segment ([A-Za-z0-9_-], ≤ 32) so Redis stays keyed by login.
• secret must equal BROKER_SECRET; broker name ≤ 64 chars.
• Anything wrong → the broker writes the raw line ERR|auth_or_unsupported_hello_version and closes. There is no hello_ok frame.
• Valid HELLO: any older socket with the same composite_id is closed; the broker immediately sends stream_on (if someone is watching) or stream_off.

The gate is global: when total subscribers (per-key + admin) go 0 → N every connected EA streams full payloads; N → 0 sends every EA back to HB. Frames are sent only on the edges. Rationale: subscriptions today are wildcard-style (per key / admin), and an admin on the dashboard usually wants all accounts live; per-account gating is a later refinement. There is no application-level ping — only 15s TCP keep-alive.

Server frames have the same shape as /dashboard/accounts/{id}/data: the Redis row + last_payload_at/updated_at formatted dd-MM-yyyy HH:mm:ss in the container TZ, broker_time_at on the broker clock, plus positions, experts, today_history. permessage-deflate (level 3, frames < 256 bytes uncompressed). 30s ping.

openssl s_client -connect fxconsole.gamelau.uk:9602 -servername fxconsole.gamelau.uk
cd monitor/AccountStatusBroker && docker compose logs -f account-status-broker
# Expect:
#   settings DB: on (<host>:<port>/<db>)
#   AccountStatusBroker pub TCP listening on 0.0.0.0:9602 (TLS)
#   Subscribed to Redis channel: report:request
#   [acct=<login>] statuspub connected composite=<composite_id> broker="…" stream=off
docker exec -it fxconsole-redis redis-cli GET "laravel-database-account_status:<login>"

Long-lived TLS-TCP channel between vps-agent (HeartbeatAgent, through stunnel on the VPS) and fxconsole, replacing HTTPS polling for heartbeats and command push; plus a WebSocket for the /vps and /deployments pages. Source: monitor/VpsStreamBroker/broker.js, monitor/VpsStreamBroker/README.md. Agent side: VPS agent.

Agents used to poll /api/vps/heartbeat + /api/vps/poll-commands over HTTPS every 5 seconds. At 100+ VPSs: ~1.5 KB/poll × 12 polls/min ≈ 18 KB/min per VPS just to say "alive"; every heartbeat boots a full Laravel app in a PHP-FPM worker; a Deploy click reaches the agent only after up to one poll interval. The broker does one TLS handshake and then exchanges tiny frames (~80 bytes).

Two separate channels: the stream carries only heartbeats, command pointers and event ACKs (no secrets besides the TLS-protected secret in HELLO); fxconsole HTTPS carries sensitive payloads (decrypted MT5 password, package zip, broker credentials) — Crypt::decryptString runs only in Laravel. If the broker dies the agent falls back to HTTPS polling; nothing breaks.

Measured after rollout: command latency from ~5s to <500ms; the UI moves queued → in_progress → success within 1s of clicking Deploy.

• Terminal events: the broker calls POST /api/internal/broker/command-ack (X-Broker-Secret, body {vps_id, cmd_id, success, log, error}) so Laravel runs the cascade (vps_commands, mt5_accounts, observers → vps:state, emails, migration stage 2; the handler is idempotent). If the callback fails, the broker runs UPDATE vps_commands itself (only when not yet terminal) and warns "mt5_accounts NOT cascaded -- check broker_internal_secret and fxconsole reachability": the account stays stuck at pending. vps_cmd is always broadcast to admins.
• Stream gate: total WS subscribers (admin + per-key) 0 → N sends stream_on to every agent (full WMI payload + logins), N → 0 sends stream_off. Only frame contents change, not cadence.
• Command push: Laravel PUBLISH laravel-database-vps:cmd {"vps_id","cmd_id","kind"} (VpsCommand::booted() on every insert). Agent not connected → nothing happens; the agent gets it over HTTPS.

Port 9606 is only exposed on the docker network; browsers use wss://<host>/vps-ws → openresty (real cert) → fxconsole-web nginx → broker (proxy_ssl_verify off). Same subscribe shape as AccountStatusBroker, so the useStreamSocket hook is shared.

Every publisher swallows Redis errors at WARN: if the broker is down the UI degrades to polling and requests don't fail. /vps and /deployments default to Live (WS) for admins; a green dot = frames flowing, pulsing amber = connecting; users without the admin token only get 5s/10s/30s/60s/Off.

On each Windows VPS, prepare-vm.ps1 installs stunnel 5.78 (C:\VpsAgent\stunnel), calls /api/agent/broker-config with X-Vps-Secret, writes broker-ca.cert + stunnel.conf (client, accept 127.0.0.1:9605, verify=2 + checkHost), adds a Startup shortcut and smoke-tests Test-NetConnection 127.0.0.1 -Port 9605. -SkipBroker skips the stunnel part. Details: VPS agent.

# From outside the LAN: prints the cert then hangs (waiting for HELLO). Ctrl-D to quit
openssl s_client -connect fxconsole.gamelau.uk:9605 -servername fxconsole.gamelau.uk

# Fake a command (the "cmd push …" log appears only with BROKER_DEBUG=1)
docker exec -it fxconsole-redis redis-cli \
  PUBLISH laravel-database-vps:cmd '{"vps_id":42,"cmd_id":99999,"kind":"test"}'

# Agent heartbeat (refreshes at the agent cadence, expires 15s after the last write)
docker exec -it fxconsole-redis redis-cli GET laravel-database-vps:agent:42:heartbeat

# Fake heartbeat: /vps/1 shows the fake numbers, reverts to DB values after 15s
docker exec -it fxconsole-redis redis-cli SETEX "laravel-database-vps:agent:1:heartbeat" 15 \
  '{"vps_id":1,"cpu_pct":42.0,"ram_pct":12.3,"disk_pct":67.0,"net_pct":1.1,"accts":3,"agent_v":"fake","ts":1717400000}'

# Watch state events
docker exec -it fxconsole-redis redis-cli SUBSCRIBE laravel-database-vps:state

WS smoke test: open /vps as admin in two tabs, click Disable on a VPS in tab 1 → tab 2's badge flips within ~100ms. If it waits for the 5s poll: check the nginx /vps-ws block, the WS listening log on 9606, the Network → WS tab for a subscribe_all frame with no error back, and SUBSCRIBE laravel-database-vps:state.

The "EA Link" TLS-TCP channel between MT5 EAs (GoldenRobber 1.42+ and future products) and fxconsole. Each running EA keeps one connection. An admin changes hot-reload config → Laravel publishes a Redis signal → the broker pushes a tiny cfg frame to the matching sockets within milliseconds → the EA pulls the new config over its HTTPS heartbeat. The broker also keeps presence in Redis (which EAs are connected). Source: monitor/EaLinkBroker/broker.js, monitor/EaLinkBroker/README.md, EA-HOTRELOAD-STANDARD.md §9, §15.3.

NDJSON UTF-8, one object per line, \n terminated. Max line 8 KiB (longer → bye line_too_long), max buffer 64 KiB per client. Unknown t values are ignored; more than 5 unparseable frames → bye bad_frames.

The HMAC key is the hex string as bytes, not the 32 decoded bytes. The raw token never crosses the socket; the broker itself only holds sha256(token). Signatures are compared with timingSafeEqual.

Link-key lookup (broker → Laravel):

Concurrent lookups for the same slug share one request, so a broker restart with 100 EAs of one product costs one call. The secret is read from MySQL settings (key = broker_internal_secret, needs DB_*) → Redis ${REDIS_PREFIX}app_config:broker_internal_secret (1s timeout) → env BROKER_INTERNAL_SECRET, 30s cache. Same secret as VpsStreamBroker. While a HELLO waits on the lookup, its deadline is extended by BROKER_INTERNAL_TIMEOUT_MS + 1s.

Registry: topic = ea/{slug}/{composite_id}/{magic}; inside a topic each socket is keyed by {node_id}#{inst}. A reconnect with the same key replaces the old socket; a different inst (leader/follower, several charts) co-exists and every one receives cfg. EA side (Include/EaLink.mqh): 2s connect timeout, handshake must finish within 10s, backoff 2→4→8…60s ±20% jitter (reset after 30s stable UP), bad_sig/unknown_ea wait 5 minutes.

cd monitor/fxconsole && docker compose up -d        # creates fxconsole_default first
cd ../EaLinkBroker
cp .env.example .env                                # fill DB_*; BROKER_INTERNAL_SECRET optional
docker compose up -d --build
docker compose logs -f ea-link-broker
# Expect:
#   EaLinkBroker listening on 0.0.0.0:9607 (TLS) hb_timeout=60s prefix="laravel-database-"
#   PSubscribed to *ea:link:cfg (subscriptions=1)

# TLS + challenge from outside (after 5s you get hello_err timeout and the connection closes)
openssl s_client -connect fxconsole.gamelau.uk:9607 -quiet

# Full auth + hb + cfg with the bundled client (or: docker compose exec ea-link-broker node test-client.js …)
node test-client.js --host fxconsole.gamelau.uk --port 9607 \
  --slug goldenrobber --token '<secret>' \
  --composite vantage::vantagemarkets-live-3::25938032 --magic 21478308 \
  --node ops-test --inst 1
# (--key <64hex> passes the link key directly instead of the token)

# In another shell: fire a signal
docker exec -it fxconsole-redis redis-cli PUBLISH laravel-database-ea:link:cfg \
  '{"slug":"goldenrobber","targets":[{"composite_id":"vantage::vantagemarkets-live-3::25938032","magic":"21478308"}],"rev":1}'
# The client prints {"t":"cfg","rev":1} and answers cfg_ack; broker logs "cfg rev=1 slug=goldenrobber targets=1 delivered=1"

# Presence
docker exec -it fxconsole-redis redis-cli HGETALL \
  'laravel-database-ea_link:presence:goldenrobber:vantage::vantagemarkets-live-3::25938032:21478308'

Restart: docker compose restart ea-link-broker. EAs get bye shutdown, reconnect with backoff and lose nothing — config is always recoverable over the HTTPS heartbeat.

Firewall: accept TCP/9607 on OVH, port-forward WAN:9607 → <docker host LAN IP>:9607 on IPFire, and add the broker hostname to Allow WebRequest on every terminal (the same list the EA's HTTPS heartbeat already needs).