Khôi phục TMap khi MT5 sậpTMap disaster recovery

Problem: the GoldenRobber EA uses Ticket Mapping to track partial closes (on a hedging account a position gets a new ticket after a partial close and loses its original comment). If MT5 crashes and another instance is opened on a different machine, the mapping is lost → the EA no longer knows which round a position belongs to → Tia/SingleRoundMode logic goes wrong.

Solution: store the TMap through 3 parallel methods:

The unique key in the cloud = magic + account. Opening MT5 on another machine with the same config (same magic, same account) makes the EA load the TMap from the cloud automatically. The local file is named by magic only, since it lives on that terminal's own machine.

flowchart LR
  A["MT5 instance A (main)<br/>GoldenRobber"]
  F[("Local file<br/>GR_TicketMap_magic.csv")]
  CF[("Cloudflare Worker → D1<br/>/tmap/save")]
  MY[("fxconsole → MySQL<br/>/easupported/goldenrobber/api/tmap/save")]
  A -- "always" --> F
  A -. "live, only when InpCloudCfDomain is set" .-> CF
  A -- "live, leader only" --> MY

Save: the local file on every call; the cloud on the InpCloudSyncIntervalSec cycle and only when the node is leader. Dashed = not used in production.

flowchart TB
  S(["OnInit on machine B"]) --> L1{"Local file has data?"}
  L1 -- "yes" --> RB
  L1 -- "empty, Sync on, live" --> C1{"CF domain set and load OK?"}
  C1 -- "yes" --> SF
  C1 -- "no" --> M1{"MyApi load OK?"}
  M1 -- "yes" --> SF["TicketMapSaveFile(): local cache"]
  M1 -- "no" --> RB
  SF --> RB["TicketMapRebuild(): add open positions missing from the map"]
  RB --> HB["First heartbeat: leader or follower"]

Loading at startup, as implemented in 1.42. TicketMapRebuild() always runs, not only when every source failed.

1. Local file        → Primary, works in backtests too
2. Cloudflare D1     → Cloud backup #1
3. MySQL REST API    → Cloud backup #2 (in case CF is down)

MT5 Instance A (main)
   ↓ saves in parallel
   ├── Local file (always)
   ├── Cloudflare Worker → D1 (live only)
   └── fxconsole API → MySQL (live only)

MT5 Instance A CRASHES

MT5 Instance B (other machine, same magic+account)
   ↓ load priority
   1. Local file (EMPTY because the machine is new)
   2. Cloudflare D1 → has data ✓ → use it
   3. If CF is down → MySQL → has data ✓ → use it
   4. Fallback: rebuild from position comments

Base URL = InpCloudMyApiDomain + /api/tmap, for example https://fxconsole.gamelau.uk/easupported/goldenrobber/api/tmap. Required header: Authorization: Bearer <secret>, where the secret is the product's token from /admin/easupported (set on the EA as InpCloudBearerToken).

• Errors: 404 Unknown EA when the slug does not exist · 401 Unauthorized when the token is empty or does not match (compared with hash_equals) · 422 when magic/account is missing or data is invalid.
• The routes run without session, cookies, CSRF or Inertia (routes/web.php, prefix easupported/{slug}/api/tmap); bearer authentication happens in the controller.
• save validation: magic string ≤32, account string ≤64, entries array; each entry has ticket, round_id, add_num (integer), is_dca_pos (boolean), pos_type (0 or 1), comment (optional, ≤255).

The EA pushes its full ticket map every InpCloudSyncIntervalSec (5–10s) even when nothing changed, and save used to delete everything + insert everything each time: 2N rows written per sync, measured at about 10 writes/second, ~873k writes/day, producing roughly 3 GB of binlog per day on a 179 MB database and eventually filling the disk.

• The server computes the fingerprint sha1(magic|account|json(normalised, sorted entries)) — independent of entry order, because the EA does not guarantee a stable order between pushes.
• The fingerprint is stored in the Redis key tmap_fp:{magic}:{account} with a TTL of 3600s: even if a fingerprint outlives its rows, the map is still rewritten at least once an hour.
• Fingerprint matches → returns "skipped":true with no DB write. Successful write → the new fingerprint is stored. clear → the fingerprint is deleted.
• Redis unreachable → fail open: the DB write still happens; a write is never skipped on a guess.
• Applies to both the per-EA endpoints (EaSupportedApiController) and the legacy endpoints (TmapApiController).

Per-EA endpoints (current). Replace <secret> with the product's token; never paste the token into docs or a shared shell history.

LEGACY Global endpoints from the original guide (your-server.com is a placeholder):

Indexes: UNIQUE(magic, account, ticket) + INDEX(magic, account). Leader-election nodes live in the tmap_nodes table (magic, account, node_id, priority, last_heartbeat).

All paths are relative to monitor/fxconsole/.

1. After pulling new code: php artisan migrate (or bash deploy.sh on the server). If config was added: php artisan config:clear.
2. Create or open the EA product on /admin/easupported and copy the URL https://fxconsole.gamelau.uk/easupported/{slug} and the secret token into InpCloudMyApiDomain / InpCloudBearerToken. The /admin/settings page no longer has a TradingMap Secret field.

LEGACY Old setup for /api/tmap/*: add TMAP_SECRET_KEY to fxconsole's .env, then migrate. The legacy endpoints still honour this value if an old client relies on it.

Folder cloudflare-tmap-worker/:

Service page: Cloudflare TMap worker.

1. Install the Wrangler CLI. A browser opens to log in to the Cloudflare account.

   cd cloudflare-tmap-worker
   npm install
   npm install -g wrangler
   wrangler login

2. Create the D1 database.

   npm run db:create

   Sample output:

   ✅ Successfully created DB 'gr-tmap'
   [[d1_databases]]
   binding = "DB"
   database_name = "gr-tmap"
   database_id = "abc123-def456-7890-..."

   Copy the database_id and paste it into wrangler.toml (replacing REPLACE_WITH_YOUR_D1_DATABASE_ID):

   [[d1_databases]]
   binding = "DB"
   database_name = "gr-tmap"
   database_id = "abc123-def456-7890-..."  # ← paste here

3. Apply the schema (create tables).

   npm run db:migrate

4. Set the secret. Enter it when prompted. It MUST equal the token the EA sends in InpCloudBearerToken. LEGACY The original guide required it to match TMAP_SECRET_KEY in fxconsole's .env and the EA input InpCloudSecretKey.

   npm run secret:set

5. Deploy. The printed URL is the value for InpCloudCfDomain (called InpCloudCfUrl in the original guide); the EA appends /tmap/{save,load,clear,heartbeat} itself.

   npm run deploy

   ✨ Deployed gr-tmap triggers (1 sec)
     https://gr-tmap.your-account.workers.dev

• D1: 5GB storage, 5M reads/day, 100k writes/day.
• Workers: 100k requests/day.

The original guide's estimate: the EA saves every 30 seconds + partial-close events ≈ 3000 requests/day → within the free tier. Note: with a 10-second cycle as in pass58, each cycle sends 1 heartbeat + 1 save, i.e. about 17,280 requests/day per instance (arithmetic estimate) — still below 100k, but well above the guide's figure.

LEGACY The input group as originally designed in the guide (before v1.38 renamed them to *Domain, and before per-product tokens):

Multi-node inputs and leader election: see Integration standard §10. Every input: Input parameters.

Open MT5 → Tools → Options → Expert Advisors → Allow WebRequest for listed URL and add the origin of every backend in use (origin only, no path). Production needs only https://fxconsole.gamelau.uk; the original guide adds both:

When InpCloudSyncEnable = true, the panel shows the HTTP code of the latest save to each backend: OK for 2xx, - when never called (or the domain is empty), otherwise the error code. Lime when every configured backend is OK, orange-red when any backend fails.

Line 1: production (CF empty). Line 2: both backends OK. Line 3: Cloudflare returned 500. Line 4: the WebRequest to MyApi failed (-1).

• TicketMapSave(): always writes the local file; calls CloudSyncPulse(). Called after a new init, an added DCA, a Tia close, and every 30 seconds in the map clean-up step in OnTick.
• CloudSyncPulse(): throttled by InpCloudSyncIntervalSec, called from both OnTick and OnTimer (so the heartbeat keeps its cycle without ticks). Each pass: heartbeat first (leader + receive config), then only the leader saves to the cloud.
• TicketMapSaveImmediate(): bypasses the throttle — after PositionClosePartial (to record the new ticket) and in OnDeinit. A follower still writes the local file but does not sync to the cloud.
• Clearing the cloud: when the map has just dropped to 0 (clean state), TicketMapClearCloud() sends DELETE to every backend with a domain.
• Save sends the whole map (replace all), never appends → no merge conflicts. The EA sends every cycle even when the map is unchanged; the server skips the DB write (deviation D5).

Comments in the block above: TicketMapSave — file every time; cloud via CloudSyncPulse (throttled by InpCloudSyncIntervalSec); TicketMapSaveFile — ALWAYS, backtests included; TicketMapHeartbeat — CF first if it has a domain, then MyApi; TicketMapSaveImmediate — critical events (partial close, OnDeinit).

Reading the block above: 1 — local file (fastest); 2 — only if local is empty, sync is on, live, and InpCloudCfDomain is set; 3 — if CF has no data or fails; when 2 or 3 succeeds, TicketMapSaveFile() caches it locally; 4 — ALWAYS runs, adding open positions missing from the map; 5 — the first heartbeat decides leader / follower before any trading.

LEGACY The original sketch from the guide (the real code differs in two ways: TicketMapRebuild() always runs as a supplement, and only the leader saves to the cloud):

When nPos == 0 (clean state): call TicketMapClear() on all 3 sources to reset. (Sketch comments: “nhanh nhất” = fastest; “Rebuild từ comment positions” = rebuild from position comments.)

Machine A: GoldenRobber, magic=21478308, account=12345, InpCloudCfDomain empty
           Every InpCloudSyncIntervalSec (pass58 = 10s): heartbeat → if leader: save to fxconsole
           (the server skips the DB write when the map is unchanged)
           Panel: Cloud: CF=- | MyApi=OK

Machine A CRASHES

Machine B: attach the EA with the same .set (same magic, account, InpCloudMyApiDomain, token)
           OnInit:
             ├─ Local file: EMPTY (new machine)
             ├─ Cloudflare: skipped (empty domain)
             ├─ MyApi: load OK ✓ → TicketMapSaveFile() caches locally
             ├─ TicketMapRebuild() fills in from comments
             └─ First heartbeat
           Leader: A stops heartbeating; once A's last heartbeat is older than
           InpCloudHeartbeatTimeoutSec (90s), B is the only alive node → B becomes leader and starts trading / saving

Machine A: GoldenRobber running with magic=21478308, account=12345
           TMap saved regularly → Cloudflare D1 + fxconsole MySQL

Machine A CRASHES (Windows crash, power cut, internet down...)

Machine B: the user attaches the EA with the same config (magic=21478308) on account 12345
           OnInit:
             ├─ Local file: EMPTY (the new machine has no file yet)
             ├─ Cloudflare D1: load OK ✓ (X entries)
             ├─ TicketMapSaveFile() to cache locally
             └─ TicketMapRebuild() fills in from comments

           → the EA keeps managing tickets as if nothing had crashed

Machine A: Save → CF error → MyApi OK ✓
           Local file + MyApi still save successfully
           Panel shows: Cloud: CF=500 | MyApi=OK (red)

Machine B (just started):
           OnInit → CF fails → falls back to MyApi → load OK ✓

Machine A: Save → both fail → only the local file saves successfully
           Panel: Cloud: CF=-1 | MyApi=-1 (red)
           The EA logs a warning and keeps running normally (local file OK)

Machine B (just started):
           OnInit → CF fails → MyApi fails → falls back to TicketMapRebuild()
           Rebuilds the map from the comments of open positions
           → earlier partial-close data is lost, but the EA still tracks current positions

In production (CF empty), “MyApi down” lands directly in this branch. If fxconsole stays down, once heartbeats have failed for more than 2 × InpCloudHeartbeatTimeoutSec the EA takes leadership itself so positions are not abandoned.

Delete entries not updated for more than 30 days:

1. Per-product token: each EA product has its own secret (easupported.secret_token). LEGACY The original guide used one identical secret in all 3 places (EA input, Cloudflare Worker secret, fxconsole .env).
2. HTTPS only — never HTTP.
3. Rotating the token: click Regenerate on /admin/easupported, then update the .set on every terminal (and the Worker secret if CF is enabled) at the same time. LEGACY Original guide: update all 3 places at the same time.
4. Whitelist the URLs in MT5 Options → Expert Advisors.
5. The account number is a string in the API — if the broker changes its format, this needs updating.
6. Never put the token in docs, logs or committed .set files; use <secret> in examples.

1. Check the WebRequest whitelist in MT5 Options.
2. Check InpCloudSyncEnable = true.
3. Read the MT5 log (Experts tab) — it will contain an error message.
4. Test the API directly with curl (examples).
5. Panel MyApi=-1: network error, or the host is not in Allow WebRequest.

• 404: the slug in InpCloudMyApiDomain does not exist on /admin/easupported.
• 401: InpCloudBearerToken differs from the product's token (perhaps Regenerate was just clicked). Copy the token into the .set again.
• Domain without /easupported/goldenrobber: the EA is calling the legacy /api/tmap/* endpoints, which use the global secret.

• TMAP_SECRET_KEY in .env differs from the EA input.
• Reload the Laravel config: php artisan config:clear.

• The secret in the EA differs from the one set via wrangler secret put.
• Set it again: npm run secret:set.

• Check that database_id in wrangler.toml is correct.
• Run npm run db:migrate to make sure the schema is applied.
• Test a save with curl and look for errors.

(“File này” = this file.) Added since the per-EA endpoints: app/Http/Controllers/Api/EaSupportedApiController.php, routes/web.php, database/migrations/2026_08_29_000007_add_secret_token_to_easupported.php (see Related files).