Repository navigation
Conversation
This was referenced Sep 10, 2026
Base automatically changed from
fix/opcua-rescan-comms-lost-freeze-source
to
main
September 29, 2026 17:57
A plugin entity's freeze-frame lived in gateway memory only, so a restart replaced it with a read of the plant as it was at the restart, under a new captured_at. The values frozen when the fault confirmed were gone. The core capture now takes an optional EntityFreezeFrameStore through its Hooks, next to it in the core and free of ROS. Every capture writes the fault's frames through to the store, replacing the code's rows, and an evicted fault loses its rows too. The stored frames are loaded on construction, before any capture: past the retained-frame bound the newest codes are kept and the older rows deleted, unreadable rows are skipped and counted. A reloaded frame is served as it was captured, captured_at and capture_origin included. The startup catch-up first drops reloaded frames whose code the fault manager no longer holds in any status, asked through a new KnownFaultCodeLister (nothing is dropped when it cannot answer). A reloaded frame whose fault has a newer first_occurred than the frame belongs to an earlier occurrence: it is read again before the bound counts anything, replaced and marked startup when the read answers, and dropped with a warning naming the fault and the entities when it does not. Each such code is asked once per catch-up, and the truncation warning names up to ten faults left without a frame. SqliteEntityFreezeFrameStore keeps the frames in a file. The node opens it from the new entity_freeze_frame.storage.path, else entity_freeze_frames.db next to triggers.storage.path; with neither, or a file it cannot open, the frames stay in memory and the log says so. StandingFault carries first_occurred_ns, parsed from the ListFaults reply with a range check.
stop.sh removes the gateway container, so faults.db, the rosbags and the entity freeze-frames went with its writable layer. start.sh now mounts the named volume ros2-medkit-opcua-state (OPCUA_DEMO_STATE_VOLUME overrides it) at /var/lib/ros2_medkit, stop.sh keeps it and says how to purge it, and the demo parameters put the freeze-frame store on it. A fault raised before stop.sh then still serves the values frozen when it confirmed after the next start.sh. The demo-faults job checks exactly that: it confirms PLC_HIGH_TEMP, runs stop.sh and start.sh, and requires the same captured_at and no capture_origin once a second alarm shows the startup catch-up is done.
run_freeze_frame_restart_tests.sh boots test_alarm_server and one gateway container that runs fault_manager_node and gateway_node with the state on a named volume, and drives the server's values through its stdin CLI. It checks that a replaced gateway container serves the frame captured at confirmation; that a fault cleared and reported again while gateway_node was down is read again on start and marked capture_origin: startup; that such a frame is dropped with a warning when the server is unreachable, and when the fault manager restarts with empty storage; and each entity_freeze_frame.storage.path case. A check that something did not happen waits for a sentinel fault the drain loop frames after the startup catch-up. The workflow runs it as its own job, and a change to the core freeze-frame capture, its store or the ROS adapter now triggers the workflow.
…ction At the retained-frame bound, a fault confirmed again kept its old place in the eviction order, so the next new fault evicted the frame just captured. After a restart the frames are ordered by captured_at, so a different fault went instead. A recapture now moves its fault to the newest end, which is the order a reload rebuilds.
The SQLite store skipped a row whose frame column is not JSON before the capture could see it, so the "unreadable row(s) skipped" warning counted only rows that were JSON but not shaped like a frame. load_all now returns such a row with a null frame, and the capture skips and counts both kinds.
A fault manager whose storage connection is lost answers ListFaults with an empty list as a success. The startup prune read that as "every fault is gone" and deleted every reloaded frame and its rows for good. The prune now runs only when the fault manager lists at least one fault. An empty list drops nothing and logs one info line. A frame whose fault the fault manager no longer holds then leaves by eviction or by the next capture of its code.
At startup a reloaded frame counted as stale when the fault manager's first_occurred was later than the frame's captured_at. That compared the clocks of two processes, so a clock skew or step either re-read a frame of the same occurrence or kept the frame of an earlier one. Only confirmed faults were judged, so a fault that cleared, confirmed again and cleared again while the gateway was down served the first occurrence's frame. Each stored row now carries the fault manager's occurrence of the fault at capture: occurrence_count and first_occurred, which the fault manager changes only when a cleared fault fails again. A confirm-edge capture takes them from the event, a catch-up capture from the ListFaults reply, both in the reply's encoding so one occurrence gives equal values. At startup every reloaded code is looked up in the all-status fault list: - same occurrence: kept, no read; - another occurrence, confirmed: read again and marked startup; - another occurrence, not confirmed: discarded with a warning that names the fault, the entities and the reason; - a row or a listed fault without an occurrence: kept. The known-fault lister now returns the listed faults with their status and occurrence, parsed by standing_faults_from_list_reply.
…eviction age The restart section of the snapshots tutorial, the entity_freeze_frame.storage.path row and the gateway README and design notes now say that a reloaded frame is judged by the occurrence stored with it, that an empty fault list drops nothing, and that the bound evicts the frame captured longest ago, counting a fault confirmed again from its new capture.
…empty fault list The freeze-frame restart script now also shows, against the real gateway and fault manager: - a fault cleared while the gateway runs keeps its frame and captured_at across a restart; - a fault cleared, confirmed again and cleared again while gateway_node is down does not serve the first occurrence's frame, and the gateway logs a warning that names the fault; - a fault manager with an empty fault list drops no stored frame, and one that holds an unrelated fault drops the frame of a fault it does not hold. The fault list helper asks for every status, so a cleared fault is seen.
The start used two ListFaults calls, a confirmed-only list for the catch-up and an all-status list for the reloaded frames, plus a pass that read stale frames again. Taken at different moments and with different filters they could disagree: a muted confirmed fault kept its stale frame for a re-read that never came, a fault that cleared between the calls was discarded and then read again from the older list, and a failed first call with a working second one kept a stale confirmed frame. Reloaded frames were also served before any of this had run. The capture now makes one call over every status, muted faults included. That answer judges each reloaded frame: the same occurrence is kept with no read, another occurrence is discarded with a warning that names the fault, the entities, both occurrence counts and the status, and a fault the list does not hold is dropped unless the list is empty. The catch-up then reads every confirmed fault that is not muted and has no frame, so a stale confirmed fault gets its startup frame through the normal path. A muted fault is not framed at start, as the fault manager publishes no confirm event for it. frames_for serves no reloaded frame until the judgement has run or the lister has given up. The frames dropped for faults the fault manager no longer holds are logged as one warning that names up to 10 codes. The known-fault lister hook is gone; the standing-fault lister returns every fault with its status, occurrence and muted flag, and a fault without a status counts as confirmed.
REQ_INTEROP_088, the entity_freeze_frame rows in the server configuration, the snapshots tutorial and the gateway README and design notes now say that a fault already confirmed at start is read and marked startup only when no frame of the same occurrence was reloaded, that one fault list of every status judges the reloaded frames before any is served, that a frame of another occurrence is discarded and a confirmed fault then framed by the catch-up, that a muted fault is not framed at start, and that frames of faults the fault manager no longer holds are dropped with a warning that names them.
The freeze-frame restart script follows the start that one fault list decides: a frame from an earlier occurrence is discarded with a warning that names the fault, the entity, both occurrence counts and the status, a confirmed fault then gets its startup frame from the catch-up, and the frames of faults the fault manager no longer holds go with one warning that names them. A new scenario runs the plugin in subscription mode with a 60 s publishing interval: right after the gateway is up, GET /data must serve the current value within 15 s, and a stale frame of a confirmed fault must be replaced at start by a startup frame that holds the current value. The plugin README describes the scenarios and quotes the warning.
In subscription mode the plugin's cache stayed empty until the subscription's first publish, up to one subscription_interval_ms after the subscriptions were created. Until then GET /data served nulls and a startup freeze-frame read found no values. With a 60 s interval the first values came 62 s after the poller started. After the plugin creates its subscriptions, at start and again after each reconnect, it now reads every subscribed point once, so the cache holds current values at once. The README says so for subscription_interval_ms.
A frame held for a fault, reloaded and judged valid or captured in this life, stayed when the fault confirmed as its next occurrence and the read of its entity gave nothing (PLC down, cold cache). The detail of the new occurrence then served the previous occurrence's values and captured_at, and the stored rows still named the previous occurrence. The capture now keeps the occurrence of every held frame. A confirm whose occurrence differs from it and whose read gives no frame drops the held frame and its rows and logs a warning that names the fault and says this occurrence has no freeze-frame. A confirm of the same occurrence, or one that carries no occurrence, keeps the frame as before.
gateway_node and the fault scope code wire the entity freeze-frame capture into the gateway, yet a change to them did not trigger the OPC UA workflow that runs the freeze-frame restart test. Both paths lists now name them, and the header comment says so.
- The REST snapshot types: a fault confirmed before the gateway started is read at start and marked startup only when no frame of the same occurrence was reloaded. - connected: false appears only when the plugin's payload carries a link flag; the OPC UA plugin's DataProvider payload carries none. - A reloaded frame is served only after the start has judged it, except when the host gives no lister or the lister throws or answers with an empty list: then the frames are kept and served unjudged. - The store file belongs to one gateway. Two gateways must not share it, since one gateway's start drops the other's frames.
The read after creating the subscriptions also ran when they were created again after a reconnect. There the alarm event pump keeps iterating the client while the poll thread reads, so a publish handled inside the read could deliver a newer value that the older read result then overwrote, with nothing to correct it until the point changed. The read now runs once at start, before the poll thread and the event pump exist. After a reconnect the cache keeps the last known values until the first publish. The README row for subscription_interval_ms says so.
bburda
force-pushed
the
fix/persist-entity-freeze-frames
branch
from
November 17, 2026 08:49
b4bf815 to
016a68e
Compare
bburda
changed the base branch from
main
to
fix/opcua-demo-runs-a-fault-manager
November 17, 2026 08:51
3 of 7 tasks
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pull Request
Summary
A plugin entity's freeze frame lived only in gateway memory. After a restart the gateway read the plant again and served those values under a new
captured_at, as if they were the values at confirmation.The core freeze-frame capture now writes its frames to a SQLite store when
entity_freeze_frame.storage.pathis set. When the path is empty andtriggers.storage.pathis set, the frames go toentity_freeze_frames.dbin the same directory. Each row records the fault manager's occurrence of the fault (occurrence_countandfirst_occurred).At start the gateway asks the fault manager once for every fault in every status, muted faults included, and judges each reloaded frame:
captured_at;capture_origin: startup;A reloaded frame is served only after that judgement. A live confirmation of a new occurrence whose read gets no values drops the previous occurrence's frame, so the fault has no frame. A recapture counts as the newest frame for the 256-fault bound.
In subscription mode the OPC UA plugin now reads every subscribed point once at start. Before,
GET /dataand the startup capture saw null values until the first publish, which can be a whole publishing interval later. After a reconnect the plugin keeps its last known values until the first publish.The OpenPLC tank demo keeps the gateway state on the named volume
ros2-medkit-opcua-stateacrossstop.shandstart.sh: the entity frames and the fault manager'sfaults.db.Stacked on #790.
Issue
Type
Testing
The new
freeze-frame-restartjob runsrun_freeze_frame_restart_tests.sh. It restarts a real gateway and fault manager against the OPC UA test server and checks:captured_at;entity_freeze_frame.storage.pathcase: a path, empty withtriggers.storage.path, both empty, and a path that cannot be opened.The
demo-faultsjob also checks that the demo's frame keeps itscaptured_atacrossstop.shandstart.sh. The workflow now also runs when the core capture, its store or their wiring ingateway_nodechange.Checklist