Skip to content

gateway: show shared and per-source fault records once per entity - #700

Draft
bburda wants to merge 23 commits into
feat/fault-identity-owner-keyfrom
feat/fault-identity-gateway-owner
Draft

bburda wants to merge 23 commits into
feat/fault-identity-owner-keyfrom
feat/fault-identity-gateway-owner

Conversation

@bburda

@bburda bburda commented Sep 25, 2026 •

Copy link
Copy Markdown
Collaborator

Pull Request

Summary

The gateway side of per-source fault records. #699 lets a fault code keep one record per reporting source
(per_source: true on the code); this PR shows those records through the SOVD fault routes. In every entity a fault
code appears once:

  • A code with one shared record (the default) looks as on main. No owner field appears in any response.
  • An App shows the record of its own source. Records reported under sub-sources of one App (<app fqn>/<suffix>)
    show as one entry.
  • A Component, Area or Function shows one entry per code for all records under it: the most advanced status, the
    highest severity, the summed occurrences, the earliest and latest occurrence, and x-medkit.records with each
    record's App, source and status. A merge of one record is that record unchanged.
  • GET {entity}/faults/{code} returns that entry, with the environment data of the record confirmed last.
    DELETE {entity}/faults/{code} clears every record of the code under the entity.
  • GET /api/v1/faults lists one item per record.
  • DELETE {entity}/faults answers 500 with failed_codes when a record could not be cleared. Before, it answered 204
    and logged a warning.

Also in this PR:

  • Recordings: a recording downloaded by id is served through an entity only when the entity's fault scope holds one
    of its records' sources.
  • Plugin API version 8: FaultProvider gains clear_fault_record(entity_id, fault_code, owner); its default calls
    clear_fault(entity_id, fault_code). Plugin source written against version 7 compiles unchanged; a pre-built
    version 7 binary is rejected by the loader.
  • OPC UA plugin: clears the record it raised, and answers its status route for the component ids it exposes.
  • sovd_service_interface: each listed Fault carries the owner of its record.
  • The fault-trigger engine reads an entity's values through its DataProvider first, so a rule on a provider-served
    entity fires.
  • Freeze frames are kept per record.
  • OpenAPI, REST reference, tutorials and READMEs describe the above.

Stacked on #699. Review that one first.


Issue


Type

  • Bug fix
  • New feature or tests
  • Breaking change
  • Documentation only

Testing

colcon build --symlink-install
colcon test --packages-select ros2_medkit_gateway ros2_medkit_opcua ros2_medkit_sovd_service_interface --parallel-workers 1
colcon test --packages-select ros2_medkit_integration_tests --ctest-args -R "fault|rosbag|owner|trigger|sse|bulk"

src/ros2_medkit_integration_tests/test/features/test_faults_owner_identity.test.py runs a real gateway and fault
manager: a shared code reported by two Apps, a per_source code from two Apps and from two sub-sources of one App,
and the entity, global and per-code routes for each.


Checklist

  • Breaking changes are clearly described (and announced in docs / changelog if needed)
  • Tests were added or updated if needed
  • Docs were updated if behavior or public API changed

// newer state of the other would delete a live fault from a client's view.
auto supersede_key = [](const QueuedEvent & queued) {
return queued.peer + '\0' + queued.event.fault.fault_code;
const auto & sources = queued.event.fault.reporting_sources;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

on_peer_eventreads onlyfault_codeback out of a relayed payload, neverreporting_sources, so for every relayed entry owneris empty here and the key collapses to (peer, code): a peer's two records of one code still coalesce into one, and a lagging client loses one owner's last update uncounted, which is exactly what this commit removes for local events. Readfault.reporting_sources(orsource_id) back into queued.event.faultinon_peer_eventand cover it with a relayed variant ofAnotherOwnersEventsDoNotSupersedeARecordsLastUpdate`.

// shows first, then the muted ones, then the cleared or healed ones, so
// neither a muted record nor one a source cleared turns the record a client
// read off the list into an ambiguous address.
auto result = fault_mgr->list_faults("", /*include_prefailed=*/true, /*include_confirmed=*/true,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Every per-code GET and DELETE now pulls the whole store (all statuses, muted included) through ListFaults before the one GetFault it used to make, and the plugin clear branch and the recording download repeat the same listing, so with cleared records retained indefinitely each request costs O(history) on routes a stream consumer hits once per event. Call get_fault_with_env(fault_code, "") first, which the manager answers only when the code names exactly one record, scope-check that record, and take the full listing only when that call does not succeed: the tiers only matter with two or more records, so nothing observable changes.

auto held = fault_mgr->list_faults("", /*include_prefailed=*/true, /*include_confirmed=*/true,
/*include_cleared=*/true, /*include_healed=*/true,
/*include_muted=*/true, /*include_clusters=*/false);
if (!held.success) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Plugin DELETE now returns 503 when ListFaults fails, but plugin GET still falls through to the provider in that case. On a plugin-only box (UDS DTCs, no fault_manager node) read works and clear of the same DTC 503s. Make both paths agree: fall through with an empty owner here, or 503 in get_fault too.

make_error(409, ERR_AMBIGUOUS_FAULT, "Fault code addresses several records in this entity",
json{{"details",
"Several sources this entity owns report this fault code, and each is its own record. "
"parameters.owners names them. Address one through the route of the app that owns it, "

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The hint points at /apps/{app_id}/faults/{fault_code}, but owners are reporting sources: a node FQN for a ROS app, and a bare component id for an external component, which has no app route. If an external component and one of its apps report the same code, the component's own record cannot be read or cleared by code at all. Return each owner's entity route (build_source_entity_map already resolves it), or let the caller pick the owner.

auto held = fault_mgr->list_faults("", /*include_prefailed=*/true, /*include_confirmed=*/true,
/*include_cleared=*/true, /*include_healed=*/true, /*include_muted=*/true,
/*include_clusters=*/false);
const json listing = held.success ? held.data : json::object();

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A failed list_faults becomes an empty listing, so requested_records and candidate_owners are empty and a bag this entity owns answers 404 "not found for this entity". Return classify_fault_failure (503) on !held.success, like the fault routes, and declare it on the download routes.

// shows first, then the muted ones, then the cleared or healed ones, so
// neither a muted record nor one a source cleared turns the record a client
// read off the list into an ambiguous address.
auto result = fault_mgr->list_faults("", /*include_prefailed=*/true, /*include_confirmed=*/true,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Every per-code GET/DELETE now lists the whole store (all entities, cleared and healed history, muted) and then does a GetFault, both under the transport's executor mutex. Cost grows with retained history. List per scope source (list_faults(source)) or add a fault_code filter to ListFaults. Same full read in the plugin clear (997), download (494) and descriptor listing (376).

}
}
if (lock_mgr) {
auto it = source_to_entity.find(owner);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Exact find(owner) misses path-child sources, which the per-entity scope gives to the parent app. A record owned by /ns/app/child is cleared through a lock on /ns/app, contrary to the new locking.rst line. Match with the same rule as source_matches_scope (exact or strict path child).

{
auto faults_result = fault_mgr->list_faults("", /*include_prefailed=*/true, /*include_confirmed=*/true,
/*include_cleared=*/true, /*include_healed=*/true,
/*include_muted=*/false, /*include_clusters=*/false);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

include_muted=false drops muted records from faults, so a muted record's legacy rows (created_at_ns == 0) get dated 0 although download serves them. Use include_muted=true, as download does.

/// The reporting source that owns this record. Together with `fault_code` it
/// addresses the record on the per-record routes, and it is the single entry
/// of `reporting_sources`.
std::optional<std::string> source_id;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A list item now has top-level source_id = owner and x-medkit.source_id = namespace path, while the detail calls the owner x-medkit.owner to avoid exactly that clash. Use owner here too.

@bburda
bburda force-pushed the feat/fault-identity-gateway-owner branch from 8fba96c to 547c411 Compare November 18, 2026 08:21
@bburda bburda changed the title gateway: address the fault record an entity owns gateway: show shared and per-source fault records once per entity Nov 18, 2026
@bburda
bburda added this pull request to stack #792 November 18, 2026 08:21
@bburda
bburda force-pushed the feat/fault-identity-gateway-owner branch from 72b93ed to b2e3c29 Compare November 18, 2026 17:43
The gateway carries the record owner through every per-record call and
resolves a per-entity fault route to the record the entity owns, answering
409 when the code names several records in its scope. Bulk-data recording
downloads, plugin clears, fault triggers, log triggers, entity locks and the
fault event stream act on one record. Sub-path sources belong to the entity
that owns the path. Plugin API version 8 adds FaultProvider::clear_fault_record.
The rosbag listing reads the fault list to find the owners in scope, so
the fake transport now lists one confirmed record per fault it holds. The
listing test also checks that an unreadable fault list answers 503, and
the REST reference says the listing reads the fault list first.
The status route answered only for the node map's current component id.
After the plugin re-derives that id from the device, /components lists
the previous id until the next discovery refresh, and the route answered
404 for it. The route now also accepts the ids the last two introspect()
calls returned. A component the plugin does not expose still gets 404.
shutdown() joins the plugin's fault-client executor and drops its clients
with their pending requests. The test now holds a clear in flight, shuts
the plugin down and destroys it, then has the stub answer: the plugin's
sink receives nothing. A refusal answered while the plugin lives, and a
control request answered after the dropped reply, show the harness sees
a reply that is delivered.
Two sources reporting one code are two records only when the fault
manager keeps that code per_source. The README now says so where it
explains per-node records and per-record recordings.
The freeze-frame overlap tests hold the standing-fault catch-up until a second
subscriber on the capture's own feed has seen the confirm, which the feed
delivers after the capture has queued it, and wait for the live frame instead
of a fixed delay. The logs trigger tests wait for the gateways to acknowledge
the published entries, pace the real entries on the entity listings, and close
each negative check with an entry logged after the others: a trigger delivers
in log order, so once that entry arrives nothing earlier can still reach it.
An app shows the record of its source and the records of its sub-sources as
one entry, and a component, area or function every record of the code under
it. An entry of several records is their merge: the most advanced status, the
highest severity, the summed occurrence count, the earliest first and latest
last occurrence, the union of reporting sources, and the description and
environment data of the confirmed record that occurred last. x-medkit.records
names each record of an entry, its app, source and status. A shared record
names no owner anywhere and looks as it did before records had owners.

GET and DELETE of a code act on every record of it under the entity, and the
DELETE clears each with its own owner; neither answers 409 any more. A
per-entity clear that cannot clear a record answers 500 with failed_codes. The
global fault list shows one item per record.

The gateway's fault data carries the record owner from Fault.owner, "*" for a
shared record, and addresses the record with it. The entity frames and the
global clear's lock check still go by the record's reporting source.
list_entity_faults fills Fault.owner from the gateway's fault data, as the
fault manager sets it: the reporting source of a per_source record, "*" for a
shared one. A peer gateway's item names a per_source record's owner as
source_id, which is used when no owner is given.
A fault list item has no top-level source_id. SOVD keeps vendor fields
inside x-medkit, so a per_source record names its app and source only in
x-medkit.records. FaultListItem drops the field and the published schema
follows.

The detail of one per_source record still names its owner as
x-medkit.owner, now taken from its single x-medkit.records entry.

sovd_service_interface reads the owner of a peer gateway's item from that
entry too.

The README fault list examples show the wire shape: items, x-medkit,
reporting_sources and a fault status.
…reference

rclcpp's AnyServiceCallback::set() stores a callable whose argument list
is not SharedPtrCallback's by assigning it to its callback variant. A
callable of (const std::shared_ptr<Request> &, const std::shared_ptr<Response> &)
converts to SharedPtrCallback and to no other alternative. The Humble,
Jazzy and Lyrical headers share that code, and the fault manager registers
its services the same way. The handlers take the same references.
The bands are uint16_t, so the comparison has no NaN case and the
simplified form clang-tidy asks for is equivalent.
…a constructor

The guard deletes its copy and move constructors, so under C++20 it is no
aggregate and the brace initialisation finds no constructor. Lyrical's
toolchain compiles the gateway tests as C++20 and stopped there. A
constructor taking the three references keeps the guard and what the test
pins unchanged on every distro.
…mble

Humble's rcl_interfaces declares the Log severity constants as byte, so
rclpy holds them as bytes and refuses them for the uint8 level field;
later distros declare them uint8. Every test publishing a /rosout entry
failed on its first publish on Humble. The constants now pass through a
helper that returns the integer on every distro.
Each demo node logs one INFO line at startup. Whether the gateway gets it
depends on whether its /rosout subscription matched the node first, and
when it does the line is an entry of the app, so the restored trigger
fires for it and writes one more marker than the test counts. The nodes
now run at --log-level warn and publish nothing, so the test's entries
are the only entries of each app.
A json returned where the function declares tl::expected<json, ...> or
std::optional<json> converts either through that type's converting
constructor or through json's own conversion operator. gcc 11 on Humble
warns under -Wconversion about the choice it makes. Constructing the
return type explicitly leaves one candidate, so the build prints no
warning from these lines; the value returned is the same.
…itten

The marker count was read as soon as two markers were listed, so a
marker written after that read went unseen. The test now waits for the
marker of the last entry it logs: one notifier worker dispatches entries
in log order and writes an entry's markers before the next, so after that
marker no trigger writes another. It then requires exactly three markers
under the root app, none under the namespaced app, and no delivered
event for an entry the test did not log.
A trigger's event stream wakes only for an event, the trigger's removal or
its 15 s keepalive. Closing the client side blocked until the next
keepalive, once per stream, and the gateway's shutdown then waited for the
same wake-up: about 45 s of the test's 61 s. The test now removes both
triggers, waits until the gateway has ended each stream, and requires
that it does.
Closing the /faults/stream response in test_08 blocked until the next
keepalive, 30 s by default. The gateway of this test sends one every
second, the minimum of sse.keepalive_interval_sec.
Seven fixed spins (14.5 s a run) stood in for "sweeps have run" and
"nothing more arrives". The test now runs the orphan sweep and restore
replay itself, the code the refresh timer runs, and closes each negative
check with a later entry: a trigger delivers in log order, so once that
entry arrives nothing logged before it can still reach the trigger. The
checks are exact: each trigger's events are the entries in log order, one
marker per firing.
test_08 read frames of the code test_077 had already raised under both
owners, so a frame for owner-b arrived even when test_08 never reported
from owner-b. It now reports a per_source code nothing else raises, so
every frame it counts comes from its own reports.
…r 1 s

The stalled fault manager case waits out fault_manager.service_timeout_sec
on every call it makes, 5 s each at the default. The test sets 1 s, scaled
by the sanitizer time factor.
The scope notification and rosbag listing tests primed their report
client and not their clear client, and the markers test made a new
report client for its one report. Each client now gets one answer from
answer_once_each before the test sends what it checks; the rosbag
listing test's handshake report goes. The markers report is still sent
once, since a second CRITICAL report would fire the faults trigger again.
… lifecycle client

Each GetState and ChangeState call made a new client and sent one
request, and a reply can be lost while the lifecycle node has not
matched that client. GetState is now sent until answered. A ChangeState
client first gets an answer to the default request, transition 0, which
a running node refuses without changing state, and then sends its one
transition.
@bburda
bburda force-pushed the feat/fault-identity-gateway-owner branch from b2e3c29 to 8a5276b Compare November 19, 2026 08:09

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants