Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
ea015fd
gateway: address the fault record an entity owns
bburda Nov 12, 2026
eb87d63
test(gateway): the recording tests' fault manager answers the fault list
bburda Nov 14, 2026
d47f99d
fix(opcua): serve x-plc-status for the component ids the plugin exposes
bburda Nov 14, 2026
e29a6cc
test(opcua): a clear reply after the plugin is destroyed is dropped
bburda Nov 14, 2026
66d561b
docs(graph_watchdog): a per-source record needs a per_source code
bburda Nov 14, 2026
ae603c2
test: the freeze-frame and logs trigger tests wait on events
bburda Nov 14, 2026
50222ef
gateway: an entity shows each fault code once
bburda Nov 14, 2026
de8078b
sovd_service_interface: each listed Fault names its record
bburda Nov 14, 2026
6f7b69c
faults: a fault item names its record only inside x-medkit
bburda Nov 14, 2026
3dc3a33
sovd_service_interface: take service requests and responses by const …
bburda Nov 14, 2026
8c9d227
opcua: write the severity band order check without a negation
bburda Nov 15, 2026
6eda791
test(gateway): build the waiting-change test's unblock guard through …
bburda Nov 18, 2026
4c89e94
test: publish integer severity levels in the logs trigger tests on Hu…
bburda Nov 18, 2026
65325c4
test: run the restored logs trigger test's nodes at warn
bburda Nov 18, 2026
726fee2
gateway: return a json as the declared expected or optional type
bburda Nov 18, 2026
949f765
test: count the restored logs trigger markers once the last one is wr…
bburda Nov 18, 2026
4291953
test: remove the restored logs triggers before closing their streams
bburda Nov 18, 2026
f2339ed
test: close the owner identity test's fault stream within a second
bburda Nov 18, 2026
76d67d8
test(gateway): the log trigger restore tests wait on events, not on time
bburda Nov 18, 2026
7aa8b4a
test: the owner identity stream case reports a code of its own
bburda Nov 18, 2026
23b033b
test: the bulk-data test's gateway gives up on the fault manager afte…
bburda Nov 18, 2026
c5c4ef5
test: three gateway fault tests get one answer per client first
bburda Nov 18, 2026
8a5276b
test(graph_watchdog): the node death e2e gets an answer from each new…
bburda Nov 18, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 8 additions & 2 deletions docs/api/locking.rst
Original file line number Diff line number Diff line change
Expand Up @@ -232,8 +232,14 @@ The One Exception: Global Fault Clear
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

``DELETE /api/v1/faults`` reads ``X-Client-Id`` like the writes above but never
answers ``409``. It walks every fault, **skips** the ones whose reporting
entity is locked by another client, clears the rest, and answers ``204``.
answers ``409``. It walks every fault RECORD, **skips** the ones whose owning
entity is locked by another client, clears the rest one record at a time with
its owner, and answers ``204``. A record belongs to an app when its owner is
the app's node FQN (the app's own id for an external app) or a path child of
it, and to an external component when its owner is the component's id or a path
child of it. The record is skipped when it belongs to an entity whose ``faults``
this client may not write, counting the locks of that entity's parent component
and area as a per-entity ``DELETE`` does.
Nothing on the response says which faults were skipped - the
``X-Medkit-Local-Only: true`` header that 204 also carries is set
unconditionally and reports that aggregated *peers* were not cleared, not that
Expand Down
16 changes: 11 additions & 5 deletions docs/api/messages.rst
Original file line number Diff line number Diff line change
Expand Up @@ -275,14 +275,20 @@ When ``skip_correlation_auto_clear`` is ``false`` (default), clearing a
root-cause fault also clears every symptom that the correlation engine
attributes to it via ``auto_clear_with_root`` rules; the cleared
symptom codes are returned in ``auto_cleared_codes``. When ``true``,
only the requested ``fault_code`` is cleared and ``auto_cleared_codes``
is empty. Either way the cleared root cause ends its relations: the symptoms it
only the requested record is cleared and ``auto_cleared_codes`` is
empty. Either way the cleared root cause ends its relations: the symptoms it
muted are unmuted, and it no longer mutes later reports. The gateway's per-entity ``DELETE
/{entity-path}/faults/{fault_code}`` route sets this to ``true`` so
that an operator with access to one entity cannot cascade-clear
correlated symptoms reported by apps in other entities. The global
``DELETE /api/v1/faults/{fault_code}`` route leaves it ``false`` so
cluster-wide clearing still works.
correlated symptoms reported by apps in other entities, and its
per-entity ``DELETE /{entity-path}/faults`` does the same for every
record it clears. The global ``DELETE /api/v1/faults`` route leaves it
``false`` so cluster-wide clearing still works. There is no
``DELETE /api/v1/faults/{fault_code}`` route: a bare fault code is not
an address, because a ``per_source`` code names as many records as there
are sources reporting it. A client that wants the cascade for one record calls
``ClearFault`` directly with that record's ``fault_code`` and
``source_id``.

.. note::

Expand Down
246 changes: 211 additions & 35 deletions docs/api/rest.rst

Large diffs are not rendered by default.

13 changes: 7 additions & 6 deletions docs/design/ros2_medkit_fault_detection/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -62,12 +62,13 @@ Transition tracking and global uniqueness
the evaluator. It returns only the signals whose active state changed since the
last call, which a plugin forwards to the fault manager as report / clear.

The tracker is keyed by ``fault_code`` alone, matching the fault manager, which
also keys and clears faults by code alone. A single tracker may therefore be
shared across many points only if every ``fault_code`` is globally unique; two
points emitting the same code would alternately raise and clear it each cycle.
Consumers that share one tracker (for example the OPC UA poller across all
node-map entries and event alarms) must enforce that uniqueness at
The tracker is keyed by ``fault_code`` alone. That is a property of the tracker,
not of the fault manager, which identifies a record by ``fault_code`` and its
owner. A single tracker may therefore be shared across many points only if every
``fault_code`` is unique within it. Two points emitting the same code would
alternately raise and clear it each cycle, inside the tracker and before any
report is sent. Consumers that share one tracker (for example the OPC UA poller
across all node-map entries and event alarms) must enforce that uniqueness at
config-load time and reject a colliding configuration before anything runs.

Raise qualifier
Expand Down
8 changes: 6 additions & 2 deletions docs/glossary.rst
Original file line number Diff line number Diff line change
Expand Up @@ -55,8 +55,12 @@ This glossary defines key terms used throughout ros2_medkit documentation.
can be polled for status or cancelled.

Fault
An error condition reported by a ROS 2 node to the fault manager.
Faults have a code, severity, message, and timestamp.
An error condition reported to the fault manager. One fault record is the
pair (fault code, owner). For a code configured ``per_source`` the owner
is the ``source_id`` the reporter used, so two sources reporting one such
code are two records, each with its own severity, message, timestamps and
status, and each cleared on its own. Every other code has one record that
all its sources share.

See: :doc:`design/ros2_medkit_fault_reporter/index`

Expand Down
2 changes: 1 addition & 1 deletion docs/requirements/specs/faults.rst
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ Faults
- ``item``: Fault details with SOVD-compliant ``status`` object (aggregatedStatus, testFailed, confirmedDTC, pendingDTC)
- ``environment_data``: Extended data records (timestamps) and snapshots array
- ``environment_data.snapshots[]``: Array of freeze_frame (topic data) and rosbag (bulk-data reference) entries
- ``x-medkit``: Extension fields (occurrence_count, reporting_sources, severity_label)
- ``x-medkit``: Extension fields (occurrence_count, reporting_sources, owner, severity_label)

.. req:: DELETE /{entity}/faults
:id: REQ_INTEROP_014
Expand Down
29 changes: 17 additions & 12 deletions docs/tutorials/fault-correlation.rst
Original file line number Diff line number Diff line change
Expand Up @@ -375,7 +375,8 @@ Response always includes:
"fault_code": "MOTOR_COMM_001",
"root_cause_code": "ESTOP_001",
"rule_id": "estop_cascade",
"delay_ms": 150
"delay_ms": 150,
"source_id": "/powertrain/motor_controller"
}
]
}
Expand Down Expand Up @@ -421,17 +422,21 @@ Response always includes:

.. note::

**Per-entity DELETE opts out of the cascade.** The same fault cleared
via ``DELETE /api/v1/{entity-path}/faults/ESTOP_001`` clears only
``ESTOP_001`` itself - ``auto_cleared_codes`` will be empty in the
response. The gateway sets ``ClearFault.srv``'s
``skip_correlation_auto_clear`` to ``true`` on per-entity routes so
that an operator with access to one entity cannot cascade-clear
correlated symptom faults reported by apps in other entities. Use
the global ``DELETE /api/v1/faults/{fault_code}`` route when you do
want the correlation cascade. Direct ``ros2 service call`` clients
can choose explicitly by setting ``skip_correlation_auto_clear`` in
the request body (see ``ros2_medkit_msgs`` ``ClearFault.srv``).
**Per-entity DELETE opts out of the cascade.** The same record cleared
via ``DELETE /api/v1/{entity-path}/faults/ESTOP_001`` clears only that
record - ``auto_cleared_codes`` will be empty in the response. The
gateway sets ``ClearFault.srv``'s ``skip_correlation_auto_clear`` to
``true`` on per-entity routes so that an operator with access to one
entity cannot cascade-clear correlated symptom faults reported by apps
in other entities.

There is no global ``DELETE /api/v1/faults/{fault_code}`` route over
HTTP: a bare ``per_source`` fault code names as many records as there
are sources reporting it, so it is not an address. ``DELETE /api/v1/faults``
clears every record and takes no code. To get the cascade for one
record, call the service directly and set ``skip_correlation_auto_clear``
to ``false`` alongside that record's ``fault_code`` and ``source_id``
(see ``ros2_medkit_msgs`` ``ClearFault.srv``).

Example: Complete Configuration
-------------------------------
Expand Down
18 changes: 11 additions & 7 deletions docs/tutorials/manifest-discovery.rst
Original file line number Diff line number Diff line change
Expand Up @@ -279,15 +279,19 @@ Runtime Linking

**Trigger entity resolution in hybrid mode**

In hybrid mode, fault and log triggers are delivered using the manifest entity
ID rather than the raw ROS node FQN. The runtime linker builds a node-to-app
mapping during the linking phase (step 4 above), and the fault manager and log
manager use this mapping to resolve incoming trigger events to the correct
manifest entity. This means fault codes, log subscriptions, and SSE events
reference stable manifest IDs (e.g., ``lidar-driver``) instead of the
ephemeral node FQN (e.g., ``/sensors/velodyne_driver``), even when nodes
In hybrid mode, fault triggers are delivered using the manifest entity ID
rather than the raw ROS node FQN. The runtime linker builds a node-to-app
mapping during the linking phase (step 4 above), and fault trigger events are
resolved through it to the correct manifest entity. This means fault codes and
SSE events reference stable manifest IDs (e.g., ``lidar-driver``) instead of
the ephemeral node FQN (e.g., ``/sensors/velodyne_driver``), even when nodes
restart and re-bind.

Log triggers do not use this mapping. A trigger on ``/{entity}/logs`` checks
each entry's logger name against the entity's log sources, the rule
``GET /{entity}/logs`` lists by, in every discovery mode. The ``logs``
collection in :doc:`/api/rest` lists the limits of that check.

ROS Binding Configuration
~~~~~~~~~~~~~~~~~~~~~~~~~

Expand Down
29 changes: 23 additions & 6 deletions docs/tutorials/plugin-system.rst
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,15 @@ Plugins implement the ``GatewayPlugin`` C++ base class plus one or more typed pr
- **OperationProvider** - per-entity operation backend (list operations, execute). Uses
the same per-entity routing model as DataProvider.
- **FaultProvider** - per-entity fault backend (list faults, get fault details, clear
faults). Uses the same per-entity routing model as DataProvider.
faults). Uses the same per-entity routing model as DataProvider. A fault record is
the pair (``fault_code``, owner). The gateway clears a code by calling
``clear_fault_record(entity_id, fault_code, owner)`` once for each record of the
code in the entity's fault scope, with that record's owner: the reporting source
of a ``per_source`` record (for a component, often one of its hosted apps), ``*``
for a shared one. Its default implementation calls
``clear_fault(entity_id, fault_code)``, so a provider that overrides only the
two-argument clear keeps compiling and keeps receiving clears by code. Override
``clear_fault_record`` to clear exactly the record the owner names.

A single plugin can implement multiple provider interfaces. For example, a "systemd" plugin
could provide both introspection (discover systemd units) and updates (manage service restarts).
Expand Down Expand Up @@ -1020,11 +1028,20 @@ Plugins export ``plugin_api_version()`` which must return the gateway's ``PLUGIN
If the version does not match, the plugin is rejected with a clear error message suggesting
a rebuild against matching gateway headers.

The current API version is **7**. It is incremented when the ``PluginContext`` vtable changes
or breaking changes are made to ``GatewayPlugin`` or provider interfaces. A change to the layout
of a type that crosses the plugin boundary, such as the entity models in ``IntrospectionInput`` /
``NewEntities``, bumps the version too. A plugin built against the old layout would read those
types wrongly, so it must be rebuilt even when its source compiles unchanged.
The current API version is **8**. It is incremented when the ``PluginContext`` vtable changes
or breaking changes are made to ``GatewayPlugin`` or provider interfaces. A new virtual method
counts even when it keeps plugin source compiling, because it changes the vtable a pre-built
plugin was compiled against. A change to the layout of a type that crosses the plugin boundary,
such as the entity models in ``IntrospectionInput`` / ``NewEntities``, bumps the version too. A
plugin built against the old layout would read those types wrongly, so it must be rebuilt even
when its source compiles unchanged. Version 8 added ``FaultProvider::clear_fault_record``: its
default implementation calls the unchanged ``clear_fault``, so v7 plugin source compiles
against v8 headers as it is, but a plugin binary built for v7 is rejected and has to be
rebuilt. Version 8 also gives ``GatewayPlugin`` a protected ``log_sink()``, a copy of the
plugin's log sink for work that can outlive the plugin (an asynchronous service reply, for
example, should log through that copy rather than capture ``this``), and makes
``set_logger()`` protected so a plugin hosted without the gateway, in a unit test for
example, can wire a sink of its own.

Build Requirements
------------------
Expand Down
5 changes: 3 additions & 2 deletions docs/tutorials/triggers-use-cases.rst
Original file line number Diff line number Diff line change
Expand Up @@ -77,8 +77,9 @@ Step 1: Create the update status trigger
Step 2: Create the failure alert trigger with log_settings
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The ``log_settings`` field configures temporary log capture when the trigger
fires, useful for recording context around a failure event.
The ``log_settings`` field writes a marker entry to the log each time the
trigger fires, useful for recording context around a failure event. The marker
fires no ``logs`` trigger.

.. code-block:: bash

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -1080,7 +1080,7 @@
"stream"
]
},
"description": "Server-Sent Events (SSE) stream for real-time fault notifications.\n\nEvents:\n- fault_confirmed: Fault transitioned to CONFIRMED status\n- fault_cleared: Fault ended (manually cleared, or auto-healed; check fault.status for CLEARED vs HEALED)\n- fault_updated: Fault data changed (occurrence_count, sources)\n\nSSE Format:\n```\nevent: fault_confirmed\ndata: {\"event_type\":\"fault_confirmed\",\"fault\":{...},\"timestamp\":1234567890.123}\n```\n\nNote: Postman has limited SSE support. For testing, use:\n- Browser: `new EventSource('http://localhost:8080/api/v1/faults/stream')`\n- curl: `curl -N {{base_url}}/faults/stream`\n\nFeatures:\n- Keepalive every 30 seconds\n- Reconnection support via Last-Event-ID header\n- Multiple simultaneous clients supported"
"description": "Server-Sent Events (SSE) stream for real-time fault notifications.\n\nEvents:\n- fault_confirmed: Fault transitioned to CONFIRMED status\n- fault_cleared: Fault ended (manually cleared, or auto-healed, check fault.status for CLEARED vs HEALED)\n- fault_updated: Record data changed (last_occurred, severity)\n\nSSE Format:\n```\nevent: fault_confirmed\ndata: {\"event_type\":\"fault_confirmed\",\"fault\":{...},\"timestamp\":1234567890.123}\n```\n\nNote: Postman has limited SSE support. For testing, use:\n- Browser: `new EventSource('http://localhost:8080/api/v1/faults/stream')`\n- curl: `curl -N {{base_url}}/faults/stream`\n\nFeatures:\n- Keepalive every 30 seconds\n- Reconnection support via Last-Event-ID header\n- Multiple simultaneous clients supported"
},
"response": []
},
Expand Down Expand Up @@ -1142,7 +1142,7 @@
"faults"
]
},
"description": "List all faults for a component (REQ_INTEROP_012). Returns component_id, source_id (namespace), faults array, and count. By default returns PREFAILED and CONFIRMED faults."
"description": "List every fault record the component owns (REQ_INTEROP_012). A record is the pair (fault_code, owner), so two sources reporting one per_source code are two items with distinct source_id. Returns component_id, faults array, and count. By default returns PREFAILED and CONFIRMED records."
},
"response": []
},
Expand Down Expand Up @@ -1189,7 +1189,7 @@
"SENSOR_OVERTEMP"
]
},
"description": "Get a specific fault by fault_code (REQ_INTEROP_013). Returns component_id and fault object with fault_code, severity, severity_label, description, status, timestamps, and reporting_sources."
"description": "Get one fault record of this component by fault_code (REQ_INTEROP_013). Returns the SOVD detail whose x-medkit names the record's owner (x-medkit.owner) beside reporting_sources, occurrence_count and severity_label. Answers 409 (x-medkit-ambiguous-fault) when the code resolves to several records of this component, with parameters.owners naming them. Records the fault list shows rank above muted ones, and muted ones above cleared or healed ones, so one active record is served even while another source's cleared record of the code is kept."
},
"response": []
},
Expand All @@ -1210,7 +1210,7 @@
"SENSOR_OVERTEMP"
]
},
"description": "Clear a fault by fault_code (REQ_INTEROP_015). Changes fault status to CLEARED. Returns success status with component_id and fault_code.\n\n**With correlation enabled:**\nIf the cleared fault is a root cause with `auto_clear_with_root=true`, the response includes `auto_cleared_codes[]` listing all symptom faults that were automatically cleared."
"description": "Clear one fault record of this component by fault_code (REQ_INTEROP_015). Changes that record's status to CLEARED and answers 204 with no body. Answers 409 (x-medkit-ambiguous-fault) when the code resolves to several records of this component, ranked the same way as the GET.\n\n**With correlation enabled:**\nThe per-entity route always sets skip_correlation_auto_clear, so symptom records correlated to the cleared one stay untouched."
},
"response": []
},
Expand Down
Loading
Loading