Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
52 commits
Select commit Hold shift + click to select a range
3f01f49
fault_manager: key every fault record by fault code and owner
bburda Nov 12, 2026
8442bdd
fault_manager: key PostgreSQL fault records by fault code and owner
bburda Nov 12, 2026
be32a4e
fault_manager: choose per fault code whether each source gets its own…
bburda Nov 12, 2026
18a20e0
fault_manager: shared records confirm CRITICAL on the report's severity
bburda Nov 12, 2026
0b1bc95
fault_manager: test the shared-record CRITICAL rule on PostgreSQL
bburda Nov 12, 2026
c02458d
integration_tests: declare the severity consistency codes per_source
bburda Nov 12, 2026
8894eff
fault_manager: correlate records across owners
bburda Nov 12, 2026
ffa3bd4
fault_manager: audit rows keep the mechanism and name the record owner
bburda Nov 12, 2026
316c76d
fault_manager: capture a code's topics once for its per-source records
bburda Nov 12, 2026
bd3fa38
fault_manager: name the owner of every record a recording covers
bburda Nov 12, 2026
258de7b
fault_manager: index snapshots by record
bburda Nov 12, 2026
8b62b73
docs: state that the owner migration is one way
bburda Nov 12, 2026
a3426c2
opcua: name the reporting entity in every clear it sends
bburda Nov 12, 2026
f04a20c
fault_manager: keep the shared snapshot capture of a code whole
bburda Nov 12, 2026
7af5454
opcua: probe and clear PLC_COMMS_LOST under the id it was reported under
bburda Nov 12, 2026
f19267c
docs: state what the correlation cascade clears
bburda Nov 12, 2026
c468d75
docs: every client must be rebuilt against the new messages
bburda Nov 12, 2026
a7a54e3
fault_manager: pin the shared record of a default code again
bburda Nov 12, 2026
1be2ed5
fault_manager: test the promises of the per-source model through the …
bburda Nov 12, 2026
3eb882f
opcua: find a per-source comms-lost record raised under the stand-in …
bburda Nov 12, 2026
12bb14b
test(opcua): a cleared alarm clears only its own per-source record
bburda Nov 12, 2026
7b0bcd5
fault_manager: a cleared root cause ends its relations whatever the c…
bburda Nov 12, 2026
0d351c3
fault_manager: a re-reported active root cause keeps the symptoms it …
bburda Nov 12, 2026
c22f2b5
test(fault_manager): a PostgreSQL database of the old layout is migra…
bburda Nov 12, 2026
88f686e
test(opcua): wait for the next probe instead of sleeping after a late…
bburda Nov 12, 2026
775fb72
test(fault_manager): the snapshot cap test no longer races topic disc…
bburda Nov 12, 2026
9754097
fault_manager: a root cause lists exactly the symptoms it mutes
bburda Nov 13, 2026
dbc0f0c
fault_manager: every end of a record releases the symptoms it muted
bburda Nov 13, 2026
a1d5bf4
opcua: report under the default component id only when the node map n…
bburda Nov 13, 2026
9936cb4
test(fault_manager): the zero-message bag test records a topic only i…
bburda Nov 13, 2026
d9c6da0
fault_manager: a show_as_single cluster mutes every member but its re…
bburda Nov 13, 2026
2c2ebb8
opcua: a bridge with a node map reports under its component id only
bburda Nov 13, 2026
e75fe64
fault_manager: a record stays muted while any live relation mutes it
bburda Nov 13, 2026
9709e41
test(fault_manager): the zero-message bag test pins that traffic else…
bburda Nov 13, 2026
f90eb62
docs: state what a root cause lists, mutes and clears for both rule k…
bburda Nov 13, 2026
f0cc31a
fault_manager: a root mutes through its own rule and its own delay
bburda Nov 13, 2026
c586771
test(fault_manager): wait for the events publisher through the node
bburda Nov 13, 2026
879e8ba
scripts: port the ROS-free cores check to the record identity API
bburda Nov 13, 2026
4b04d71
test(fault_manager): the snapshot tests wait for topic discovery befo…
bburda Nov 13, 2026
ada9c91
test(fault_manager): the snapshot queue test fails if the silent topi…
bburda Nov 13, 2026
c8461c2
msgs: every Fault names the record it describes
bburda Nov 14, 2026
4535403
test: no null dereference warnings in the Release build of the fault …
bburda Nov 14, 2026
fcc6a5a
test(fault_manager): get an answer on each client before the restart …
bburda Nov 18, 2026
5bf0288
opcua: make the first reconnect wait a setting
bburda Nov 18, 2026
74ee0fc
test(fault_manager): the refused clear test waits on a later event
bburda Nov 18, 2026
daaf79f
test(fault_manager): the snapshot queue test ends each slow capture i…
bburda Nov 18, 2026
f38df61
opcua: state the first reconnect wait when the poller starts
bburda Nov 18, 2026
4082e5d
test(opcua): the comms-lost test talks to the fault manager from one …
bburda Nov 18, 2026
f6b0572
test(fault_manager): get an answer and an event before a test acts
bburda Nov 18, 2026
1a0b408
test: the integration tests get one answer per fault manager client f…
bburda Nov 18, 2026
d22b809
test(opcua): both fault manager tests use one in-process client
bburda Nov 18, 2026
cfefb52
test(fault_manager): the rosbag tests wait for the history they assert
bburda Nov 18, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
71 changes: 59 additions & 12 deletions docs/api/messages.rst
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,17 @@ Messages
Fault.msg
~~~~~~~~~

Core fault data model representing an aggregated fault condition.
Core fault data model representing one fault record.

A record is identified by the pair (``fault_code``, owner). A code listed with
``per_source: true`` in ``fault_thresholds.yaml`` has one record per reporting source, owned by
the ``source_id`` of the ``ReportFault`` call. Every other code has one record shared by all
its sources, owned by the reserved id ``*``. Each record has its own status, debounce counter,
``occurrence_count``, severity and timestamps, and is cleared on its own.

``owner`` names the record a ``Fault`` describes. Passed back as ``source_id``, it addresses exactly
that record, also when a stored shared record and a per-source record of one source carry the same
``reporting_sources`` after the code changed from shared to ``per_source``.

.. code-block:: text

Expand Down Expand Up @@ -49,9 +59,14 @@ Core fault data model representing an aggregated fault condition.
# Current fault status (PREFAILED, PREPASSED, CONFIRMED, HEALED, CLEARED)
string status

# List of source identifiers that have reported this fault
# Every source that reported this record: one for a per-source record, all of
# them for a shared one.
string[] reporting_sources

# Owner of the record this Fault describes: the reporting source of a
# per-source record, "*" for a shared record.
string owner

**Severity Constants:**

.. list-table::
Expand Down Expand Up @@ -146,14 +161,18 @@ prefix (for example ``/robot1/fault_manager/events``).
MutedFaultInfo.msg
~~~~~~~~~~~~~~~~~~

Information about correlated (muted) symptom faults.
Information about a correlated (muted) symptom record. One entry per muted RECORD:
a root cause mutes the matching symptom records of every owner, so two per-source records
muted on one ``fault_code`` produce two entries carrying that code, told apart by
``source_id``.

.. code-block:: text

string fault_code # The muted symptom's fault code
string root_cause_code # Root cause that triggered muting
string rule_id # Correlation rule ID that matched
uint32 delay_ms # Time delay from root cause [ms]
string source_id # Owner of the muted record (``*`` for a shared record)

ClusterInfo.msg
~~~~~~~~~~~~~~~
Expand All @@ -172,6 +191,12 @@ Auto-detected fault cluster information.
uint32 count # Number of faults
builtin_interfaces/Time first_at # First fault timestamp
builtin_interfaces/Time last_at # Last fault timestamp
string[] fault_owners # Owner of each member, parallel to fault_codes
string representative_owner # Owner of the representative record

A cluster holds the matching records of every owner, so a member is the record
``(fault_codes[i], fault_owners[i])``. Pass the owner as ``source_id`` to the
single-record services. The owner of a shared record is ``*``.

Services
--------
Expand All @@ -189,7 +214,8 @@ Report a fault event to the FaultManager.
uint8 event_type # EVENT_FAILED (0) or EVENT_PASSED (1)
uint8 severity # Fault.SEVERITY_* constant (for FAILED events)
string description # Human-readable description
string source_id # Fully qualified node name (e.g., "/powertrain/temp_sensor")
string source_id # Fully qualified node name (e.g., "/powertrain/temp_sensor").
# Owns the record of a per_source code. "*" is reserved and refused.

**Response:**

Expand All @@ -215,29 +241,43 @@ Report a fault event to the FaultManager.
ClearFault.srv
~~~~~~~~~~~~~~

Clear/acknowledge a fault.
Clear/acknowledge one fault record.

**Request:**

.. code-block:: text

string fault_code # Fault code to clear
bool skip_correlation_auto_clear # Opt out of correlation cascade clear
string source_id # Owner of the record, or a source that reports into it

**Response:**

.. code-block:: text

bool success # True if fault was found and cleared
bool success # True if the record was found and cleared
string message # Status message or error description
string[] auto_cleared_codes # Symptoms auto-cleared with root cause

``source_id`` names the owner of the record to clear (``*`` for a shared record). When no
record has that owner, it names the record that source reports into. Leaving it empty is
unscoped: the call applies only when exactly one record carries the ``fault_code``, and fails
otherwise with ``success=false`` and a ``message`` beginning ``ambiguous:`` that lists the owners.
An ambiguous call clears nothing. ``GetFault``, ``GetSnapshots`` and ``GetRosbag`` carry
the same field and resolve the same way. On ``GetRosbag`` it scopes the ``fault_code``
lookup only, and the ``recording_id`` path ignores it.

``GetRosbag`` answers with ``fault_codes`` and ``owners``, parallel arrays with one entry per
fault record the recording is attached to. A burst of several per-source records of one code
lists the code once per owner. A shared record is one entry carrying the owner ``*``.

When ``skip_correlation_auto_clear`` is ``false`` (default), clearing a
root-cause fault also clears every symptom that the correlation engine
attributes to it via ``auto_clear_with_root`` rules; the cleared
symptom codes are returned in ``auto_cleared_codes``. When ``true``,
only the requested ``fault_code`` is cleared and ``auto_cleared_codes``
is empty. The gateway's per-entity ``DELETE
is empty. Either way the cleared root cause ends its relations: the symptoms it
muted are unmuted, and it no longer mutes later reports. The gateway's per-entity ``DELETE
/{entity-path}/faults/{fault_code}`` route sets this to ``true`` so
that an operator with access to one entity cannot cascade-clear
correlated symptoms reported by apps in other entities. The global
Expand All @@ -246,11 +286,18 @@ cluster-wide clearing still works.

.. note::

Added in ``ros2_medkit_msgs`` post-0.4.0. Adding a request field
changes the service type hash, so out-of-tree callers that invoke
``/fault_manager/clear_fault`` directly via ``ros2 service call`` or
a generated client must rebuild against the new ``ros2_medkit_msgs``
release to keep talking to ``fault_manager``.
``skip_correlation_auto_clear`` was added in ``ros2_medkit_msgs`` post-0.4.0,
and ``source_id`` after it (the same field was added to ``GetFault``,
``GetSnapshots`` and ``GetRosbag``). Adding a field changes the type hash, and
a field added inside an array element changes the wire layout, so every client
of the fault manager's services and topics must be rebuilt against this
``ros2_medkit_msgs``. This includes out-of-tree callers that use ``ros2
service call`` or a generated client. The case that breaks without an error is
``ListFaults``: ``MutedFaultInfo.source_id`` and ``ClusterInfo.fault_owners`` and
``representative_owner`` sit inside the elements of the ``muted_faults`` and
``clusters`` arrays, CDR has no per-element length, and a client built against
an earlier ``ros2_medkit_msgs`` misreads any response that carries a muted or
clustered entry. Trailing fields do not keep old clients working.

ListFaults.srv
~~~~~~~~~~~~~~
Expand Down
Loading
Loading