diff --git a/docs/api/locking.rst b/docs/api/locking.rst index 50c382633..88907c5bf 100644 --- a/docs/api/locking.rst +++ b/docs/api/locking.rst @@ -232,8 +232,14 @@ The One Exception: Global Fault Clear ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ``DELETE /api/v1/faults`` reads ``X-Client-Id`` like the writes above but never -answers ``409``. It walks every fault, **skips** the ones whose reporting -entity is locked by another client, clears the rest, and answers ``204``. +answers ``409``. It walks every fault RECORD, **skips** the ones whose owning +entity is locked by another client, clears the rest one record at a time with +its owner, and answers ``204``. A record belongs to an app when its owner is +the app's node FQN (the app's own id for an external app) or a path child of +it, and to an external component when its owner is the component's id or a path +child of it. The record is skipped when it belongs to an entity whose ``faults`` +this client may not write, counting the locks of that entity's parent component +and area as a per-entity ``DELETE`` does. Nothing on the response says which faults were skipped - the ``X-Medkit-Local-Only: true`` header that 204 also carries is set unconditionally and reports that aggregated *peers* were not cleared, not that diff --git a/docs/api/messages.rst b/docs/api/messages.rst index 7e14a432c..4eeb6daeb 100644 --- a/docs/api/messages.rst +++ b/docs/api/messages.rst @@ -275,14 +275,20 @@ When ``skip_correlation_auto_clear`` is ``false`` (default), clearing a root-cause fault also clears every symptom that the correlation engine attributes to it via ``auto_clear_with_root`` rules; the cleared symptom codes are returned in ``auto_cleared_codes``. When ``true``, -only the requested ``fault_code`` is cleared and ``auto_cleared_codes`` -is empty. Either way the cleared root cause ends its relations: the symptoms it +only the requested record is cleared and ``auto_cleared_codes`` is +empty. Either way the cleared root cause ends its relations: the symptoms it muted are unmuted, and it no longer mutes later reports. The gateway's per-entity ``DELETE /{entity-path}/faults/{fault_code}`` route sets this to ``true`` so that an operator with access to one entity cannot cascade-clear -correlated symptoms reported by apps in other entities. The global -``DELETE /api/v1/faults/{fault_code}`` route leaves it ``false`` so -cluster-wide clearing still works. +correlated symptoms reported by apps in other entities, and its +per-entity ``DELETE /{entity-path}/faults`` does the same for every +record it clears. The global ``DELETE /api/v1/faults`` route leaves it +``false`` so cluster-wide clearing still works. There is no +``DELETE /api/v1/faults/{fault_code}`` route: a bare fault code is not +an address, because a ``per_source`` code names as many records as there +are sources reporting it. A client that wants the cascade for one record calls +``ClearFault`` directly with that record's ``fault_code`` and +``source_id``. .. note:: diff --git a/docs/api/rest.rst b/docs/api/rest.rst index a27580577..5da87781d 100644 --- a/docs/api/rest.rst +++ b/docs/api/rest.rst @@ -1384,23 +1384,79 @@ Query and manage faults. .. note:: - **Per-entity fault scope (``/{entity-path}/faults`` routes).** The gateway keys - faults by ``fault_code`` only, and a fault's ``reporting_sources`` set is the - union of every app that has reported that code. Per-entity routes apply a - strict all-sources scope check: a fault is in scope for an entity iff **every** - entry in ``reporting_sources`` is an app owned by that entity (exact FQN - match, or strict path-child). - - This means a ``fault_code`` reported by apps in two different entities - (for example ``SENSOR_TIMEOUT`` reported by both the lidar and the - temperature sensor app) is **not** visible or clearable through either - entity's per-entity routes - per-fault routes return ``404``, collection - responses omit it, and per-entity ``DELETE`` skips it. To see, list, or - clear such shared faults use the global ``GET /api/v1/faults`` / - ``DELETE /api/v1/faults`` routes. + **Per-entity fault scope (``/{entity-path}/faults`` routes).** A fault record + is the pair (``fault_code``, owner). For a code the fault manager keeps + ``per_source`` (see ``fault_thresholds.yaml``), the owner is the ``source_id`` + the reporter used and the single entry of the record's ``reporting_sources``, + so two apps reporting one such ``fault_code`` are two records, each with its + own status, occurrence count and timestamps, and each cleared on its own. Any + other code has one shared record that every source reports into. A record is + in scope for an entity when each of its reporting sources equals one of the + entity's owned sources or is a path child of one (``/...``). The owned + sources are: + + - an app: its node FQN, or its own id when the app is external + - a component: the owned sources of its hosted apps, plus its own id when the + component is external + - an area: the owned sources of every component in the area and in its + subareas + - a function: the owned sources of its host apps and host components + + The same rule, applied to the record whatever its status, decides which + ``/{entity-path}/faults`` triggers see the record's events, which entity + ``x-medkit`` names in ``GET /api/v1/faults/stream``, which locks the global + ``DELETE /api/v1/faults`` honours, and which recordings an entity lists (with + one fallback, see List Bulk Data Items). The last path segment of an owner is + never read as an entity id. + + An entity shows each ``fault_code`` once. The entry of a code covers every + record of it in the entity's scope: on an app, the record of its own source + and the records of its sub-sources (``/``); on a component, an + area or a function, every record of the code under it. An entry of one record + is that record. An entry of several records is their merge: + + - ``status``: the most advanced, ``CONFIRMED`` > ``PREFAILED`` > ``PREPASSED`` + > ``HEALED`` > ``CLEARED`` + - ``severity``: the highest + - ``occurrence_count``: the sum + - ``first_occurred``: the earliest; ``last_occurred`` and ``last_passed``: the + latest + - ``reporting_sources``: the union + - ``description`` and the detail's environment data: from the ``CONFIRMED`` + record that occurred last, else from the record that occurred last + + ``x-medkit.records`` on a list item and on a detail names each record the + entry covers: ``app_id`` and ``source_id`` (the owner) for a ``per_source`` + record, ``reporting_sources`` for a shared one, and its ``status``. A fault + item carries vendor fields only inside ``x-medkit``, so it has no top-level + ``source_id``. The detail of one ``per_source`` record also names its owner as + ``x-medkit.owner``. A shared record names no owner anywhere: alone in an + entry, its item and its detail look as they did before records had owners. + + A ``per_source`` ``SENSOR_TIMEOUT`` reported by both the lidar app and the + temperature sensor app therefore appears on each app's own ``/faults`` page as + that app's record, and on the component hosting both as one entry with two + ``x-medkit.records``. + + ``GET /{entity-path}/faults/{fault_code}`` serves the entry of the code over + every record of it in scope, whatever its status and muted or not, and + ``DELETE /{entity-path}/faults/{fault_code}`` clears each of those records + with its own owner. ``DELETE /{entity-path}/faults`` clears each record the + entity's fault list shows, with its own owner. Like that list, it leaves muted + records alone; the per-code ``DELETE`` clears a muted record. When the fault + manager refuses to clear a record, the bulk route answers ``500`` with the + codes it could not clear in ``parameters.failed_codes``, and so does the + per-code route when the code covers several records. The records it could + clear stay cleared. + + A fault LIST's ``x-medkit.source_id`` is something else: the addressed + entity's own namespace path, which is empty for an external app. Never read it + as an owner. ``GET /api/v1/faults`` - List all faults across the system. + List all faults across the system, one item per record. A ``per_source`` code + reported by two sources is two items, each with an ``x-medkit.records`` entry + naming its app and source. A shared record is one item that names no owner. ``GET /api/v1/components/{id}/faults`` List all faults for an entity. @@ -1493,6 +1549,14 @@ Query and manage faults. "x-medkit": { "occurrence_count": 3, "reporting_sources": ["/powertrain/motor_controller"], + "owner": "/powertrain/motor_controller", + "records": [ + { + "app_id": "motor_controller", + "source_id": "/powertrain/motor_controller", + "status": "CONFIRMED" + } + ], "severity_label": "ERROR" } } @@ -1553,11 +1617,19 @@ Query and manage faults. ``DELETE /api/v1/components/{id}/faults/{fault_code}`` Clear a fault. + - **200:** Fault cleared by the plugin that serves this entity's faults, with + the plugin's acknowledgement as the body - **204:** Fault cleared - **400:** ``fault_code`` empty or longer than 256 characters - **404:** Fault not found, reported by an app outside this entity's scope, or declined by the fault manager - - **503:** Fault manager unavailable + - **409:** The entity is locked by another client + - **500:** The code covers several records and the fault manager refused to + clear at least one; ``parameters.failed_codes`` names the code + - **503:** Fault manager unavailable. This holds for an entity whose faults a + plugin serves as well: the gateway reads the fault manager to learn which + record the code names before it asks the plugin, and when that read fails + it does not ask the plugin at all. .. note:: @@ -1577,10 +1649,14 @@ Query and manage faults. Clear all faults for an entity. Accepts the optional ``?status=`` query parameter (same values as ``GET /faults``). - Without it, clears pending and confirmed faults. + Without it, clears pending and confirmed faults. Muted records are not cleared, + because the entity's fault list does not show them. Clear one with its per-code + ``DELETE``. - **204:** Faults cleared (or none to clear) - **400:** Invalid status parameter + - **500:** The fault manager refused to clear at least one record; + ``parameters.failed_codes`` names their codes - **503:** Fault manager unavailable ``DELETE /api/v1/faults`` @@ -1800,10 +1876,19 @@ List all bulk-data items in a category for the entity. For ``rosbags``, the descriptor ``id`` is the recording id - the bag directory name - and it is what the download URL takes. There is **one descriptor per recording**, not one per fault: faults confirmed in one burst share a single -recording, and ``x-medkit.fault_codes`` lists every fault attached to it. A -recording therefore reports its size once. One fault code can appear on several -descriptors, one per occurrence it kept, told apart by ``creation_date``, which -is the time that recording was made. +recording, and ``x-medkit.fault_codes`` lists the code of every record of the +entity attached to it. A recording therefore reports its size once. One fault +code can appear on several descriptors, one per occurrence it kept, told apart +by ``creation_date``, which is the time that recording was made. + +An entity lists a recording when the owner of a record it was captured for is +in the entity's fault scope (see Faults Endpoints), whatever the record's status +and including a muted record. An entity whose scope resolves no reporting +source at all, for example a manifest component that hosts no app, matches +owners against its own node FQN, or its namespace when it has no FQN, instead. +A function that hosts nothing lists no recording. The listing reads the fault +list first, then the recordings of each owner in scope. When either read fails, +it answers ``503`` rather than an empty or partial list. ``x-medkit.storage_files`` is how many storage files the recording is held in, read from the recording's own metadata. A ``1`` is the normal case and says the @@ -1913,6 +1998,15 @@ and OpenAPI 3.1 has no way to say "bytes" - ``format: binary`` was an OpenAPI 3.0 idiom that 3.1 dropped when it aligned with JSON Schema 2020-12. A schema-free media type entry is the accurate description. +**Which recordings an entity serves.** A recording belongs to the fault records +it is attached to, each a (``fault_code``, owner) pair, and a burst attaches +several. An entity serves a recording when the owner of one of those records is +in the entity's fault scope, the scope its fault list uses. A fault code the +entity also owns a record of is not enough: +two apps reporting one ``per_source`` code each download their own recording and get ``404`` on +the other's, whether or not either record has been cleared, while the component +hosting both serves both. + **Range requests.** A request carrying a ``Range`` header is answered with **206 Partial Content** and a ``Content-Range: bytes -/`` header instead of ``200``; the body is the requested slice. Several ranges in @@ -1945,11 +2039,27 @@ ranged request carries the whole error document. - **200 OK**: File content - **206 Partial Content**: The byte range requested via ``Range``, with ``Content-Range`` -- **404 Not Found**: Entity, category, or bulk-data ID not found +- **404 Not Found**: Entity, category, or bulk-data ID not found, or a + recording none of this entity's records is attached to +- **409 Conflict**: ``x-medkit-ambiguous-fault``. A ``rosbags`` URL whose last + segment is a fault code rather than a recording id resolves that code to one + record in the entity's scope. The candidates come in three tiers, and the + first tier holding any record of the code decides: the ``PREFAILED`` and + ``CONFIRMED`` records that are not muted, then the muted ones of those + statuses, then the ``CLEARED``, ``HEALED`` and ``PREPASSED`` ones. Several + candidates in that tier name none of them, and ``parameters.owners`` names + their owners. Address the recording by + its own id instead: the ``rosbags`` listing of this entity carries it as each + descriptor's ``id``, and a fault detail links it as + ``environment_data.snapshots[].bulk_data_uri``. - **416 Range Not Satisfiable**: No range of the ``Range`` header starts inside the file, with ``Content-Range: bytes */``. Also sent when the header could not be parsed, which is not specific to this endpoint - see :ref:`rest-range-rejection`. +- **503 Service Unavailable**: ``rosbags`` only. The fault manager did not + answer a call the download needs: the record list, the recording lookup, or + the rosbag list of an owner that could hold the recording while no other + owner in scope does. A ``404`` would say the recording does not exist. Upload Bulk Data ~~~~~~~~~~~~~~~~ @@ -2637,13 +2747,55 @@ against each change and only fire when the condition is met. hierarchy-scoped monitoring. Area-level triggers catch changes from all descendant entities within the area. +A fault event reaches a trigger on ``/{entity-path}/faults`` when the record's +owner is in the entity's fault scope (see Faults Endpoints) - the rule by which +``GET /{entity-path}/faults?status=all`` lists the record. Status does not +matter: confirm, update, clear and heal events are all scoped this way, and the +payload's ``status`` tells a clear (``CLEARED``) from a heal (``HEALED``). The +trigger's condition then decides whether the event fires. + +Changes of every collection reach the triggers through one gateway-wide queue +of up to 10000 changes. Past that bound the oldest change is dropped and +logged, and no trigger sees it. + **Observable resource collections:** - ``data`` - Topic data changes (driven by ``TriggerTopicSubscriber``) - ``faults`` - Fault state transitions (created, updated, cleared) - ``operations`` - Operation execution completions - ``updates`` - Software update status changes -- ``logs`` - Log entries matching configured severity (x-medkit extension) +- ``logs`` - Log entries (x-medkit extension). A trigger checks each entry by + the rule ``GET /{entity}/logs`` lists by: the logger name is one of the + entity's log sources, in slash or dot form, and the entry's numeric level is + at or above the entity's ``severity_filter``. The check runs when the gateway + dispatches the change, against the entity's scope and ``severity_filter`` at + that moment, not when the entry arrived. Limits: + + - An entry the log buffer's distinct-logger cap drops fires nothing. + - A change the gateway-wide queue drops fires nothing. After such an + overflow, a waiting entry logged at a level between the named ones can be + compared by its severity name instead of its level: the gateway keeps + those exact levels for the last 20000 such entries, dropped ones included. + - A trigger keeps at most 100 events that no client has read and drops the + oldest. + - With a ``LogProvider`` plugin, ``GET /{entity}/logs`` returns what the + plugin returns. The trigger still checks only the entries the gateway takes + from ``/rosout`` (in observer mode also those the plugin keeps out of the + log buffer). In full-ingestion mode the gateway takes no entries from + ``/rosout``. + - A persistent trigger restored before its entity is discovered checks, + once the entity is discovered, the entries ``GET /{entity}/logs`` lists at + that moment (at most the entity's ``max_entries``) other than markers, then + the entries that follow. If that listing fails, for example in a + ``LogProvider`` plugin, the next sweep or entry tries again. + + The entry a trigger's ``log_settings`` writes when it fires (the marker) + fires no ``logs`` trigger: not its own trigger, not another trigger on the + same entity or on an entity above or below it. The gateway knows the marker + by how it was written, so a ``/rosout`` entry with the same text still fires. + The marker is logged under the trigger's entity id as logger name, and + ``GET /{entity}/logs`` lists it where that name is one of the entity's log + sources, for example on an app bound to a root-namespace node of that name. Create Trigger ~~~~~~~~~~~~~~ @@ -2727,11 +2879,14 @@ Create Trigger duration. Must be a positive integer. Omit for no expiry. * - ``log_settings`` - No - - Temporary log entry injected when the trigger fires. - Accepts ``severity`` (log level: ``debug``, ``info``, ``warning``, - ``error``, ``fatal``; default: ``info``) and ``marker`` (descriptive - message text; default: ``"Trigger fired"``). The log entry includes - trigger metadata (trigger ID, condition type, resource URI). + - Log entry (the marker) written to the gateway's log buffer each time + the trigger fires. Accepts ``severity`` (log level: ``debug``, + ``info``, ``warning``, ``error``, ``fatal``; default: ``info``) and + ``marker`` (descriptive message text; default: ``"Trigger fired"``). + The log entry includes trigger metadata (trigger ID, condition type, + resource URI). ``GET /{entity}/logs`` lists the marker where its + logger name, the trigger's entity id, is one of the entity's log + sources; it fires no ``logs`` trigger. **Response 201 Created:** @@ -3031,6 +3186,14 @@ plugin is loaded. Without it the routes stay mounted and answer ``501`` use, so a client can tell "this build has no threshold engine" apart from "no such app or rule". +A rule reads its value from whichever source the owning plugin serves the app's +``/data`` through: the plugin's data provider first, its own vendor data route +only when it exposes no provider. Rule evaluation and the create-time data-point +check use the same resolution in the same order, so a point that validates on +create is one the engine can read. A rule the engine cannot read holds its state +rather than firing on a stale value, which is also what it does while the app's +link reports itself down. + ``GET /api/v1/apps/{app_id}/fault-triggers`` List the app's rules. The owning app is the one in the path; it is not repeated in the item, and neither is the engine's internal cross latch. @@ -3062,8 +3225,9 @@ such app or rule". ``data_name`` the app does not expose (when enumerable); ``404`` (``entity-not-found``) when the app itself was never discovered; ``409`` (``precondition-not-fulfilled``) when the ``fault_code`` is already used by - another rule - fault codes are global to the fault store, so two rules - sharing one would fight over the same fault. + another rule on ANY app - the engine keeps one rule per code across every + app, so a code another app's rule already claims is refused. The error names + the rule and the app that holds it. ``DELETE /api/v1/apps/{app_id}/fault-triggers/{trigger_id}`` Remove a rule (``204``). A fault currently asserted by the rule is cleared; @@ -3618,6 +3782,12 @@ Vendor-specific ``x-medkit-*`` codes are enveloped: the response carries * - ``vendor-error`` - varies - A vendor-specific failure; read ``vendor_code`` for the real code + * - ``x-medkit-ambiguous-fault`` + - 409 + - The fault code in a ``rosbags`` URL addresses several fault records + inside the addressed entity, because several of its reporting sources + report that code and each is its own record. ``parameters.owners`` names + them. Address the recording by its own id. * - ``x-medkit-plugin-error`` - 400-599 - Plugin provider returned an error. Status varies by plugin. Message truncated to 512 chars. @@ -4058,11 +4228,17 @@ Other extensions beyond SOVD: - ``DELETE /faults`` - Clear all faults globally - ``GET /faults/stream`` - SSE real-time fault notifications. Each event payload carries an optional ``x-medkit`` SOVD payload-extension object with ``entity_type`` and ``entity_id`` - fields when the gateway can resolve the fault's first reporting source back to an entity, - so consumers can hit ``/{entity_type}/{entity_id}/bulk-data/rosbags/{fault_code}`` directly - without enumerating entities - that address serves the fault's newest recording. To reach an - older one, list ``/bulk-data/rosbags`` and use the descriptor ``id``. Resolution is snapshotted at event arrival; the entire - ``x-medkit`` object is omitted when no entity can be resolved. + fields naming the entity whose fault scope holds the record's owner: the app, or else the + external component. With nested node paths the app owning the longest source is named, and a + tie goes to the smallest id. Consumers can hit + ``/{entity_type}/{entity_id}/bulk-data/rosbags/{fault_code}`` directly without enumerating + entities - that address serves the fault's newest recording. To reach an older one, list + ``/bulk-data/rosbags`` and use the descriptor ``id``. The named entity can hold several records + of the code, for example records owned by two paths below one app's node. That address then + answers ``409`` ``x-medkit-ambiguous-fault``, and the recording is reached by its ``id`` from + the entity's ``/bulk-data/rosbags`` listing. Resolution is snapshotted at event + arrival, so an event replayed from the buffer after discovery removed that entity still names + it. The entire ``x-medkit`` object is omitted when no entity owns the record. - ``/health`` - Health check with discovery pipeline diagnostics - ``/version-info`` - Gateway version information - ``/docs`` - OpenAPI capability description diff --git a/docs/design/ros2_medkit_fault_detection/index.rst b/docs/design/ros2_medkit_fault_detection/index.rst index 894f408eb..46e47f489 100644 --- a/docs/design/ros2_medkit_fault_detection/index.rst +++ b/docs/design/ros2_medkit_fault_detection/index.rst @@ -62,12 +62,13 @@ Transition tracking and global uniqueness the evaluator. It returns only the signals whose active state changed since the last call, which a plugin forwards to the fault manager as report / clear. -The tracker is keyed by ``fault_code`` alone, matching the fault manager, which -also keys and clears faults by code alone. A single tracker may therefore be -shared across many points only if every ``fault_code`` is globally unique; two -points emitting the same code would alternately raise and clear it each cycle. -Consumers that share one tracker (for example the OPC UA poller across all -node-map entries and event alarms) must enforce that uniqueness at +The tracker is keyed by ``fault_code`` alone. That is a property of the tracker, +not of the fault manager, which identifies a record by ``fault_code`` and its +owner. A single tracker may therefore be shared across many points only if every +``fault_code`` is unique within it. Two points emitting the same code would +alternately raise and clear it each cycle, inside the tracker and before any +report is sent. Consumers that share one tracker (for example the OPC UA poller +across all node-map entries and event alarms) must enforce that uniqueness at config-load time and reject a colliding configuration before anything runs. Raise qualifier diff --git a/docs/glossary.rst b/docs/glossary.rst index 6e816d98c..ab27214ba 100644 --- a/docs/glossary.rst +++ b/docs/glossary.rst @@ -55,8 +55,12 @@ This glossary defines key terms used throughout ros2_medkit documentation. can be polled for status or cancelled. Fault - An error condition reported by a ROS 2 node to the fault manager. - Faults have a code, severity, message, and timestamp. + An error condition reported to the fault manager. One fault record is the + pair (fault code, owner). For a code configured ``per_source`` the owner + is the ``source_id`` the reporter used, so two sources reporting one such + code are two records, each with its own severity, message, timestamps and + status, and each cleared on its own. Every other code has one record that + all its sources share. See: :doc:`design/ros2_medkit_fault_reporter/index` diff --git a/docs/requirements/specs/faults.rst b/docs/requirements/specs/faults.rst index bc83eae8e..4d9670e60 100644 --- a/docs/requirements/specs/faults.rst +++ b/docs/requirements/specs/faults.rst @@ -21,7 +21,7 @@ Faults - ``item``: Fault details with SOVD-compliant ``status`` object (aggregatedStatus, testFailed, confirmedDTC, pendingDTC) - ``environment_data``: Extended data records (timestamps) and snapshots array - ``environment_data.snapshots[]``: Array of freeze_frame (topic data) and rosbag (bulk-data reference) entries - - ``x-medkit``: Extension fields (occurrence_count, reporting_sources, severity_label) + - ``x-medkit``: Extension fields (occurrence_count, reporting_sources, owner, severity_label) .. req:: DELETE /{entity}/faults :id: REQ_INTEROP_014 diff --git a/docs/tutorials/fault-correlation.rst b/docs/tutorials/fault-correlation.rst index 316334509..310c76722 100644 --- a/docs/tutorials/fault-correlation.rst +++ b/docs/tutorials/fault-correlation.rst @@ -375,7 +375,8 @@ Response always includes: "fault_code": "MOTOR_COMM_001", "root_cause_code": "ESTOP_001", "rule_id": "estop_cascade", - "delay_ms": 150 + "delay_ms": 150, + "source_id": "/powertrain/motor_controller" } ] } @@ -421,17 +422,21 @@ Response always includes: .. note:: - **Per-entity DELETE opts out of the cascade.** The same fault cleared - via ``DELETE /api/v1/{entity-path}/faults/ESTOP_001`` clears only - ``ESTOP_001`` itself - ``auto_cleared_codes`` will be empty in the - response. The gateway sets ``ClearFault.srv``'s - ``skip_correlation_auto_clear`` to ``true`` on per-entity routes so - that an operator with access to one entity cannot cascade-clear - correlated symptom faults reported by apps in other entities. Use - the global ``DELETE /api/v1/faults/{fault_code}`` route when you do - want the correlation cascade. Direct ``ros2 service call`` clients - can choose explicitly by setting ``skip_correlation_auto_clear`` in - the request body (see ``ros2_medkit_msgs`` ``ClearFault.srv``). + **Per-entity DELETE opts out of the cascade.** The same record cleared + via ``DELETE /api/v1/{entity-path}/faults/ESTOP_001`` clears only that + record - ``auto_cleared_codes`` will be empty in the response. The + gateway sets ``ClearFault.srv``'s ``skip_correlation_auto_clear`` to + ``true`` on per-entity routes so that an operator with access to one + entity cannot cascade-clear correlated symptom faults reported by apps + in other entities. + + There is no global ``DELETE /api/v1/faults/{fault_code}`` route over + HTTP: a bare ``per_source`` fault code names as many records as there + are sources reporting it, so it is not an address. ``DELETE /api/v1/faults`` + clears every record and takes no code. To get the cascade for one + record, call the service directly and set ``skip_correlation_auto_clear`` + to ``false`` alongside that record's ``fault_code`` and ``source_id`` + (see ``ros2_medkit_msgs`` ``ClearFault.srv``). Example: Complete Configuration ------------------------------- diff --git a/docs/tutorials/manifest-discovery.rst b/docs/tutorials/manifest-discovery.rst index 83e0e9f35..b0d56b157 100644 --- a/docs/tutorials/manifest-discovery.rst +++ b/docs/tutorials/manifest-discovery.rst @@ -279,15 +279,19 @@ Runtime Linking **Trigger entity resolution in hybrid mode** - In hybrid mode, fault and log triggers are delivered using the manifest entity - ID rather than the raw ROS node FQN. The runtime linker builds a node-to-app - mapping during the linking phase (step 4 above), and the fault manager and log - manager use this mapping to resolve incoming trigger events to the correct - manifest entity. This means fault codes, log subscriptions, and SSE events - reference stable manifest IDs (e.g., ``lidar-driver``) instead of the - ephemeral node FQN (e.g., ``/sensors/velodyne_driver``), even when nodes + In hybrid mode, fault triggers are delivered using the manifest entity ID + rather than the raw ROS node FQN. The runtime linker builds a node-to-app + mapping during the linking phase (step 4 above), and fault trigger events are + resolved through it to the correct manifest entity. This means fault codes and + SSE events reference stable manifest IDs (e.g., ``lidar-driver``) instead of + the ephemeral node FQN (e.g., ``/sensors/velodyne_driver``), even when nodes restart and re-bind. + Log triggers do not use this mapping. A trigger on ``/{entity}/logs`` checks + each entry's logger name against the entity's log sources, the rule + ``GET /{entity}/logs`` lists by, in every discovery mode. The ``logs`` + collection in :doc:`/api/rest` lists the limits of that check. + ROS Binding Configuration ~~~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/docs/tutorials/plugin-system.rst b/docs/tutorials/plugin-system.rst index 1fb338158..aa2012e73 100644 --- a/docs/tutorials/plugin-system.rst +++ b/docs/tutorials/plugin-system.rst @@ -26,7 +26,15 @@ Plugins implement the ``GatewayPlugin`` C++ base class plus one or more typed pr - **OperationProvider** - per-entity operation backend (list operations, execute). Uses the same per-entity routing model as DataProvider. - **FaultProvider** - per-entity fault backend (list faults, get fault details, clear - faults). Uses the same per-entity routing model as DataProvider. + faults). Uses the same per-entity routing model as DataProvider. A fault record is + the pair (``fault_code``, owner). The gateway clears a code by calling + ``clear_fault_record(entity_id, fault_code, owner)`` once for each record of the + code in the entity's fault scope, with that record's owner: the reporting source + of a ``per_source`` record (for a component, often one of its hosted apps), ``*`` + for a shared one. Its default implementation calls + ``clear_fault(entity_id, fault_code)``, so a provider that overrides only the + two-argument clear keeps compiling and keeps receiving clears by code. Override + ``clear_fault_record`` to clear exactly the record the owner names. A single plugin can implement multiple provider interfaces. For example, a "systemd" plugin could provide both introspection (discover systemd units) and updates (manage service restarts). @@ -1020,11 +1028,20 @@ Plugins export ``plugin_api_version()`` which must return the gateway's ``PLUGIN If the version does not match, the plugin is rejected with a clear error message suggesting a rebuild against matching gateway headers. -The current API version is **7**. It is incremented when the ``PluginContext`` vtable changes -or breaking changes are made to ``GatewayPlugin`` or provider interfaces. A change to the layout -of a type that crosses the plugin boundary, such as the entity models in ``IntrospectionInput`` / -``NewEntities``, bumps the version too. A plugin built against the old layout would read those -types wrongly, so it must be rebuilt even when its source compiles unchanged. +The current API version is **8**. It is incremented when the ``PluginContext`` vtable changes +or breaking changes are made to ``GatewayPlugin`` or provider interfaces. A new virtual method +counts even when it keeps plugin source compiling, because it changes the vtable a pre-built +plugin was compiled against. A change to the layout of a type that crosses the plugin boundary, +such as the entity models in ``IntrospectionInput`` / ``NewEntities``, bumps the version too. A +plugin built against the old layout would read those types wrongly, so it must be rebuilt even +when its source compiles unchanged. Version 8 added ``FaultProvider::clear_fault_record``: its +default implementation calls the unchanged ``clear_fault``, so v7 plugin source compiles +against v8 headers as it is, but a plugin binary built for v7 is rejected and has to be +rebuilt. Version 8 also gives ``GatewayPlugin`` a protected ``log_sink()``, a copy of the +plugin's log sink for work that can outlive the plugin (an asynchronous service reply, for +example, should log through that copy rather than capture ``this``), and makes +``set_logger()`` protected so a plugin hosted without the gateway, in a unit test for +example, can wire a sink of its own. Build Requirements ------------------ diff --git a/docs/tutorials/triggers-use-cases.rst b/docs/tutorials/triggers-use-cases.rst index fb68119b6..379a7c610 100644 --- a/docs/tutorials/triggers-use-cases.rst +++ b/docs/tutorials/triggers-use-cases.rst @@ -77,8 +77,9 @@ Step 1: Create the update status trigger Step 2: Create the failure alert trigger with log_settings ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -The ``log_settings`` field configures temporary log capture when the trigger -fires, useful for recording context around a failure event. +The ``log_settings`` field writes a marker entry to the log each time the +trigger fires, useful for recording context around a failure event. The marker +fires no ``logs`` trigger. .. code-block:: bash diff --git a/postman/collections/ros2-medkit-gateway.postman_collection.json b/postman/collections/ros2-medkit-gateway.postman_collection.json index 6a3b69f17..c04c33887 100644 --- a/postman/collections/ros2-medkit-gateway.postman_collection.json +++ b/postman/collections/ros2-medkit-gateway.postman_collection.json @@ -1080,7 +1080,7 @@ "stream" ] }, - "description": "Server-Sent Events (SSE) stream for real-time fault notifications.\n\nEvents:\n- fault_confirmed: Fault transitioned to CONFIRMED status\n- fault_cleared: Fault ended (manually cleared, or auto-healed; check fault.status for CLEARED vs HEALED)\n- fault_updated: Fault data changed (occurrence_count, sources)\n\nSSE Format:\n```\nevent: fault_confirmed\ndata: {\"event_type\":\"fault_confirmed\",\"fault\":{...},\"timestamp\":1234567890.123}\n```\n\nNote: Postman has limited SSE support. For testing, use:\n- Browser: `new EventSource('http://localhost:8080/api/v1/faults/stream')`\n- curl: `curl -N {{base_url}}/faults/stream`\n\nFeatures:\n- Keepalive every 30 seconds\n- Reconnection support via Last-Event-ID header\n- Multiple simultaneous clients supported" + "description": "Server-Sent Events (SSE) stream for real-time fault notifications.\n\nEvents:\n- fault_confirmed: Fault transitioned to CONFIRMED status\n- fault_cleared: Fault ended (manually cleared, or auto-healed, check fault.status for CLEARED vs HEALED)\n- fault_updated: Record data changed (last_occurred, severity)\n\nSSE Format:\n```\nevent: fault_confirmed\ndata: {\"event_type\":\"fault_confirmed\",\"fault\":{...},\"timestamp\":1234567890.123}\n```\n\nNote: Postman has limited SSE support. For testing, use:\n- Browser: `new EventSource('http://localhost:8080/api/v1/faults/stream')`\n- curl: `curl -N {{base_url}}/faults/stream`\n\nFeatures:\n- Keepalive every 30 seconds\n- Reconnection support via Last-Event-ID header\n- Multiple simultaneous clients supported" }, "response": [] }, @@ -1142,7 +1142,7 @@ "faults" ] }, - "description": "List all faults for a component (REQ_INTEROP_012). Returns component_id, source_id (namespace), faults array, and count. By default returns PREFAILED and CONFIRMED faults." + "description": "List every fault record the component owns (REQ_INTEROP_012). A record is the pair (fault_code, owner), so two sources reporting one per_source code are two items with distinct source_id. Returns component_id, faults array, and count. By default returns PREFAILED and CONFIRMED records." }, "response": [] }, @@ -1189,7 +1189,7 @@ "SENSOR_OVERTEMP" ] }, - "description": "Get a specific fault by fault_code (REQ_INTEROP_013). Returns component_id and fault object with fault_code, severity, severity_label, description, status, timestamps, and reporting_sources." + "description": "Get one fault record of this component by fault_code (REQ_INTEROP_013). Returns the SOVD detail whose x-medkit names the record's owner (x-medkit.owner) beside reporting_sources, occurrence_count and severity_label. Answers 409 (x-medkit-ambiguous-fault) when the code resolves to several records of this component, with parameters.owners naming them. Records the fault list shows rank above muted ones, and muted ones above cleared or healed ones, so one active record is served even while another source's cleared record of the code is kept." }, "response": [] }, @@ -1210,7 +1210,7 @@ "SENSOR_OVERTEMP" ] }, - "description": "Clear a fault by fault_code (REQ_INTEROP_015). Changes fault status to CLEARED. Returns success status with component_id and fault_code.\n\n**With correlation enabled:**\nIf the cleared fault is a root cause with `auto_clear_with_root=true`, the response includes `auto_cleared_codes[]` listing all symptom faults that were automatically cleared." + "description": "Clear one fault record of this component by fault_code (REQ_INTEROP_015). Changes that record's status to CLEARED and answers 204 with no body. Answers 409 (x-medkit-ambiguous-fault) when the code resolves to several records of this component, ranked the same way as the GET.\n\n**With correlation enabled:**\nThe per-entity route always sets skip_correlation_auto_clear, so symptom records correlated to the cleared one stay untouched." }, "response": [] }, diff --git a/scripts/check_cores_without_ros/main.cpp b/scripts/check_cores_without_ros/main.cpp index a162ef9ff..92967ca8c 100644 --- a/scripts/check_cores_without_ros/main.cpp +++ b/scripts/check_cores_without_ros/main.cpp @@ -154,17 +154,27 @@ class NoFaultTransport : public gw::FaultServiceTransport { hook(fault_code, source_id); return {true, {}, {}, gw::FaultFailure::Declined}; } - // Once set, every list answers one fault whose occurrence_count grows with each call. + // Once set, every list answers one fault whose occurrence_count grows with each call. The codes `serve_detail` + // made answer are listed too, so a per-entity fault route can resolve their records. gw::FaultResult list_faults(const std::string & /*source_id*/, bool /*include_prefailed*/, bool /*include_confirmed*/, bool /*include_cleared*/, bool /*include_healed*/, bool /*include_muted*/, bool /*include_clusters*/) override { - if (!reports_faults.load()) { + gw::json faults = gw::json::array(); + { + std::lock_guard lock(detail_mutex_); + for (const auto & [code, source] : detail_sources_) { + faults.push_back(served_fault(code, source)); + } + } + if (reports_faults.load()) { + faults.push_back({{"fault_code", "CHECK_FAULT"}, + {"occurrence_count", ++list_calls}, + {"reporting_sources", gw::json::array({"hosted_app"})}}); + } + if (faults.empty()) { return none(); } - gw::json fault = {{"fault_code", "CHECK_FAULT"}, - {"occurrence_count", ++list_calls}, - {"reporting_sources", gw::json::array({"hosted_app"})}}; - return {true, {{"faults", gw::json::array({fault})}}, {}, gw::FaultFailure::Declined}; + return {true, {{"faults", faults}}, {}, gw::FaultFailure::Declined}; } gw::FaultWithEnvJsonResult get_fault_with_env(const std::string & fault_code, const std::string & /*source_id*/) override { @@ -173,12 +183,10 @@ class NoFaultTransport : public gw::FaultServiceTransport { if (it == detail_sources_.end()) { return {false, "not supported", {}, gw::FaultFailure::Unavailable}; } - const gw::json fault = {{"fault_code", fault_code}, - {"severity", 2}, - {"status", "CONFIRMED"}, - {"occurrence_count", 1}, - {"reporting_sources", gw::json::array({it->second})}}; - return {true, {}, {{"fault", fault}, {"environment_data", gw::json::object()}}, gw::FaultFailure::Declined}; + return {true, + {}, + {{"fault", served_fault(fault_code, it->second)}, {"environment_data", gw::json::object()}}, + gw::FaultFailure::Declined}; } // Makes the detail of `fault_code` answer, reported by `source`. void serve_detail(const std::string & fault_code, const std::string & source) { @@ -188,17 +196,19 @@ class NoFaultTransport : public gw::FaultServiceTransport { gw::FaultResult get_fault(const std::string & /*fault_code*/, const std::string & /*source_id*/) override { return none(); } - gw::FaultResult clear_fault(const std::string & /*fault_code*/, bool /*skip_correlation_auto_clear*/) override { + gw::FaultResult clear_fault(const std::string & /*fault_code*/, const std::string & /*source_id*/, + bool /*skip_correlation_auto_clear*/) override { if (!on_report_set()) { return none(); } ++clears; return {true, {}, {}, gw::FaultFailure::Declined}; } - gw::FaultResult get_snapshots(const std::string & /*fault_code*/, const std::string & /*topic*/) override { + gw::FaultResult get_snapshots(const std::string & /*fault_code*/, const std::string & /*source_id*/, + const std::string & /*topic*/) override { return none(); } - gw::FaultResult get_rosbag(const std::string & /*fault_code*/) override { + gw::FaultResult get_rosbag(const std::string & /*id*/, const std::string & /*source_id*/) override { return none(); } gw::FaultResult list_rosbags(const std::string & /*entity_fqn*/) override { @@ -235,6 +245,14 @@ class NoFaultTransport : public gw::FaultServiceTransport { static gw::FaultResult none() { return {false, {}, "not supported", gw::FaultFailure::Unavailable}; } + // The record `serve_detail` makes answer: confirmed, owned by `source`. + static gw::json served_fault(const std::string & fault_code, const std::string & source) { + return {{"fault_code", fault_code}, + {"severity", 2}, + {"status", "CONFIRMED"}, + {"occurrence_count", 1}, + {"reporting_sources", gw::json::array({source})}}; + } }; // Answers every sample with a value that grows by one per call. @@ -871,7 +889,7 @@ std::pair share_one_event_source() { stream_done = true; reader.join(); const bool framed = wait_until([&] { - return !capture->frames_for("SHARED_1").empty(); + return !capture->frames_for("SHARED_1", "plugin_app").empty(); }); return {sse_seen.load(), framed}; } @@ -1232,7 +1250,7 @@ bool check_triggers() { }); ok &= check(fault_fired, "a confirmation from the host's event source fires the fault trigger"); ok &= check(capture && wait_until([&] { - return !capture->frames_for("TRIGGER_FAULT").empty(); + return !capture->frames_for("TRIGGER_FAULT", "plugin_app").empty(); }), "the freeze-frame capture on the same source holds a frame for the confirmation"); if (fault_fired) { @@ -1475,7 +1493,7 @@ bool check_plugin_context_and_fault_triggers() { stream_done = true; stream.join(); ok &= check(wait_until([&] { - return !capture->frames_for("SPEED_HIGH").empty(); + return !capture->frames_for("SPEED_HIGH", "plugin_app").empty(); }), "the freeze-frame capture gets the same confirmation"); const auto detail = client.Get("/api/v1/apps/plugin_app/faults/SPEED_HIGH"); @@ -1826,7 +1844,7 @@ static int run() { plugin.speed = 10; events.confirm("FF_1", "plugin_app"); ok &= check(wait_until([&] { - return !capture->frames_for("FF_1").empty(); + return !capture->frames_for("FF_1", "plugin_app").empty(); }), "a fault confirmation for a plugin-served entity is captured"); plugin.speed = 99; @@ -1843,10 +1861,10 @@ static int run() { events.confirm("FF_HOSTED", "hosted_app"); events.confirm("FF_2", "plugin_app"); ok &= check(wait_until([&] { - return !capture->frames_for("FF_2").empty(); + return !capture->frames_for("FF_2", "plugin_app").empty(); }), "a second fault is captured"); - ok &= check(capture->frames_for("FF_HOSTED").empty() && + ok &= check(capture->frames_for("FF_HOSTED", "hosted_app").empty() && freeze_frame_of(detail_of("hosted_app", "FF_HOSTED").second, "hosted_app").is_null(), "an entity no plugin serves gets no freeze_frame"); @@ -1854,13 +1872,14 @@ static int run() { plugin.speed = 30; events.confirm("FF_3", "plugin_app"); ok &= check(wait_until([&] { - return !capture->frames_for("FF_3").empty(); + return !capture->frames_for("FF_3", "plugin_app").empty(); }), "a third fault is captured"); - ok &= check(capture->frames_for("FF_1").empty() && + ok &= check(capture->frames_for("FF_1", "plugin_app").empty() && freeze_frame_of(detail_of("plugin_app", "FF_1").second, "plugin_app").is_null(), "past max_faults the oldest fault's frames are evicted"); - ok &= check(!capture->frames_for("FF_2").empty() && !capture->frames_for("FF_3").empty(), + ok &= check(!capture->frames_for("FF_2", "plugin_app").empty() && + !capture->frames_for("FF_3", "plugin_app").empty(), "the newer faults keep their frames"); // A re-confirmation takes a new frame. diff --git a/src/ros2_medkit_fault_detection/README.md b/src/ros2_medkit_fault_detection/README.md index 3902dd3cb..115614e88 100644 --- a/src/ros2_medkit_fault_detection/README.md +++ b/src/ros2_medkit_fault_detection/README.md @@ -34,7 +34,10 @@ std::vector transitions = tracker.apply(signals); // raises + `evaluate` reports the full set of faults a rule governs (each flagged active or inactive). `FaultTransitionTracker` keeps the last-known state per `fault_code` and emits only the raise/clear edges, which a plugin forwards to -the fault manager as report/clear. +the fault manager as report/clear under the entity that owns the point. The +code-only key is the tracker's own: a fault manager record is identified by +`fault_code` and its owner, so one tracker per set of points sharing +a code, or unique codes within one tracker. ## Waiting before a raise diff --git a/src/ros2_medkit_gateway/CMakeLists.txt b/src/ros2_medkit_gateway/CMakeLists.txt index 3fd25feb7..224cd6ed5 100644 --- a/src/ros2_medkit_gateway/CMakeLists.txt +++ b/src/ros2_medkit_gateway/CMakeLists.txt @@ -455,6 +455,12 @@ if(BUILD_TESTING) medkit_add_gtest(test_plugin_notify_integration test/test_plugin_notify_integration.cpp) target_link_libraries(test_plugin_notify_integration gateway_ros2) + medkit_add_gtest(test_log_change_owner test/test_log_change_owner.cpp TIMEOUT 180) + target_link_libraries(test_log_change_owner gateway_ros2) + + medkit_add_gtest(test_log_trigger_restore test/test_log_trigger_restore.cpp TIMEOUT 120) + target_link_libraries(test_log_trigger_restore gateway_ros2) + medkit_add_gtest(test_plugin_context_aggregation test/test_plugin_context_aggregation.cpp) target_link_libraries(test_plugin_context_aggregation gateway_ros2) @@ -731,6 +737,9 @@ if(BUILD_TESTING) medkit_add_gtest(test_sse_fault_handler test/test_sse_fault_handler.cpp) target_link_libraries(test_sse_fault_handler gateway_ros2) medkit_target_dependencies(test_sse_fault_handler rclcpp ros2_medkit_msgs) + medkit_add_gtest(test_fault_handlers_plugin_clear test/test_fault_handlers_plugin_clear.cpp) + target_link_libraries(test_fault_handlers_plugin_clear gateway_ros2) + medkit_target_dependencies(test_fault_handlers_plugin_clear rclcpp ros2_medkit_msgs) # Zero-config entity freeze-frame capture tests medkit_add_gtest(test_entity_freeze_frame_capture test/test_entity_freeze_frame_capture.cpp) diff --git a/src/ros2_medkit_gateway/README.md b/src/ros2_medkit_gateway/README.md index 34ba4ce23..a8ba39ae8 100644 --- a/src/ros2_medkit_gateway/README.md +++ b/src/ros2_medkit_gateway/README.md @@ -100,6 +100,28 @@ All endpoints are prefixed with `/api/v1` for API versioning. - `DELETE /api/v1/{entity}/{id}/triggers/{trigger_id}` - Delete a trigger - `GET /api/v1/{entity}/{id}/triggers/{trigger_id}/events` - SSE stream of trigger events +A fault event reaches a trigger on `/{entity}/{id}/faults` when the record's owner is owned by the +entity, as an app's node FQN, an external app's or external component's bare id, or a path child of +one: the rule by which `GET /{entity}/{id}/faults?status=all` lists the record. Status does not +matter, so clear and heal events are scoped the same way; the payload's `status` is `CLEARED` or +`HEALED`. The rule is the same in every discovery mode and at every level (an area includes its +subareas, a function the apps of the components it hosts). The trigger's condition then decides +whether the event fires. Changes reach the triggers through one gateway-wide queue of up to 10000 +changes; past that the oldest change is dropped and logged. + +A trigger on `/{entity}/{id}/logs` checks each entry by the rule `GET /{entity}/{id}/logs` lists by: the +logger name is one of the entity's log sources (slash or dot form) and the entry's numeric level is at or above +the entity's `severity_filter`. The check runs when the change is dispatched. The entry a trigger's +`log_settings` writes (the marker) fires no `logs` trigger; `GET /{entity}/{id}/logs` lists it where its +logger name, the trigger's entity id, is one of the entity's log sources. The entries a trigger never sees +(buffer cap, change queue bound, `LogProvider` plugins) and the replay of a restored trigger are listed under +the `logs` collection in the REST API reference. + +A plugin that subscribes to `logs` changes through `PluginContext::get_resource_change_notifier()` gets, as +the change's entity id, the one app whose log scope holds the logger (the sources `GET /apps/{id}/logs` +queries), or an empty id when no app or more than one does. The logger name is `context.node` in the change +value. A marker reaches plugins the same way. + ### Fault-Trigger Endpoints (threshold rules) - `GET /api/v1/apps/{app_id}/fault-triggers` - List threshold rules for an app @@ -978,7 +1000,17 @@ curl -X DELETE http://localhost:8080/api/v1/components/temp_sensor/configuration ### Faults Endpoints -Faults represent errors or warnings reported by system components. The gateway provides access to faults stored in `ros2_medkit_fault_manager`. +Faults represent errors or warnings reported by system components. The gateway provides access to fault records stored in `ros2_medkit_fault_manager`. + +One record is the pair (`fault_code`, owner). For a code the fault manager keeps `per_source` (see `fault_thresholds.yaml`), the owner is the `source_id` the reporter used and the one entry of `reporting_sources`, so two sources reporting one such `fault_code` are two records, each with its own status, occurrence count and timestamps, and each cleared on its own. Any other code has one shared record that every source reports into. + +An entity shows each `fault_code` once. The entry of a code covers every record of it in the entity's scope: on an app, the record of its own source and of its sub-sources (`/`); on a component, an area or a function, every record of the code under it. An entry of several records is their merge: the most advanced status (`CONFIRMED` > `PREFAILED` > `PREPASSED` > `HEALED` > `CLEARED`), the highest severity, the summed occurrence count, the earliest first and the latest last occurrence, the union of reporting sources, and the description and environment data of the `CONFIRMED` record that occurred last (else of the record that occurred last). `x-medkit.records` names each record of an entry: `app_id` and `source_id` for a `per_source` record, `reporting_sources` for a shared one, and its `status`. A fault item carries vendor fields only inside `x-medkit`, so it has no top-level `source_id`. The detail of one `per_source` record also names its owner as `x-medkit.owner`; a shared record names no owner anywhere. `GET /api/v1/faults` is not an entity and lists one item per record. + +`GET` and `DELETE` on `/api/v1/{entity-path}/faults/{fault_code}` act on every record of the code in the entity's scope, whatever its status and muted or not; the `DELETE` clears each with its own owner. `DELETE /api/v1/{entity-path}/faults` clears each record the entity's fault list shows. Like that list, it leaves muted records alone. When the fault manager refuses to clear a record, the bulk route, and the per-code route for a code that covers several records, answer `500` with `parameters.failed_codes`. + +A recording belongs to the fault records it is attached to, each a (`fault_code`, owner) pair. `GET /api/v1/{entity-path}/bulk-data/rosbags/{recording_id}` serves it when the owner of one of those records is in the entity's fault scope, the scope its fault list uses. Owning a record of the same code is not enough: two apps reporting one `per_source` code each download only their own recording, cleared or not, and the component hosting both serves both. When the fault manager does not answer a call that decides this, the download answers `503`, not `404`. + +A fault list's `x-medkit.source_id` is not an owner: it is the addressed entity's own namespace path. - `GET /api/v1/faults` - List all faults across the system (convenience API for dashboards) - `GET /api/v1/faults/stream` - Real-time fault event stream via Server-Sent Events (SSE) @@ -1006,20 +1038,20 @@ curl http://localhost:8080/api/v1/faults?status=all **Response (200 OK):** ```json { - "faults": [ + "items": [ { "fault_code": "NAVIGATION_PATH_BLOCKED", "severity": 2, "severity_label": "ERROR", "description": "Navigation failed - cannot find valid path to goal", - "source_id": "/nav2_fault_reporter", - "status": "pending", + "reporting_sources": ["/nav2_fault_reporter"], + "status": "PREFAILED", "first_occurred": 1735830000, "last_occurred": 1735830000, "occurrence_count": 1 } ], - "count": 1 + "x-medkit": {"count": 1, "muted_count": 0, "cluster_count": 0} } ``` @@ -1041,8 +1073,8 @@ Real-time fault event stream using Server-Sent Events (SSE). Clients receive ins - **Real-time notifications**: Events pushed instantly when fault state changes - **Automatic reconnection**: Supports `Last-Event-ID` header for seamless reconnection - **Keepalive**: Sends `:keepalive` comment every 30 seconds to prevent timeouts -- **Event buffer**: Buffers up to 100 recent events for reconnecting clients. Under overflow the buffer evicts entries every live client has already received, then `fault_updated` entries superseded by a newer event for the same fault code - a lagging client still converges on the current state of every fault and loses no status transition. Anything beyond that (a superseded transition, or an event with no newer sibling) is genuinely lost for lagging clients: those losses are counted and logged as drops, and an affected client should refetch `GET /api/v1/faults` to resynchronize -- **Entity context (SOVD payload extension)**: When the gateway can resolve the fault's first reporting source back to an entity, the payload carries an `x-medkit` object with `entity_type` and `entity_id` fields so consumers can hit `/{entity_type}/{entity_id}/bulk-data/rosbags/{fault_code}` directly without enumerating entities. That address serves the fault's newest recording; to reach an older one, list `/bulk-data/rosbags` and use the descriptor `id` (the recording id) +- **Event buffer**: Buffers up to 100 recent events for reconnecting clients. Under overflow the buffer evicts entries every live client has already received, then `fault_updated` entries superseded by a newer event for the same fault record (the same code and owner) - a lagging client still converges on the current state of every fault and loses no status transition. Anything beyond that (a superseded transition, or an event with no newer sibling) is genuinely lost for lagging clients: those losses are counted and logged as drops, and an affected client should refetch `GET /api/v1/faults` to resynchronize +- **Entity context (SOVD payload extension)**: When an entity's fault scope holds the record's owner, the payload carries an `x-medkit` object with `entity_type` and `entity_id` fields so consumers can hit `/{entity_type}/{entity_id}/bulk-data/rosbags/{fault_code}` directly without enumerating entities. That address serves the fault's newest recording; to reach an older one, list `/bulk-data/rosbags` and use the descriptor `id` (the recording id). When the named entity holds several records of the code (for example two owners below one app's node), the address answers `409` `x-medkit-ambiguous-fault` and only the recording id reaches the recording - **Correlation payload**: when a root-cause event auto-clears correlated symptom faults, the payload carries their codes in `auto_cleared_codes` (omitted when empty); those symptoms get no event of their own **Event Types:** @@ -1068,7 +1100,7 @@ event: fault_cleared data: {"event_type":"fault_cleared","fault":{"fault_code":"MOTOR_OVERHEAT",...},"timestamp":1735830060.456,"x-medkit":{"entity_type":"apps","entity_id":"motor_controller"}} ``` -**SOVD payload extension `x-medkit.entity_*`** (non-standard, SOVD-compatible): Resolution is best-effort and snapshotted at event arrival. Manifest and hybrid discovery use the linking result; runtime-only discovery falls back to the FQN's last segment and only emits the fields when an App with that ID exists in the cache. When no entity can be resolved (no reporting sources, orphan source, etc.), the entire `x-medkit` object is omitted and the consumer must fall back to discovery. +**SOVD payload extension `x-medkit.entity_*`** (non-standard, SOVD-compatible): Resolution uses the per-entity fault scope rule, the same in every discovery mode and for every event type including `fault_cleared`. The named entity is the app that owns the record's owner, either as its node FQN (the bare id for an external app) or as a path child of it (`/...`), or else the external component whose id is the owner or a path parent of it. When nested node paths give several such apps, the one with the longest owned source is named, and a tie goes to the smallest id. The last path segment of the owner is never read as an app id. Resolution is snapshotted at event arrival, so an event replayed from the buffer after discovery removed that entity still names it. When no entity owns the record, the entire `x-medkit` object is omitted and the consumer must fall back to discovery. **Response (503 Service Unavailable - Client Limit Reached):** ```json @@ -1086,7 +1118,7 @@ This replays any buffered events with ID > 5, then continues streaming new event #### GET /api/v1/components/{component_id}/faults -List all faults for a specific component. +List one entry per fault code the component owns, through the apps it hosts plus, for an external component, its own id. **Query Parameters:** - `status` - Filter by fault status: `pending`, `confirmed`, `cleared`, `all` (default: `pending` + `confirmed`) @@ -1099,10 +1131,8 @@ curl http://localhost:8080/api/v1/components/nav2_controller/faults **Response (200 OK):** ```json { - "component_id": "nav2_controller", - "source_id": "/nav2_controller", - "faults": [...], - "count": 1 + "items": [...], + "x-medkit": {"entity_id": "nav2_controller", "aggregation_level": "component", "aggregated": true, "count": 1} } ``` @@ -1704,7 +1734,7 @@ A program that hosts the core gets the cyclic-subscription samplers and the bulk - **Lock expiry.** `LockReaper(locks, subscriptions, interval_sec, logger)` (`core/managers/lock_reaper.hpp`) calls `LockManager::cleanup_expired()` every `interval_sec` (`LockConfig::cleanup_interval`, default 30, raised to 1 when lower) on a thread it owns. For each expired lock whose scope is empty or includes `cyclic-subscriptions` it removes the cyclic subscriptions of the locked entity. `subscriptions` may be null. The destructor and `stop()` join the thread. Without a reaper a host keeps expired locks until it calls `cleanup_expired()` itself. The ROS 2 node runs the same class. - **Scripts.** `make_script_host(config, plugin_backend)` (`core/managers/script_manager.hpp`) returns a `ScriptManager` and, when no plugin backend is given and `ScriptsConfig::scripts_dir` is not empty, the `DefaultScriptProvider` (fork/execvp) behind it. `ScriptsConfig` (`core/script_types.hpp`) has the defaults of the `scripts.*` parameters, and `entries` carries predefined scripts. The host returns `host.manager` from `GatewayServices::script_manager()` and keeps the `ScriptHost` alive as long as the services. With neither a backend nor a directory the script routes answer 501. - **Aggregation.** `AggregationHost(config, listen_port, mdns_name, logger)` (`core/aggregation/aggregation_host.hpp`) builds the `AggregationManager` from an `AggregationConfig` (the defaults of the `aggregation.*` parameters; the host range-checks values it reads from its own configuration) and, when `announce` or `discover` is set, the mDNS discovery that adds and removes peers and ignores this gateway's own announcements. The host passes `host.manager()` to `RESTServer::set_aggregation_manager()` and calls `host.merge_peers(areas, components, apps, functions)` on every entity refresh, before it fills the entity cache. The call merges the peers' entities into the four vectors and publishes the routing table, the leaf warnings and the per-entity peer contributors. It does nothing while there is no peer. Call `host.shutdown()` before `RESTServer::stop()`. -- **Triggers.** `make_trigger_host(config, notifier, conditions, wiring)` (`core/trigger_host.hpp`) builds the `SqliteTriggerStore` and the `TriggerManager`, wires the entity hierarchy, entity existence and topic lookups to `wiring.cache`, and restores persistent triggers. `TriggerHostConfig` has the names and defaults of the `triggers.*` parameters (`enabled`, `max_triggers`, `on_restart_behavior`, `storage_path`), and `data_poll_interval_ms` (default 1000, clamped to 100-60000 ms), which the node does not read. `register_builtin_conditions()` fills the `ConditionRegistry`. The host passes the manager to `RESTServer::set_trigger_handlers()` and returns it from `GatewayServices::trigger_manager()`. A trigger past `max_triggers` answers 503. Where its events come from is chosen by the wiring: `fault_events` (a `FaultEventSource`; it serves several subscribers, so this can be the source the host returns from `GatewayServices::fault_events()`) starts a `TriggerFaultFeed` that subscribes to it and turns each fault event into a `faults` notification, and a fault trigger fires on it. The feed's subscription ends when the host is destroyed. A data trigger reads from `topic_transport` (a `TopicSubscriptionTransport`, the ROS 2 node's choice) when its topic is known, else from `polled_plugins`: a `PollingDataItemSource` that reads the item through `read_plugin_data_item`, as `GET /{entity}/data/{item}` does once per `data_poll_interval_ms`, starting at once. The item name is the trigger `resource` path percent-decoded as the GET path is. It delivers the `data` field of the body (the whole body when there is none), so a trigger `path` is relative to the value, as it is for a topic. A host without topics sets only `polled_plugins`; a host with neither gets data triggers that never fire. The trigger's condition runs on every sample, so `EnterRange` fires once at the crossing and `OnChange` at each difference. An item that cannot be read in a round delivers nothing in that round. Keep the host alive as long as the services. The ROS 2 node uses the same builder with its topic transport and feeds fault triggers through `TriggerFaultSubscriber`, a `Ros2FaultEventSource` under a `TriggerFaultFeed`. +- **Triggers.** `make_trigger_host(config, notifier, conditions, wiring)` (`core/trigger_host.hpp`) builds the `SqliteTriggerStore` and the `TriggerManager`, wires the entity hierarchy, entity existence (by the trigger's entity type) and topic lookups to `wiring.cache`, sets the `faults` and `logs` scope functions and the `logs` restore replay (the `logs` ones read `wiring.log_manager`), and restores persistent triggers. `TriggerHostConfig` has the names and defaults of the `triggers.*` parameters (`enabled`, `max_triggers`, `on_restart_behavior`, `storage_path`), and `data_poll_interval_ms` (default 1000, clamped to 100-60000 ms), which the node does not read. `register_builtin_conditions()` fills the `ConditionRegistry`. The host passes the manager to `RESTServer::set_trigger_handlers()` and returns it from `GatewayServices::trigger_manager()`. A trigger past `max_triggers` answers 503. Where its events come from is chosen by the wiring: `fault_events` (a `FaultEventSource`; it serves several subscribers, so this can be the source the host returns from `GatewayServices::fault_events()`) starts a `TriggerFaultFeed` that subscribes to it and turns each fault event into a `faults` notification, and a fault trigger fires on it. The notification's entity is the app whose fault scope holds the record's owner (`make_fault_owner_resolver`), or none. The feed's subscription ends when the host is destroyed. A data trigger reads from `topic_transport` (a `TopicSubscriptionTransport`, the ROS 2 node's choice) when its topic is known, else from `polled_plugins`: a `PollingDataItemSource` that reads the item through `read_plugin_data_item`, as `GET /{entity}/data/{item}` does once per `data_poll_interval_ms`, starting at once. The item name is the trigger `resource` path percent-decoded as the GET path is. It delivers the `data` field of the body (the whole body when there is none), so a trigger `path` is relative to the value, as it is for a topic. A host without topics sets only `polled_plugins`; a host with neither gets data triggers that never fire. The trigger's condition runs on every sample, so `EnterRange` fires once at the crossing and `OnChange` at each difference. An item that cannot be read in a round delivers nothing in that round. Keep the host alive as long as the services. The ROS 2 node uses the same builder with its topic transport and feeds fault triggers through `TriggerFaultSubscriber`, a `Ros2FaultEventSource` under a `TriggerFaultFeed`. - **Plugin context.** `make_core_plugin_context(services, wiring)` (`core/plugins/core_plugin_context.hpp`) builds the context for plugins from `GatewayServices`: entities and the entity snapshot from `entity_cache()`, locks from `lock_manager()`, faults from `fault_manager()`, samplers from `sampler_registry()`. The host passes it to `PluginManager::set_context()` and keeps it alive as long as the plugins. `CorePluginContextWiring` carries what `GatewayServices` does not: getters for the change notifier and the condition registry (a null getter or a null result gives the plugin a null), a getter for the `AggregationManager` (the fault listings add the peers' faults), a callback for `notify_entities_changed()` (none: the call is ignored) and a sink for notices. The context is a `RosPluginContext`, so `as_ros_plugin_context()` is valid for every plugin, and its `node()` returns null: a plugin that needs ROS checks for it and declines to start. Plugins that use no ROS (a provider, a route, an introspection source) run unchanged. `PluginManager::load_plugins()` loads them from shared objects as on the node. The ROS 2 node derives its context from the same class and overrides only `node()`. - **Fault-trigger engine.** `make_fault_trigger_host(config, wiring)` (`core/fault_trigger_host.hpp`) builds the threshold-rule engine behind `/apps/{id}/fault-triggers` and the thread that evaluates its rules. `FaultTriggerHostConfig` has the names and defaults of the `fault_triggers.*` parameters (`enabled`, `poll_interval_ms`, raised to 50 ms and capped at `INT_MAX`, `storage_path`). The wiring has the entity cache, the `PluginManager` the rules read their values from (through the plugin's `x-plc-data` route), and the `FaultManager` a crossed rule reports to and clears through. The factory returns null when the configuration is disabled or no plugin is loaded. The host returns the engine from `GatewayServices::fault_trigger_engine()`; without it the routes answer 501. The engine consumes no fault events. The fault it reports confirms in the fault manager and reaches the fault SSE stream, the freeze-frame capture and fault triggers through the host's event source. Destroy the host before the plugins and the fault manager go. The ROS 2 node uses the same builder. - **Lifecycle.** `RESTServer` takes a `LifecycleStateReader` or null. A host without one passes null. `GET /{entity}/status` then reports `ready` for an online app and `notReady` for an offline one, also when the app exposes the lifecycle services, because graph presence is the only signal left. A component is ready unless every app it hosts is offline. `PUT /{entity}/status/{action}` answers 501 unless a plugin `LifecycleProvider` serves the entity. diff --git a/src/ros2_medkit_gateway/config/examples/fault_owner_identity_manifest.yaml b/src/ros2_medkit_gateway/config/examples/fault_owner_identity_manifest.yaml new file mode 100644 index 000000000..f44544460 --- /dev/null +++ b/src/ros2_medkit_gateway/config/examples/fault_owner_identity_manifest.yaml @@ -0,0 +1,59 @@ +# SOVD System Manifest: several reporters of one fault code +# ========================================================= +# Hybrid-mode manifest with three external apps hosted by one component in one +# area. They report the same fault_code to the fault_manager under their own +# entity ids. With that code configured per_source, each is a record of its own +# inside the component's and the area's fault scope. +# +# owner-a2 extends owner-a's id, so a prefix match would hand it owner-a's +# sub-sources (owner-a/x). It owns none of them. +# +# The apps are external: they report under their bare entity ids rather than a +# ROS FQN, which is also how a non-ROS asset bridged in by a protocol plugin +# reports. +# +# Used by: test/features/test_faults_owner_identity.test.py, +# test_faults_entity_clear_failure.test.py, test_rosbag_owner_download.test.py + +manifest_version: "1.0" + +metadata: + name: "fault-owner-identity" + version: "1.0.0" + description: "Two external apps under one component sharing a fault code" + +config: + unmanifested_nodes: warn + inherit_runtime_resources: true + +areas: + - id: shared-code-cell + name: "Shared Code Cell" + namespace: /shared_code_cell + description: "Cell hosting two assets that report the same fault code" + +components: + - id: shared-code-hub + name: "Shared Code Hub" + type: "controller" + area: shared-code-cell + description: "Hosts every reporting asset, so every record lands in one scope" + +apps: + - id: owner-a + name: "Owner A" + external: true + is_located_on: shared-code-hub + description: "Reports SHARED_CODE under its own entity id" + + - id: owner-b + name: "Owner B" + external: true + is_located_on: shared-code-hub + description: "Reports the same SHARED_CODE under its own entity id" + + - id: owner-a2 + name: "Owner A2" + external: true + is_located_on: shared-code-hub + description: "Its id extends owner-a's; it owns none of owner-a's sources" diff --git a/src/ros2_medkit_gateway/design/index.rst b/src/ros2_medkit_gateway/design/index.rst index 29543915c..fa6cb70a8 100644 --- a/src/ros2_medkit_gateway/design/index.rst +++ b/src/ros2_medkit_gateway/design/index.rst @@ -90,7 +90,8 @@ store from it: * ``make_trigger_host`` (``core/trigger_host.hpp``) builds the trigger store and the ``TriggerManager`` from a ``TriggerHostConfig`` (the ``triggers.*`` names and defaults, plus ``data_poll_interval_ms``), wires the entity lookups - to the entity cache and restores persistent triggers. Fault triggers are fed + to the entity cache, sets the ``faults`` and ``logs`` scope functions and the + ``logs`` restore replay, and restores persistent triggers. Fault triggers are fed by a ``TriggerFaultFeed`` that holds one ``FaultEventSubscription`` on a ``FaultEventSource``, so it can share the source with the fault SSE stream and the freeze-frame capture. A data trigger with a known topic reads @@ -723,19 +724,35 @@ It consists of five main components: - Evaluates conditions using registered ``ConditionEvaluator`` instances via the ``ConditionRegistry`` - Uses O(1) dispatch indexing by ``{collection, entity_id}`` for efficient notification matching - Supports entity hierarchy matching (area-level triggers catch descendant changes) + - A collection can route by a scope function on the change value instead of the change's entity id. + ``faults`` uses one: the record's owner against the entity's fault sources. ``logs`` uses one: the + entry's ``context.node`` against the entity's log sources and its level against the entity's + ``severity_filter``, checked at dispatch + - A persistent ``logs`` trigger restored before its entity is discovered replays, once, what + ``GET /{entity}/logs`` lists when the entity appears, less markers, and skips the live changes for + those entries + - The marker a firing writes through ``log_settings`` (``LogManager::add_log_entry``) is notified to every + subscription but the manager's own, so no trigger fires for it; ``LogManager::get_logs`` reports which + listed entries are markers, and the replay leaves them out + - The orphan sweep and the replay look the entity up by the trigger's entity type, so a Component + that shares an App's id does not stand in for the App - Manages pending events for SSE stream pickup with per-trigger mutexes - Persists triggers via the ``TriggerStore`` interface - Data-trigger ROS subscriptions are routed through ``TopicSubscriptionTransport`` (``Ros2TopicSubscriptionTransport`` adapter wraps ``TriggerTopicSubscriber``) 2. **ResourceChangeNotifier** ``[gateway_core]`` - Async notification hub for resource changes. - - Producers (FaultManager, DataAccessManager, UpdateManager, OperationManager, LogManager) call ``notify()`` + - Producers (FaultManager, DataAccessManager, UpdateManager, OperationManager, LogManager) call ``notify()``; + ``notify_except()`` skips one subscription - Observers (TriggerManager) register callbacks with filters - ``notify()`` is non-blocking - pushes to an internal queue processed by a dedicated worker thread - Filters support collection, entity_id, and resource_path matching - ``unsubscribe(id)`` from another thread returns once that callback is not running, and the callback is never called again, so an observer can be destroyed right after it unsubscribes. From inside a notifier callback it returns at once, the current invocation continues, and the callback is not called again after it returns. + - The queue holds 10000 changes and drops the oldest past that + - A ``logs`` change names the one app whose log scope holds the logger, or no entity; the logger name is + ``context.node`` in the value - **Teardown contract:** producers hold a non-owning ``ResourceChangeNotifier *``, so every producer must be quiesced before the notifier is destroyed. ``notify()``'s internal drain guard only protects calls that are already in progress - it cannot help a call that starts after the object is freed. Most producers notify from diff --git a/src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/faults/fault_json.hpp b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/faults/fault_json.hpp index 253b54967..ba6fa8e07 100644 --- a/src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/faults/fault_json.hpp +++ b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/faults/fault_json.hpp @@ -21,7 +21,8 @@ namespace ros2_medkit_gateway { /// Fault JSON object served by /faults and the fault stream. Times are seconds -/// as doubles; ``last_passed`` is present only when set. +/// as doubles; ``last_passed`` is present only when set. ``source_id`` is the first +/// reporting source, the value the per-record routes take. nlohmann::json fault_record_to_json(const ros2_medkit::fault_detection::FaultRecord & fault); } // namespace ros2_medkit_gateway diff --git a/src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/faults/fault_scope.hpp b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/faults/fault_scope.hpp index f97102fbf..8600d5d1b 100644 --- a/src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/faults/fault_scope.hpp +++ b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/faults/fault_scope.hpp @@ -14,10 +14,16 @@ #pragma once +#include +#include +#include #include #include #include +#include +#include +#include "ros2_medkit_gateway/core/discovery/models/app.hpp" #include "ros2_medkit_gateway/core/models/entity_types.hpp" namespace ros2_medkit_gateway { @@ -26,11 +32,18 @@ class ThreadSafeEntityCache; namespace faults { -/// Resolve the single fault-reporting source an app owns, or "" when it owns -/// none. An external app (a plugin-introspected asset with no ROS binding) -/// reports under its bare entity id; every other app owns its `effective_fqn()`. -/// An unbound non-external app owns nothing - granting it its bare id would let -/// it claim faults it never reported. +/// True when `source` equals `owned` or is a path child of it (`/...`). +/// The one rule for "this reporting source belongs to this owner". +bool source_in_scope(const std::string & source, const std::string & owned); + +/// The fault-reporting source `app` owns, or "" when it owns none. An external +/// app (a plugin-introspected asset with no ROS binding) reports under its bare +/// entity id; every other app owns its `effective_fqn()`. An unbound +/// non-external app owns nothing - granting it its bare id would let it claim +/// faults it never reported. +std::string app_owned_source(const App & app); + +/// `app_owned_source` of the cached app `app_id`, or "" when it is not cached. std::string resolve_app_source_fqn(const ThreadSafeEntityCache & cache, const std::string & app_id); /// Resolve the set of fault-reporting sources that fall within an entity's @@ -57,5 +70,103 @@ bool fault_in_source_scope(const nlohmann::json & fault, const std::set & source_fqns); +/// One fault record paired with the reporting source that owns it. +struct ScopedFault { + nlohmann::json fault; + std::string owner; +}; + +/// The first reporting source of `fault`, or "" when the record names none. It +/// is the owner of a per_source record, and a source that reaches a shared one. +std::string record_owner(const nlohmann::json & fault); + +/// Every record of `fault_code` in `faults_array` whose owner lies inside +/// `source_fqns`, ordered by owner so the answer does not depend on the order +/// the store listed them in. A rosbag download asks each of their owners +/// whether it holds the recording. +std::vector records_of_code_in_scope(const nlohmann::json & faults_array, const std::string & fault_code, + const std::set & source_fqns); + +/// The records of `fault_code` inside `source_fqns` that a recording URL +/// carrying a fault code resolves over. +/// +/// `listing` is a fault-manager listing read with every status and with muted +/// records included: its `faults` array holds every record, and each +/// `muted_faults` entry names one the correlation engine hides by its +/// `fault_code` and `source_id`. +/// +/// The candidates come in three tiers, and the first tier holding any record of +/// the code decides: +/// 1. the records the entity's default fault list shows: PREFAILED and +/// CONFIRMED, not muted +/// 2. the muted records of those statuses, which that list would show but for +/// the correlation engine +/// 3. the records the list shows only when asked for status=cleared or +/// status=healed: CLEARED, HEALED and PREPASSED, muted or not +/// +/// So a record hidden as a symptom, or one a source cleared, stays addressable +/// by its code while it is the only one, and never makes the record a client +/// read off the list ambiguous. +std::vector addressable_records(const nlohmann::json & listing, const std::string & fault_code, + const std::set & source_fqns); + +/// The owner the fault manager keeps a record under: a per_source record's +/// reporting source, or "*" for a shared record. An item that names no owner +/// (an older fault manager, a peer gateway) is addressed by its first +/// reporting source. +std::string stored_owner(const nlohmann::json & fault); + +/// True for a shared record, the one record of a code that every source +/// reports into. An item that names no owner is shared. +bool is_shared_record(const nlohmann::json & fault); + +/// Names the App whose scope holds a reporting source, "" when none does. +using AppOfSource = std::function; + +/// One record of an entity's fault entry, as `x-medkit.records` lists it: the +/// App and source of a per_source record, the reporting sources of a shared +/// one, and its status. +nlohmann::json record_ref(const nlohmann::json & record, const AppOfSource & app_of); + +/// One record as a fault item: the fault manager's item without its internal +/// `owner` and `source_id`. A per_source record names its app and source in +/// `x-medkit.records`; a shared record has no `x-medkit`. +nlohmann::json record_item(const nlohmann::json & record, const AppOfSource & app_of); + +/// The record whose environment data an entry of several records serves: the +/// CONFIRMED record that occurred last, else the record that occurred last. +/// `records` must not be empty. +const nlohmann::json & representative_record(const std::vector & records); + +/// One fault entry of an entity: `record_item` of its only record, else the +/// merge of its records. The merge takes the most advanced status, the highest +/// severity, the summed occurrence count, the earliest first and the latest +/// last occurrence, the union of reporting sources and the representative +/// record's description, and lists every record in `x-medkit.records`. +/// `records` must not be empty and must share one fault code. +nlohmann::json entry_item(const std::vector & records, const AppOfSource & app_of); + +/// The records of `faults` grouped by fault code, in the order each code first +/// appears, each group ordered by owner. +std::vector> group_by_code(const nlohmann::json & faults); + +/// Owned source of every cached app, rebuilt when the cache generation moves. +/// Thread-safe. One instance per caller that resolves record owners per event. +class AppSourceIndex { + public: + /// Id of the app whose scope holds `owner`, the record's reporting source, + /// i.e. whose `GET /apps/{id}/faults?status=all` lists the record. With + /// nested node paths the app owning the longest source wins, then the + /// smallest id. "" when no app holds it. + std::string find_owner(const ThreadSafeEntityCache & cache, const std::string & owner); + + private: + std::mutex mutex_; + bool built_{false}; + uint64_t generation_{0}; + /// (owned source, app id), one entry per app that owns a source. + std::vector> sources_; +}; + } // namespace faults } // namespace ros2_medkit_gateway diff --git a/src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/faults/freeze_frame_capture.hpp b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/faults/freeze_frame_capture.hpp index 70e8729fe..a1c0b72b3 100644 --- a/src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/faults/freeze_frame_capture.hpp +++ b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/faults/freeze_frame_capture.hpp @@ -104,7 +104,7 @@ class FreezeFrameCapture : public FreezeFrameSource { FreezeFrameCapture(FreezeFrameCapture &&) = delete; FreezeFrameCapture & operator=(FreezeFrameCapture &&) = delete; - std::vector frames_for(const std::string & fault_code) const override; + std::vector frames_for(const std::string & fault_code, const std::string & owner) const override; /// Parses a ListFaults reply body into standing faults. Nullopt when the body has no /// "faults" array. Items that lack a string "fault_code" or an array "reporting_sources" @@ -129,8 +129,8 @@ class FreezeFrameCapture : public FreezeFrameSource { /// Guards frames_, insertion_order_ and fallback_logged_. mutable std::mutex mutex_; - /// Keyed by fault code only: cross-entity isolation rests on the fault manager keeping - /// reporting_sources append-only for a code and on `get_fault` gating by source scope. + /// Keyed by the record the frames belong to, `(fault_code, owner)`, rendered as + /// `fault_code + '\0' + owner`, which no id can contain. std::unordered_map> frames_; std::deque insertion_order_; ///< eviction order (FIFO) std::unordered_set fallback_logged_; ///< fault codes already warned about (bounded) diff --git a/src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/faults/freeze_frame_source.hpp b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/faults/freeze_frame_source.hpp index 3ca53cdad..681a1eb87 100644 --- a/src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/faults/freeze_frame_source.hpp +++ b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/faults/freeze_frame_source.hpp @@ -62,8 +62,9 @@ class FreezeFrameSource { FreezeFrameSource(FreezeFrameSource &&) = delete; FreezeFrameSource & operator=(FreezeFrameSource &&) = delete; - /// Frames captured for a fault code (empty when none). Thread-safe. - virtual std::vector frames_for(const std::string & fault_code) const = 0; + /// Frames captured for one fault record, addressed by its code and the + /// reporting source that owns it (empty when none). Thread-safe. + virtual std::vector frames_for(const std::string & fault_code, const std::string & owner) const = 0; /// Build the compact {resource_id: value} dict from a DataProvider::list_data /// response. Items without a "value" field map to null; a response without an diff --git a/src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/faults/trigger_fault_feed.hpp b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/faults/trigger_fault_feed.hpp index ef06a9d0b..aca286464 100644 --- a/src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/faults/trigger_fault_feed.hpp +++ b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/faults/trigger_fault_feed.hpp @@ -24,17 +24,19 @@ namespace ros2_medkit_gateway { /// Turns the events of a `FaultEventSource` into "faults" notifications for the trigger manager. /// -/// The entity of an event is the first reporting source, mapped by the resolver when it knows the source and by its -/// last path segment otherwise. The resource path is "/" plus the fault code; the value is the fault JSON plus -/// `event_type`. The feed holds one subscription on the source, so other consumers can subscribe to the same source. +/// Which triggers a change reaches is decided from the record's owner (its first reporting source) by the trigger +/// manager's "faults" scope function, not from the change's entity id. The entity id is the app the resolver maps the +/// owner to, or empty when no app owns it, and is informational. The resource path is "/" plus the fault code; the +/// value is the fault JSON plus `event_type`. An event without an owner is dropped. The feed holds one subscription on +/// the source, so other consumers can subscribe to the same source. class TriggerFaultFeed { public: - /// Maps a reporting source to an entity id. Returns an empty string for a source it does not know. - using NodeToEntityFn = std::function; + /// Maps a record's owner (its reporting source) to the id of the app whose scope holds it, or "" when no app does. + using FaultOwnerFn = std::function; /// Subscribes to `source` at once. `source` and `notifier` must outlive the feed. /// Destroying the feed ends its subscription and waits for a running callback. - TriggerFaultFeed(FaultEventSource & source, ResourceChangeNotifier & notifier, NodeToEntityFn resolver = {}); + TriggerFaultFeed(FaultEventSource & source, ResourceChangeNotifier & notifier, FaultOwnerFn resolver = {}); ~TriggerFaultFeed(); TriggerFaultFeed(const TriggerFaultFeed &) = delete; TriggerFaultFeed & operator=(const TriggerFaultFeed &) = delete; @@ -42,13 +44,13 @@ class TriggerFaultFeed { TriggerFaultFeed & operator=(TriggerFaultFeed &&) = delete; /// Replaces the resolver. Call it before events can arrive: it is not synchronised with delivery. - void set_node_to_entity_resolver(NodeToEntityFn resolver); + void set_fault_owner_resolver(FaultOwnerFn resolver); private: void on_event(const ros2_medkit::fault_detection::FaultEventRecord & event); ResourceChangeNotifier & notifier_; - NodeToEntityFn resolver_; + FaultOwnerFn resolver_; FaultEventSubscription subscription_; }; diff --git a/src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/http/error_codes.hpp b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/http/error_codes.hpp index 1c0ae37de..2e64666b8 100644 --- a/src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/http/error_codes.hpp +++ b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/http/error_codes.hpp @@ -171,6 +171,11 @@ constexpr const char * ERR_SCRIPT_NOT_RUNNING = "x-medkit-script-not-running"; constexpr const char * ERR_SCRIPT_CONCURRENCY_LIMIT = "x-medkit-concurrency-limit"; constexpr const char * ERR_SCRIPT_FILE_TOO_LARGE = "x-medkit-script-too-large"; +/// A recording URL carrying a fault code addresses several records in the +/// entity's scope, so it names no single recording. Answered as 409 with the +/// code and the owners in `params`. +constexpr const char * ERR_AMBIGUOUS_FAULT = "x-medkit-ambiguous-fault"; + /// Plugin provider returned an error (used for DataProvider/OperationProvider/FaultProvider errors) constexpr const char * ERR_PLUGIN_ERROR = "x-medkit-plugin-error"; diff --git a/src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/http/handlers/bulkdata_handlers.hpp b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/http/handlers/bulkdata_handlers.hpp index 7e7809b4d..18422ac60 100644 --- a/src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/http/handlers/bulkdata_handlers.hpp +++ b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/http/handlers/bulkdata_handlers.hpp @@ -17,11 +17,13 @@ #include #include #include +#include #include #include #include #include +#include "ros2_medkit_gateway/core/managers/fault_manager.hpp" #include "ros2_medkit_gateway/dto/bulkdata.hpp" #include "ros2_medkit_gateway/http/handlers/handler_context.hpp" #include "ros2_medkit_gateway/http/response_types.hpp" @@ -159,8 +161,7 @@ namespace detail { * deployments grouping topics rather than nodes); FUNCTION never falls back * (pure aggregated view). * - * @param cache Entity cache to resolve hosted apps in (used for FUNCTION / - * COMPONENT only) + * @param cache Entity cache the entity's owned sources resolve in * @param entity Entity information * @return Vector of source filter strings (empty if no valid filters) */ @@ -183,25 +184,42 @@ std::vector compute_bulkdata_source_filters(const ThreadSafeEntityC std::string rosbag_recording_id(const std::string & file_path); /** - * @brief Fault codes a rosbag download is authorized against. + * @brief Fault codes of the records a rosbag recording is attached to. * - * A recording is shared by every fault of a burst, so ownership is the union - * over those faults rather than a single code: the entity that owns any one of - * them may download the bag. That grants nothing new - before recordings had - * their own identity, each of those faults already addressed its own copy of - * the same bytes - it only renames the door. + * A recording is shared by every fault of a burst, so it belongs to several + * records, each a (fault code, owner) pair. These are the codes of those + * records. The download authorizes on the pairs, not on the codes: the codes + * only say which of the entity's records to ask about, and + * ``rosbag_rows_hold_recording`` says whether one of those records' owners + * really holds the recording. A code alone would let an entity owning any + * record of the code download another owner's recording of it. * * When the wire carries no ``fault_codes`` the response came from a peer that - * predates the field, where the addressed id *was* the fault code; authorizing - * against the requested id then reproduces the previous check exactly. + * predates the field, where the addressed id *was* the fault code, so the + * requested id stands in for it. * * @param rosbag_data Rosbag response from the fault manager * @param requested_id The ``{file_id}`` path segment the client asked for - * @return Non-empty list of fault codes to test against the entity's scope + * @return Non-empty list of fault codes whose in-scope records to check */ std::vector rosbag_attached_fault_codes(const nlohmann::json & rosbag_data, const std::string & requested_id); +/** + * @brief Does one owner's rosbag listing hold the recording that was served? + * + * ListRosbags answers the rows whose record is owned by exactly the source it + * was asked for, one row per (fault code, recording) link. A row naming the + * served recording therefore proves that a record of that owner is attached to + * it. A row names it by the same recording id, or, when either side predates + * recording ids, by the same bag path. + * + * @param rows The ``rosbags`` array of one owner's ListRosbags answer + * @param served Rosbag response from the fault manager for the download + * @return True when a row names the served recording + */ +bool rosbag_rows_hold_recording(const nlohmann::json & rows, const nlohmann::json & served); + /** * @brief Did the fault manager read the URL segment as a FAULT CODE rather than a * recording id? @@ -280,13 +298,60 @@ std::optional rosbag_served_bytes(const std::string & bag_path); * ``x-medkit.storage_files`` is ``rosbag_storage().file_count``, omitted when * that is nullopt. Each row's metadata is read once. * - * @param rows Rosbag rows as returned by the fault manager - * @param faults_by_code Faults keyed by code, for timestamp enrichment + * @param rows Rosbag rows as returned by the fault manager, each stamped with + * the ``owner`` it was listed under + * @param faults_by_record Fault records keyed by ``record_map_key``, for + * timestamp enrichment * @return One descriptor per distinct recording */ std::vector fold_rosbag_rows_into_descriptors(const std::vector & rows, - const std::unordered_map & faults_by_code); + const std::unordered_map & faults_by_record); + +/** + * @brief Key one fault record for the descriptor date lookup. + * + * A record is the pair (owner, fault code), and the two owners of one + * per_source code have their own ``first_occurred``. + */ +std::string record_map_key(const std::string & owner, const std::string & fault_code); + +/** + * @brief Rosbag descriptors for ``GET /{entity}/bulk-data/rosbags``. + * + * A recording is listed when the owner of a record it was captured for is in + * ``scope`` (``faults::fault_in_source_scope``), whatever the record's status, + * muted records included. The fault manager matches a recording's owner + * exactly, so it is asked once per owner of an in-scope record. + * + * @param fault_mgr Fault manager to list faults and rosbags from + * @param scope The entity's rosbag scope (``compute_bulkdata_source_filters``) + * @return Descriptors, or a 503 when any fault manager call fails: an empty or + * partial list would misreport the recordings that exist + */ +tl::expected, ErrorInfo> +list_scoped_rosbag_descriptors(FaultManager & fault_mgr, const std::set & scope); + +/** + * @brief The recording ``GET /{entity}/bulk-data/rosbags/{id}`` serves. + * + * ``bulk_data_id`` is a recording id, or a fault code on the compatibility + * path, which resolves to the newest recording of the entity's one record of + * that code. The recording is served when a record in ``scope`` holds it. + * + * @param fault_mgr Fault manager to read records and recordings from + * @param scope The entity's rosbag scope (``compute_bulkdata_source_filters``) + * @param bulk_data_id The ``{id}`` path segment + * @param entity_id Entity id for the error parameters + * @return GetRosbag's answer for the served recording, or the error: 404 when + * no record in scope holds it, 409 when a fault code names several + * records, 503 when GetRosbag, the record list or a needed rosbag list + * gets no answer + */ +tl::expected resolve_scoped_rosbag(FaultManager & fault_mgr, + const std::set & scope, + const std::string & bulk_data_id, + const std::string & entity_id); } // namespace detail diff --git a/src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/logs/log_scope.hpp b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/logs/log_scope.hpp index 9730a3cfe..91bf71934 100644 --- a/src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/logs/log_scope.hpp +++ b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/logs/log_scope.hpp @@ -14,10 +14,16 @@ #pragma once +#include +#include #include +#include #include #include #include +#include +#include +#include #include "ros2_medkit_gateway/core/models/entity_types.hpp" @@ -28,6 +34,65 @@ class ThreadSafeEntityCache; namespace logs { +/// Log sources of one entity, as the log routes query them. +struct LogScope { + /// Matched exactly (slash or dot form). + std::vector exact; + /// Namespace prefix, COMPONENT and AREA only. Empty when there is none. + std::string prefix; +}; + +/// Build the scope `query_entity_logs` queries. `resolved_sources` (optional +/// out) receives the exact-match source set. +LogScope resolve_log_scope(const ThreadSafeEntityCache & cache, SovdEntityType type, const std::string & entity_id, + const std::string & entity_fqn, std::set * resolved_sources = nullptr); + +/// True when entries of `logger_name` fall inside `scope` +/// (`LogManager::logger_matches` per source). +bool log_scope_contains(const LogScope & scope, const std::string & logger_name); + +/// The fqn the log routes pass for an entity: component fqn, area namespace, +/// app effective fqn. Empty for FUNCTION and for an entity not in the cache. +std::string entity_log_fqn(const ThreadSafeEntityCache & cache, SovdEntityType type, const std::string & entity_id); + +/// True when the log routes' rule admits an entry of `logger_name` at +/// `level`: the logger is in the entity's scope and `level` is at or above the +/// entity's configured severity_filter. With a LogProvider plugin, GET +/// returns the plugin's entries instead. +bool entity_lists_log_entry(const LogManager & log_mgr, const ThreadSafeEntityCache & cache, SovdEntityType type, + const std::string & entity_id, const std::string & logger_name, uint8_t level); + +/// `entity_lists_log_entry` for the value of a "logs" change: the logger name +/// is `context.node`, the level is `LogManager::entry_level`. False for a value +/// that carries no logger name. +bool entity_lists_log_change(const LogManager & log_mgr, const ThreadSafeEntityCache & cache, SovdEntityType type, + const std::string & entity_id, const nlohmann::json & entry); + +/// Entries a restored logs trigger on the entity replays: what +/// `GET /{entity}/logs` lists now, oldest first, less the trigger markers +/// (`LogManager::add_log_entry`). nullopt while the entity of that type is not +/// in the cache or the listing fails; the replay is then retried by the next +/// sweep or live change. +std::optional> restore_replay_entries(LogManager & log_mgr, + const ThreadSafeEntityCache & cache, + SovdEntityType type, const std::string & entity_id); + +/// Loggers the log scope of each cached app holds, rebuilt when the cache +/// generation moves. Thread-safe. +class LoggerOwnerIndex { + public: + /// Id of the one app whose log scope holds `logger_name`, or "" when no app + /// or more than one does. + std::string find_owner(const ThreadSafeEntityCache & cache, const std::string & logger_name); + + private: + std::mutex mutex_; + bool built_{false}; + uint64_t generation_{0}; + /// Logger name in slash and dot form -> owning app id, "" when shared. + std::unordered_map owners_; +}; + /// Query every log entry within an entity's scope. /// /// Exact-match sources come from `faults::resolve_entity_source_fqns`, so the @@ -42,12 +107,14 @@ namespace logs { /// ascending and re-capped to the entity's configured max_entries. /// /// `resolved_sources` (optional out) receives the exact-match source set for -/// response metadata. Shared by the HTTP log handlers and the cyclic +/// response metadata. `marker_ids` (optional out) receives the ids of the +/// listed trigger markers. Shared by the HTTP log handlers and the cyclic /// subscription "logs" sampler so both agree on entity -> log-source scoping. tl::expected query_entity_logs(LogManager & log_mgr, const ThreadSafeEntityCache & cache, SovdEntityType type, const std::string & entity_id, const std::string & entity_fqn, const std::string & min_severity, - const std::string & context_filter, std::set * resolved_sources = nullptr); + const std::string & context_filter, std::set * resolved_sources = nullptr, + std::unordered_set * marker_ids = nullptr); } // namespace logs } // namespace ros2_medkit_gateway diff --git a/src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/managers/fault_manager.hpp b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/managers/fault_manager.hpp index 1386d1daf..09e1c4ac6 100644 --- a/src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/managers/fault_manager.hpp +++ b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/managers/fault_manager.hpp @@ -66,31 +66,40 @@ class FaultManager { const std::string & source_id); /// Get all faults, optionally filtered by component (prefix match on source_id). + /// The filter is applied gateway-side: ListFaults carries no source field. FaultResult list_faults(const std::string & source_id = "", bool include_prefailed = true, bool include_confirmed = true, bool include_cleared = false, bool include_healed = false, bool include_muted = false, bool include_clusters = false); - /// Get a specific fault by code with environment data, returned as JSON. + /// Get one fault record with environment data, returned as JSON. + /// A record is the pair (fault_code, owner), and `source_id` names the owner and + /// is sent in the request. Empty is unscoped and the fault manager refuses it + /// when several records carry the code. /// `data` carries `{ "fault": {...}, "environment_data": {...} }`. The /// rosbag-snapshot bulk_data_uri is intentionally NOT included; per-request /// URL building belongs to the handler that knows the entity path. FaultWithEnvJsonResult get_fault_with_env(const std::string & fault_code, const std::string & source_id = ""); - /// Get a specific fault by code (JSON result - "fault" body only). + /// Get one fault record (JSON result - "fault" body only). FaultResult get_fault(const std::string & fault_code, const std::string & source_id = ""); - /// Clear a fault. When `skip_correlation_auto_clear` is true the fault - /// manager will not cascade-clear correlated symptom fault codes - per-entity - /// DELETE routes set this to keep their clear inside the entity boundary. - FaultResult clear_fault(const std::string & fault_code, bool skip_correlation_auto_clear = false); + /// Clear one fault record, addressed by code and owning source. When + /// `skip_correlation_auto_clear` is true the fault manager will not + /// cascade-clear correlated symptom fault codes - per-entity DELETE routes + /// set this to keep their clear inside the entity boundary. + FaultResult clear_fault(const std::string & fault_code, const std::string & source_id, + bool skip_correlation_auto_clear = false); - /// Get snapshots for a fault (optional topic filter). - FaultResult get_snapshots(const std::string & fault_code, const std::string & topic = ""); + /// Get snapshots of one fault record (optional topic filter). + FaultResult get_snapshots(const std::string & fault_code, const std::string & source_id, + const std::string & topic = ""); - /// Get rosbag file info for a fault. - FaultResult get_rosbag(const std::string & fault_code); + /// Get rosbag file info for a recording id, or for one fault record when the + /// id is a fault code. `source_id` scopes the fault-code lookup only. + FaultResult get_rosbag(const std::string & id, const std::string & source_id = ""); - /// Get all rosbag files for an entity (batch operation). + /// Get the rosbag files of the records `entity_fqn` owns, matched exactly + /// (batch operation, no prefix matching). FaultResult list_rosbags(const std::string & entity_fqn); /// Check if fault manager services are available. diff --git a/src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/managers/log_manager.hpp b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/managers/log_manager.hpp index 3c792ca97..acdd684cf 100644 --- a/src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/managers/log_manager.hpp +++ b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/managers/log_manager.hpp @@ -33,14 +33,13 @@ #include "ros2_medkit_gateway/core/log_types.hpp" #include "ros2_medkit_gateway/core/providers/log_provider.hpp" #include "ros2_medkit_gateway/core/providers/log_provider_registry.hpp" +#include "ros2_medkit_gateway/core/resource_change_notifier.hpp" #include "ros2_medkit_gateway/core/transports/log_source.hpp" namespace ros2_medkit_gateway { using json = nlohmann::json; -class ResourceChangeNotifier; // forward declaration - /** * @brief Manages log collection via the default ring-buffer backend. * @@ -81,6 +80,9 @@ class LogManager { /// when the gateway adapter forwards to /rosout. May be null in tests. using LogSink = std::function; + /// Maps a logger name to the entity id its "logs" change carries. + using LoggerOwnerFn = std::function; + /** * @brief Construct LogManager * @@ -120,11 +122,14 @@ class LogManager { * @param min_severity Additional severity filter from query parameter. Empty = no override. * @param context_filter Substring filter applied to log entry's name (logger name). Empty = no filter. * @param entity_id Entity ID for config lookup. Empty = use defaults. + * @param marker_ids Optional out: receives the ids of the returned entries that add_log_entry() + * wrote. A LogProvider plugin's entries are never markers. * @return JSON array of LogEntry objects sorted by id ascending, capped at entity config max_entries. */ tl::expected get_logs(const std::vector & node_fqns, bool prefix_match, const std::string & min_severity, const std::string & context_filter, - const std::string & entity_id); + const std::string & entity_id, + std::unordered_set * marker_ids = nullptr); /// Get current log configuration for entity (returns defaults if unconfigured) tl::expected get_config(const std::string & entity_id) const; @@ -137,32 +142,40 @@ class LogManager { const std::optional & max_entries); /** - * @brief Programmatically add a log entry (e.g. from trigger log_settings) + * @brief Add the marker entry of a trigger's log_settings * * Creates a LogEntry and pushes it to the internal ring buffer using the - * same path as on_log_entry(). If a ResourceChangeNotifier is set, emits a - * "logs" CREATED notification so triggers can observe log changes. + * same path as on_log_entry(). If a ResourceChangeNotifier is set, emits the + * same "logs" CREATED change a source entry does, to every subscription but + * `skipped_subscription`. get_logs() reports the entry as a marker. * * @param entity_id Entity to associate the log with (used as logger name) * @param severity SOVD severity string (debug, info, warning, error, fatal) * @param message Human-readable log message * @param metadata Additional JSON metadata stored in the message (appended) + * @param skipped_subscription Notifier subscription the change skips; 0 = none */ void add_log_entry(const std::string & entity_id, const std::string & severity, const std::string & message, - const nlohmann::json & metadata); + const nlohmann::json & metadata, NotifierSubscriptionId skipped_subscription = 0); /// Set the ResourceChangeNotifier for emitting log change events. /// Called by GatewayNode after both LogManager and the notifier are available. + /// Each stored entry emits one "logs" CREATED change: entity_id from the + /// logger owner function, the entry as value, its logger name in + /// value.context.node. An entry the buffer cap drops emits none. void set_notifier(ResourceChangeNotifier * notifier); - /// Callback that maps a logger FQN to a manifest entity ID. - /// Returns empty string if the FQN cannot be resolved. - using NodeToEntityFn = std::function; + /// Set the function that gives a "logs" change its entity_id from the logger + /// name. Without one the entity_id is "". Call before set_notifier(). + void set_logger_owner_fn(LoggerOwnerFn fn); - /// Set the node-to-entity resolver for trigger notifications. - /// When set, on_log_entry() resolves logger names to manifest entity IDs before - /// notifying the ResourceChangeNotifier. - void set_node_to_entity_resolver(NodeToEntityFn resolver); + /// Numeric level of the entry in a "logs" change value. The value names the + /// severity only, so for an entry logged at a level between the named ones + /// the level is looked up by entry id. The lookup keeps as many such entries + /// as the notifier can hold waiting for dispatch + /// (ResourceChangeNotifier::max_in_flight), counting changes the queue drops; + /// an older one falls back to the severity name. 0 without a severity. + uint8_t entry_level(const nlohmann::json & entry) const; // ---- Static utilities (no middleware required - safe in unit tests) ---- @@ -181,6 +194,10 @@ class LogManager { /// Strip leading '/' from a node FQN for ring-buffer key normalization static std::string normalize_fqn(const std::string & fqn); + /// True when get_logs() with `node_fqn` covers entries of `logger_name`: the + /// FQN in slash or dot form, and with `prefix_match` any logger below it. + static bool logger_matches(const std::string & logger_name, const std::string & node_fqn, bool prefix_match); + // ---- Test injection (for unit tests - do not use in production code) ---- /** @@ -209,6 +226,15 @@ class LogManager { /// dispatch. void on_log_entry(const LogEntry & entry); + /// Emit the "logs" CREATED change for a stored entry, if a notifier is set. + void notify_entry(const LogEntry & entry, NotifierSubscriptionId skipped_subscription = 0); + + /// Drop the oldest entry of `buf`. Caller holds buffers_mutex_. + void pop_oldest_locked(std::deque & buf); + + /// Keep the level of an entry whose level has no severity name. + void remember_unnamed_level(const LogEntry & entry); + /// Get the effective LogProvider: plugin if registered, else nullptr (use local buffer) LogProvider * effective_provider() const; @@ -223,9 +249,7 @@ class LogManager { std::shared_ptr source_; LogProviderRegistry * provider_registry_; ResourceChangeNotifier * notifier_ = nullptr; - /// Write-once: must be set before the source starts delivering entries. - /// After that, only read from source callbacks (no mutex needed). - NodeToEntityFn node_to_entity_resolver_; + LoggerOwnerFn logger_owner_fn_; size_t max_buffer_size_; LogSink log_sink_; @@ -239,6 +263,8 @@ class LogManager { // Total number of entries that were rejected because adding a new node // would exceed the buffer cap. Guarded by buffers_mutex_. std::size_t dropped_total_{0}; + // Ids of buffered entries that add_log_entry() wrote. Guarded by buffers_mutex_. + std::unordered_set marker_ids_; mutable std::mutex buffers_mutex_; // Per-entity configuration keyed by entity_id @@ -247,6 +273,11 @@ class LogManager { // Monotonically increasing entry ID (never resets; starts at 1) std::atomic next_id_{1}; + + // Exact levels of notified entries with no severity name, oldest first. + std::unordered_map unnamed_levels_; + std::deque unnamed_level_order_; + mutable std::mutex unnamed_levels_mutex_; }; } // namespace ros2_medkit_gateway diff --git a/src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/managers/trigger_manager.hpp b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/managers/trigger_manager.hpp index f54a6f1bb..7b7bafe59 100644 --- a/src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/managers/trigger_manager.hpp +++ b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/managers/trigger_manager.hpp @@ -162,8 +162,38 @@ class TriggerManager { /// Set the entity hierarchy resolver. Called by GatewayNode after cache is available. void set_entity_children_fn(EntityChildrenFn fn); + /// Decides from the change payload whether a change concerns a trigger's + /// entity. Arguments: trigger entity type ("apps", ...), trigger entity id, + /// change value. + using CollectionScopeFn = + std::function; + + /// Route changes on `collection` by `fn` alone: a change reaches a trigger iff + /// `fn` accepts it; the change's entity id and the entity hierarchy are not + /// consulted. GatewayNode sets it for "faults", where ownership follows the + /// record's owner, and for "logs", where it follows the logger name in the + /// entry. + void set_collection_scope_fn(const std::string & collection, CollectionScopeFn fn); + + /// Items an entity lists in a collection, oldest first, or nullopt while + /// they cannot be listed: the entity is not in the cache or the listing + /// failed. nullopt keeps the replay pending. Arguments: entity type + /// ("apps", ...), entity id. + using RestoreReplayFn = std::function>(const std::string & entity_type, + const std::string & entity_id)>; + + /// A trigger restored on `collection` evaluates, once, the items `fn` lists + /// when its entity is first in the cache, and then skips live changes whose + /// value "id" it replayed. Changes that arrived before discovery reach it + /// this way. Set before load_persistent_triggers(); GatewayNode sets it for + /// "logs". + void set_restore_replay_fn(const std::string & collection, RestoreReplayFn fn); + /// Set the LogManager for trigger log_settings integration. /// Called by GatewayNode after both TriggerManager and LogManager are available. + /// The marker entry a firing writes reaches every notifier subscription but + /// this manager's, so it fires no trigger. This holds because GatewayNode + /// creates one TriggerManager per notifier. void set_log_manager(LogManager * log_manager); /// Resolves a resource_path to a full ROS 2 topic name for a given entity. @@ -230,6 +260,11 @@ class TriggerManager { std::deque pending_events; static constexpr size_t kMaxPendingEvents = 100; std::atomic event_counter{0}; + /// Restored trigger whose restore replay has not run. Read without `mtx` + /// by the sweep; the replay clears it under `mtx`. + std::atomic replay_pending{false}; + /// Value ids the restore replay evaluated. + std::unordered_set replayed_ids; }; /// Generate the next trigger ID. @@ -246,8 +281,21 @@ class TriggerManager { void remove_from_dispatch_index(const std::string & trigger_id, const std::string & collection, const std::string & entity_id); - /// Check if the notification entity matches the trigger's entity (direct or hierarchy). - bool matches_entity(const TriggerState & state, const std::string & notification_entity_id) const; + /// Check if a change concerns the trigger's entity: by the collection's scope + /// function when one is set, otherwise by direct or hierarchy id match. + bool matches_entity(const TriggerState & state, const ResourceChange & change) const; + + /// Whether the trigger's entity is in discovery, by the entity-exists function. + bool entity_exists(const TriggerState & state) const; + + /// Evaluate `value` against the trigger's condition and queue an event when + /// it fires. Caller holds state.mtx. nullopt when the trigger's path does not + /// resolve in `value` or its condition type has no evaluator. + std::optional evaluate_locked(TriggerState & state, const nlohmann::json & value); + + /// Run the pending restore replay if the entity is in the cache. Caller + /// holds state.mtx. Returns true when a one-shot trigger fired and ended. + bool replay_restored_locked(TriggerState & state); /// Check if a trigger has expired. bool is_expired(const TriggerState & state) const; @@ -287,9 +335,11 @@ class TriggerManager { // Removal callback OnRemovedCallback on_removed_; - // Hierarchy resolver + // Hierarchy resolver and per-collection scope functions mutable std::mutex hierarchy_mutex_; EntityChildrenFn entity_children_fn_; + std::unordered_map collection_scope_fns_; + std::unordered_map restore_replay_fns_; // Data trigger topic transport (shared, optional). The adapter side owns // the rclcpp resources; the manager only stores per-trigger handles. @@ -311,11 +361,6 @@ class TriggerManager { mutable std::mutex entity_exists_mutex_; EntityExistsFn entity_exists_fn_; - // Recursive loop prevention: set when evaluating a trigger fires a log entry - // that would re-enter on_resource_change(). Only accessed from the notifier - // worker thread (single-threaded dispatch), so no synchronization needed. - bool evaluating_trigger_{false}; - // Deferred resolution for data triggers whose topic was not discoverable when // the trigger entered this process - created through the API before the topic // existed, or restored from the store. Periodically retried via diff --git a/src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/models/thread_safe_entity_cache.hpp b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/models/thread_safe_entity_cache.hpp index 04d801939..87fd42247 100644 --- a/src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/models/thread_safe_entity_cache.hpp +++ b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/models/thread_safe_entity_cache.hpp @@ -294,6 +294,8 @@ class ThreadSafeEntityCache { bool has_component(const std::string & id) const; bool has_app(const std::string & id) const; bool has_function(const std::string & id) const; + /// The has_* check for `type`; false for SERVER and UNKNOWN. + bool has_entity(SovdEntityType type, const std::string & id) const; // --- Topic type lookup (O(1)) --- /** diff --git a/src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/plugins/gateway_plugin.hpp b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/plugins/gateway_plugin.hpp index bb24aa58e..566ea6296 100644 --- a/src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/plugins/gateway_plugin.hpp +++ b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/plugins/gateway_plugin.hpp @@ -122,16 +122,29 @@ class GatewayPlugin { } } + /// A copy of the log sink, for work that can outlive the call that started it. + /// + /// An asynchronous reply or a detached task that logs through `log_warn()` + /// holds `this`, and the plugin can be destroyed while that work runs. + /// Capturing this copy instead keeps the work from touching the plugin. The + /// copy is empty while no sink is wired, so check it before calling it. + std::function log_sink() const { + return log_fn_; + } + + /// Wire the log sink. PluginManager does this for every plugin it loads, + /// before configure(). A plugin hosted without a PluginManager, in a unit + /// test for example, wires its own through a subclass. The sink is not + /// synchronized with the log calls, so wire it while no plugin thread logs. + void set_logger(std::function fn) { + log_fn_ = std::move(fn); + } + private: friend class PluginManager; // Sets log_fn_ after construction /// Logging callback set by PluginManager. Routes to rclcpp::get_logger("plugin."). std::function log_fn_; - - /// Called by PluginManager to wire up logging - void set_logger(std::function fn) { - log_fn_ = std::move(fn); - } }; } // namespace ros2_medkit_gateway diff --git a/src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/plugins/plugin_manager.hpp b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/plugins/plugin_manager.hpp index 95b5534ac..d1fdeb378 100644 --- a/src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/plugins/plugin_manager.hpp +++ b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/plugins/plugin_manager.hpp @@ -235,6 +235,29 @@ class PluginManager : public LogProviderRegistry { std::optional fetch_entity_data_via_route(const std::string & entity_id, const std::string & item = ""); + /** + * @brief The entity's current data content, from whichever source its owning + * plugin actually serves /data through. + * + * The owning plugin's `DataProvider::list_data` first, and only when the + * plugin exposes no provider for that entity, the plugin's own data route via + * `fetch_entity_data_via_route`. That is the order the /data endpoint itself + * resolves in, so a caller reading through this sees exactly what a client + * reading the entity would see. + * + * Reading the route first instead is not a fallback, it is a different + * answer: a plugin that serves /data through a provider and registers no + * vendor route returns nothing at all, and every caller downstream reads that + * as "this entity has no data right now". + * + * A throwing provider is treated as an absent one and falls through to the + * route: an in-process plugin call must not take the caller's loop down. + * + * @return Parsed content on success. Nullopt when the entity is not + * plugin-owned, or neither source can answer. + */ + std::optional fetch_entity_data_content(const std::string & entity_id); + /// Whether the entity's owning plugin registered a GET data route /// (x-plc-data) that fetch_entity_data_via_route() could dispatch. Cheap /// route-table check, no handler invocation - capability advertising uses diff --git a/src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/plugins/plugin_types.hpp b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/plugins/plugin_types.hpp index 1580b5a52..cdf889369 100644 --- a/src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/plugins/plugin_types.hpp +++ b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/plugins/plugin_types.hpp @@ -40,11 +40,22 @@ namespace ros2_medkit_gateway { /// so a pre-compiled v6 `.so` is rejected. Out-of-tree plugins must be /// recompiled against v7 headers; in-tree plugins that `return /// PLUGIN_API_VERSION` pick up the bump automatically. +/// - v8: FaultProvider::clear_fault_record(entity_id, fault_code, owner), which +/// the gateway now calls to clear one fault record. Its default +/// implementation calls the unchanged two-argument clear_fault(), so +/// plugin SOURCE written against v7 compiles unchanged against v8 +/// headers (source-compatible). The new virtual changes the +/// FaultProvider vtable, so a pre-compiled v7 `.so` is rejected by the +/// strict equality check and must be recompiled against v8 headers. +/// GatewayPlugin also gains log_sink(), a copy of the log sink for work +/// that can outlive the plugin, and set_logger() becomes protected so a +/// plugin hosted without a PluginManager can wire its own. Both are +/// non-virtual and change no layout. /// /// The version is bumped when the PluginContext vtable changes, when a /// GatewayPlugin or provider interface changes incompatibly, and when a type /// that crosses the plugin boundary changes layout. -constexpr int PLUGIN_API_VERSION = 7; +constexpr int PLUGIN_API_VERSION = 8; /// Log severity levels for plugin logging callback enum class PluginLogLevel { kInfo, kWarn, kError }; diff --git a/src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/providers/fault_provider.hpp b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/providers/fault_provider.hpp index 2f23d85f1..2f9b92aba 100644 --- a/src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/providers/fault_provider.hpp +++ b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/providers/fault_provider.hpp @@ -89,6 +89,31 @@ class FaultProvider { /// @param fault_code Fault code to clear virtual tl::expected clear_fault(const std::string & entity_id, const std::string & fault_code) = 0; + + /// Clear one fault record, named by its code and the source that owns it. + /// + /// This is what the gateway calls for `DELETE /{entity}/faults/{code}` and + /// for each item of `DELETE /{entity}/faults`. A fault record is the pair + /// (fault_code, owner), so the code alone does not name one when several + /// sources report it. + /// + /// The default implementation calls `clear_fault(entity_id, fault_code)` and + /// drops the owner, so a provider written against the two-argument contract + /// compiles unchanged and keeps clearing by code. A provider that can scope + /// its clear to one record overrides this method. + /// + /// @param entity_id SOVD entity ID the request addressed + /// @param fault_code Fault code to clear + /// @param owner Reporting source that owns the record, as the gateway resolved + /// it in the entity's fault scope. It is NOT the entity id: a component + /// owns the records its hosted apps reported, so the owner is the app. + /// Empty only when the fault manager holds no record of this code (a + /// record the provider's own backend keeps, so the provider decides). + virtual tl::expected + clear_fault_record(const std::string & entity_id, const std::string & fault_code, const std::string & owner) { + static_cast(owner); + return clear_fault(entity_id, fault_code); + } }; } // namespace ros2_medkit_gateway diff --git a/src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/resource_change_notifier.hpp b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/resource_change_notifier.hpp index fd1e9a611..9647c5276 100644 --- a/src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/resource_change_notifier.hpp +++ b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/resource_change_notifier.hpp @@ -37,7 +37,7 @@ enum class ChangeType { CREATED, UPDATED, DELETED }; /// Describes a single resource change event. struct ResourceChange { std::string collection; ///< "faults", "updates", "data", "x-custom" - std::string entity_id; ///< "temp_sensor" + std::string entity_id; ///< "temp_sensor"; for "logs" the one owning app, or "" std::string resource_path; ///< "fault_003" or "temperature" nlohmann::json value; ///< The changed item (not full collection) ChangeType change_type; @@ -90,6 +90,11 @@ class ResourceChangeNotifier { void notify(const std::string & collection, const std::string & entity_id, const std::string & resource_path, const nlohmann::json & value, ChangeType change_type = ChangeType::UPDATED); + /// notify() for every subscription except `skipped`. 0 skips none. + void notify_except(NotifierSubscriptionId skipped, const std::string & collection, const std::string & entity_id, + const std::string & resource_path, const nlohmann::json & value, + ChangeType change_type = ChangeType::UPDATED); + /// Shut down the worker thread. After this, notify() is a no-op. /// Safe to call multiple times. void shutdown(); @@ -101,12 +106,21 @@ class ResourceChangeNotifier { /// Must be called before concurrent use (typically from GatewayNode init). void set_max_queue_size(size_t max_size); + /// Most changes that can wait for dispatch at once: the worker takes the + /// whole queue as one batch, and the queue refills while it dispatches. + size_t max_in_flight() const; + private: struct SubscriptionEntry { NotifierFilter filter; NotifierCallback callback; }; + struct QueuedChange { + ResourceChange change; + NotifierSubscriptionId skipped{0}; ///< Subscription not called; 0 = none. + }; + /// Check if a change matches a filter. static bool matches_filter(const NotifierFilter & filter, const ResourceChange & change); @@ -126,9 +140,9 @@ class ResourceChangeNotifier { std::condition_variable running_cv_; // Async work queue - std::mutex queue_mutex_; + mutable std::mutex queue_mutex_; std::condition_variable queue_cv_; - std::deque queue_; + std::deque queue_; size_t max_queue_size_{kDefaultMaxQueueSize}; size_t overflow_drop_count_{0}; ///< Accumulated drops since last overflow log diff --git a/src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/transports/fault_service_transport.hpp b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/transports/fault_service_transport.hpp index 97f908424..767c6d260 100644 --- a/src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/transports/fault_service_transport.hpp +++ b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/transports/fault_service_transport.hpp @@ -43,20 +43,30 @@ class FaultServiceTransport { bool include_cleared, bool include_healed, bool include_muted, bool include_clusters) = 0; + /// One fault record with its environment data. `source_id` names the source + /// that owns the record and is sent in the request, so the fault manager + /// resolves the record rather than the gateway filtering an arbitrary one. virtual FaultWithEnvJsonResult get_fault_with_env(const std::string & fault_code, const std::string & source_id) = 0; virtual FaultResult get_fault(const std::string & fault_code, const std::string & source_id) = 0; - /// Clear a fault by its fault_code. + /// Clear one fault record, addressed by `fault_code` and the `source_id` of + /// the source that owns it. An empty `source_id` is unscoped and the fault + /// manager refuses it when several records carry the code. /// `skip_correlation_auto_clear`, when true, asks the fault manager to NOT /// cascade-clear correlated symptom fault codes. Per-entity DELETE routes /// set this to true so the clear cannot reach faults reported by apps /// outside the addressed entity via the correlation graph. - virtual FaultResult clear_fault(const std::string & fault_code, bool skip_correlation_auto_clear = false) = 0; + virtual FaultResult clear_fault(const std::string & fault_code, const std::string & source_id, + bool skip_correlation_auto_clear = false) = 0; - virtual FaultResult get_snapshots(const std::string & fault_code, const std::string & topic) = 0; + virtual FaultResult get_snapshots(const std::string & fault_code, const std::string & source_id, + const std::string & topic) = 0; - virtual FaultResult get_rosbag(const std::string & fault_code) = 0; + /// Rosbag info for one recording, or for one fault record when `id` is a + /// fault code. `source_id` names the record's owner and scopes the fault-code + /// lookup only. The recording-id path ignores it. + virtual FaultResult get_rosbag(const std::string & id, const std::string & source_id) = 0; virtual FaultResult list_rosbags(const std::string & entity_fqn) = 0; diff --git a/src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/trigger_host.hpp b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/trigger_host.hpp index 05dc919db..6e2ad2684 100644 --- a/src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/trigger_host.hpp +++ b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/trigger_host.hpp @@ -77,8 +77,9 @@ struct TriggerHost { /// Registers OnChange, OnChangeTo, EnterRange and LeaveRange. void register_builtin_conditions(ConditionRegistry & conditions); -/// The entity a reporting source (a node FQN) belongs to, or an empty string. A leading slash is optional. -std::string resolve_trigger_entity(const ThreadSafeEntityCache & cache, const std::string & node_fqn); +/// Maps a fault record's owner to the id of the app whose scope holds it, the app whose +/// `GET /apps/{id}/faults?status=all` lists the record, or "" when none does. `cache` must outlive the resolver. +TriggerFaultFeed::FaultOwnerFn make_fault_owner_resolver(const ThreadSafeEntityCache & cache); /// The topic of `entity_id` a data trigger on `resource_path` binds to: the first one whose name equals the path or /// ends with it. Empty when there is none. `resource_path` starts with the separator slash, and the item after it may diff --git a/src/ros2_medkit_gateway/include/ros2_medkit_gateway/dto/faults.hpp b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/dto/faults.hpp index 64e07a49d..a4d9f9b07 100644 --- a/src/ros2_medkit_gateway/include/ros2_medkit_gateway/dto/faults.hpp +++ b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/dto/faults.hpp @@ -37,13 +37,51 @@ namespace ros2_medkit_gateway { namespace dto { // ============================================================================= -// FaultListItem - flat fault list item emitted by fault_msg_conversions.cpp -// (fault_to_json shape), wrapped in Collection for list endpoints. +// FaultRecordRef - one fault record an entry covers, in x-medkit.records +// (faults::record_ref). A per_source record has app_id and source_id, a shared +// record has reporting_sources. +// ============================================================================= +struct FaultRecordRef { + /// The app whose scope holds the record's source. Absent when no app does. + std::optional app_id; + /// The owner of a per_source record. + std::optional source_id; + /// The sources of a shared record. + std::optional> reporting_sources; + std::string status; +}; + +template <> +inline constexpr auto dto_fields = + std::make_tuple(field("app_id", &FaultRecordRef::app_id), field("source_id", &FaultRecordRef::source_id), + field("reporting_sources", &FaultRecordRef::reporting_sources), + field("status", &FaultRecordRef::status)); + +template <> +inline constexpr std::string_view dto_name = "FaultRecordRef"; + +// ============================================================================= +// FaultListItemXMedkit - x-medkit of a fault list item: the records it covers +// ============================================================================= +struct FaultListItemXMedkit { + std::optional> records; +}; + +template <> +inline constexpr auto dto_fields = + std::make_tuple(field("records", &FaultListItemXMedkit::records)); + +template <> +inline constexpr std::string_view dto_name = "FaultListItemXMedkit"; + +// ============================================================================= +// FaultListItem - flat fault list item emitted by faults::record_item and +// faults::entry_item, wrapped in Collection for list endpoints. // -// Wire keys (exact, from fault_msg_conversions.cpp): +// Wire keys: // fault_code, severity, description, first_occurred, last_occurred, // last_passed (absent = never passed), occurrence_count, status, -// reporting_sources, severity_label +// reporting_sources, severity_label, x-medkit // ============================================================================= struct FaultListItem { std::string fault_code; @@ -55,7 +93,8 @@ struct FaultListItem { std::optional occurrence_count; std::string status; std::optional> reporting_sources; - std::optional severity_label; // enum: INFO|WARN|ERROR|CRITICAL|UNKNOWN + std::optional severity_label; // enum: INFO|WARN|ERROR|CRITICAL|UNKNOWN + std::optional x_medkit; // wire key: "x-medkit" }; template <> @@ -65,7 +104,8 @@ inline constexpr auto dto_fields = std::make_tuple( field("last_occurred", &FaultListItem::last_occurred), field("last_passed", &FaultListItem::last_passed), field("occurrence_count", &FaultListItem::occurrence_count), field("status", &FaultListItem::status), field("reporting_sources", &FaultListItem::reporting_sources), - field_enum("severity_label", &FaultListItem::severity_label, kFaultSeverityLabelValues)); + field_enum("severity_label", &FaultListItem::severity_label, kFaultSeverityLabelValues), + field("x-medkit", &FaultListItem::x_medkit)); template <> inline constexpr std::string_view dto_name = "FaultListItem"; @@ -242,11 +282,19 @@ inline constexpr std::string_view dto_name = "FaultListAggX // FaultXMedkit - x-medkit vendor extension inside FaultDetail // // Wire keys (from build_sovd_fault_response): -// occurrence_count, reporting_sources, severity_label, status_raw +// occurrence_count, reporting_sources, owner, records, severity_label, status_raw +// +// The key is `owner`, not `source_id`: a fault LIST's x-medkit carries +// `source_id` for the addressed entity's namespace path (FaultListXMedkit). // ============================================================================= struct FaultXMedkit { std::optional occurrence_count; std::optional> reporting_sources; + /// The owner of a per_source record. Absent for a shared record and for an + /// entry of several records. + std::optional owner; + /// The records the detail covers. + std::optional> records; std::optional severity_label; std::optional status_raw; }; @@ -254,8 +302,8 @@ struct FaultXMedkit { template <> inline constexpr auto dto_fields = std::make_tuple(field("occurrence_count", &FaultXMedkit::occurrence_count), - field("reporting_sources", &FaultXMedkit::reporting_sources), - field("severity_label", &FaultXMedkit::severity_label), + field("reporting_sources", &FaultXMedkit::reporting_sources), field("owner", &FaultXMedkit::owner), + field("records", &FaultXMedkit::records), field("severity_label", &FaultXMedkit::severity_label), field("status_raw", &FaultXMedkit::status_raw)); template <> @@ -513,11 +561,10 @@ struct SchemaWriter { {"additionalProperties", true}, {"x-medkit-opaque", true}, {"description", - "Acknowledgement of a clear, shaped by whoever owns the entity. The ROS 2 path answers " - "`{\"code\": , \"cleared\": true}`; a plugin answers with its backend's own " - "acknowledgement - UDS clear response codes, vendor warnings, residual fault state - and the " - "gateway emits it verbatim. Treat the 2xx status, not a body field, as the signal that the clear " - "succeeded."}}; + "Acknowledgement of a clear, shaped by whoever owns the entity. The ROS 2 path answers 204 with " + "no body at all. A plugin answers with its backend's own acknowledgement (UDS clear response " + "codes, vendor warnings, residual fault state) and the gateway emits it verbatim. Treat the 2xx " + "status, not a body field, as the signal that the clear succeeded."}}; } }; diff --git a/src/ros2_medkit_gateway/include/ros2_medkit_gateway/dto/registry.hpp b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/dto/registry.hpp index dac110ef7..9896a2154 100644 --- a/src/ros2_medkit_gateway/include/ros2_medkit_gateway/dto/registry.hpp +++ b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/dto/registry.hpp @@ -52,10 +52,11 @@ using AllDtos = std::tuple, - Collection, Collection, Collection, FaultListItem, - Collection, Collection, - FaultListXMedkit, FaultListAggXMedkit, FaultStatus, FaultItem, FaultEnvironmentData, FaultXMedkit, - FaultDetail, FaultListResult, FaultDetailResult, FaultClearResult, + Collection, Collection, Collection, FaultRecordRef, + FaultListItemXMedkit, FaultListItem, Collection, + Collection, FaultListXMedkit, FaultListAggXMedkit, FaultStatus, + FaultItem, FaultEnvironmentData, FaultXMedkit, FaultDetail, FaultListResult, FaultDetailResult, + FaultClearResult, // Fault-trigger (threshold-rule) domain DTOs FaultTriggerRule, FaultTriggerRuleCreateRequest, FaultTriggerRuleList, XMedkitOperationItem, XMedkitOperationExecution, OperationItem, Collection, OperationDetail, OperationExecution, diff --git a/src/ros2_medkit_gateway/include/ros2_medkit_gateway/dto/sse_frames.hpp b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/dto/sse_frames.hpp index 2d827783d..381983cc0 100644 --- a/src/ros2_medkit_gateway/include/ros2_medkit_gateway/dto/sse_frames.hpp +++ b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/dto/sse_frames.hpp @@ -97,10 +97,9 @@ inline constexpr std::string_view dto_name = "TriggerEventFra // ----------------------------------------------------------------------------- // FaultStreamXMedkit - the `x-medkit` object on a fault-stream frame. // -// Present only when the fault's reporting source resolves to a known entity. -// It is a hint for addressing the fault's bulk-data, not an ownership claim: -// a debounced fault can have several co-reporters and this names the -// lexicographically first. +// Names the entity whose fault scope holds the record's owner, resolved when +// the event arrives: the app, else the external component. Absent when no +// entity owns it. // ----------------------------------------------------------------------------- struct FaultStreamXMedkit { std::string entity_type; @@ -108,11 +107,12 @@ struct FaultStreamXMedkit { }; template <> -inline constexpr auto dto_fields = std::make_tuple( - field("entity_type", &FaultStreamXMedkit::entity_type, "`areas`, `components`, `apps` or `functions`."), - field("entity_id", &FaultStreamXMedkit::entity_id, - "Entity to address for this fault's rosbag: " - "`GET /{entity_type}/{entity_id}/bulk-data/rosbags/{fault_code}`.")); +inline constexpr auto dto_fields = + std::make_tuple(field("entity_type", &FaultStreamXMedkit::entity_type, + "`apps`, or `components` for an external component that owns the record."), + field("entity_id", &FaultStreamXMedkit::entity_id, + "Entity to address for this fault's rosbag: " + "`GET /{entity_type}/{entity_id}/bulk-data/rosbags/{fault_code}`.")); template <> inline constexpr std::string_view dto_name = "FaultStreamXMedkit"; @@ -145,8 +145,8 @@ inline constexpr auto dto_fields = std::make_tuple( "Event time as seconds since the Unix epoch, with nanosecond precision in the fraction. Note this " "is a number, where the subscription and trigger frames send an ISO 8601 string."), field("x-medkit", &FaultStreamEvent::x_medkit, - "Where to fetch this fault's bulk-data. Absent when the reporting source resolves to no known " - "entity.")); + "The entity whose fault scope holds the record's owner, where the record's bulk-data lives. " + "Absent when no entity owns it.")); template <> inline constexpr std::string_view dto_name = "FaultStreamEvent"; diff --git a/src/ros2_medkit_gateway/include/ros2_medkit_gateway/entity_freeze_frame_capture.hpp b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/entity_freeze_frame_capture.hpp index 92674f1f3..2dcabaec2 100644 --- a/src/ros2_medkit_gateway/include/ros2_medkit_gateway/entity_freeze_frame_capture.hpp +++ b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/entity_freeze_frame_capture.hpp @@ -120,7 +120,7 @@ class EntityFreezeFrameCapture : public FreezeFrameSource { EntityFreezeFrameCapture(EntityFreezeFrameCapture &&) = delete; EntityFreezeFrameCapture & operator=(EntityFreezeFrameCapture &&) = delete; - std::vector frames_for(const std::string & fault_code) const override; + std::vector frames_for(const std::string & fault_code, const std::string & owner) const override; /// See FreezeFrameCapture::standing_faults_from_list_reply. static std::optional> standing_faults_from_list_reply(const nlohmann::json & data) { diff --git a/src/ros2_medkit_gateway/include/ros2_medkit_gateway/http/handlers/fault_handlers.hpp b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/http/handlers/fault_handlers.hpp index bc4727104..76d70637c 100644 --- a/src/ros2_medkit_gateway/include/ros2_medkit_gateway/http/handlers/fault_handlers.hpp +++ b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/http/handlers/fault_handlers.hpp @@ -17,10 +17,12 @@ #include #include #include +#include #include #include #include +#include "ros2_medkit_gateway/core/faults/fault_scope.hpp" #include "ros2_medkit_gateway/core/faults/fault_types.hpp" #include "ros2_medkit_gateway/core/faults/freeze_frame_source.hpp" #include "ros2_medkit_gateway/dto/faults.hpp" @@ -142,7 +144,9 @@ class FaultHandlers { * the handler appends `bulk_data_uri = entity_path + "/bulk-data/rosbags/" + fault_code`. * - "x-medkit" extensions with occurrence_count, severity_label, etc. * - * @param fault_json Per-fault JSON (as produced by the transport adapter). + * @param fault_json A fault item, `faults::record_item` or `faults::entry_item`. + * Its `x-medkit.records` is kept; the source of a single per_source + * record becomes `x-medkit.owner`. * @param env_data_json Environment-data JSON (as produced by the transport). * @param entity_path Entity path used to construct rosbag bulk_data_uri. * @return SOVD-compliant FaultDetail DTO @@ -159,22 +163,15 @@ class FaultHandlers { * or is a strict path-child (i.e. `/<...>`), so similarly named nodes * like `/ns/node` and `/ns/node_extra` are not conflated. * - * The "all sources must match" semantic (rather than "any source") is - * deliberate: it blocks two cross-entity escalation paths. - * - * 1. GET would otherwise return a response whose `reporting_sources` and - * environment data carry identities of nodes the caller has no - * business reading. - * 2. DELETE would otherwise let a viewer of entity A clear the aggregated - * fault record for a fault that entity B also reports, because the - * underlying `ClearFault.srv` has no scope argument. + * A per_source record names one reporting source; a shared record names + * every source that reported it, and is in the entity's scope only when all + * of them are. * * An empty scope set, an empty `reporting_sources` array, a missing * `reporting_sources` field, or any non-string source entry all return * false - there is no vacuous "all match" case. * - * Public for direct unit testing; called by `get_fault`, `clear_fault`, - * and indirectly via the per-entity collection routes. + * Public for direct unit testing. Called through the per-entity routes. */ static bool fault_in_source_scope(const nlohmann::json & fault, const std::set & source_fqns); @@ -219,8 +216,43 @@ class FaultHandlers { static nlohmann::json merge_entity_freeze_frames(nlohmann::json env_data, const std::vector & frames); + /** + * @brief Map every owned source to every entity that owns it, for the global + * clear's lock check: each app's source (an external app's bare id) and each + * external component's bare id. Two apps can bind one node path. The clear + * matches a record's owner against each key and its path children. + * + * Public static for direct unit testing. Called by `clear_all_faults_global`. + */ + static std::unordered_map> + build_source_entity_map(const ThreadSafeEntityCache & cache); + private: + /** + * @brief Every record of `fault_code` in `entity`'s scope, ordered by owner. + * + * Lists every status with muted records included: the per-code routes act + * on the code's records whatever their status. An empty result is not an + * error; the caller answers 404. A fault manager that cannot be read is the + * error, classified with `failure_summary`. + */ + tl::expected, ErrorInfo> + records_of_code(const EntityInfo & entity_info, const std::string & fault_code, const std::string & failure_summary); + + /** + * @brief The detail of the entry `records` form (not empty, one code): the + * record itself, or their merge with the environment data of + * `faults::representative_record`. Fails as the fault manager's read fails. + */ + tl::expected fault_detail(const std::vector & records, + const std::string & fault_code, + const std::string & entity_path); + + /// Names the app whose scope holds a reporting source, for `x-medkit.records`. + faults::AppOfSource app_of_source(); + HandlerContext & ctx_; + faults::AppSourceIndex app_index_; }; } // namespace handlers diff --git a/src/ros2_medkit_gateway/include/ros2_medkit_gateway/http/handlers/sse_fault_handler.hpp b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/http/handlers/sse_fault_handler.hpp index b0d30727a..d6d44e6ab 100644 --- a/src/ros2_medkit_gateway/include/ros2_medkit_gateway/http/handlers/sse_fault_handler.hpp +++ b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/http/handlers/sse_fault_handler.hpp @@ -31,6 +31,7 @@ #include "ros2_medkit_fault_detection/fault_types.hpp" #include "ros2_medkit_gateway/core/aggregation/peer_fault_relay.hpp" #include "ros2_medkit_gateway/core/faults/fault_event_source.hpp" +#include "ros2_medkit_gateway/core/faults/fault_scope.hpp" #include "ros2_medkit_gateway/core/http/sse_client_tracker.hpp" #include "ros2_medkit_gateway/http/handlers/handler_context.hpp" #include "ros2_medkit_gateway/http/response_types.hpp" @@ -68,10 +69,11 @@ namespace handlers { * - Replay buffer of up to 100 events. Eviction order under overflow: * 1. entries every live client has already been sent (free), * 2. fault_updated entries superseded by a newer event for the same fault - * code (coalesced - a lagging client still converges on the correct - * current state, and no status transition is erased), + * record, the same code and owner (coalesced - a lagging client still + * converges on the correct current state, and no status transition is + * erased), * 3. transition entries (confirmed / cleared) superseded by a newer event - * for the same code - the current state survives but the transition + * for the same record - the current state survives but the transition * history is lost, so this IS counted and logged as a drop, * 4. the oldest entry, owed and not superseded - counted and logged too. */ @@ -156,9 +158,9 @@ class SSEFaultHandler { /** * @brief Events genuinely lost: evicted while still owed to a live client, - * except fault_updated entries a newer same-code event supersedes. Includes - * superseded transitions - their history is gone even though the current - * state survives. + * except fault_updated entries a newer event of the same record supersedes. + * Includes superseded transitions - their history is gone even though the + * current state survives. * * Rotation of the replay buffer with no client attached is NOT a drop. */ @@ -259,7 +261,8 @@ class SSEFaultHandler { /// live client is attached. Caller holds queue_mutex_. std::optional delivered_watermark_locked() const; - /// Oldest buffered entry whose state a later same-code entry supersedes. + /// Oldest buffered entry whose state a later entry of the same record (fault + /// code and owner, from the same peer) supersedes. /// With `updates_only` set, only fault_updated entries qualify (erasing them /// cannot erase a status transition). Entries carrying auto_cleared_codes /// are never chosen: that correlation payload exists nowhere else. Caller @@ -284,18 +287,18 @@ class SSEFaultHandler { /// Format a fault event as SSE message static std::string format_sse_event(const QueuedEvent & queued); - /// Resolve the owning entity for a fault, snapshotting the cache. Manifest - /// / hybrid mode uses the cache's node-to-app index; runtime mode falls - /// back to the FQN's last segment, and for collision-disambiguated runtime - /// apps also to ``_`` per - /// ros2_runtime_introspection.cpp's renaming rule. Only emits a value when - /// an App with the resolved id actually exists in the cache - returns - /// ``std::nullopt`` otherwise so the consumer falls back to discovery. + /// Resolve the record's owning entity when the event arrives: the app whose + /// scope holds the owner (``faults::AppSourceIndex``), else the external + /// component that does. A buffered event keeps this answer on replay, even + /// after discovery drops the entity. ``std::nullopt`` when no entity owns it. std::optional resolve_entity_context(const ros2_medkit::fault_detection::FaultRecord & fault) const; HandlerContext & ctx_; std::shared_ptr client_tracker_; + /// Owned source of each cached app, for resolve_entity_context. + mutable faults::AppSourceIndex app_sources_; + /// Relays the peers' fault streams into this one on an aggregating gateway. /// Always constructed; it opens nothing when this gateway has no peers. std::unique_ptr peer_relay_; diff --git a/src/ros2_medkit_gateway/include/ros2_medkit_gateway/ros2/conversions/fault_msg_conversions.hpp b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/ros2/conversions/fault_msg_conversions.hpp index a4ec46ff3..caaf6de6c 100644 --- a/src/ros2_medkit_gateway/include/ros2_medkit_gateway/ros2/conversions/fault_msg_conversions.hpp +++ b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/ros2/conversions/fault_msg_conversions.hpp @@ -27,7 +27,8 @@ namespace ros2_medkit_gateway::ros2::conversions { ros2_medkit::fault_detection::FaultRecord from_msg(const ros2_medkit_msgs::msg::Fault & msg); ros2_medkit::fault_detection::FaultEventRecord from_msg(const ros2_medkit_msgs::msg::FaultEvent & msg); -/// Fault message to JSON through fault_record_to_json. +/// Fault message to JSON through fault_record_to_json, plus the record's `owner` +/// ("*" for a shared record) when the message names one. nlohmann::json fault_to_json(const ros2_medkit_msgs::msg::Fault & fault); /// Convert a ros2_medkit_msgs::msg::EnvironmentData to JSON. diff --git a/src/ros2_medkit_gateway/include/ros2_medkit_gateway/ros2/transports/ros2_fault_service_transport.hpp b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/ros2/transports/ros2_fault_service_transport.hpp index 6c408ca5e..37f5ef2bf 100644 --- a/src/ros2_medkit_gateway/include/ros2_medkit_gateway/ros2/transports/ros2_fault_service_transport.hpp +++ b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/ros2/transports/ros2_fault_service_transport.hpp @@ -74,11 +74,13 @@ class Ros2FaultServiceTransport : public FaultServiceTransport { FaultResult get_fault(const std::string & fault_code, const std::string & source_id) override; - FaultResult clear_fault(const std::string & fault_code, bool skip_correlation_auto_clear) override; + FaultResult clear_fault(const std::string & fault_code, const std::string & source_id, + bool skip_correlation_auto_clear) override; - FaultResult get_snapshots(const std::string & fault_code, const std::string & topic) override; + FaultResult get_snapshots(const std::string & fault_code, const std::string & source_id, + const std::string & topic) override; - FaultResult get_rosbag(const std::string & fault_code) override; + FaultResult get_rosbag(const std::string & id, const std::string & source_id) override; FaultResult list_rosbags(const std::string & entity_fqn) override; diff --git a/src/ros2_medkit_gateway/include/ros2_medkit_gateway/trigger_fault_subscriber.hpp b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/trigger_fault_subscriber.hpp index 21e4b3d13..aa8386dea 100644 --- a/src/ros2_medkit_gateway/include/ros2_medkit_gateway/trigger_fault_subscriber.hpp +++ b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/trigger_fault_subscriber.hpp @@ -41,13 +41,13 @@ class TriggerFaultSubscriber { */ TriggerFaultSubscriber(rclcpp::Node * node, ResourceChangeNotifier & notifier); - /// Callback to resolve ROS node FQN to manifest entity ID. - /// Returns manifest entity ID, or empty string if not resolvable. - using NodeToEntityFn = TriggerFaultFeed::NodeToEntityFn; + /// Maps a record's owner (its reporting source) to the id of the app whose + /// scope holds it, or "" when no app does. + using FaultOwnerFn = TriggerFaultFeed::FaultOwnerFn; - /// Set the node-to-entity resolver for manifest entity ID lookup. + /// Set the fault owner resolver used for the change's entity id. /// Write-once: call it before the ROS executor starts spinning. - void set_node_to_entity_resolver(NodeToEntityFn resolver); + void set_fault_owner_resolver(FaultOwnerFn resolver); ~TriggerFaultSubscriber() = default; TriggerFaultSubscriber(const TriggerFaultSubscriber &) = delete; diff --git a/src/ros2_medkit_gateway/src/core/builtin_samplers.cpp b/src/ros2_medkit_gateway/src/core/builtin_samplers.cpp index 5812c999c..c32b6bf38 100644 --- a/src/ros2_medkit_gateway/src/core/builtin_samplers.cpp +++ b/src/ros2_medkit_gateway/src/core/builtin_samplers.cpp @@ -122,20 +122,7 @@ Sampled sample_logs(GatewayServices & services, const std::string & entity_id) { } // Same scoping as GET /{entity}/logs via the shared helper (external app -> bare id, external // component owns its own id, plus the namespace-prefix merge for COMPONENT / AREA). - std::string entity_fqn; - if (entity_ref->type == SovdEntityType::COMPONENT) { - if (auto comp = cache.get_component(entity_id)) { - entity_fqn = comp->fqn; - } - } else if (entity_ref->type == SovdEntityType::AREA) { - if (auto area = cache.get_area(entity_id)) { - entity_fqn = area->namespace_path; - } - } else if (entity_ref->type == SovdEntityType::APP) { - if (auto app = cache.get_app(entity_id)) { - entity_fqn = app->effective_fqn(); - } - } + const auto entity_fqn = logs::entity_log_fqn(cache, entity_ref->type, entity_id); auto result = logs::query_entity_logs(*log_mgr, cache, entity_ref->type, entity_id, entity_fqn, "", ""); if (!result.has_value()) { return tl::make_unexpected(result.error()); diff --git a/src/ros2_medkit_gateway/src/core/fault_trigger_engine.cpp b/src/ros2_medkit_gateway/src/core/fault_trigger_engine.cpp index 2e376a27b..ffbb22907 100644 --- a/src/ros2_medkit_gateway/src/core/fault_trigger_engine.cpp +++ b/src/ros2_medkit_gateway/src/core/fault_trigger_engine.cpp @@ -176,16 +176,18 @@ tl::expected> FaultTriggerEngine:: } std::lock_guard lock(mutex_); - // fault_code is the PRIMARY KEY of the fault-manager store, so two rules - // sharing one code would fight over the same stored fault (one rule's clear - // erases the other's assertion) and make per-app delete/clear ambiguous. + // One rule per fault_code across every app, checked with no app filter. A code + // is how an operator names a rule's condition, and one code asserted by two + // rules cannot be read back to either of them. Unless the code is per_source + // in the fault manager, two rules on two apps would also share one record. const auto dup = std::find_if(rules_.begin(), rules_.end(), [&](const FaultTriggerRule & r) { return r.fault_code == rule.fault_code; }); if (dup != rules_.end()) { - return tl::make_unexpected( - std::make_pair(409, "fault_code '" + rule.fault_code + "' is already used by rule '" + dup->id + "' on app '" + - dup->app_id + "' - fault codes are global to the fault store, pick a distinct one")); + return tl::make_unexpected(std::make_pair( + 409, "fault_code '" + rule.fault_code + "' is already used by rule '" + dup->id + "' on app '" + dup->app_id + + "'. The trigger engine keeps one rule per fault code across every app, so " + "that a code names exactly one rule's condition. Pick a distinct fault_code.")); } rule.id = "ftr_" + std::to_string(next_seq_++); rules_.push_back(rule); diff --git a/src/ros2_medkit_gateway/src/core/fault_trigger_host.cpp b/src/ros2_medkit_gateway/src/core/fault_trigger_host.cpp index 1a6c04f6e..a120d3dcb 100644 --- a/src/ros2_medkit_gateway/src/core/fault_trigger_host.cpp +++ b/src/ros2_medkit_gateway/src/core/fault_trigger_host.cpp @@ -82,10 +82,10 @@ FaultTriggerHost::FaultTriggerHost(const FaultTriggerHostConfig & config, int po FaultManager * faults = wiring.faults; const ThreadSafeEntityCache * cache = wiring.cache; - // Same in-process x-plc-data route the freeze-frame capture uses, so a rule sees exactly the value the /data - // endpoint would serve. + // The owning plugin's DataProvider first, its vendor route only as a fallback: the content the /data endpoint would + // serve. A route-first read never sees an app whose plugin serves /data through a provider alone. auto fetcher = [plugins](const std::string & app_id, const std::string & data_name) -> std::optional { - const auto content = plugins->fetch_entity_data_via_route(app_id); + const auto content = plugins->fetch_entity_data_content(app_id); // A down link serves frozen last-known values. Returning nullopt holds the rule state instead of firing on a stale // number for the whole outage. if (!content || !FreezeFrameSource::content_has_live_data(*content) || @@ -106,17 +106,18 @@ FaultTriggerHost::FaultTriggerHost(const FaultTriggerHostConfig & config, int po } }; - // A rule's auto-clear is local to the rule: skip the correlation cascade, so clearing one rule's fault never wipes - // correlated faults owned by anything else. fault_code is unique across rules, so the app scope is in the code. - auto clear = [faults](const std::string & /*app_id*/, const std::string & fault_code) { + // The clear names the owner the report used (app_id), so it never reaches another source's record of the code. A + // rule's auto-clear is local to the rule: skip the correlation cascade, so clearing one rule's fault never wipes + // correlated faults owned by anything else. + auto clear = [faults](const std::string & app_id, const std::string & fault_code) { if (faults) { - faults->clear_fault(fault_code, /*skip_correlation_auto_clear=*/true); + faults->clear_fault(fault_code, app_id, /*skip_correlation_auto_clear=*/true); } }; - // Same route as the fetcher: "exists" means exactly "the fetcher could ever read it". + // Same source as the fetcher, in the same order: "exists" means exactly "the fetcher could ever read it". auto data_point_names = [plugins](const std::string & app_id) -> std::optional> { - const auto content = plugins->fetch_entity_data_via_route(app_id); + const auto content = plugins->fetch_entity_data_content(app_id); if (!content || !FreezeFrameSource::content_has_live_data(*content)) { return std::nullopt; } diff --git a/src/ros2_medkit_gateway/src/core/faults/fault_json.cpp b/src/ros2_medkit_gateway/src/core/faults/fault_json.cpp index 794170bd2..43c52e699 100644 --- a/src/ros2_medkit_gateway/src/core/faults/fault_json.cpp +++ b/src/ros2_medkit_gateway/src/core/faults/fault_json.cpp @@ -49,6 +49,11 @@ nlohmann::json fault_record_to_json(const fd::FaultRecord & fault) { j["occurrence_count"] = fault.occurrence_count; j["status"] = fault.status; j["reporting_sources"] = fault.reporting_sources; + // The first reporting source. The per-record routes take it back as source_id, + // so a client addresses the record without reaching into the array. + if (!fault.reporting_sources.empty()) { + j["source_id"] = fault.reporting_sources.front(); + } switch (fault.severity) { case fd::severity::kInfo: diff --git a/src/ros2_medkit_gateway/src/core/faults/fault_scope.cpp b/src/ros2_medkit_gateway/src/core/faults/fault_scope.cpp index 0f377026d..914c6003c 100644 --- a/src/ros2_medkit_gateway/src/core/faults/fault_scope.cpp +++ b/src/ros2_medkit_gateway/src/core/faults/fault_scope.cpp @@ -14,6 +14,9 @@ #include "ros2_medkit_gateway/core/faults/fault_scope.hpp" +#include +#include +#include #include #include @@ -91,37 +94,88 @@ void collect_function_app_fqns(const ThreadSafeEntityCache & cache, const std::s } } +/// True when the entity's fault list shows `fault` when no `status` is asked +/// for. That list reads the fault manager with parse_fault_status_param's +/// defaults, pending and confirmed, which the transport sends as PREFAILED and +/// CONFIRMED. CLEARED, HEALED and PREPASSED appear only under status=cleared, +/// status=healed or status=all. +bool shown_by_default_list(const nlohmann::json & fault) { + const auto status = fault.value("status", std::string{}); + return status == "PREFAILED" || status == "CONFIRMED"; +} + bool source_matches_scope(const std::string & src, const std::set & scope_fqns) { - for (const auto & fqn : scope_fqns) { - if (src == fqn) { - return true; - } - if (src.size() > fqn.size() && src.compare(0, fqn.size(), fqn) == 0 && src[fqn.size()] == '/') { - return true; - } + return std::any_of(scope_fqns.begin(), scope_fqns.end(), [&src](const std::string & fqn) { + return source_in_scope(src, fqn); + }); +} + +/// The owner the fault manager gives a shared record (Fault.msg `owner`). +constexpr const char * kSharedOwner = "*"; + +/// Rank of a record status: the higher, the more advanced the fault. +int status_rank(const std::string & status) { + if (status == "CONFIRMED") { + return 5; } - return false; + if (status == "PREFAILED") { + return 4; + } + if (status == "PREPASSED") { + return 3; + } + if (status == "HEALED") { + return 2; + } + if (status == "CLEARED") { + return 1; + } + return 0; +} + +double number_or(const nlohmann::json & j, const char * key, double fallback) { + const auto it = j.find(key); + return it != j.end() && it->is_number() ? it->get() : fallback; +} + +std::string owner_field(const nlohmann::json & fault) { + const auto it = fault.find("owner"); + return it != fault.end() && it->is_string() ? it->get() : std::string{}; } } // namespace -std::string resolve_app_source_fqn(const ThreadSafeEntityCache & cache, const std::string & app_id) { - auto app = cache.get_app(app_id); - if (!app) { - return ""; +bool source_in_scope(const std::string & source, const std::string & owned) { + if (owned.empty()) { + return false; + } + if (source == owned) { + return true; } + return source.size() > owned.size() && source.compare(0, owned.size(), owned) == 0 && source[owned.size()] == '/'; +} + +std::string app_owned_source(const App & app) { // External assets introspected by a protocol plugin (e.g. a PLC over OPC UA) // report faults to the fault_manager under their own entity id, so that id is // their sole fault-scope owner. This must be checked before effective_fqn(): // a manifest may declare a stray ros_binding on an external app, and the // derived FQN would otherwise shadow the bare id and drop its faults from // every rollup (#517, neighbor case). - if (app->external.value_or(false)) { - return app_id; + if (app.external.value_or(false)) { + return app.id; } // A non-external app that merely failed to bind owns nothing: granting an // unbound ROS app its bare id would let it claim faults it never reported. - return app->effective_fqn(); + return app.effective_fqn(); +} + +std::string resolve_app_source_fqn(const ThreadSafeEntityCache & cache, const std::string & app_id) { + auto app = cache.get_app(app_id); + if (!app) { + return ""; + } + return app_owned_source(*app); } std::set resolve_entity_source_fqns(const ThreadSafeEntityCache & cache, SovdEntityType type, @@ -180,5 +234,261 @@ nlohmann::json filter_faults_by_sources(const nlohmann::json & faults_array, return filtered; } +std::string record_owner(const nlohmann::json & fault) { + if (!fault.contains("reporting_sources") || !fault["reporting_sources"].is_array()) { + return {}; + } + const auto & sources = fault["reporting_sources"]; + if (sources.empty() || !sources.front().is_string()) { + return {}; + } + return sources.front().get(); +} + +std::vector records_of_code_in_scope(const nlohmann::json & faults_array, const std::string & fault_code, + const std::set & source_fqns) { + std::vector records; + if (!faults_array.is_array()) { + return records; + } + for (const auto & fault : faults_array) { + if (!fault.is_object() || fault.value("fault_code", std::string{}) != fault_code) { + continue; + } + // The scope predicate the list routes filter with. It decides which entity + // a record belongs to, not whether that entity's list shows it. + if (!fault_in_source_scope(fault, source_fqns)) { + continue; + } + records.push_back(ScopedFault{fault, record_owner(fault)}); + } + std::sort(records.begin(), records.end(), [](const ScopedFault & a, const ScopedFault & b) { + return a.owner < b.owner; + }); + return records; +} + +std::vector addressable_records(const nlohmann::json & listing, const std::string & fault_code, + const std::set & source_fqns) { + if (!listing.is_object()) { + return {}; + } + auto records = records_of_code_in_scope(listing.value("faults", nlohmann::json::array()), fault_code, source_fqns); + + // Owners whose record of this code is muted. An entry is one muted record, + // so the owner has to match as well as the code: muting one owner's record + // never hides another owner's record of the same code. + std::set muted_owners; + const auto muted_it = listing.find("muted_faults"); + if (muted_it != listing.end() && muted_it->is_array()) { + for (const auto & entry : *muted_it) { + if (entry.is_object() && entry.value("fault_code", std::string{}) == fault_code) { + muted_owners.insert(entry.value("source_id", std::string{})); + } + } + } + + // Three tiers, and the first one holding any record decides. The records the + // entity's default fault list shows come first. Then the muted ones, which + // that list would show but for the correlation engine. Then everything the + // list hides for its status: CLEARED, HEALED and PREPASSED, muted or not. + // Without the last split a record one source cleared long ago stayed a + // candidate beside the record the list shows, and the code answered 409 for + // as long as the cleared record was kept. + std::vector shown; + std::vector muted; + std::vector inactive; + for (auto & record : records) { + if (!shown_by_default_list(record.fault)) { + inactive.push_back(std::move(record)); + } else if (muted_owners.count(record.owner) > 0) { + muted.push_back(std::move(record)); + } else { + shown.push_back(std::move(record)); + } + } + if (!shown.empty()) { + return shown; + } + if (!muted.empty()) { + return muted; + } + return inactive; +} + +std::string stored_owner(const nlohmann::json & fault) { + auto owner = owner_field(fault); + return owner.empty() ? record_owner(fault) : owner; +} + +bool is_shared_record(const nlohmann::json & fault) { + const auto owner = owner_field(fault); + return owner.empty() || owner == kSharedOwner; +} + +nlohmann::json record_ref(const nlohmann::json & record, const AppOfSource & app_of) { + nlohmann::json ref = nlohmann::json::object(); + if (is_shared_record(record)) { + ref["reporting_sources"] = record.value("reporting_sources", nlohmann::json::array()); + } else { + const auto owner = stored_owner(record); + auto app_id = app_of ? app_of(owner) : std::string{}; + if (!app_id.empty()) { + ref["app_id"] = std::move(app_id); + } + ref["source_id"] = owner; + } + ref["status"] = record.value("status", std::string{}); + return ref; +} + +nlohmann::json record_item(const nlohmann::json & record, const AppOfSource & app_of) { + nlohmann::json item = record; + item.erase("owner"); + item.erase("source_id"); + if (!is_shared_record(record)) { + item["x-medkit"] = {{"records", nlohmann::json::array({record_ref(record, app_of)})}}; + } + return item; +} + +const nlohmann::json & representative_record(const std::vector & records) { + const nlohmann::json * best = &records.front(); + const auto better = [](const nlohmann::json & a, const nlohmann::json & b) { + const bool a_confirmed = a.value("status", std::string{}) == "CONFIRMED"; + const bool b_confirmed = b.value("status", std::string{}) == "CONFIRMED"; + if (a_confirmed != b_confirmed) { + return a_confirmed; + } + return number_or(a, "last_occurred", 0.0) > number_or(b, "last_occurred", 0.0); + }; + for (const auto & record : records) { + if (better(record, *best)) { + best = &record; + } + } + return *best; +} + +nlohmann::json entry_item(const std::vector & records, const AppOfSource & app_of) { + if (records.size() == 1) { + return record_item(records.front(), app_of); + } + const auto & representative = representative_record(records); + const nlohmann::json * most_severe = &records.front(); + const nlohmann::json * most_advanced = &records.front(); + std::set sources; + uint64_t occurrences = 0; + std::optional first_occurred; + std::optional last_occurred; + std::optional last_passed; + nlohmann::json refs = nlohmann::json::array(); + for (const auto & record : records) { + if (record.value("severity", 0) > most_severe->value("severity", 0)) { + most_severe = &record; + } + if (status_rank(record.value("status", std::string{})) > + status_rank(most_advanced->value("status", std::string{}))) { + most_advanced = &record; + } + occurrences += record.value("occurrence_count", uint64_t{0}); + for (const auto & source : record.value("reporting_sources", nlohmann::json::array())) { + if (source.is_string()) { + sources.insert(source.get()); + } + } + if (record.contains("first_occurred")) { + const double t = number_or(record, "first_occurred", 0.0); + first_occurred = first_occurred ? std::min(*first_occurred, t) : t; + } + if (record.contains("last_occurred")) { + const double t = number_or(record, "last_occurred", 0.0); + last_occurred = last_occurred ? std::max(*last_occurred, t) : t; + } + if (record.contains("last_passed")) { + const double t = number_or(record, "last_passed", 0.0); + last_passed = last_passed ? std::max(*last_passed, t) : t; + } + refs.push_back(record_ref(record, app_of)); + } + + nlohmann::json item = nlohmann::json::object(); + item["fault_code"] = representative.value("fault_code", std::string{}); + item["description"] = representative.value("description", std::string{}); + item["severity"] = most_severe->value("severity", 0); + item["severity_label"] = most_severe->value("severity_label", std::string{"UNKNOWN"}); + item["status"] = most_advanced->value("status", std::string{}); + item["occurrence_count"] = occurrences; + item["reporting_sources"] = std::vector(sources.begin(), sources.end()); + if (first_occurred) { + item["first_occurred"] = *first_occurred; + } + if (last_occurred) { + item["last_occurred"] = *last_occurred; + } + if (last_passed) { + item["last_passed"] = *last_passed; + } + item["x-medkit"] = {{"records", std::move(refs)}}; + return item; +} + +std::vector> group_by_code(const nlohmann::json & faults) { + std::vector> groups; + if (!faults.is_array()) { + return groups; + } + std::unordered_map index; + for (const auto & fault : faults) { + if (!fault.is_object()) { + continue; + } + const auto [it, inserted] = index.emplace(fault.value("fault_code", std::string{}), groups.size()); + if (inserted) { + groups.emplace_back(); + } + groups[it->second].push_back(fault); + } + for (auto & group : groups) { + std::stable_sort(group.begin(), group.end(), [](const nlohmann::json & a, const nlohmann::json & b) { + return stored_owner(a) < stored_owner(b); + }); + } + return groups; +} + +std::string AppSourceIndex::find_owner(const ThreadSafeEntityCache & cache, const std::string & owner) { + if (owner.empty()) { + return ""; + } + std::lock_guard lock(mutex_); + // Read before collecting: a mutation in between leaves a newer generation, + // so the next call rebuilds instead of keeping a stale index. + const auto generation = cache.generation(); + if (!built_ || generation != generation_) { + sources_.clear(); + for (const auto & app : cache.get_apps()) { + auto owned = app_owned_source(app); + if (!owned.empty()) { + sources_.emplace_back(std::move(owned), app.id); + } + } + generation_ = generation; + built_ = true; + } + + const std::pair * best = nullptr; + for (const auto & entry : sources_) { + if (!source_in_scope(owner, entry.first)) { + continue; + } + if (best == nullptr || entry.first.size() > best->first.size() || + (entry.first.size() == best->first.size() && entry.second < best->second)) { + best = &entry; + } + } + return best != nullptr ? best->second : ""; +} + } // namespace faults } // namespace ros2_medkit_gateway diff --git a/src/ros2_medkit_gateway/src/core/faults/freeze_frame_capture.cpp b/src/ros2_medkit_gateway/src/core/faults/freeze_frame_capture.cpp index b6c117e78..e47867d36 100644 --- a/src/ros2_medkit_gateway/src/core/faults/freeze_frame_capture.cpp +++ b/src/ros2_medkit_gateway/src/core/faults/freeze_frame_capture.cpp @@ -64,9 +64,20 @@ FreezeFrameCapture::~FreezeFrameCapture() { } } -std::vector FreezeFrameCapture::frames_for(const std::string & fault_code) const { +namespace { + +/// The storage key for one record. '\0' cannot appear in a fault code or in a +/// reporting source, so no pair of ids can collide with another pair. +std::string record_key(const std::string & fault_code, const std::string & owner) { + return fault_code + '\0' + owner; +} + +} // namespace + +std::vector FreezeFrameCapture::frames_for(const std::string & fault_code, + const std::string & owner) const { std::lock_guard lock(mutex_); - auto it = frames_.find(fault_code); + auto it = frames_.find(record_key(fault_code, owner)); return it != frames_.end() ? it->second : std::vector{}; } @@ -215,16 +226,20 @@ void FreezeFrameCapture::capture_standing_faults() { MEDKIT_LOG_WARN(logger_, "Entity freeze-frame startup catch-up failed: standing-fault lister threw"); return; } - // Codes with a confirm already queued belong to the drain loop: capturing - // them here too would read the plugin twice for one confirm. - std::unordered_set queued_codes; + // Records with a confirm already queued belong to the drain loop: capturing + // them here too would read the plugin twice for one confirm. The key is the + // record and not the code, because another source's record of the same code + // has its own confirm and its own frame. + std::unordered_set queued_records; { std::lock_guard lock(queue_mutex_); if (stop_) { return; } for (const auto & queued : queue_) { - queued_codes.insert(queued.fault.fault_code); + for (const auto & source : queued.fault.reporting_sources) { + queued_records.insert(record_key(queued.fault.fault_code, source)); + } } } size_t framed = 0; @@ -236,8 +251,14 @@ void FreezeFrameCapture::capture_standing_faults() { if (fault.fault_code.empty() || fault.reporting_sources.empty()) { continue; } - if (queued_codes.count(fault.fault_code) != 0) { - continue; + std::vector pending_sources; + for (const auto & source : fault.reporting_sources) { + if (queued_records.count(record_key(fault.fault_code, source)) == 0) { + pending_sources.push_back(source); + } + } + if (pending_sources.empty()) { + continue; // every record of this code is already on the drain loop } if (framed >= max_faults_) { ++over_cap; // storing more would FIFO-evict this catch-up's own frames @@ -246,7 +267,7 @@ void FreezeFrameCapture::capture_standing_faults() { fd::FaultEventRecord event; event.event_type = fd::fault_event_type::kConfirmed; event.fault.fault_code = fault.fault_code; - event.fault.reporting_sources = fault.reporting_sources; + event.fault.reporting_sources = std::move(pending_sources); if (capture_for_event(event, /*startup_catchup=*/true)) { ++framed; } @@ -328,15 +349,20 @@ bool FreezeFrameCapture::capture_for_event(const fd::FaultEventRecord & event, b } } + // One entry per source, under the source that reported it, because + // frames_for asks for one owner at a time. std::lock_guard lock(mutex_); - if (frames_.find(fault_code) == frames_.end()) { - insertion_order_.push_back(fault_code); - while (frames_.size() >= max_faults_ && !insertion_order_.empty()) { - frames_.erase(insertion_order_.front()); - insertion_order_.pop_front(); + for (auto & frame : frames) { + const auto key = record_key(fault_code, frame.entity_id); + if (frames_.find(key) == frames_.end()) { + insertion_order_.push_back(key); + while (frames_.size() >= max_faults_ && !insertion_order_.empty()) { + frames_.erase(insertion_order_.front()); + insertion_order_.pop_front(); + } } + frames_[key] = std::vector{frame}; } - frames_[fault_code] = std::move(frames); MEDKIT_LOG_DEBUG(logger_, "Captured entity freeze-frame(s) for fault '%s'", fault_code.c_str()); return true; diff --git a/src/ros2_medkit_gateway/src/core/faults/trigger_fault_feed.cpp b/src/ros2_medkit_gateway/src/core/faults/trigger_fault_feed.cpp index b6fcca639..519c0f8e8 100644 --- a/src/ros2_medkit_gateway/src/core/faults/trigger_fault_feed.cpp +++ b/src/ros2_medkit_gateway/src/core/faults/trigger_fault_feed.cpp @@ -22,8 +22,7 @@ namespace ros2_medkit_gateway { namespace fd = ros2_medkit::fault_detection; -TriggerFaultFeed::TriggerFaultFeed(FaultEventSource & source, ResourceChangeNotifier & notifier, - NodeToEntityFn resolver) +TriggerFaultFeed::TriggerFaultFeed(FaultEventSource & source, ResourceChangeNotifier & notifier, FaultOwnerFn resolver) : notifier_(notifier) , resolver_(std::move(resolver)) , subscription_(source.subscribe([this](const fd::FaultEventRecord & event) { @@ -35,29 +34,23 @@ TriggerFaultFeed::~TriggerFaultFeed() { subscription_.reset(); } -void TriggerFaultFeed::set_node_to_entity_resolver(NodeToEntityFn resolver) { +void TriggerFaultFeed::set_fault_owner_resolver(FaultOwnerFn resolver) { resolver_ = std::move(resolver); } void TriggerFaultFeed::on_event(const fd::FaultEventRecord & event) { + // A record without an owner is in no entity's scope. + if (event.fault.reporting_sources.empty() || event.fault.reporting_sources.front().empty()) { + return; + } + const auto & owner = event.fault.reporting_sources.front(); + nlohmann::json fault_json = fault_record_to_json(event.fault); fault_json["event_type"] = event.event_type; - // The resolver maps a source to a manifest entity id. Without a mapping the last path segment is the entity id. std::string entity_id; - if (!event.fault.reporting_sources.empty()) { - const auto & source = event.fault.reporting_sources[0]; - if (resolver_) { - entity_id = resolver_(source); - } - if (entity_id.empty()) { - const auto last_slash = source.rfind('/'); - entity_id = (last_slash != std::string::npos) ? source.substr(last_slash + 1) : source; - } - } - // No entity: a catch-all match would be unintended. - if (entity_id.empty()) { - return; + if (resolver_) { + entity_id = resolver_(owner); } // Both fault_cleared and a heal remove the fault from the active set. The payload "status" tells them apart. diff --git a/src/ros2_medkit_gateway/src/core/logs/log_scope.cpp b/src/ros2_medkit_gateway/src/core/logs/log_scope.cpp index 90af99e8c..d6519d73f 100644 --- a/src/ros2_medkit_gateway/src/core/logs/log_scope.cpp +++ b/src/ros2_medkit_gateway/src/core/logs/log_scope.cpp @@ -40,37 +40,157 @@ unsigned long long entry_id_num(const json & item) { } // namespace -tl::expected query_entity_logs(LogManager & log_mgr, const ThreadSafeEntityCache & cache, - SovdEntityType type, const std::string & entity_id, - const std::string & entity_fqn, const std::string & min_severity, - const std::string & context_filter, - std::set * resolved_sources) { +LogScope resolve_log_scope(const ThreadSafeEntityCache & cache, SovdEntityType type, const std::string & entity_id, + const std::string & entity_fqn, std::set * resolved_sources) { auto sources = faults::resolve_entity_source_fqns(cache, type, entity_id); if (resolved_sources) { *resolved_sources = sources; } - std::vector exact(sources.begin(), sources.end()); + LogScope scope; + scope.exact.assign(sources.begin(), sources.end()); // Legacy APP fallback: an app that is not in the cache (or resolves to no // source) keeps the entity.fqn exact query it always had. - if (type == SovdEntityType::APP && exact.empty() && !entity_fqn.empty()) { - exact.push_back(entity_fqn); + if (type == SovdEntityType::APP && scope.exact.empty() && !entity_fqn.empty()) { + scope.exact.push_back(entity_fqn); } // Functions are pure aggregated views - no namespace fallback. APP has a // single source. Only grouping entities own a namespace prefix. - const bool run_prefix = (type == SovdEntityType::COMPONENT || type == SovdEntityType::AREA) && !entity_fqn.empty(); + if (type == SovdEntityType::COMPONENT || type == SovdEntityType::AREA) { + scope.prefix = entity_fqn; + } + return scope; +} + +bool log_scope_contains(const LogScope & scope, const std::string & logger_name) { + for (const auto & fqn : scope.exact) { + if (LogManager::logger_matches(logger_name, fqn, false)) { + return true; + } + } + return !scope.prefix.empty() && LogManager::logger_matches(logger_name, scope.prefix, true); +} + +std::string entity_log_fqn(const ThreadSafeEntityCache & cache, SovdEntityType type, const std::string & entity_id) { + switch (type) { + case SovdEntityType::COMPONENT: + if (auto comp = cache.get_component(entity_id)) { + return comp->fqn; + } + break; + case SovdEntityType::AREA: + if (auto area = cache.get_area(entity_id)) { + return area->namespace_path; + } + break; + case SovdEntityType::APP: + if (auto app = cache.get_app(entity_id)) { + return app->effective_fqn(); + } + break; + case SovdEntityType::FUNCTION: + case SovdEntityType::SERVER: + case SovdEntityType::UNKNOWN: + break; + } + return ""; +} + +std::optional> restore_replay_entries(LogManager & log_mgr, const ThreadSafeEntityCache & cache, + SovdEntityType type, const std::string & entity_id) { + if (!cache.has_entity(type, entity_id)) { + return std::nullopt; + } + std::unordered_set markers; + auto listed = query_entity_logs(log_mgr, cache, type, entity_id, entity_log_fqn(cache, type, entity_id), "", "", + nullptr, &markers); + if (!listed || !listed->is_array()) { + return std::nullopt; + } + std::vector entries; + for (const auto & item : *listed) { + if (markers.count(item.value("id", "")) == 0) { + entries.push_back(item); + } + } + return entries; +} + +std::string LoggerOwnerIndex::find_owner(const ThreadSafeEntityCache & cache, const std::string & logger_name) { + std::lock_guard lock(mutex_); + // Read before collecting: a mutation in between leaves a newer generation, + // so the next call rebuilds. + const auto generation = cache.generation(); + if (!built_ || generation != generation_) { + owners_.clear(); + for (const auto & app : cache.get_apps()) { + // An app's log scope is its one source, matched exactly in either form. + const auto source = LogManager::normalize_fqn(faults::resolve_app_source_fqn(cache, app.id)); + if (source.empty()) { + continue; + } + std::string dotted = source; + std::replace(dotted.begin(), dotted.end(), '/', '.'); + for (const auto & form : {source, dotted}) { + auto [it, inserted] = owners_.emplace(form, app.id); + if (!inserted && it->second != app.id) { + it->second.clear(); + } + } + } + generation_ = generation; + built_ = true; + } + auto it = owners_.find(logger_name); + return it != owners_.end() ? it->second : ""; +} + +bool entity_lists_log_entry(const LogManager & log_mgr, const ThreadSafeEntityCache & cache, SovdEntityType type, + const std::string & entity_id, const std::string & logger_name, uint8_t level) { + const auto scope = resolve_log_scope(cache, type, entity_id, entity_log_fqn(cache, type, entity_id)); + if (!log_scope_contains(scope, logger_name)) { + return false; + } + // get_logs() drops entries below the entity's severity_filter. + auto cfg = log_mgr.get_config(entity_id); + return cfg && level >= LogManager::severity_to_level(cfg->severity_filter); +} + +bool entity_lists_log_change(const LogManager & log_mgr, const ThreadSafeEntityCache & cache, SovdEntityType type, + const std::string & entity_id, const json & entry) { + if (!entry.is_object()) { + return false; + } + const auto ctx = entry.find("context"); + if (ctx == entry.end() || !ctx->is_object()) { + return false; + } + const auto node = ctx->find("node"); + if (node == ctx->end() || !node->is_string()) { + return false; + } + return entity_lists_log_entry(log_mgr, cache, type, entity_id, node->get(), log_mgr.entry_level(entry)); +} + +tl::expected query_entity_logs(LogManager & log_mgr, const ThreadSafeEntityCache & cache, + SovdEntityType type, const std::string & entity_id, + const std::string & entity_fqn, const std::string & min_severity, + const std::string & context_filter, + std::set * resolved_sources, + std::unordered_set * marker_ids) { + const auto scope = resolve_log_scope(cache, type, entity_id, entity_fqn, resolved_sources); json merged = json::array(); - if (!exact.empty()) { - auto result = log_mgr.get_logs(exact, false, min_severity, context_filter, entity_id); + if (!scope.exact.empty()) { + auto result = log_mgr.get_logs(scope.exact, false, min_severity, context_filter, entity_id, marker_ids); if (!result) { return tl::make_unexpected(result.error()); } merged = std::move(*result); } - if (run_prefix) { - auto result = log_mgr.get_logs({entity_fqn}, true, min_severity, context_filter, entity_id); + if (!scope.prefix.empty()) { + auto result = log_mgr.get_logs({scope.prefix}, true, min_severity, context_filter, entity_id, marker_ids); if (!result) { return tl::make_unexpected(result.error()); } diff --git a/src/ros2_medkit_gateway/src/core/managers/fault_manager.cpp b/src/ros2_medkit_gateway/src/core/managers/fault_manager.cpp index c406cbac6..5edade383 100644 --- a/src/ros2_medkit_gateway/src/core/managers/fault_manager.cpp +++ b/src/ros2_medkit_gateway/src/core/managers/fault_manager.cpp @@ -41,16 +41,18 @@ FaultResult FaultManager::get_fault(const std::string & fault_code, const std::s return transport_->get_fault(fault_code, source_id); } -FaultResult FaultManager::clear_fault(const std::string & fault_code, bool skip_correlation_auto_clear) { - return transport_->clear_fault(fault_code, skip_correlation_auto_clear); +FaultResult FaultManager::clear_fault(const std::string & fault_code, const std::string & source_id, + bool skip_correlation_auto_clear) { + return transport_->clear_fault(fault_code, source_id, skip_correlation_auto_clear); } -FaultResult FaultManager::get_snapshots(const std::string & fault_code, const std::string & topic) { - return transport_->get_snapshots(fault_code, topic); +FaultResult FaultManager::get_snapshots(const std::string & fault_code, const std::string & source_id, + const std::string & topic) { + return transport_->get_snapshots(fault_code, source_id, topic); } -FaultResult FaultManager::get_rosbag(const std::string & fault_code) { - return transport_->get_rosbag(fault_code); +FaultResult FaultManager::get_rosbag(const std::string & id, const std::string & source_id) { + return transport_->get_rosbag(id, source_id); } FaultResult FaultManager::list_rosbags(const std::string & entity_fqn) { diff --git a/src/ros2_medkit_gateway/src/core/managers/log_manager.cpp b/src/ros2_medkit_gateway/src/core/managers/log_manager.cpp index 0095df780..e1a1eabe1 100644 --- a/src/ros2_medkit_gateway/src/core/managers/log_manager.cpp +++ b/src/ros2_medkit_gateway/src/core/managers/log_manager.cpp @@ -16,6 +16,7 @@ #include #include +#include #include #include #include @@ -91,6 +92,19 @@ std::string LogManager::normalize_fqn(const std::string & fqn) { return fqn; } +bool LogManager::logger_matches(const std::string & logger_name, const std::string & node_fqn, bool prefix_match) { + // Logger names use '.' (e.g. "powertrain.engine.temp_sensor"), entity FQNs + // use '/'. Both forms match. + const std::string norm = normalize_fqn(node_fqn); + std::string norm_dot = norm; + std::replace(norm_dot.begin(), norm_dot.end(), '/', '.'); + if (logger_name == norm || logger_name == norm_dot) { + return true; + } + // A prefix covers only whole segments: "ns/" or "ns.". + return prefix_match && (logger_name.rfind(norm + "/", 0) == 0 || logger_name.rfind(norm_dot + ".", 0) == 0); +} + json LogManager::entry_to_json(const LogEntry & e) { // Format as ISO 8601 UTC std::time_t t = static_cast(e.stamp_sec); @@ -185,6 +199,7 @@ void LogManager::on_log_entry(const LogEntry & source_entry) { } } + bool buffered = false; if (!suppress_buffer) { std::lock_guard lock(buffers_mutex_); // Cap the number of distinct node buffers to prevent unbounded growth. @@ -202,35 +217,32 @@ void LogManager::on_log_entry(const LogEntry & source_entry) { auto & buf = buffers_[entry.name]; buf.push_back(entry); if (buf.size() > max_buffer_size_) { - buf.pop_front(); + pop_oldest_locked(buf); } + buffered = true; } } - // Notify triggers about the log event (resolve logger name to entity ID) - if (notifier_) { - std::string entity_id; - - // Try resolving via node_to_app mapping (manifest/hybrid mode) - if (node_to_entity_resolver_) { - // entry.name has no leading '/' (rcl convention), but the mapping - // may use FQNs with leading '/' - try both forms - entity_id = node_to_entity_resolver_("/" + entry.name); - if (entity_id.empty()) { - entity_id = node_to_entity_resolver_(entry.name); - } - } - - // Fallback: extract last path segment (runtime_only mode) - if (entity_id.empty()) { - auto last_slash = entry.name.rfind('/'); - entity_id = (last_slash != std::string::npos) ? entry.name.substr(last_slash + 1) : entry.name; - } + // A dropped entry is listed nowhere, so it emits no change. + if (buffered || suppress_buffer) { + notify_entry(entry); + } +} - if (!entity_id.empty()) { - notifier_->notify("logs", entity_id, "", entry_to_json(entry), ChangeType::CREATED); - } +void LogManager::notify_entry(const LogEntry & entry, NotifierSubscriptionId skipped_subscription) { + if (!notifier_) { + return; + } + if (severity_to_level(level_to_severity(entry.level)) != entry.level) { + remember_unnamed_level(entry); } + const std::string owner = logger_owner_fn_ ? logger_owner_fn_(entry.name) : std::string{}; + notifier_->notify_except(skipped_subscription, "logs", owner, "", entry_to_json(entry), ChangeType::CREATED); +} + +void LogManager::pop_oldest_locked(std::deque & buf) { + marker_ids_.erase(buf.front().id); + buf.pop_front(); } // --------------------------------------------------------------------------- @@ -251,7 +263,7 @@ void LogManager::inject_entry_for_testing(LogEntry entry) { auto & buf = buffers_[entry.name]; buf.push_back(std::move(entry)); if (buf.size() > max_buffer_size_) { - buf.pop_front(); + pop_oldest_locked(buf); } } @@ -265,7 +277,7 @@ std::size_t LogManager::dropped_entries_count() const noexcept { // --------------------------------------------------------------------------- void LogManager::add_log_entry(const std::string & entity_id, const std::string & severity, const std::string & message, - const nlohmann::json & metadata) { + const nlohmann::json & metadata, NotifierSubscriptionId skipped_subscription) { std::string normalized = normalize_fqn(entity_id); // Build a LogEntry using the same structure as source-emitted entries. @@ -299,15 +311,13 @@ void LogManager::add_log_entry(const std::string & entity_id, const std::string std::lock_guard lock(buffers_mutex_); auto & buf = buffers_[entry.name]; buf.push_back(entry); + marker_ids_.insert(entry.id); if (buf.size() > max_buffer_size_) { - buf.pop_front(); + pop_oldest_locked(buf); } } - // Notify observers if a notifier is set - if (notifier_) { - notifier_->notify("logs", normalized, "", entry_to_json(entry), ChangeType::CREATED); - } + notify_entry(entry, skipped_subscription); } // --------------------------------------------------------------------------- @@ -318,8 +328,47 @@ void LogManager::set_notifier(ResourceChangeNotifier * notifier) { notifier_ = notifier; } -void LogManager::set_node_to_entity_resolver(NodeToEntityFn resolver) { - node_to_entity_resolver_ = std::move(resolver); +void LogManager::set_logger_owner_fn(LoggerOwnerFn fn) { + logger_owner_fn_ = std::move(fn); +} + +void LogManager::remember_unnamed_level(const LogEntry & entry) { + // Without a queue overflow, a change waiting for dispatch has fewer newer + // changes than the notifier holds in flight, so its level is still here when + // it is judged. Changes the queue drops count too. + const size_t capacity = notifier_->max_in_flight(); + std::lock_guard lock(unnamed_levels_mutex_); + unnamed_levels_[entry.id] = entry.level; + unnamed_level_order_.push_back(entry.id); + while (unnamed_level_order_.size() > capacity) { + unnamed_levels_.erase(unnamed_level_order_.front()); + unnamed_level_order_.pop_front(); + } +} + +uint8_t LogManager::entry_level(const json & entry) const { + if (!entry.is_object()) { + return 0; + } + const auto id = entry.find("id"); + if (id != entry.end() && id->is_string()) { + // Ids are "log_" (entry_to_json). + const auto & text = id->get_ref(); + if (text.rfind("log_", 0) == 0) { + const char * digits = text.c_str() + 4; + char * end = nullptr; + const long long n = std::strtoll(digits, &end, 10); + if (end != digits && *end == '\0') { + std::lock_guard lock(unnamed_levels_mutex_); + auto it = unnamed_levels_.find(n); + if (it != unnamed_levels_.end()) { + return it->second; + } + } + } + } + const auto severity = entry.find("severity"); + return severity != entry.end() && severity->is_string() ? severity_to_level(severity->get()) : 0; } // --------------------------------------------------------------------------- @@ -339,8 +388,8 @@ LogProvider * LogManager::effective_provider() const { tl::expected LogManager::get_logs(const std::vector & node_fqns, bool prefix_match, const std::string & min_severity, - const std::string & context_filter, - const std::string & entity_id) { + const std::string & context_filter, const std::string & entity_id, + std::unordered_set * marker_ids) { // Delegate to plugin if one is registered if (auto * provider = effective_provider()) { // Normalize FQNs before passing to plugin (strip leading '/') @@ -383,31 +432,14 @@ tl::expected LogManager::get_logs(const std::vector collected; + std::unordered_set collected_markers; { std::lock_guard lock(buffers_mutex_); for (const auto & [buf_name, buf] : buffers_) { - bool matches = false; - for (const auto & fqn : node_fqns) { - const std::string norm = normalize_fqn(fqn); - // ROS 2 logger names use '.' as separator (e.g. "powertrain.engine.temp_sensor") - // while entity FQNs use '/' (e.g. "powertrain/engine/temp_sensor"). - // Try both slash-format and dot-format so the default ring buffer matches either. - std::string norm_dot = norm; - std::replace(norm_dot.begin(), norm_dot.end(), '/', '.'); - if (prefix_match) { - // Match exactly OR as a namespace prefix. - // Slash-format: prefix must be followed by '/' - // Dot-format: prefix must be followed by '.' - matches = (buf_name == norm) || (buf_name.rfind(norm + "/", 0) == 0) || (buf_name == norm_dot) || - (buf_name.rfind(norm_dot + ".", 0) == 0); - } else { - matches = (buf_name == norm) || (buf_name == norm_dot); - } - if (matches) { - break; - } - } + const bool matches = std::any_of(node_fqns.begin(), node_fqns.end(), [&](const std::string & fqn) { + return logger_matches(buf_name, fqn, prefix_match); + }); if (!matches) { continue; } @@ -419,6 +451,9 @@ tl::expected LogManager::get_logs(const std::vector 0) { + collected_markers.insert(entry.id); + } collected.push_back(entry); } } @@ -437,6 +472,9 @@ tl::expected LogManager::get_logs(const std::vector 0) { + marker_ids->insert(items.back()["id"].get()); + } } return items; } diff --git a/src/ros2_medkit_gateway/src/core/managers/trigger_manager.cpp b/src/ros2_medkit_gateway/src/core/managers/trigger_manager.cpp index 7b4e72e07..2aaa2be4c 100644 --- a/src/ros2_medkit_gateway/src/core/managers/trigger_manager.cpp +++ b/src/ros2_medkit_gateway/src/core/managers/trigger_manager.cpp @@ -122,6 +122,16 @@ void TriggerManager::set_entity_children_fn(EntityChildrenFn fn) { entity_children_fn_ = std::move(fn); } +void TriggerManager::set_collection_scope_fn(const std::string & collection, CollectionScopeFn fn) { + std::lock_guard lock(hierarchy_mutex_); + collection_scope_fns_[collection] = std::move(fn); +} + +void TriggerManager::set_restore_replay_fn(const std::string & collection, RestoreReplayFn fn) { + std::lock_guard lock(hierarchy_mutex_); + restore_replay_fns_[collection] = std::move(fn); +} + void TriggerManager::set_log_manager(LogManager * log_manager) { log_manager_ = log_manager; } @@ -262,9 +272,15 @@ void TriggerManager::set_entity_exists_fn(EntityExistsFn fn) { entity_exists_fn_ = std::move(fn); } +bool TriggerManager::entity_exists(const TriggerState & state) const { + std::lock_guard lock(entity_exists_mutex_); + return entity_exists_fn_ && entity_exists_fn_(state.info.entity_id, state.info.entity_type); +} + void TriggerManager::sweep_orphaned_triggers() { // Phase 1: collect orphaned triggers under lock std::vector> orphaned; // trigger id, entity id + std::vector>> to_replay; WarnLogFn warn_fn; { std::lock_guard lock(triggers_mutex_); @@ -280,6 +296,9 @@ void TriggerManager::sweep_orphaned_triggers() { } if (entity_exists_fn_(state->info.entity_id, state->info.entity_type)) { state->entity_seen.store(true); + if (state->replay_pending.load()) { + to_replay.emplace_back(id, state); + } continue; } // An entity this gateway has never seen has not disappeared from its @@ -308,22 +327,47 @@ void TriggerManager::sweep_orphaned_triggers() { "' is not in the discovery cache."); } } + + // Phase 3: restored triggers whose entity is now discovered replay what it + // lists, unless a live change already made them do so. + for (const auto & [id, state] : to_replay) { + if (shutdown_flag_.load()) { + return; + } + bool ended = false; + { + std::lock_guard lock(state->mtx); + if (state->info.status != TriggerStatus::ACTIVE || !state->active.load()) { + continue; + } + ended = replay_restored_locked(*state); + state->cv.notify_all(); + } + if (ended) { + remove(id); + } + } } -bool TriggerManager::matches_entity(const TriggerState & state, const std::string & notification_entity_id) const { +bool TriggerManager::matches_entity(const TriggerState & state, const ResourceChange & change) const { + std::lock_guard lock(hierarchy_mutex_); + auto scope_it = collection_scope_fns_.find(change.collection); + if (scope_it != collection_scope_fns_.end() && scope_it->second) { + return scope_it->second(state.info.entity_type, state.info.entity_id, change.value); + } + // Direct match - if (state.info.entity_id == notification_entity_id) { + if (state.info.entity_id == change.entity_id) { return true; } // Hierarchy match: check if notification entity is a descendant - std::lock_guard lock(hierarchy_mutex_); if (!entity_children_fn_) { return false; } auto children = entity_children_fn_(state.info.entity_id, state.info.entity_type); - return std::find(children.begin(), children.end(), notification_entity_id) != children.end(); + return std::find(children.begin(), children.end(), change.entity_id) != children.end(); } // --------------------------------------------------------------------------- @@ -790,6 +834,10 @@ size_t TriggerManager::load_persistent_triggers() { // not read "missing from the cache" as "gone" until discovery has had // its say. state->entity_seen.store(false); + { + std::lock_guard replay_lock(hierarchy_mutex_); + state->replay_pending.store(restore_replay_fns_.count(state->info.collection) > 0); + } // Restore previous value state if available auto state_result = store_.load_state(state->info.id); @@ -889,15 +937,125 @@ size_t TriggerManager::load_persistent_triggers() { // Resource change evaluation // --------------------------------------------------------------------------- -void TriggerManager::on_resource_change(const ResourceChange & change) { - if (shutdown_flag_.load()) { - return; +std::optional TriggerManager::evaluate_locked(TriggerState & state, const nlohmann::json & value) { + // Extract value via JSON Pointer - skip evaluation if pointer doesn't match + nlohmann::json extracted_value; + if (state.info.path.empty()) { + extracted_value = value; + } else { + try { + auto ptr = nlohmann::json::json_pointer(state.info.path); + if (!value.contains(ptr)) { + return std::nullopt; + } + extracted_value = value[ptr]; + } catch (...) { + return std::nullopt; // Malformed pointer - skip evaluation + } + } + + auto evaluator = conditions_.get(state.info.condition_type); + if (!evaluator) { + return std::nullopt; + } + + std::optional prev_opt; + if (state.has_previous_value) { + prev_opt = state.previous_value; + } + const bool fired = evaluator->evaluate(prev_opt, extracted_value, state.info.condition_params); + state.previous_value = std::move(extracted_value); + state.has_previous_value = true; + + if (fired) { + // Event counter feeds the SSE id: field for reconnection. + state.event_counter.fetch_add(1); + nlohmann::json envelope; + envelope["timestamp"] = to_iso8601(std::chrono::system_clock::now()); + envelope["payload"] = value; + // Bounded: the oldest undelivered event is dropped. + state.pending_events.push_back(std::move(envelope)); + if (state.pending_events.size() > TriggerState::kMaxPendingEvents) { + state.pending_events.pop_front(); + } + } + return fired; +} + +bool TriggerManager::replay_restored_locked(TriggerState & state) { + // The sweep and the dispatch worker both call this; the first one replays. + if (!state.replay_pending.load()) { + return false; + } + RestoreReplayFn replay_fn; + { + std::lock_guard lock(hierarchy_mutex_); + auto it = restore_replay_fns_.find(state.info.collection); + if (it != restore_replay_fns_.end()) { + replay_fn = it->second; + } + } + if (!replay_fn) { + state.replay_pending.store(false); + return false; + } + auto items = replay_fn(state.info.entity_type, state.info.entity_id); + if (!items) { + return false; // Entity not in the cache yet. + } + state.replay_pending.store(false); + + bool evaluated_any = false; + bool ended = false; + for (const auto & item : *items) { + if (item.is_object()) { + auto id_it = item.find("id"); + if (id_it != item.end() && id_it->is_string()) { + state.replayed_ids.insert(id_it->get()); + } + } + auto evaluated = evaluate_locked(state, item); + if (!evaluated) { + continue; + } + evaluated_any = true; + if (!*evaluated) { + continue; + } + if (state.info.log_settings.has_value() && log_manager_) { + const auto & ls = *state.info.log_settings; + try { + log_manager_->add_log_entry(state.info.entity_id, ls.value("severity", "info"), + ls.value("marker", "Trigger fired"), + {{"trigger_id", state.info.id}, + {"condition_type", state.info.condition_type}, + {"resource", state.info.resource_uri}}, + notifier_sub_id_); + } catch (...) { + // Don't let log entry failure prevent trigger delivery + } + } + if (!state.info.multishot) { + state.info.status = TriggerStatus::TERMINATED; + state.active.store(false); + ended = true; + break; + } + } + + if (evaluated_any && config_.on_restart_behavior == "restore" && state.info.persistent) { + (void)store_.save_state(state.info.id, state.previous_value); + } + if (ended && state.info.persistent) { + nlohmann::json fields; + fields["status"] = "TERMINATED"; + (void)store_.update(state.info.id, fields); } + return ended; +} - // I5 fix: prevent recursive trigger evaluation. The notifier worker is - // single-threaded, so if log_manager_->add_log_entry() triggers a notification - // that re-enters on_resource_change(), we break the cycle here. - if (evaluating_trigger_) { +void TriggerManager::on_resource_change(const ResourceChange & change) { + if (shutdown_flag_.load()) { return; } @@ -958,10 +1116,16 @@ void TriggerManager::on_resource_change(const ResourceChange & change) { // Release sub_lock before calling matches_entity (it acquires hierarchy_mutex_) sub_lock.unlock(); - if (!matches_entity(*state, change.entity_id)) { + if (!matches_entity(*state, change)) { continue; } + // A restored trigger whose entity a live change finds has seen it, as it + // would at the next sweep; the sweep can then remove it once it is gone. + if (!state->entity_seen.load() && entity_exists(*state)) { + state->entity_seen.store(true); + } + sub_lock.lock(); // Re-check status after re-acquiring lock @@ -969,62 +1133,43 @@ void TriggerManager::on_resource_change(const ResourceChange & change) { continue; } - // Extract value via JSON Pointer - skip evaluation if pointer doesn't match - nlohmann::json extracted_value; - if (state->info.path.empty()) { - extracted_value = change.value; - } else { - try { - auto ptr = nlohmann::json::json_pointer(state->info.path); - if (change.value.contains(ptr)) { - extracted_value = change.value[ptr]; - } else { - continue; // Path not found in this change - skip evaluation - } - } catch (...) { - continue; // Malformed pointer - skip evaluation + // A restored trigger replays what its entity lists before its first live + // change, and skips the live changes it replayed. While the replay stays + // pending the live change is not judged: the replay that succeeds lists + // it, after the entries logged before it. + if (state->replay_pending.load()) { + const bool ended = replay_restored_locked(*state); + state->cv.notify_all(); + if (ended) { + terminated_triggers.push_back(trigger_id); + continue; + } + if (state->replay_pending.load()) { + continue; + } + } + if (!state->replayed_ids.empty() && change.value.is_object()) { + auto id_it = change.value.find("id"); + if (id_it != change.value.end() && id_it->is_string() && state->replayed_ids.count(id_it->get())) { + continue; } } - // Get the evaluator - auto evaluator = conditions_.get(state->info.condition_type); - if (!evaluator) { + auto evaluated = evaluate_locked(*state, change.value); + if (!evaluated) { continue; } - - // Build optional previous value - std::optional prev_opt; - if (state->has_previous_value) { - prev_opt = state->previous_value; + const bool fired = *evaluated; + const bool should_persist_state = (config_.on_restart_behavior == "restore" && state->info.persistent); + nlohmann::json previous_value_copy; + if (should_persist_state) { + previous_value_copy = state->previous_value; } - // Evaluate condition - bool fired = evaluator->evaluate(prev_opt, extracted_value, state->info.condition_params); - - // Update previous value - state->previous_value = extracted_value; - state->has_previous_value = true; - if (fired) { - // Build EventEnvelope - // Increment event counter (used by SSE id: field for reconnection) - state->event_counter.fetch_add(1); - nlohmann::json envelope; - envelope["timestamp"] = to_iso8601(std::chrono::system_clock::now()); - envelope["payload"] = change.value; - - // C2 fix: push to bounded deque instead of overwriting single optional. - // Drop oldest events when queue is full to prevent unbounded growth. - state->pending_events.push_back(std::move(envelope)); - if (state->pending_events.size() > TriggerState::kMaxPendingEvents) { - state->pending_events.pop_front(); - } - // I6 fix: capture data under lock, then release before doing I/O. // This prevents store_.save_state() and log_manager_->add_log_entry() // from blocking SSE consumers that need state->mtx. - bool should_persist_state = (config_.on_restart_behavior == "restore" && state->info.persistent); - const auto & previous_value_copy = extracted_value; bool should_terminate = !state->info.multishot; auto tid_copy = state->info.id; auto log_settings_copy = state->info.log_settings; @@ -1059,23 +1204,20 @@ void TriggerManager::on_resource_change(const ResourceChange & change) { (void)store_.update(tid_copy, fields); } - // I5 fix: RAII guard prevents recursive notification from log triggers + // The marker skips this manager's subscription, so it fires no trigger. if (log_settings_copy.has_value() && log_manager_) { auto & ls = *log_settings_copy; - evaluating_trigger_ = true; try { log_manager_->add_log_entry( entity_id_copy, ls.value("severity", "info"), ls.value("marker", "Trigger fired"), - {{"trigger_id", tid_copy}, {"condition_type", condition_type_copy}, {"resource", resource_uri_copy}}); + {{"trigger_id", tid_copy}, {"condition_type", condition_type_copy}, {"resource", resource_uri_copy}}, + notifier_sub_id_); } catch (...) { // Don't let log entry failure prevent trigger delivery } - evaluating_trigger_ = false; } } else { // Condition didn't fire - persist state if needed (I6: still move I/O outside lock) - bool should_persist_state = (config_.on_restart_behavior == "restore" && state->info.persistent); - const auto & previous_value_copy = extracted_value; auto tid_copy = trigger_id; sub_lock.unlock(); diff --git a/src/ros2_medkit_gateway/src/core/models/thread_safe_entity_cache.cpp b/src/ros2_medkit_gateway/src/core/models/thread_safe_entity_cache.cpp index 39c58e94c..238702d5f 100644 --- a/src/ros2_medkit_gateway/src/core/models/thread_safe_entity_cache.cpp +++ b/src/ros2_medkit_gateway/src/core/models/thread_safe_entity_cache.cpp @@ -452,6 +452,23 @@ bool ThreadSafeEntityCache::has_function(const std::string & id) const { return slot && functions_.is_live(*slot); } +bool ThreadSafeEntityCache::has_entity(SovdEntityType type, const std::string & id) const { + switch (type) { + case SovdEntityType::AREA: + return has_area(id); + case SovdEntityType::COMPONENT: + return has_component(id); + case SovdEntityType::APP: + return has_app(id); + case SovdEntityType::FUNCTION: + return has_function(id); + case SovdEntityType::SERVER: + case SovdEntityType::UNKNOWN: + break; + } + return false; +} + // ============================================================================ // Reader methods - Find any entity // ============================================================================ diff --git a/src/ros2_medkit_gateway/src/core/resource_change_notifier.cpp b/src/ros2_medkit_gateway/src/core/resource_change_notifier.cpp index b2616b31c..8a6fb15a2 100644 --- a/src/ros2_medkit_gateway/src/core/resource_change_notifier.cpp +++ b/src/ros2_medkit_gateway/src/core/resource_change_notifier.cpp @@ -55,6 +55,12 @@ void ResourceChangeNotifier::unsubscribe(NotifierSubscriptionId id) { void ResourceChangeNotifier::notify(const std::string & collection, const std::string & entity_id, const std::string & resource_path, const nlohmann::json & value, ChangeType change_type) { + notify_except(0, collection, entity_id, resource_path, value, change_type); +} + +void ResourceChangeNotifier::notify_except(NotifierSubscriptionId skipped, const std::string & collection, + const std::string & entity_id, const std::string & resource_path, + const nlohmann::json & value, ChangeType change_type) { // Prevent destruction while we're inside this function. The counter is // incremented BEFORE checking the shutdown flag (both seq_cst) so that // shutdown() is guaranteed to see the increment and wait for us to finish. @@ -89,7 +95,7 @@ void ResourceChangeNotifier::notify(const std::string & collection, const std::s { std::lock_guard lock(queue_mutex_); - queue_.push_back(std::move(change)); + queue_.push_back(QueuedChange{std::move(change), skipped}); // Enforce bounded queue: drop oldest entries when limit is exceeded. // Log only once per overflow batch to avoid log flooding. @@ -157,7 +163,7 @@ void ResourceChangeNotifier::worker_loop() { worker_id_.store(std::this_thread::get_id()); std::vector targets; while (true) { - std::vector batch; + std::vector batch; { std::unique_lock lock(queue_mutex_); @@ -174,12 +180,13 @@ void ResourceChangeNotifier::worker_loop() { queue_.clear(); } - for (const auto & change : batch) { + for (const auto & queued : batch) { + const auto & change = queued.change; targets.clear(); { std::lock_guard lock(sub_mutex_); for (const auto & [id, entry] : subscriptions_) { - if (matches_filter(entry->filter, change)) { + if (id != queued.skipped && matches_filter(entry->filter, change)) { targets.push_back(id); } } @@ -227,4 +234,9 @@ void ResourceChangeNotifier::set_max_queue_size(size_t max_size) { max_queue_size_ = max_size; } +size_t ResourceChangeNotifier::max_in_flight() const { + std::lock_guard lock(queue_mutex_); + return 2 * max_queue_size_; +} + } // namespace ros2_medkit_gateway diff --git a/src/ros2_medkit_gateway/src/core/trigger_host.cpp b/src/ros2_medkit_gateway/src/core/trigger_host.cpp index c97550643..6b33a7be3 100644 --- a/src/ros2_medkit_gateway/src/core/trigger_host.cpp +++ b/src/ros2_medkit_gateway/src/core/trigger_host.cpp @@ -17,11 +17,14 @@ #include #include #include +#include #include #include #include #include "ros2_medkit_gateway/core/data/polling_data_item_source.hpp" +#include "ros2_medkit_gateway/core/faults/fault_scope.hpp" +#include "ros2_medkit_gateway/core/logs/log_scope.hpp" #include "ros2_medkit_gateway/core/sqlite_trigger_store.hpp" namespace ros2_medkit_gateway { @@ -50,6 +53,23 @@ std::vector apps_under(const ThreadSafeEntityCache & cache, const s return {}; } +/// Entity type of a trigger, which stores it as its route segment ("apps", ...). +SovdEntityType entity_type_from_collection_segment(const std::string & segment) { + if (segment == "apps") { + return SovdEntityType::APP; + } + if (segment == "components") { + return SovdEntityType::COMPONENT; + } + if (segment == "areas") { + return SovdEntityType::AREA; + } + if (segment == "functions") { + return SovdEntityType::FUNCTION; + } + return SovdEntityType::UNKNOWN; +} + } // namespace void register_builtin_conditions(ConditionRegistry & conditions) { @@ -59,12 +79,11 @@ void register_builtin_conditions(ConditionRegistry & conditions) { conditions.register_condition("LeaveRange", std::make_shared()); } -std::string resolve_trigger_entity(const ThreadSafeEntityCache & cache, const std::string & node_fqn) { - auto entity_id = cache.resolve_node_to_app(node_fqn); - if (entity_id.empty() && !node_fqn.empty() && node_fqn[0] == '/') { - entity_id = cache.resolve_node_to_app(node_fqn.substr(1)); - } - return entity_id; +TriggerFaultFeed::FaultOwnerFn make_fault_owner_resolver(const ThreadSafeEntityCache & cache) { + auto owners = std::make_shared(); + return [&cache, owners](const std::string & owner) { + return owners->find_owner(cache, owner); + }; } std::string resolve_trigger_topic(const ThreadSafeEntityCache & cache, const std::string & entity_id, @@ -126,24 +145,53 @@ TriggerHost make_trigger_host(const TriggerHostConfig & config, ResourceChangeNo host.manager->set_entity_children_fn([cache](const std::string & entity_id, const std::string & entity_type) { return apps_under(*cache, entity_id, entity_type); }); - host.manager->set_entity_exists_fn([cache](const std::string & entity_id, const std::string & /*entity_type*/) { - return cache->find_entity(entity_id).has_value(); + // Orphan sweep: an entity of the trigger's type, not any entity with its id. + host.manager->set_entity_exists_fn([cache](const std::string & entity_id, const std::string & entity_type) { + return cache->has_entity(entity_type_from_collection_segment(entity_type), entity_id); }); host.manager->set_resolve_topic_fn([cache](const std::string & entity_id, const std::string & resource_path) { return resolve_trigger_topic(*cache, entity_id, resource_path); }); host.manager->set_warn_log_fn(warn); host.manager->set_unresolved_timeout(wiring.unresolved_timeout); - if (wiring.log_manager != nullptr) { - host.manager->set_log_manager(wiring.log_manager); + LogManager * log_manager = wiring.log_manager; + if (log_manager != nullptr) { + host.manager->set_log_manager(log_manager); } + // A fault event matches an entity when its scope holds the record's owner, the rule of + // GET /{entity}/faults?status=all. Clear and heal events included. + host.manager->set_collection_scope_fn("faults", [cache](const std::string & entity_type, + const std::string & entity_id, const nlohmann::json & fault) { + return faults::fault_in_source_scope( + fault, faults::resolve_entity_source_fqns(*cache, entity_type_from_collection_segment(entity_type), entity_id)); + }); + // A log trigger checks each entry by the rule GET /{entity}/logs lists by: the entity's log scope and + // severity_filter, read at dispatch, against the logger name and level the entry carries. + host.manager->set_collection_scope_fn("logs", [cache, log_manager](const std::string & entity_type, + const std::string & entity_id, + const nlohmann::json & entry) { + return log_manager != nullptr && + logs::entity_lists_log_change(*log_manager, *cache, entity_type_from_collection_segment(entity_type), + entity_id, entry); + }); + // A restored log trigger may wait for its entity's discovery while the entity logs. Once the entity is in the cache + // the trigger replays what GET /{entity}/logs lists, less trigger markers, so those entries reach it too. Set before + // the restore below. + host.manager->set_restore_replay_fn( + "logs", + [cache, log_manager](const std::string & entity_type, + const std::string & entity_id) -> std::optional> { + if (log_manager == nullptr) { + return std::vector{}; + } + return logs::restore_replay_entries(*log_manager, *cache, entity_type_from_collection_segment(entity_type), + entity_id); + }); host.restored = host.manager->load_persistent_triggers(); if (wiring.fault_events != nullptr) { host.fault_feed = - std::make_unique(*wiring.fault_events, notifier, [cache](const std::string & node_fqn) { - return resolve_trigger_entity(*cache, node_fqn); - }); + std::make_unique(*wiring.fault_events, notifier, make_fault_owner_resolver(*cache)); } return host; } diff --git a/src/ros2_medkit_gateway/src/entity_freeze_frame_capture.cpp b/src/ros2_medkit_gateway/src/entity_freeze_frame_capture.cpp index e7ddc0054..538267662 100644 --- a/src/ros2_medkit_gateway/src/entity_freeze_frame_capture.cpp +++ b/src/ros2_medkit_gateway/src/entity_freeze_frame_capture.cpp @@ -156,8 +156,9 @@ EntityFreezeFrameCapture::EntityFreezeFrameCapture(rclcpp::Node * node, ros2_com EntityFreezeFrameCapture::~EntityFreezeFrameCapture() = default; -std::vector EntityFreezeFrameCapture::frames_for(const std::string & fault_code) const { - return capture_->frames_for(fault_code); +std::vector EntityFreezeFrameCapture::frames_for(const std::string & fault_code, + const std::string & owner) const { + return capture_->frames_for(fault_code, owner); } } // namespace ros2_medkit_gateway diff --git a/src/ros2_medkit_gateway/src/gateway_node.cpp b/src/ros2_medkit_gateway/src/gateway_node.cpp index af9c2abae..199cb1a98 100644 --- a/src/ros2_medkit_gateway/src/gateway_node.cpp +++ b/src/ros2_medkit_gateway/src/gateway_node.cpp @@ -34,6 +34,7 @@ #include "ros2_medkit_gateway/core/discovery/merge_types.hpp" #include "ros2_medkit_gateway/core/discovery/refresh_debounce.hpp" #include "ros2_medkit_gateway/core/entity_validation.hpp" +#include "ros2_medkit_gateway/core/logs/log_scope.hpp" #include "ros2_medkit_gateway/core/thread_pool_config.hpp" #include "ros2_medkit_gateway/fault_manager_paths.hpp" #include "ros2_medkit_gateway/param_utils.hpp" @@ -1001,6 +1002,12 @@ GatewayNode::GatewayNode(const rclcpp::NodeOptions & options) : Node("ros2_medki // Wire LogManager <-> TriggerManager for log_settings integration if (log_mgr_) { + // A "logs" change names the one app whose log scope holds the logger, or + // no entity. + auto owners = std::make_shared(); + log_mgr_->set_logger_owner_fn([this, owners](const std::string & logger_name) { + return owners->find_owner(get_thread_safe_cache(), logger_name); + }); log_mgr_->set_notifier(resource_change_notifier_.get()); } @@ -1011,19 +1018,8 @@ GatewayNode::GatewayNode(const rclcpp::NodeOptions & options) : Node("ros2_medki trigger_host_.manager->retry_unresolved_triggers(); }); - // Wire node-to-entity resolver for trigger notifications (#305). - // The resolver looks up node FQNs in the ThreadSafeEntityCache's node_to_app - // mapping (populated from linking result). Captures 'this' - the lambda is - // called from ROS subscription callbacks on the same node's executor. - auto node_resolver = [this](const std::string & ros_fqn) -> std::string { - return resolve_trigger_entity(thread_safe_cache_, ros_fqn); - }; - if (trigger_fault_subscriber_) { - trigger_fault_subscriber_->set_node_to_entity_resolver(node_resolver); - } - if (log_mgr_) { - log_mgr_->set_node_to_entity_resolver(node_resolver); + trigger_fault_subscriber_->set_fault_owner_resolver(make_fault_owner_resolver(thread_safe_cache_)); } RCLCPP_INFO(get_logger(), "Trigger subsystem: enabled (max=%d, storage=%s)", trigger_host_.max_triggers, diff --git a/src/ros2_medkit_gateway/src/http/handlers/bulkdata_handlers.cpp b/src/ros2_medkit_gateway/src/http/handlers/bulkdata_handlers.cpp index 9012cfd71..2be4ecb05 100644 --- a/src/ros2_medkit_gateway/src/http/handlers/bulkdata_handlers.cpp +++ b/src/ros2_medkit_gateway/src/http/handlers/bulkdata_handlers.cpp @@ -277,6 +277,31 @@ std::vector rosbag_attached_fault_codes(const nlohmann::json & rosb return {requested_id}; } +bool rosbag_rows_hold_recording(const nlohmann::json & rows, const nlohmann::json & served) { + if (!rows.is_array() || !served.is_object()) { + return false; + } + const std::string served_id = served.value("recording_id", std::string{}); + const std::string served_path = served.value("file_path", std::string{}); + for (const auto & row : rows) { + if (!row.is_object()) { + continue; + } + const std::string row_id = row.value("recording_id", std::string{}); + if (!served_id.empty() && !row_id.empty()) { + if (row_id == served_id) { + return true; + } + continue; + } + // One side predates recording ids. The bag path is the recording then. + if (!served_path.empty() && row.value("file_path", std::string{}) == served_path) { + return true; + } + } + return false; +} + bool rosbag_resolved_by_fault_code(const nlohmann::json & rosbag_data, const std::string & requested_id) { const std::string resolved = rosbag_data.value("recording_id", ""); // An absent id means a peer that predates the field; it answers by fault code @@ -284,9 +309,14 @@ bool rosbag_resolved_by_fault_code(const nlohmann::json & rosbag_data, const std return !resolved.empty() && resolved != requested_id; } +std::string record_map_key(const std::string & owner, const std::string & fault_code) { + // '\0' cannot appear in either id, so the two halves cannot run together. + return owner + '\0' + fault_code; +} + std::vector fold_rosbag_rows_into_descriptors(const std::vector & rows, - const std::unordered_map & faults_by_code) { + const std::unordered_map & faults_by_record) { struct RecordingEntry { std::string recording_id; std::string format; @@ -320,7 +350,8 @@ fold_rosbag_rows_into_descriptors(const std::vector & rows, // fault only for a row that predates the field. int64_t created_at_ns = row.value("created_at_ns", int64_t{0}); if (created_at_ns == 0) { - if (auto it = faults_by_code.find(fault_code); it != faults_by_code.end()) { + const auto key = record_map_key(row.value("owner", std::string{}), fault_code); + if (auto it = faults_by_record.find(key); it != faults_by_record.end()) { const double first_occurred = it->second.value("first_occurred", 0.0); created_at_ns = static_cast(first_occurred * 1'000'000'000); } @@ -387,6 +418,169 @@ fold_rosbag_rows_into_descriptors(const std::vector & rows, return descriptors; } +tl::expected, ErrorInfo> +list_scoped_rosbag_descriptors(FaultManager & fault_mgr, const std::set & scope) { + // An empty scope owns no record, so the fault manager is not asked. + if (scope.empty()) { + return std::vector{}; + } + // Every status and muted records: a recording outlives its record's active state. + auto faults_result = fault_mgr.list_faults("", /*include_prefailed=*/true, /*include_confirmed=*/true, + /*include_cleared=*/true, /*include_healed=*/true, + /*include_muted=*/true); + if (!faults_result.success) { + return tl::unexpected(make_error(503, ERR_SERVICE_UNAVAILABLE, "Failed to get faults", + nlohmann::json{{"details", faults_result.error_message}})); + } + + // The owners of the records in scope, and each record for the date fallback. + std::unordered_map faults_by_record; + std::set owners; + for (const auto & fault_json : faults_result.data.value("faults", nlohmann::json::array())) { + if (!fault_json.contains("fault_code") || !faults::fault_in_source_scope(fault_json, scope)) { + continue; + } + const auto owner = faults::record_owner(fault_json); + faults_by_record[record_map_key(owner, fault_json["fault_code"].get())] = fault_json; + owners.insert(owner); + } + + // ListRosbags matches the owner exactly, so each owner is asked on its own. + // A row does not carry its owner; it is stamped here. + std::vector rows; + for (const auto & owner : owners) { + auto rosbags_result = fault_mgr.list_rosbags(owner); + if (!rosbags_result.success) { + return tl::unexpected(make_error(503, ERR_SERVICE_UNAVAILABLE, "Recordings could not be listed", + nlohmann::json{{"details", rosbags_result.error_message}})); + } + for (auto row : rosbags_result.data.value("rosbags", nlohmann::json::array())) { + row["owner"] = owner; + row["manifest"] = faults::recording_has_manifest(row.value("file_path", std::string{})); + rows.push_back(std::move(row)); + } + } + return fold_rosbag_rows_into_descriptors(rows, faults_by_record); +} + +tl::expected resolve_scoped_rosbag(FaultManager & fault_mgr, + const std::set & scope, + const std::string & bulk_data_id, + const std::string & entity_id) { + // Every record this entity owns, in one read, every status and muted ones + // too. Both the code resolution and the authorization below ask which + // RECORDS a code addresses, not whether a code exists: two sources reporting + // one per_source code are two records, and an unscoped read of that code is ambiguous + // and answers nothing at all. A muted or cleared record keeps its + // recordings, so leaving it out would 404 them. Reading the list once also + // replaces one GetFault round trip per attached code. + // A failed read is 503: an empty list would turn every recording into a 404. + auto held = fault_mgr.list_faults("", /*include_prefailed=*/true, /*include_confirmed=*/true, + /*include_cleared=*/true, /*include_healed=*/true, /*include_muted=*/true, + /*include_clusters=*/false); + if (!held.success) { + return tl::unexpected( + make_error(503, ERR_SERVICE_UNAVAILABLE, "Failed to get faults", json{{"details", held.error_message}})); + } + const json & listing = held.data; + const json all_faults = listing.value("faults", json::array()); + + // A compatibility URL carries a fault code, so the entity's own records of + // that code say which owner to ask the recording for. The same rule as the + // fault routes picks them: the records the entity's fault list shows, then + // its muted ones, then its cleared or healed ones, the first of those that + // holds any. An id that is really a recording id matches none and the owner + // stays empty, which is what the recording path ignores anyway. + // + // Several candidates means the URL names none of them, and each owner keeps + // its own recordings. Serving the lowest-sorting owner's bag would hand the + // caller another owner's bytes under a URL that never said whose they were, + // so this refuses the same way the fault routes do. The recording-id form + // is unaffected: it addresses the bag directly and needs no owner. + auto requested_records = faults::addressable_records(listing, bulk_data_id, scope); + if (requested_records.size() > 1) { + std::vector owners; + owners.reserve(requested_records.size()); + for (const auto & record : requested_records) { + owners.push_back(record.owner); + } + return tl::unexpected( + make_error(409, ERR_AMBIGUOUS_FAULT, "Fault code addresses several records in this entity", + json{{"details", + "Several sources this entity owns report this fault code, and each record keeps its own " + "recordings. parameters.owners names them. Address the recording by its own id: the " + "rosbags listing of this entity (GET .../bulk-data/rosbags) carries it as each " + "descriptor's id, and a fault detail links it as " + "environment_data.snapshots[].bulk_data_uri."}, + {"entity_id", entity_id}, + {"fault_code", bulk_data_id}, + {"owners", owners}})); + } + const std::string requested_owner = requested_records.empty() ? std::string{} : requested_records.front().owner; + + // No answer is 503. An answered "no such recording" is 404. + auto rosbag_result = fault_mgr.get_rosbag(bulk_data_id, requested_owner); + if (!rosbag_result.success && rosbag_result.failure == FaultFailure::Unavailable) { + return tl::unexpected(make_error(503, ERR_SERVICE_UNAVAILABLE, "Failed to get rosbag", + json{{"details", rosbag_result.error_message}})); + } + if (!rosbag_result.success || !rosbag_result.data.contains("file_path")) { + return tl::unexpected( + make_error(404, ERR_RESOURCE_NOT_FOUND, "Bulk-data not found", json{{"bulk_data_id", bulk_data_id}})); + } + + // The compatibility path needs the REQUESTED code in scope, not just some code + // the recording is attached to. A burst shares one bag, so authorizing on the + // union alone answers 200 for a fault code this entity does not own - the bytes + // are ones it could already fetch under its own code, but the 200 itself tells + // the caller that another fault shares its recording. + if (rosbag_resolved_by_fault_code(rosbag_result.data, bulk_data_id) && requested_records.empty()) { + return tl::unexpected( + make_error(404, ERR_RESOURCE_NOT_FOUND, "Bulk-data not found for this entity", json{{"entity_id", entity_id}})); + } + + const auto attached_codes = rosbag_attached_fault_codes(rosbag_result.data, bulk_data_id); + + // The bag belongs to this entity when one of the records it is attached to, + // a (fault code, owner) pair, has its owner in scope. The code alone does not + // say that: two sources reporting one per_source code are two records with their own + // recordings, and owning one of them, cleared or not, does not make the + // other's recording this entity's. GetRosbag names the attached codes but + // not their owners, so the candidates are the owners of this entity's own + // records of those codes, and ListRosbags, which answers exactly the rows + // one owner holds, says whether the served recording is among them: a + // recording is downloadable exactly when a listing under one of those owners + // carries it. The owners are matched with the fault list's scope rule, so + // an owner below one of the entity's sources counts too. + std::set candidate_owners; + for (const auto & code : attached_codes) { + for (const auto & record : faults::records_of_code_in_scope(all_faults, code, scope)) { + if (!record.owner.empty()) { + candidate_owners.insert(record.owner); + } + } + } + // One owner holding it is enough. A failed listing leaves the answer + // unknown, so without a holder it is 503, not 404. + std::optional failed_listing; + for (const auto & owner : candidate_owners) { + auto held_rows = fault_mgr.list_rosbags(owner); + if (!held_rows.success) { + failed_listing = held_rows.error_message; + continue; + } + if (rosbag_rows_hold_recording(held_rows.data.value("rosbags", json::array()), rosbag_result.data)) { + return tl::expected{rosbag_result.data}; + } + } + if (failed_listing) { + return tl::unexpected( + make_error(503, ERR_SERVICE_UNAVAILABLE, "Failed to list rosbags", json{{"details", *failed_listing}})); + } + return tl::unexpected( + make_error(404, ERR_RESOURCE_NOT_FOUND, "Bulk-data not found for this entity", json{{"entity_id", entity_id}})); +} + std::vector compute_bulkdata_source_filters(const ThreadSafeEntityCache & cache, const EntityInfo & entity) { // One rule for every entity type: the same fault-scope resolution that drives @@ -489,45 +683,16 @@ BulkDataHandlers::list_descriptors(const http::TypedRequest & req) { if (category == "rosbags" && rosbags_served(ctx_)) { // === Rosbags: served via FaultManager === - auto fault_mgr = ctx_.services().fault_manager(); - - // Get source filters for this entity (single for most, multiple for functions). - // Functions aggregate rosbags from all hosting apps. auto source_filters = get_source_filters(entity); - - // Collect faults across all source filters for timestamp enrichment - std::unordered_map fault_map; - for (const auto & source_filter : source_filters) { - auto faults_result = fault_mgr->list_faults(source_filter); - if (faults_result.success && faults_result.data.contains("faults")) { - for (const auto & fault_json : faults_result.data["faults"]) { - if (fault_json.contains("fault_code")) { - std::string fc = fault_json["fault_code"].get(); - fault_map[fc] = fault_json; - } - } - } - } - - // Collect rosbags across all source filters. A listing that could not read every - // one of them is refused rather than served short: a 200 here is a complete list. - std::vector all_rosbags; - for (const auto & source_filter : source_filters) { - auto rosbags_result = fault_mgr->list_rosbags(source_filter); - if (!rosbags_result.success) { - return tl::unexpected(make_error(503, ERR_SERVICE_UNAVAILABLE, "Recordings could not be listed", - json{{"details", rosbags_result.error_message}})); - } - if (rosbags_result.data.contains("rosbags")) { - for (auto rosbag : rosbags_result.data["rosbags"]) { - rosbag["manifest"] = faults::recording_has_manifest(rosbag.value("file_path", std::string{})); - all_rosbags.push_back(std::move(rosbag)); - } - } + auto items = detail::list_scoped_rosbag_descriptors( + *ctx_.services().fault_manager(), std::set(source_filters.begin(), source_filters.end())); + if (!items) { + auto err = items.error(); + err.params[entity.id_field] = entity.id; + return tl::unexpected(std::move(err)); } - dto::Collection rosbag_response; - rosbag_response.items = detail::fold_rosbag_rows_into_descriptors(all_rosbags, fault_map); + rosbag_response.items = std::move(*items); return rosbag_response; } @@ -604,58 +769,23 @@ http::Result BulkDataHandlers::download(const http::TypedR // recording of this fault", which is what a fault-code URL has always returned. // Both routes end up holding a recording before anything is authorized, so there // is exactly one authorization semantic, not two. - auto fault_mgr = ctx_.services().fault_manager(); - - auto rosbag_result = fault_mgr->get_rosbag(bulk_data_id); - if (!rosbag_result.success || !rosbag_result.data.contains("file_path")) { - return tl::unexpected( - make_error(404, ERR_RESOURCE_NOT_FOUND, "Bulk-data not found", json{{"bulk_data_id", bulk_data_id}})); - } - - // Security check: the bag belongs to this entity when ANY fault it was captured - // for is within the entity's source scope. Union rather than a single code - // because a burst shares one recording, and each of those faults already had its - // own downloadable copy of it before - so this grants nothing new, it only - // renames the door. Tested with the shared boundary-aware matcher: the - // transport's get_fault(code, source) check is a raw prefix match, so app id - // "plc" would otherwise claim the assets of "plc_line1". auto source_filters = get_source_filters(entity); - std::set scope(source_filters.begin(), source_filters.end()); - - // The compatibility path needs the REQUESTED code in scope, not just some code - // the recording is attached to. A burst shares one bag, so authorizing on the - // union alone answers 200 for a fault code this entity does not own - the bytes - // are ones it could already fetch under its own code, but the 200 itself tells - // the caller that another fault shares its recording. build_sovd_fault_response - // refuses to mix sources for the same reason. When the id really was a recording - // id there is no requested code and the union is the whole answer. - if (detail::rosbag_resolved_by_fault_code(rosbag_result.data, bulk_data_id)) { - auto requested = fault_mgr->get_fault(bulk_data_id, ""); - if (!requested.success || !faults::fault_in_source_scope(requested.data, scope)) { - return tl::unexpected(make_error(404, ERR_RESOURCE_NOT_FOUND, "Bulk-data not found for this entity", - json{{"entity_id", path_info->entity_id}})); - } - } - - const auto attached_codes = detail::rosbag_attached_fault_codes(rosbag_result.data, bulk_data_id); - - const bool authorized = std::any_of(attached_codes.begin(), attached_codes.end(), [&](const std::string & code) { - auto fault_result = fault_mgr->get_fault(code, ""); - return fault_result.success && faults::fault_in_source_scope(fault_result.data, scope); - }); - if (!authorized) { - return tl::unexpected(make_error(404, ERR_RESOURCE_NOT_FOUND, "Bulk-data not found for this entity", - json{{"entity_id", path_info->entity_id}})); + auto served = detail::resolve_scoped_rosbag(*ctx_.services().fault_manager(), + std::set(source_filters.begin(), source_filters.end()), + bulk_data_id, path_info->entity_id); + if (!served) { + return tl::unexpected(served.error()); } + const json & rosbag_data = *served; - std::string file_path = rosbag_result.data["file_path"].get(); + std::string file_path = rosbag_data.at("file_path").get(); // Default to sqlite3 (the historical FaultManager default) for bags predating the // format field; metadata normally carries the real one persisted at capture time. - std::string format = rosbag_result.data.value("format", "sqlite3"); + std::string format = rosbag_data.value("format", "sqlite3"); mimetype = get_rosbag_mimetype(format); // Named after the recording that was actually served, which for a compatibility // URL is not the segment the client sent. - filename = rosbag_result.data.value("recording_id", bulk_data_id) + "." + format; + filename = rosbag_data.value("recording_id", bulk_data_id) + "." + format; // Only the storage file is served, not metadata.yaml. When the listing size differs from // Content-Length, see "One recording, one size" in docs/api/rest.rst. diff --git a/src/ros2_medkit_gateway/src/http/handlers/fault_handlers.cpp b/src/ros2_medkit_gateway/src/http/handlers/fault_handlers.cpp index 76f78412a..31de9b23c 100644 --- a/src/ros2_medkit_gateway/src/http/handlers/fault_handlers.cpp +++ b/src/ros2_medkit_gateway/src/http/handlers/fault_handlers.cpp @@ -207,6 +207,27 @@ dto::FaultListResult wrap_list_result(json payload) { return out; } +/// One item per fault code of `faults`, the records already scoped to the entity. +json entity_entries(const json & faults, const faults::AppOfSource & app_of) { + json items = json::array(); + for (const auto & records : faults::group_by_code(faults)) { + items.push_back(faults::entry_item(records, app_of)); + } + return items; +} + +/// The 404 of a per-code route when no record of the code is in the entity's scope. +ErrorInfo fault_not_in_scope(const std::string & id_field, const std::string & entity_id, + const std::string & fault_code) { + return make_error(404, ERR_RESOURCE_NOT_FOUND, "Fault not found", + json{{"details", + "Fault is not in scope for this entity: every reporting source must be one of the entity's " + "owned apps, and a mixed-source fault that includes any out-of-entity reporter is rejected " + "to prevent cross-entity disclosure"}, + {id_field, entity_id}, + {"fault_code", fault_code}}); +} + /// Wrap a built JSON detail payload in a typed FaultDetailResult envelope. dto::FaultDetailResult wrap_detail_result(json payload) { dto::FaultDetailResult out; @@ -232,6 +253,89 @@ ErrorInfo FaultHandlers::classify_fault_failure(FaultFailure failure, const std: return make_error(503, ERR_SERVICE_UNAVAILABLE, unavailable_summary, params); } +std::unordered_map> +FaultHandlers::build_source_entity_map(const ThreadSafeEntityCache & cache) { + std::unordered_map> source_to_entities; + for (const auto & app : cache.get_apps()) { + // The same resolution the fault scope uses: an external app's bare id, any + // other app's effective FQN, and nothing for an unbound non-external app. + auto source = faults::resolve_app_source_fqn(cache, app.id); + if (!source.empty()) { + source_to_entities[source].insert(app.id); + } + } + for (const auto & component : cache.get_components()) { + // Only an external component claims its bare id as a reporting source. + if (component.external.value_or(false) && !component.id.empty()) { + source_to_entities[component.id].insert(component.id); + } + } + return source_to_entities; +} + +tl::expected, ErrorInfo> FaultHandlers::records_of_code(const EntityInfo & entity_info, + const std::string & fault_code, + const std::string & failure_summary) { + auto * fault_mgr = ctx_.services().fault_manager(); + if (fault_mgr == nullptr) { + return tl::make_unexpected(make_error(503, ERR_SERVICE_UNAVAILABLE, failure_summary, + json{{entity_info.id_field, entity_info.id}, {"fault_code", fault_code}})); + } + auto result = fault_mgr->list_faults("", /*include_prefailed=*/true, /*include_confirmed=*/true, + /*include_cleared=*/true, /*include_healed=*/true, /*include_muted=*/true, + /*include_clusters=*/false); + if (!result.success) { + return tl::make_unexpected(classify_fault_failure(result.failure, result.error_message, failure_summary, + entity_info.id_field, entity_info.id, fault_code)); + } + const auto & cache = ctx_.services().entity_cache(); + const auto source_fqns = HandlerContext::resolve_entity_source_fqns(cache, entity_info); + std::vector records; + for (const auto & fault : result.data.value("faults", json::array())) { + if (fault.is_object() && fault.value("fault_code", std::string{}) == fault_code && + faults::fault_in_source_scope(fault, source_fqns)) { + records.push_back(fault); + } + } + std::stable_sort(records.begin(), records.end(), [](const json & a, const json & b) { + return faults::stored_owner(a) < faults::stored_owner(b); + }); + return records; +} + +faults::AppOfSource FaultHandlers::app_of_source() { + return [this](const std::string & source) { + return app_index_.find_owner(ctx_.services().entity_cache(), source); + }; +} + +tl::expected FaultHandlers::fault_detail(const std::vector & records, + const std::string & fault_code, + const std::string & entity_path) { + auto * fault_mgr = ctx_.services().fault_manager(); + if (fault_mgr == nullptr) { + return tl::make_unexpected(FaultResult{false, json{}, "Fault manager unavailable", FaultFailure::Unavailable}); + } + const auto & representative = records.size() == 1 ? records.front() : faults::representative_record(records); + const auto owner = faults::stored_owner(representative); + auto result = fault_mgr->get_fault_with_env(fault_code, owner); + if (!result.success) { + return tl::make_unexpected(FaultResult{false, json{}, result.error_message, result.failure}); + } + const auto app_of = app_of_source(); + // One record is served as the fault manager reads it now, several as their merge. + const json item = records.size() == 1 ? faults::record_item(result.data.value("fault", json::object()), app_of) + : faults::entry_item(records, app_of); + json env_data_json = result.data.value("environment_data", json::object()); + if (auto * capture = ctx_.services().freeze_frames()) { + // Entity frames are kept per reporting source, not per stored owner. + env_data_json = merge_entity_freeze_frames(std::move(env_data_json), + capture->frames_for(fault_code, faults::record_owner(representative))); + } + return wrap_detail_result( + dto::JsonWriter::write(build_sovd_fault_response(item, env_data_json, entity_path))); +} + bool FaultHandlers::fault_in_source_scope(const json & fault, const std::set & source_fqns) { // Thin wrapper preserving the public static API; the scope logic now lives in // the neutral core helper shared with the ROS 2 plugin-context fault path. @@ -439,6 +543,22 @@ dto::FaultDetail FaultHandlers::build_sovd_fault_response(const json & fault_jso if (!reporting_sources.empty()) { xm.reporting_sources = std::move(reporting_sources); } + if (const auto it = fault_json.find("x-medkit"); it != fault_json.end() && it->is_object()) { + std::vector records; + for (const auto & ref : it->value("records", json::array())) { + if (auto parsed = dto::JsonReader::read(ref)) { + records.push_back(std::move(*parsed)); + } + } + // A detail of one per_source record names its owner. Named `owner`, not + // `source_id`: a fault LIST's x-medkit uses `source_id` for the entity's namespace. + if (records.size() == 1 && records.front().source_id) { + xm.owner = records.front().source_id; + } + if (!records.empty()) { + xm.records = std::move(records); + } + } xm.severity_label = severity_to_label(severity); xm.status_raw = status; detail.x_medkit = std::move(xm); @@ -482,8 +602,12 @@ http::Result FaultHandlers::list_all_faults(const http::Ty make_error(503, ERR_SERVICE_UNAVAILABLE, "Failed to get faults", json{{"details", result.error_message}})); } - // Format: items array at top level - json response = {{"items", result.data["faults"]}}; + // One item per record, each in the view the per-entity routes give a record. + json response = {{"items", json::array()}}; + const auto app_of = app_of_source(); + for (const auto & record : result.data.value("faults", json::array())) { + response["items"].push_back(faults::record_item(record, app_of)); + } // x-medkit extension for ros2_medkit-specific fields (typed DTO) dto::FaultListXMedkit xm; @@ -631,8 +755,8 @@ http::Result FaultHandlers::list_faults(const http::TypedR const auto & cache = ctx_.services().entity_cache(); auto source_fqns = HandlerContext::resolve_entity_source_fqns(cache, entity_info); - json filtered_faults = faults::filter_faults_by_sources(result.data["faults"], source_fqns); - json response = {{"items", filtered_faults}}; + json response = {{"items", entity_entries(faults::filter_faults_by_sources(result.data["faults"], source_fqns), + app_of_source())}}; // x-medkit extension (typed DTO) dto::FaultListAggXMedkit xm; @@ -678,8 +802,8 @@ http::Result FaultHandlers::list_faults(const http::TypedR const auto & cache = ctx_.services().entity_cache(); auto app_fqns = HandlerContext::resolve_entity_source_fqns(cache, entity_info); - json filtered_faults = faults::filter_faults_by_sources(result.data["faults"], app_fqns); - json response = {{"items", filtered_faults}}; + json response = {{"items", entity_entries(faults::filter_faults_by_sources(result.data["faults"], app_fqns), + app_of_source())}}; dto::FaultListAggXMedkit xm; xm.entity_id = entity_id; @@ -717,8 +841,8 @@ http::Result FaultHandlers::list_faults(const http::TypedR json{{"details", result.error_message}, {entity_info.id_field, entity_id}})); } - json filtered_faults = faults::filter_faults_by_sources(result.data["faults"], app_fqns); - json response = {{"items", filtered_faults}}; + json response = { + {"items", entity_entries(faults::filter_faults_by_sources(result.data["faults"], app_fqns), app_of_source())}}; // x-medkit extension for ros2_medkit-specific fields (typed DTO) dto::FaultListXMedkit xm; @@ -796,26 +920,14 @@ http::Result FaultHandlers::get_fault(const http::TypedR // native path): skip the plugin delegation entirely and serve the // fault_manager's enriched record through the shared path below. if (fault_prov != nullptr) { - // Prefer the fault_manager's environment-enriched record when this entity - // OWNS the fault (it was reported to the fault_manager under an app the - // entity owns, per the fault-scope resolution). This carries the - // freeze-frame/rosbag snapshots, consistent with the non-plugin detail - // path. Fall through to the plugin's own provider for faults the - // fault_manager does not hold (e.g. on-demand UDS DTCs). - if (auto * fault_mgr = ctx_.services().fault_manager(); fault_mgr != nullptr) { - auto mgr_result = fault_mgr->get_fault_with_env(fault_code, ""); - if (mgr_result.success) { - const auto & owned_fault_json = mgr_result.data.value("fault", json::object()); - const auto & cache = ctx_.services().entity_cache(); - auto source_fqns = HandlerContext::resolve_entity_source_fqns(cache, entity_info); - if (FaultHandlers::fault_in_source_scope(owned_fault_json, source_fqns)) { - json env_data_json = mgr_result.data.value("environment_data", json::object()); - if (auto * capture = ctx_.services().freeze_frames()) { - env_data_json = merge_entity_freeze_frames(std::move(env_data_json), capture->frames_for(fault_code)); - } - auto detail = build_sovd_fault_response(owned_fault_json, env_data_json, entity_path_info->entity_path); - return wrap_detail_result(dto::JsonWriter::write(detail)); - } + // Records of the code the fault_manager holds in this entity's scope are + // served from it, environment data included, as on the native path. A + // code it does not hold there (e.g. an on-demand UDS DTC) goes to the + // plugin's own provider. + if (auto records = records_of_code(entity_info, fault_code, "Failed to get fault"); + records && !records->empty()) { + if (auto detail = fault_detail(*records, fault_code, entity_path_info->entity_path)) { + return std::move(*detail); } } @@ -839,37 +951,20 @@ http::Result FaultHandlers::get_fault(const http::TypedR } } - auto fault_mgr = ctx_.services().fault_manager(); - - auto result = fault_mgr->get_fault_with_env(fault_code, ""); - if (!result.success) { - return tl::make_unexpected(classify_fault_failure(result.failure, result.error_message, "Failed to get fault", - entity_info.id_field, entity_id, fault_code)); + auto records = records_of_code(entity_info, fault_code, "Failed to get fault"); + if (!records) { + return tl::make_unexpected(records.error()); } - - // Build SOVD-compliant response from the transport-supplied JSON shape. - const auto & fault_json = result.data.value("fault", json::object()); - - const auto & cache = ctx_.services().entity_cache(); - auto source_fqns = HandlerContext::resolve_entity_source_fqns(cache, entity_info); - if (!FaultHandlers::fault_in_source_scope(fault_json, source_fqns)) { - return tl::make_unexpected( - make_error(404, ERR_RESOURCE_NOT_FOUND, "Fault not found", - json{{"details", - "Fault is not in scope for this entity: every reporting source must be one of the entity's " - "owned apps, and a mixed-source fault that includes any out-of-entity reporter is rejected " - "to prevent cross-entity disclosure"}, - {entity_info.id_field, entity_id}, - {"fault_code", fault_code}})); + if (records->empty()) { + return tl::make_unexpected(fault_not_in_scope(entity_info.id_field, entity_id, fault_code)); } - - json env_data_json = result.data.value("environment_data", json::object()); - if (auto * capture = ctx_.services().freeze_frames()) { - env_data_json = merge_entity_freeze_frames(std::move(env_data_json), capture->frames_for(fault_code)); + auto detail = fault_detail(*records, fault_code, entity_path_info->entity_path); + if (!detail) { + return tl::make_unexpected(classify_fault_failure(detail.error().failure, detail.error().error_message, + "Failed to get fault", entity_info.id_field, entity_id, + fault_code)); } - auto detail = build_sovd_fault_response(fault_json, env_data_json, entity_path_info->entity_path); - - return wrap_detail_result(dto::JsonWriter::write(detail)); + return std::move(*detail); } catch (const std::exception & e) { MEDKIT_LOG_ERROR(HandlerContext::logger(), "Error in get_fault for entity '%s', fault '%s': %s", entity_id.c_str(), fault_code.c_str(), e.what()); @@ -929,83 +1024,116 @@ FaultHandlers::clear_fault(const http::TypedRequest & req) { // native path): fall through to the scope-checked fault_manager clear // below like any other entity. if (fault_prov != nullptr) { - // Cross-entity clear guard: the plugin clear_fault forwards the code to - // the fault_manager with no scope check, so without this a - // DELETE /{A}/faults/{code} could clear a fault owned by entity B. When - // the fault_manager holds this fault, require it to be in this entity's - // source scope before delegating; reject out-of-scope. Faults the - // fault_manager does not hold (plugin-internal, e.g. on-demand UDS DTCs) - // fall through to the plugin provider unchanged. Mirrors the ownership - // check in the plugin get_fault branch and the non-plugin clear path. - if (auto * fault_mgr = ctx_.services().fault_manager(); fault_mgr != nullptr) { - auto mgr_result = fault_mgr->get_fault_with_env(fault_code, ""); - if (mgr_result.success) { - const auto & owned_fault_json = mgr_result.data.value("fault", json::object()); - const auto & cache = ctx_.services().entity_cache(); - auto source_fqns = HandlerContext::resolve_entity_source_fqns(cache, entity_info); - if (!FaultHandlers::fault_in_source_scope(owned_fault_json, source_fqns)) { - return tl::make_unexpected( - make_error(404, ERR_RESOURCE_NOT_FOUND, "Fault not found", - json{{"details", - "Fault is not in scope for this entity: every reporting source must be one of the " - "entity's owned apps, and a mixed-source fault that includes any out-of-entity " - "reporter is rejected to prevent cross-entity clear"}, - {entity_info.id_field, entity_id}, - {"fault_code", fault_code}})); + // Cross-entity clear guard: the plugin clear names its own entity, but the + // record it reaches is addressed by code. When the fault_manager holds the + // code at all, only the records in this entity's scope are cleared, each + // with its owner. A fault manager that cannot be read answers 503: the + // route cannot tell a record it holds from one it does not. + auto * fault_mgr = ctx_.services().fault_manager(); + if (fault_mgr == nullptr) { + return tl::make_unexpected(make_error(503, ERR_SERVICE_UNAVAILABLE, "Failed to clear fault", + json{{entity_info.id_field, entity_id}, {"fault_code", fault_code}})); + } + auto held = fault_mgr->list_faults("", /*include_prefailed=*/true, /*include_confirmed=*/true, + /*include_cleared=*/true, /*include_healed=*/true, + /*include_muted=*/true, /*include_clusters=*/false); + if (!held.success) { + return tl::make_unexpected(classify_fault_failure(held.failure, held.error_message, "Failed to clear fault", + entity_info.id_field, entity_id, fault_code)); + } + // Each record of the code in this entity's scope, with its own owner. A + // code the fault manager does not hold at all is the plugin's own, and + // goes with an empty owner for the provider to resolve. + std::vector owners; + const auto & all = held.data.value("faults", json::array()); + const bool store_holds_code = std::any_of(all.begin(), all.end(), [&](const json & fault) { + return fault.is_object() && fault.value("fault_code", std::string{}) == fault_code; + }); + if (store_holds_code) { + const auto & cache = ctx_.services().entity_cache(); + const auto source_fqns = HandlerContext::resolve_entity_source_fqns(cache, entity_info); + for (const auto & fault : all) { + if (fault.is_object() && fault.value("fault_code", std::string{}) == fault_code && + faults::fault_in_source_scope(fault, source_fqns)) { + owners.push_back(faults::stored_owner(fault)); } } + if (owners.empty()) { + return tl::make_unexpected(fault_not_in_scope(entity_info.id_field, entity_id, fault_code)); + } + std::sort(owners.begin(), owners.end()); + } else { + owners.emplace_back(); } - try { - auto result = fault_prov->clear_fault(entity_id, fault_code); - if (!result) { + std::optional cleared; + size_t failed = 0; + for (const auto & owner : owners) { + try { + auto result = fault_prov->clear_fault_record(entity_id, fault_code, owner); + if (!result) { + if (owners.size() == 1) { + return tl::make_unexpected(make_plugin_error(result.error().http_status, result.error().message, + json{{"entity_id", entity_id}})); + } + ++failed; + } else if (!cleared) { + cleared = std::move(*result); + } + } catch (const std::exception & e) { + MEDKIT_LOG_ERROR(HandlerContext::logger(), "Plugin FaultProvider threw for entity '%s': %s", + entity_id.c_str(), e.what()); return tl::make_unexpected( - make_plugin_error(result.error().http_status, result.error().message, json{{"entity_id", entity_id}})); + make_plugin_error(500, "Plugin threw exception", json{{"entity_id", entity_id}})); + } catch (...) { + MEDKIT_LOG_ERROR(HandlerContext::logger(), "Plugin FaultProvider threw unknown exception for entity '%s'", + entity_id.c_str()); + return tl::make_unexpected( + make_plugin_error(500, "Plugin threw unknown exception", json{{"entity_id", entity_id}})); } - return Outcome{std::move(*result)}; - } catch (const std::exception & e) { - MEDKIT_LOG_ERROR(HandlerContext::logger(), "Plugin FaultProvider threw for entity '%s': %s", - entity_id.c_str(), e.what()); - return tl::make_unexpected(make_plugin_error(500, "Plugin threw exception", json{{"entity_id", entity_id}})); - } catch (...) { - MEDKIT_LOG_ERROR(HandlerContext::logger(), "Plugin FaultProvider threw unknown exception for entity '%s'", - entity_id.c_str()); - return tl::make_unexpected( - make_plugin_error(500, "Plugin threw unknown exception", json{{"entity_id", entity_id}})); } + if (failed > 0) { + return tl::make_unexpected(make_plugin_error( + 500, "Failed to clear " + std::to_string(failed) + " of " + std::to_string(owners.size()) + " records", + json{{"entity_id", entity_id}, {"failed_codes", json::array({fault_code})}})); + } + return Outcome{std::move(*cleared)}; } } - auto fault_mgr = ctx_.services().fault_manager(); - - // Verify the fault is in this entity's scope BEFORE clearing. - auto get_result = fault_mgr->get_fault_with_env(fault_code, ""); - if (!get_result.success) { - return tl::make_unexpected(classify_fault_failure(get_result.failure, get_result.error_message, - "Failed to clear fault", entity_info.id_field, entity_id, - fault_code)); + auto * fault_mgr = ctx_.services().fault_manager(); + auto records = records_of_code(entity_info, fault_code, "Failed to clear fault"); + if (!records) { + return tl::make_unexpected(records.error()); } - - const auto & cache = ctx_.services().entity_cache(); - auto source_fqns = HandlerContext::resolve_entity_source_fqns(cache, entity_info); - const auto & fault_json = get_result.data.value("fault", json::object()); - if (!FaultHandlers::fault_in_source_scope(fault_json, source_fqns)) { - return tl::make_unexpected( - make_error(404, ERR_RESOURCE_NOT_FOUND, "Fault not found", - json{{"details", - "Fault is not in scope for this entity: every reporting source must be one of the entity's " - "owned apps, and a mixed-source fault that includes any out-of-entity reporter is rejected " - "to prevent cross-entity disclosure"}, - {entity_info.id_field, entity_id}, - {"fault_code", fault_code}})); + if (records->empty()) { + return tl::make_unexpected(fault_not_in_scope(entity_info.id_field, entity_id, fault_code)); } - - auto result = fault_mgr->clear_fault(fault_code, /*skip_correlation_auto_clear=*/true); - if (!result.success) { - return tl::make_unexpected(classify_fault_failure(result.failure, result.error_message, "Failed to clear fault", - entity_info.id_field, entity_id, fault_code)); + // Every record of the code under this entity, each with its own owner. The + // correlation cascade stays off so this clear cannot reach other entities. + std::vector failures; + for (const auto & record : *records) { + auto result = fault_mgr->clear_fault(fault_code, faults::stored_owner(record), + /*skip_correlation_auto_clear=*/true); + if (!result.success) { + failures.push_back(std::move(result)); + } + } + if (failures.empty()) { + return Outcome{http::NoContent{}}; + } + if (records->size() == 1) { + return tl::make_unexpected(classify_fault_failure(failures.front().failure, failures.front().error_message, + "Failed to clear fault", entity_info.id_field, entity_id, + fault_code)); } - return Outcome{http::NoContent{}}; + return tl::make_unexpected(make_error(500, ERR_INTERNAL_ERROR, + "Failed to clear " + std::to_string(failures.size()) + " of " + + std::to_string(records->size()) + " records", + json{{"details", failures.front().error_message}, + {entity_info.id_field, entity_id}, + {"fault_code", fault_code}, + {"failed_codes", json::array({fault_code})}})); } catch (const std::exception & e) { MEDKIT_LOG_ERROR(HandlerContext::logger(), "Error in clear_fault for entity '%s', fault '%s': %s", entity_id.c_str(), fault_code.c_str(), e.what()); @@ -1058,7 +1186,14 @@ http::Result FaultHandlers::clear_all_faults(const http::TypedR if (code.empty()) { continue; } - auto clear_result = fault_prov->clear_fault(entity_id, code); + // Each listed item names its own record. Sending the entity id + // for all of them addresses at most one owner's record and + // silently leaves the others standing. + auto owner = fault.value("source_id", std::string{}); + if (owner.empty()) { + owner = faults::record_owner(fault); + } + auto clear_result = fault_prov->clear_fault_record(entity_id, code, owner); if (!clear_result) { failed_codes.push_back(code); } @@ -1089,6 +1224,9 @@ http::Result FaultHandlers::clear_all_faults(const http::TypedR // resolves through `HandlerContext::resolve_entity_source_fqns` so the // area BFS, function-hosting-component expansion, and wildcard-app // empty-set behavior stay consistent across all four fault routes. + // The defaults leave muted records out, as the entity's fault list does. + // That is deliberate: this route clears what the list shows, and a muted + // record is cleared by its own per-code DELETE, which resolves it. auto result = fault_mgr->list_faults(""); if (!result.success) { return tl::make_unexpected( @@ -1099,22 +1237,28 @@ http::Result FaultHandlers::clear_all_faults(const http::TypedR auto entity_fqns = HandlerContext::resolve_entity_source_fqns(cache, entity_info); json faults_to_clear = faults::filter_faults_by_sources(result.data["faults"], entity_fqns); - // Clear each matching fault. Use `skip_correlation_auto_clear=true` for - // the same reason as the single-fault DELETE: keep this entity's clear - // from cascading into correlated symptoms reported by other entities. - if (faults_to_clear.is_array()) { - for (const auto & fault : faults_to_clear) { - if (!fault.contains("fault_code")) { - continue; - } - std::string code = fault["fault_code"].get(); - auto clear_result = fault_mgr->clear_fault(code, /*skip_correlation_auto_clear=*/true); - if (!clear_result.success) { - MEDKIT_LOG_WARN(HandlerContext::logger(), "Failed to clear fault '%s' for entity '%s': %s", code.c_str(), - entity_id.c_str(), clear_result.error_message.c_str()); - } + // Each in-scope record with its own owner. The correlation cascade stays off, + // as on the per-code DELETE, so this clear cannot reach other entities. + std::set failed_codes; + for (const auto & fault : faults_to_clear) { + if (!fault.contains("fault_code")) { + continue; + } + const std::string code = fault["fault_code"].get(); + const std::string owner = faults::stored_owner(fault); + auto clear_result = fault_mgr->clear_fault(code, owner, /*skip_correlation_auto_clear=*/true); + if (!clear_result.success) { + MEDKIT_LOG_WARN(HandlerContext::logger(), "Failed to clear fault '%s' of source '%s' for entity '%s': %s", + code.c_str(), owner.c_str(), entity_id.c_str(), clear_result.error_message.c_str()); + failed_codes.insert(code); } } + if (!failed_codes.empty()) { + return tl::make_unexpected( + make_error(500, ERR_INTERNAL_ERROR, "Failed to clear " + std::to_string(failed_codes.size()) + " fault(s)", + json{{entity_info.id_field, entity_id}, + {"failed_codes", std::vector(failed_codes.begin(), failed_codes.end())}})); + } return http::NoContent{}; } catch (const std::exception & e) { MEDKIT_LOG_ERROR(HandlerContext::logger(), "Error in clear_all_faults for entity '%s': %s", entity_id.c_str(), @@ -1149,42 +1293,39 @@ FaultHandlers::clear_all_faults_global(const http::TypedRequest & req) { json{{"details", faults_result.error_message}})); } - // Build FQN-to-entity-ID map for lock checking + // Owned source -> owning entities, for lock checking: every app's source + // and every external component's bare id. A record belongs to an entity + // when its owner is that source or a path child of it, the rule the + // per-entity routes scope records by. auto * lock_mgr = ctx_.services().lock_manager(); - std::unordered_map fqn_to_entity; + std::unordered_map> source_to_entities; if (lock_mgr) { - const auto & cache = ctx_.services().entity_cache(); - for (const auto & app : cache.get_apps()) { - auto fqn = app.effective_fqn(); - if (!fqn.empty()) { - fqn_to_entity[fqn] = app.id; - } - } + source_to_entities = build_source_entity_map(ctx_.services().entity_cache()); } auto client_id = req.header("X-Client-Id").value_or(std::string{}); - // Clear each fault, skipping those on locked entities + // Clear each RECORD, each with its owner, skipping records whose owning + // entity is locked by another client. if (faults_result.data.contains("faults") && faults_result.data["faults"].is_array()) { for (const auto & fault : faults_result.data["faults"]) { if (!fault.contains("fault_code")) { continue; } - // Check if any reporting source is on a locked entity + const std::string owner = faults::stored_owner(fault); + const std::string source = faults::record_owner(fault); + // Skipped when any entity whose scope holds the record's first source is + // locked by another client. check_access adds that entity's parent chain. bool blocked = false; - if (lock_mgr && fault.contains("reporting_sources")) { - for (const auto & src : fault["reporting_sources"]) { - auto src_str = src.get(); - auto it = fqn_to_entity.find(src_str); - if (it != fqn_to_entity.end()) { - auto access = lock_mgr->check_access(it->second, client_id, "faults"); - if (!access.allowed) { - blocked = true; - break; - } - } - } + if (lock_mgr) { + const auto denied = [&](const std::string & entity_id) { + return !lock_mgr->check_access(entity_id, client_id, "faults").allowed; + }; + blocked = std::any_of(source_to_entities.begin(), source_to_entities.end(), [&](const auto & entry) { + return faults::source_in_scope(source, entry.first) && + std::any_of(entry.second.begin(), entry.second.end(), denied); + }); } if (blocked) { @@ -1192,10 +1333,10 @@ FaultHandlers::clear_all_faults_global(const http::TypedRequest & req) { } std::string code = fault["fault_code"].get(); - auto clear_result = fault_mgr->clear_fault(code); + auto clear_result = fault_mgr->clear_fault(code, owner); if (!clear_result.success) { - MEDKIT_LOG_WARN(HandlerContext::logger(), "Failed to clear fault '%s': %s", code.c_str(), - clear_result.error_message.c_str()); + MEDKIT_LOG_WARN(HandlerContext::logger(), "Failed to clear fault '%s' of source '%s': %s", code.c_str(), + owner.c_str(), clear_result.error_message.c_str()); } } } diff --git a/src/ros2_medkit_gateway/src/http/handlers/sse_fault_handler.cpp b/src/ros2_medkit_gateway/src/http/handlers/sse_fault_handler.cpp index de51eb1e1..08ff37ce6 100644 --- a/src/ros2_medkit_gateway/src/http/handlers/sse_fault_handler.cpp +++ b/src/ros2_medkit_gateway/src/http/handlers/sse_fault_handler.cpp @@ -146,12 +146,16 @@ std::optional SSEFaultHandler::delivered_watermark_locked() const { } std::deque::iterator SSEFaultHandler::find_superseded_locked(bool updates_only) { - // Keyed on the peer as well as the code: a fault code is unique on the - // gateway that raised it and nowhere else, so two peers reporting the same - // code are reporting two faults, and treating one as the newer state of the - // other would delete a live fault from a client's view. + // Keyed on the record, not the code: a record is (fault_code, owner), so two + // sources reporting one per_source code are two faults, and one owner's newer event says + // nothing about the other owner's record. Keyed on the peer as well: a fault + // is unique on the gateway that raised it and nowhere else, so two peers + // reporting the same record are reporting two faults. Treating either as the + // newer state of the other would delete a live fault from a client's view. auto supersede_key = [](const QueuedEvent & queued) { - return queued.peer + '\0' + queued.event.fault.fault_code; + const auto & sources = queued.event.fault.reporting_sources; + const std::string owner = sources.empty() ? std::string{} : sources.front(); + return queued.peer + '\0' + queued.event.fault.fault_code + '\0' + owner; }; std::unordered_map newest_index; for (std::size_t i = 0; i < event_queue_.size(); ++i) { @@ -212,8 +216,8 @@ SSEFaultHandler::EvictionStats SSEFaultHandler::evict_to_capacity_locked() { ++stats.coalesced; continue; } - // Something a live client is owed has to go. Prefer an entry a newer - // same-code event supersedes: the current state still reaches the client + // Something a live client is owed has to go. Prefer an entry a newer event + // of the same record supersedes: the current state still reaches the client // even though the transition history does not. Either way it is a real, // counted loss. auto victim = find_superseded_locked(/*updates_only=*/false); @@ -695,66 +699,34 @@ std::string SSEFaultHandler::format_sse_event(const QueuedEvent & queued) { std::optional SSEFaultHandler::resolve_entity_context(const fd::FaultRecord & fault) const { - if (fault.reporting_sources.empty()) { + // The first reporting source stands for the record: its owner, or for a + // record that several sources share, one of them. + if (fault.reporting_sources.empty() || fault.reporting_sources.front().empty()) { return std::nullopt; } - // reporting_sources is a set; debounced faults can carry several co-reporters - // (e.g. node_a and node_b raising the same fault_code). .front() picks the - // lexicographically-first FQN, not a defined owner - any co-reporter's - // rosbag is fetchable, so this remains a valid hint, just not authoritative. - const auto & raw_fqn = fault.reporting_sources.front(); - if (raw_fqn.empty()) { - return std::nullopt; - } - + const auto & owner = fault.reporting_sources.front(); const auto & cache = ctx_.services().entity_cache(); - // Manifest / hybrid mode: the linking step populated node_to_app with the - // ROS FQN -> manifest app id mapping. Try both FQN forms (with and without - // the leading '/'), mirroring gateway_node's node_resolver lambda. - std::string entity_id = cache.resolve_node_to_app(raw_fqn); - if (entity_id.empty() && raw_fqn.front() == '/') { - entity_id = cache.resolve_node_to_app(raw_fqn.substr(1)); - } - - // Runtime fallback: synthetic apps are created with id = ROS node name - // (the FQN's last segment) when there is no namespace collision, or - // ``_`` (slashes in the namespace replaced with '_') when - // multiple nodes share the same name. See ros2_runtime_introspection.cpp. - // Only accept a candidate that actually exists as an App in the cache so - // we never point consumers at a 404. - if (entity_id.empty()) { - auto last_slash = raw_fqn.rfind('/'); - auto name = (last_slash != std::string::npos) ? raw_fqn.substr(last_slash + 1) : raw_fqn; - if (!name.empty() && cache.has_app(name)) { - entity_id = std::move(name); - } else if (last_slash != std::string::npos && last_slash > 0) { - // Try the collision-disambiguated form: ns prefix (sans leading '/'), - // slashes replaced with '_', then '_' + name. - auto ns_prefix = raw_fqn.substr(1, last_slash - 1); - std::replace(ns_prefix.begin(), ns_prefix.end(), '/', '_'); - auto namespaced = ns_prefix + "_" + name; - if (cache.has_app(namespaced)) { - entity_id = std::move(namespaced); - } - } + // The documented use of x-medkit is the rosbag address under the named + // entity, whose download is authorized by the fault scope rule. So the named + // app is the one whose scope holds the owner, never a namesake of a segment. + auto entity_id = app_sources_.find_owner(cache, owner); + if (!entity_id.empty()) { + return EntityContext{"apps", std::move(entity_id)}; } - if (entity_id.empty()) { - MEDKIT_LOG_DEBUG(HandlerContext::logger(), - "SSE fault event: no entity match for reporting source '%s' (fault_code='%s'); " - "omitting x-medkit extension", - raw_fqn.c_str(), fault.fault_code.c_str()); - return std::nullopt; + // A protocol bridge raises its faults under an external component's bare id + // or a path below it. Entity ids hold no '/', so the first segment is the + // only candidate. A runtime host component claims no bare id. + const auto head = owner.substr(0, owner.find('/')); + if (auto component = cache.get_component(head); component && component->external.value_or(false)) { + return EntityContext{"components", head}; } - // entity_type is hardcoded "apps" because apps are the leaf reporters in - // SOVD - reporting_sources always carries ROS node FQNs which map to apps. - // Components own faults transitively via their hosted apps; consumers can - // walk up the hierarchy via /apps/ -> belongs_to if they need the - // owning component. Manifest-only components without a bound node have no - // FQN match here and fall back to plain discovery - by design. - return EntityContext{"apps", std::move(entity_id)}; + MEDKIT_LOG_DEBUG(HandlerContext::logger(), + "SSE fault event: no entity owns '%s' (fault_code='%s'); omitting x-medkit extension", owner.c_str(), + fault.fault_code.c_str()); + return std::nullopt; } } // namespace handlers diff --git a/src/ros2_medkit_gateway/src/http/handlers/trigger_handlers.cpp b/src/ros2_medkit_gateway/src/http/handlers/trigger_handlers.cpp index f1200f4d6..90b6d7b72 100644 --- a/src/ros2_medkit_gateway/src/http/handlers/trigger_handlers.cpp +++ b/src/ros2_medkit_gateway/src/http/handlers/trigger_handlers.cpp @@ -228,21 +228,12 @@ TriggerHandlers::post_trigger(const http::TypedRequest & req, dto::TriggerCreate // (the retry budget is sized against it in gateway_node), and a // genuinely topic-less point surfaces through the retry-expiry warning. auto * pmgr = ctx_.services().plugin_manager(); + // Provider first, vendor route as the fallback - the shared resolution + // the fault-trigger engine's own enumeration uses, so "exists" here and + // "the fetcher could read it" there cannot drift apart. std::optional content; if (pmgr != nullptr) { - if (auto * data_prov = pmgr->get_data_provider_for_entity(entity_id)) { - try { - if (auto result = data_prov->list_data(entity_id)) { - content = result->content; - } - } catch (...) { - // Enumeration is advisory here - a throwing provider must not - // break trigger creation, it just skips the strict check. - } - } - if (!content) { - content = pmgr->fetch_entity_data_via_route(entity_id); - } + content = pmgr->fetch_entity_data_content(entity_id); } // Same liveness guard as the fault-trigger enumeration in gateway_node: // a bridge with a dead PLC link answers 200 with connected=false and an diff --git a/src/ros2_medkit_gateway/src/http/rest_server.cpp b/src/ros2_medkit_gateway/src/http/rest_server.cpp index 66ca1ac03..09317d74f 100644 --- a/src/ros2_medkit_gateway/src/http/rest_server.cpp +++ b/src/ros2_medkit_gateway/src/http/rest_server.cpp @@ -1203,8 +1203,8 @@ void RESTServer::setup_routes() { // path while the typed router still owns wire framing. The single-fault // DELETE uses `del_alternates` so the ROS // path returns 204 while the plugin path keeps its 200 + ack-body shape. - // The bulk-clear DELETE returns NoContent unconditionally - the legacy - // plugin branch also emitted 204 after iterating the per-fault clears. + // The bulk-clear DELETE returns NoContent, or 500 with `failed_codes` when + // the clear of a record fails. reg.get(entity_path + "/faults", [this](http::TypedRequest req) -> http::Result { return fault_handlers_->list_faults(req); @@ -1247,8 +1247,8 @@ void RESTServer::setup_routes() { .description(std::string("Clears a specific fault for this ") + et.singular + ".") // FaultHandlers::clear_fault -> validate_lock_access("faults"). .lock_guarded() - // 503 when the fault store cannot be read - clear_fault reads the fault - // before clearing it, so it answers the same status the read does. + // 503 when the fault store cannot be read - clear_fault reads the + // records before clearing them, so it answers the status the read does. .errors({503}) .operation_id(std::string("clear") + capitalize(et.singular) + "Fault"); @@ -1370,6 +1370,12 @@ void RESTServer::setup_routes() { .requires_role(UserRole::VIEWER) .summary(std::string("Download bulk-data file for ") + et.singular) .description("Downloads a bulk-data file (binary content).") + // 409 x-medkit-ambiguous-fault: a rosbag URL carrying a fault code + // (the form that predates recording ids) resolves that code to one + // record the way the fault routes do, and several candidate records + // in this entity's scope name none of them. 503: a rosbag when the + // fault manager does not answer the ownership check. + .errors({409, 503}) .operation_id(std::string("download") + capitalize(et.singular) + "BulkData"); register_recording_routes(entity_path, et.singular); @@ -2159,6 +2165,8 @@ void RESTServer::setup_routes() { .requires_role(UserRole::VIEWER) .summary("Download bulk-data file for subarea") .description("Downloads a bulk-data file for a subarea.") + // 409 x-medkit-ambiguous-fault and 503, as on the four top-level download routes. + .errors({409, 503}) .operation_id("downloadSubareaBulkData"); register_recording_routes("/areas/{area_id}/subareas/{subarea_id}", "subarea"); @@ -2196,6 +2204,8 @@ void RESTServer::setup_routes() { .requires_role(UserRole::VIEWER) .summary("Download bulk-data file for subcomponent") .description("Downloads a bulk-data file for a subcomponent.") + // 409 x-medkit-ambiguous-fault and 503, as on the four top-level download routes. + .errors({409, 503}) .operation_id("downloadSubcomponentBulkData"); register_recording_routes("/components/{component_id}/subcomponents/{subcomponent_id}", "subcomponent"); diff --git a/src/ros2_medkit_gateway/src/plugins/plugin_manager.cpp b/src/ros2_medkit_gateway/src/plugins/plugin_manager.cpp index 405e8296d..80384e037 100644 --- a/src/ros2_medkit_gateway/src/plugins/plugin_manager.cpp +++ b/src/ros2_medkit_gateway/src/plugins/plugin_manager.cpp @@ -434,6 +434,24 @@ bool PluginManager::has_entity_data_route(const std::string & entity_id) { return false; } +std::optional PluginManager::fetch_entity_data_content(const std::string & entity_id) { + // Rationale is on the declaration: provider first, vendor route only when the + // owner exposes no provider for this entity. + if (auto * data_prov = get_data_provider_for_entity(entity_id)) { + try { + if (auto result = data_prov->list_data(entity_id)) { + return std::optional{result->content}; + } + } catch (const std::exception & e) { + MEDKIT_LOG_WARN(logger(), "DataProvider threw for entity '%s': %s. Trying its route.", entity_id.c_str(), + e.what()); + } catch (...) { + MEDKIT_LOG_WARN(logger(), "DataProvider threw for entity '%s'. Trying its route.", entity_id.c_str()); + } + } + return fetch_entity_data_via_route(entity_id); +} + std::optional PluginManager::fetch_entity_data_via_route(const std::string & entity_id, const std::string & item) { const std::string full_path = diff --git a/src/ros2_medkit_gateway/src/ros2/conversions/fault_msg_conversions.cpp b/src/ros2_medkit_gateway/src/ros2/conversions/fault_msg_conversions.cpp index c4903d201..b0dce0b57 100644 --- a/src/ros2_medkit_gateway/src/ros2/conversions/fault_msg_conversions.cpp +++ b/src/ros2_medkit_gateway/src/ros2/conversions/fault_msg_conversions.cpp @@ -42,6 +42,7 @@ fd::FaultRecord from_msg(const ros2_medkit_msgs::msg::Fault & msg) { record.occurrence_count = msg.occurrence_count; record.status = msg.status; record.reporting_sources = msg.reporting_sources; + record.owner = msg.owner; return record; } @@ -55,7 +56,13 @@ fd::FaultEventRecord from_msg(const ros2_medkit_msgs::msg::FaultEvent & msg) { } nlohmann::json fault_to_json(const ros2_medkit_msgs::msg::Fault & fault) { - return fault_record_to_json(from_msg(fault)); + auto j = fault_record_to_json(from_msg(fault)); + // Internal to the gateway: the per-record calls address the record with it, + // and the HTTP items drop it (faults::record_item). + if (!fault.owner.empty()) { + j["owner"] = fault.owner; + } + return j; } nlohmann::json environment_data_to_json(const ros2_medkit_msgs::msg::EnvironmentData & env_data) { diff --git a/src/ros2_medkit_gateway/src/ros2/transports/ros2_fault_service_transport.cpp b/src/ros2_medkit_gateway/src/ros2/transports/ros2_fault_service_transport.cpp index f097ec7e6..7c84c79a0 100644 --- a/src/ros2_medkit_gateway/src/ros2/transports/ros2_fault_service_transport.cpp +++ b/src/ros2_medkit_gateway/src/ros2/transports/ros2_fault_service_transport.cpp @@ -258,10 +258,14 @@ FaultResult Ros2FaultServiceTransport::list_faults(const std::string & source_id if (include_muted && !response->muted_faults.empty()) { json muted_array = json::array(); for (const auto & muted : response->muted_faults) { + // source_id is what makes the entry addressable: a muted record is one + // record, and its code alone names as many as there are sources + // reporting it. muted_array.push_back({{"fault_code", muted.fault_code}, {"root_cause_code", muted.root_cause_code}, {"rule_id", muted.rule_id}, - {"delay_ms", muted.delay_ms}}); + {"delay_ms", muted.delay_ms}, + {"source_id", muted.source_id}}); } result.data["muted_faults"] = muted_array; } @@ -296,6 +300,11 @@ FaultWithEnvJsonResult Ros2FaultServiceTransport::get_fault_with_env(const std:: auto request = std::make_shared(); request->fault_code = fault_code; + // The record is (fault_code, source_id), so the owner goes in the request and + // the fault manager resolves the record. Filtering the answer here instead + // would mean asking for an arbitrary record and then hoping it is the one the + // caller meant. + request->source_id = source_id; auto response = invoke_fault_service( get_fault_client_, request, executor_, executor_mutex_, std::chrono::duration(service_timeout_sec_), @@ -312,22 +321,6 @@ FaultWithEnvJsonResult Ros2FaultServiceTransport::get_fault_with_env(const std:: return result; } - // Verify source_id if provided (prefix match against any reporting source). - if (!source_id.empty()) { - bool matches = false; - for (const auto & src : response->fault.reporting_sources) { - if (src.rfind(source_id, 0) == 0) { - matches = true; - break; - } - } - if (!matches) { - result.success = false; - result.error_message = "Fault not found for source: " + source_id; - return result; - } - } - result.success = true; result.data = {{"fault", conversions::fault_to_json(response->fault)}, {"environment_data", conversions::environment_data_to_json(response->environment_data)}}; @@ -350,11 +343,13 @@ FaultResult Ros2FaultServiceTransport::get_fault(const std::string & fault_code, return result; } -FaultResult Ros2FaultServiceTransport::clear_fault(const std::string & fault_code, bool skip_correlation_auto_clear) { +FaultResult Ros2FaultServiceTransport::clear_fault(const std::string & fault_code, const std::string & source_id, + bool skip_correlation_auto_clear) { FaultResult result; auto request = std::make_shared(); request->fault_code = fault_code; + request->source_id = source_id; request->skip_correlation_auto_clear = skip_correlation_auto_clear; auto response = invoke_fault_service( @@ -379,11 +374,13 @@ FaultResult Ros2FaultServiceTransport::clear_fault(const std::string & fault_cod return result; } -FaultResult Ros2FaultServiceTransport::get_snapshots(const std::string & fault_code, const std::string & topic) { +FaultResult Ros2FaultServiceTransport::get_snapshots(const std::string & fault_code, const std::string & source_id, + const std::string & topic) { FaultResult result; auto request = std::make_shared(); request->fault_code = fault_code; + request->source_id = source_id; request->topic = topic; auto response = invoke_fault_service( @@ -414,16 +411,18 @@ FaultResult Ros2FaultServiceTransport::get_snapshots(const std::string & fault_c return result; } -FaultResult Ros2FaultServiceTransport::get_rosbag(const std::string & fault_code) { +FaultResult Ros2FaultServiceTransport::get_rosbag(const std::string & id, const std::string & source_id) { FaultResult result; // The parameter is the bulk-data id, which is a recording id. It is sent in BOTH // fields: the fault manager prefers recording_id and falls back to fault_code, // which is what keeps a pre-#620 URL (and every existing .test.py that calls this - // service with a fault code) working unchanged. + // service with a fault code) working unchanged. source_id scopes that fallback + // to one record. The recording_id path ignores it. auto request = std::make_shared(); - request->recording_id = fault_code; - request->fault_code = fault_code; + request->recording_id = id; + request->fault_code = id; + request->source_id = source_id; auto response = invoke_fault_service( get_rosbag_client_, request, executor_, executor_mutex_, std::chrono::duration(service_timeout_sec_), diff --git a/src/ros2_medkit_gateway/src/trigger_fault_subscriber.cpp b/src/ros2_medkit_gateway/src/trigger_fault_subscriber.cpp index dca012715..7aacf9ce6 100644 --- a/src/ros2_medkit_gateway/src/trigger_fault_subscriber.cpp +++ b/src/ros2_medkit_gateway/src/trigger_fault_subscriber.cpp @@ -26,8 +26,8 @@ TriggerFaultSubscriber::TriggerFaultSubscriber(rclcpp::Node * node, ResourceChan build_fault_manager_events_topic(node).c_str()); } -void TriggerFaultSubscriber::set_node_to_entity_resolver(NodeToEntityFn resolver) { - feed_.set_node_to_entity_resolver(std::move(resolver)); +void TriggerFaultSubscriber::set_fault_owner_resolver(FaultOwnerFn resolver) { + feed_.set_fault_owner_resolver(std::move(resolver)); } } // namespace ros2_medkit_gateway diff --git a/src/ros2_medkit_gateway/test/test_bulkdata_handlers.cpp b/src/ros2_medkit_gateway/test/test_bulkdata_handlers.cpp index 5f8cf4824..a2601659e 100644 --- a/src/ros2_medkit_gateway/test/test_bulkdata_handlers.cpp +++ b/src/ros2_medkit_gateway/test/test_bulkdata_handlers.cpp @@ -16,11 +16,14 @@ #include #include +#include #include #include #include #include #include +#include +#include #include #include #include @@ -39,7 +42,9 @@ #include "ros2_medkit_gateway/core/http/handlers/bulkdata_handlers.hpp" #include "ros2_medkit_gateway/core/http/http_utils.hpp" #include "ros2_medkit_gateway/core/managers/bulk_data_store.hpp" +#include "ros2_medkit_gateway/core/managers/fault_manager.hpp" #include "ros2_medkit_gateway/core/models/thread_safe_entity_cache.hpp" +#include "ros2_medkit_gateway/core/transports/fault_service_transport.hpp" using namespace ros2_medkit_gateway; // No `using json = nlohmann::json` here: the namespace pulled in above already @@ -120,15 +125,28 @@ TEST_F(BulkDataHandlersTest, RecordingIdToleratesTrailingSlashAndEmptyPath) { namespace { -json rosbag_row(const std::string & fault_code, const std::string & recording_id, uint64_t size_bytes = 1024) { - return json{{"fault_code", fault_code}, {"recording_id", recording_id}, {"file_path", "/var/bags/" + recording_id}, - {"format", "mcap"}, {"duration_sec", 5.0}, {"size_bytes", size_bytes}}; +/// The listing stamps each row with the reporting source it was listed under, +/// which is the owner of the record the recording belongs to. +json rosbag_row(const std::string & fault_code, const std::string & recording_id, uint64_t size_bytes = 1024, + const std::string & owner = "app_a") { + return json{{"fault_code", fault_code}, + {"recording_id", recording_id}, + {"file_path", "/var/bags/" + recording_id}, + {"format", "mcap"}, + {"duration_sec", 5.0}, + {"size_bytes", size_bytes}, + {"owner", owner}}; } json fault_at(double first_occurred) { return json{{"first_occurred", first_occurred}}; } +/// Key one record for the date lookup the same way the handler does. +std::string record_key(const std::string & owner, const std::string & fault_code) { + return handlers::detail::record_map_key(owner, fault_code); +} + /// A row carrying the recording's own timestamp, which is what the fault manager /// sends now. json rosbag_row_made_at(const std::string & fault_code, const std::string & recording_id, int64_t created_at_ns) { @@ -188,8 +206,8 @@ TEST_F(BulkDataHandlersTest, ARecordingIsDatedByTheEarliestFaultOfItsBurst) { // Downstream faults confirm after the root cause, and the recording covers // the whole burst, so the earliest is the honest creation date. const std::vector rows{rosbag_row("DOWNSTREAM", "fault_ROOT_9"), rosbag_row("ROOT", "fault_ROOT_9")}; - const std::unordered_map faults{{"DOWNSTREAM", fault_at(1700000900.0)}, - {"ROOT", fault_at(1700000000.0)}}; + const std::unordered_map faults{{record_key("app_a", "DOWNSTREAM"), fault_at(1700000900.0)}, + {record_key("app_a", "ROOT"), fault_at(1700000000.0)}}; const auto descriptors = handlers::detail::fold_rosbag_rows_into_descriptors(rows, faults); ASSERT_EQ(descriptors.size(), 1u); @@ -202,7 +220,7 @@ TEST_F(BulkDataHandlersTest, EachRecordingOfOneFaultIsDatedByItsOwnCapture) { // is exactly what tells the occurrences apart. const std::vector rows{rosbag_row_made_at("FLAP", "fault_FLAP_2", int64_t{1700000900} * 1'000'000'000), rosbag_row_made_at("FLAP", "fault_FLAP_1", int64_t{1700000000} * 1'000'000'000)}; - const std::unordered_map faults{{"FLAP", fault_at(1700000000.0)}}; + const std::unordered_map faults{{record_key("app_a", "FLAP"), fault_at(1700000000.0)}}; const auto descriptors = handlers::detail::fold_rosbag_rows_into_descriptors(rows, faults); ASSERT_EQ(descriptors.size(), 2u); @@ -211,6 +229,23 @@ TEST_F(BulkDataHandlersTest, EachRecordingOfOneFaultIsDatedByItsOwnCapture) { EXPECT_NE(descriptors[0].creation_date, descriptors[1].creation_date); } +// Two sources reporting one code are two records with their own first_occurred. +// Keying the date lookup on the code alone let whichever record was listed last +// date the other owner's recordings. +TEST_F(BulkDataHandlersTest, ARecordingIsDatedByItsOwnOwnersRecord) { + const std::vector rows{rosbag_row("SHARED_CODE", "rec_a", 1024, "app_a"), + rosbag_row("SHARED_CODE", "rec_b", 1024, "app_b")}; + const std::unordered_map faults{{record_key("app_a", "SHARED_CODE"), fault_at(1700000000.0)}, + {record_key("app_b", "SHARED_CODE"), fault_at(1700009999.0)}}; + + const auto descriptors = handlers::detail::fold_rosbag_rows_into_descriptors(rows, faults); + + ASSERT_EQ(descriptors.size(), 2u); + EXPECT_EQ(descriptors[0].creation_date, format_timestamp_ns(int64_t{1700000000} * 1'000'000'000)); + EXPECT_EQ(descriptors[1].creation_date, format_timestamp_ns(int64_t{1700009999} * 1'000'000'000)) + << "app_b's recording was dated from app_a's record of the same code"; +} + TEST_F(BulkDataHandlersTest, AnAcknowledgedFaultsRecordingsKeepTheirRealDate) { // list_faults excludes cleared faults by default, so an acknowledged fault is // absent from the map while its rows are still listed. Reading the date off the @@ -1645,9 +1680,11 @@ TEST_F(BulkDataSourceFiltersTest, FunctionWithComponentHostResolvesComponentApps // get_fault(code, source) semantics) would let app id "plc" claim the bag of // "plc_line1". // === Download authorization tests === -// A recording is shared by a whole burst, so ownership is the union over its -// attached faults. The scope matcher itself is unchanged and pinned below; what -// is new is which codes get fed to it. +// A recording is shared by a whole burst, so ownership is the union over the +// records it is attached to, each a (fault code, owner) pair. The scope matcher +// itself is unchanged and pinned below. The attached codes say which of the +// entity's records to ask about, and the owner's own rosbag rows say whether it +// holds the recording. TEST_F(BulkDataSourceFiltersTest, AttachedFaultCodesComeFromTheRecordingNotTheUrl) { const nlohmann::json rosbag = {{"file_path", "/var/bags/fault_ROOT_1"}, @@ -1677,6 +1714,44 @@ TEST_F(BulkDataSourceFiltersTest, AttachedFaultCodesFallBackOnAnEmptyOrMalformed EXPECT_EQ(handlers::detail::rosbag_attached_fault_codes(not_an_array, "X"), (std::vector{"X"})); } +// One owner's ListRosbags rows prove its records are attached to the recording +// only when a row names that recording. Another recording of the same code is +// another owner's, or another occurrence, and proves nothing. +TEST_F(BulkDataSourceFiltersTest, AnOwnersRowsHoldARecordingOnlyWhenARowNamesIt) { + const nlohmann::json served = {{"file_path", "/var/bags/rec_pump"}, {"recording_id", "rec_pump"}}; + const nlohmann::json tank_rows = nlohmann::json::array( + {nlohmann::json{{"fault_code", "SHARED"}, {"recording_id", "rec_tank"}, {"file_path", "/var/bags/rec_tank"}}}); + const nlohmann::json pump_rows = nlohmann::json::array( + {nlohmann::json{{"fault_code", "SHARED"}, {"recording_id", "rec_pump"}, {"file_path", "/var/bags/rec_pump"}}}); + + EXPECT_FALSE(handlers::detail::rosbag_rows_hold_recording(tank_rows, served)) + << "tank's recording of the same code is not pump's recording"; + EXPECT_TRUE(handlers::detail::rosbag_rows_hold_recording(pump_rows, served)); + EXPECT_FALSE(handlers::detail::rosbag_rows_hold_recording(nlohmann::json::array(), served)); + EXPECT_FALSE(handlers::detail::rosbag_rows_hold_recording(nlohmann::json::object(), served)); +} + +// A peer that predates recording ids names a bag only by its path, on either +// side. The path then identifies the recording. Two ids that differ never match +// through their paths. +TEST_F(BulkDataSourceFiltersTest, ARecordingIsMatchedByItsPathWhenEitherSideHasNoId) { + const nlohmann::json row_without_id = + nlohmann::json::array({nlohmann::json{{"fault_code", "SHARED"}, {"file_path", "/var/bags/rec_pump"}}}); + const nlohmann::json served_with_id = {{"file_path", "/var/bags/rec_pump"}, {"recording_id", "rec_pump"}}; + const nlohmann::json served_without_id = {{"file_path", "/var/bags/rec_pump"}}; + const nlohmann::json row_with_id = nlohmann::json::array( + {nlohmann::json{{"fault_code", "SHARED"}, {"recording_id", "rec_pump"}, {"file_path", "/var/bags/rec_pump"}}}); + + EXPECT_TRUE(handlers::detail::rosbag_rows_hold_recording(row_without_id, served_with_id)); + EXPECT_TRUE(handlers::detail::rosbag_rows_hold_recording(row_with_id, served_without_id)); + EXPECT_FALSE(handlers::detail::rosbag_rows_hold_recording(row_without_id, {{"file_path", "/var/bags/rec_tank"}})); + + const nlohmann::json same_path_other_id = nlohmann::json::array( + {nlohmann::json{{"fault_code", "SHARED"}, {"recording_id", "rec_other"}, {"file_path", "/var/bags/rec_pump"}}}); + EXPECT_FALSE(handlers::detail::rosbag_rows_hold_recording(same_path_other_id, served_with_id)) + << "two recording ids are two recordings"; +} + TEST_F(BulkDataSourceFiltersTest, AFaultCodeUrlIsRecognisedAsTheCompatibilityPath) { // The segment named a fault; the answer is that fault's newest recording, whose // id is something else. Authorizing on the union alone would 200 a code the @@ -1788,3 +1863,298 @@ TEST_F(BulkDataSourceFiltersTest, DownloadOwnershipScopeRejectsPrefixSiblingApp) nlohmann::json child_fault = {{"reporting_sources", {"plc/axis1"}}}; EXPECT_TRUE(faults::fault_in_source_scope(child_fault, scope)); } + +// ============================================================================= +// list_scoped_rosbag_descriptors: which recordings an entity lists, and what a +// failing fault manager turns the listing into. +// ============================================================================= + +namespace { + +/// Fault manager stand-in answering list_faults and list_rosbags from fixed +/// records. list_rosbags matches the owner exactly, like the fault manager. +class ListingTransport : public FaultServiceTransport { + public: + ListingTransport() = default; + ~ListingTransport() override = default; + ListingTransport(const ListingTransport &) = delete; + ListingTransport & operator=(const ListingTransport &) = delete; + ListingTransport(ListingTransport &&) = delete; + ListingTransport & operator=(ListingTransport &&) = delete; + + FaultResult report_fault(const std::string & /*fault_code*/, uint8_t /*severity*/, + const std::string & /*description*/, const std::string & /*source_id*/) override { + return {}; + } + FaultResult list_faults(const std::string & /*source_id*/, bool prefailed, bool confirmed, bool cleared, bool healed, + bool muted, bool /*include_clusters*/) override { + ++list_faults_calls; + asked_all_states = prefailed && confirmed && cleared && healed && muted; + FaultResult r; + r.success = faults_ok; + r.data = {{"faults", faults}}; + r.error_message = faults_ok ? "" : "list_faults timed out"; + return r; + } + FaultWithEnvJsonResult get_fault_with_env(const std::string & /*fault_code*/, + const std::string & /*source_id*/) override { + return {}; + } + FaultResult get_fault(const std::string & /*fault_code*/, const std::string & /*source_id*/) override { + return {}; + } + FaultResult clear_fault(const std::string & /*fault_code*/, const std::string & /*source_id*/, + bool /*skip_correlation_auto_clear*/) override { + return {}; + } + FaultResult get_snapshots(const std::string & /*fault_code*/, const std::string & /*source_id*/, + const std::string & /*topic*/) override { + return {}; + } + FaultResult get_rosbag(const std::string & /*id*/, const std::string & /*source_id*/) override { + FaultResult r; + r.success = !served.is_null(); + r.data = served; + if (!r.success) { + r.failure = rosbag_failure; + } + return r; + } + FaultResult list_rosbags(const std::string & owner) override { + ++list_rosbags_calls[owner]; + FaultResult r; + r.success = owner != failing_owner; + r.error_message = r.success ? "" : "list_rosbags timed out"; + nlohmann::json rows = nlohmann::json::array(); + for (const auto & [row_owner, row] : rosbags) { + if (row_owner == owner) { + rows.push_back(row); + } + } + r.data = {{"rosbags", rows}}; + return r; + } + bool wait_for_services(std::chrono::duration /*timeout*/) override { + return true; + } + bool is_available() const override { + return true; + } + + bool faults_ok{true}; + bool asked_all_states{false}; + int list_faults_calls{0}; + std::string failing_owner; + /// ListRosbags calls per owner. + std::map list_rosbags_calls; + nlohmann::json faults = nlohmann::json::array(); + /// (owner, row) pairs; a row of the wire format does not carry its owner. + std::vector> rosbags; + /// GetRosbag's answer; null fails the call with `rosbag_failure`. + nlohmann::json served; + FaultFailure rosbag_failure{FaultFailure::Declined}; +}; + +nlohmann::json listing_record(const std::string & code, const std::string & owner) { + return {{"fault_code", code}, {"reporting_sources", {owner}}}; +} + +std::pair listing_row(const std::string & code, const std::string & owner) { + return {owner, + {{"fault_code", code}, {"recording_id", "fault_" + code + "_1"}, {"file_path", "/tmp/fault_" + code + "_1"}}}; +} + +std::set listed_codes(const std::vector & descriptors) { + std::set codes; + for (const auto & d : descriptors) { + for (const auto & code : d.x_medkit->at("fault_codes")) { + codes.insert(code.get()); + } + } + return codes; +} + +} // namespace + +TEST(ListScopedRosbagDescriptorsTest, ListsBagsOfRecordsWhoseOwnerIsInScope) { + auto transport = std::make_shared(); + transport->faults = nlohmann::json::array({ + listing_record("EXACT", "/x/detector"), + listing_record("SUB", "/x/detector/diag"), + // A second record of the same owner: the owner is still asked once. + listing_record("SUB_TOO", "/x/detector/diag"), + listing_record("DEEP", "/x/detector/diag/deep"), + listing_record("FOREIGN", "/z/other"), + listing_record("PREFIX", "/x/detector_extra"), + }); + transport->rosbags = { + listing_row("EXACT", "/x/detector"), listing_row("SUB", "/x/detector/diag"), + listing_row("SUB_TOO", "/x/detector/diag"), listing_row("DEEP", "/x/detector/diag/deep"), + listing_row("FOREIGN", "/z/other"), listing_row("PREFIX", "/x/detector_extra"), + }; + FaultManager mgr(transport); + + auto result = handlers::detail::list_scoped_rosbag_descriptors(mgr, {"/x/detector"}); + ASSERT_TRUE(result.has_value()) << result.error().message; + EXPECT_EQ(listed_codes(*result), (std::set{"DEEP", "EXACT", "SUB", "SUB_TOO"})); + EXPECT_TRUE(transport->asked_all_states); + EXPECT_EQ(transport->list_rosbags_calls, + (std::map{{"/x/detector", 1}, {"/x/detector/diag", 1}, {"/x/detector/diag/deep", 1}})); +} + +TEST(ListScopedRosbagDescriptorsTest, FaultListFailureIsAnErrorNotAnEmptyList) { + auto transport = std::make_shared(); + transport->faults_ok = false; + FaultManager mgr(transport); + + auto result = handlers::detail::list_scoped_rosbag_descriptors(mgr, {"/x/detector"}); + ASSERT_FALSE(result.has_value()); + EXPECT_EQ(result.error().http_status, 503); + EXPECT_EQ(result.error().code, ERR_SERVICE_UNAVAILABLE); +} + +TEST(ListScopedRosbagDescriptorsTest, EmptyScopeListsNothingWithoutAskingTheFaultManager) { + // A function that hosts nothing: its listing is empty whether or not the fault manager answers. + auto transport = std::make_shared(); + transport->faults_ok = false; + FaultManager mgr(transport); + + auto result = handlers::detail::list_scoped_rosbag_descriptors(mgr, {}); + ASSERT_TRUE(result.has_value()) << result.error().message; + EXPECT_TRUE(result->empty()); + EXPECT_EQ(transport->list_faults_calls, 0); + EXPECT_TRUE(transport->list_rosbags_calls.empty()); +} + +TEST(ListScopedRosbagDescriptorsTest, OneFailedRosbagQueryIsAnErrorNotAPartialList) { + auto transport = std::make_shared(); + transport->faults = nlohmann::json::array({ + listing_record("EXACT", "/x/detector"), + listing_record("SUB", "/x/detector/diag"), + }); + transport->rosbags = {listing_row("EXACT", "/x/detector"), listing_row("SUB", "/x/detector/diag")}; + transport->failing_owner = "/x/detector/diag"; + FaultManager mgr(transport); + + auto result = handlers::detail::list_scoped_rosbag_descriptors(mgr, {"/x/detector"}); + ASSERT_FALSE(result.has_value()); + EXPECT_EQ(result.error().http_status, 503); + EXPECT_EQ(result.error().code, ERR_SERVICE_UNAVAILABLE); +} + +// ============================================================================= +// resolve_scoped_rosbag: which recording a download serves, and what a failing +// fault manager turns the download into. +// ============================================================================= + +namespace { + +/// GetRosbag's answer for recording `fault__1`, attached to `code`. +nlohmann::json served_recording(const std::string & code) { + return {{"fault_codes", {code}}, + {"recording_id", "fault_" + code + "_1"}, + {"file_path", "/tmp/fault_" + code + "_1"}, + {"format", "mcap"}}; +} + +/// A transport where `code` has one record per owner, and each owner holds the recording. +std::shared_ptr transport_holding(const std::string & code, const std::vector & owners) { + auto transport = std::make_shared(); + for (const auto & owner : owners) { + transport->faults.push_back(listing_record(code, owner)); + transport->rosbags.push_back(listing_row(code, owner)); + } + transport->served = served_recording(code); + return transport; +} + +} // namespace + +TEST(ResolveScopedRosbagTest, ServesARecordingASubPathOwnerHolds) { + auto transport = transport_holding("SUB", {"/x/detector/diag"}); + FaultManager mgr(transport); + + auto result = handlers::detail::resolve_scoped_rosbag(mgr, {"/x/detector"}, "fault_SUB_1", "detector"); + ASSERT_TRUE(result.has_value()) << result.error().message; + EXPECT_EQ(result->value("recording_id", ""), "fault_SUB_1"); +} + +TEST(ResolveScopedRosbagTest, ARecordingNoOwnerInScopeHoldsIsNotFound) { + auto transport = transport_holding("FOREIGN", {"/z/other"}); + FaultManager mgr(transport); + + auto result = handlers::detail::resolve_scoped_rosbag(mgr, {"/x/detector"}, "fault_FOREIGN_1", "detector"); + ASSERT_FALSE(result.has_value()); + EXPECT_EQ(result.error().http_status, 404); +} + +TEST(ResolveScopedRosbagTest, FaultListFailureIsUnavailableNotNotFound) { + auto transport = transport_holding("SUB", {"/x/detector/diag"}); + transport->faults_ok = false; + FaultManager mgr(transport); + + auto result = handlers::detail::resolve_scoped_rosbag(mgr, {"/x/detector"}, "fault_SUB_1", "detector"); + ASSERT_FALSE(result.has_value()); + EXPECT_EQ(result.error().http_status, 503); + EXPECT_EQ(result.error().code, ERR_SERVICE_UNAVAILABLE); +} + +TEST(ResolveScopedRosbagTest, OwnersRosbagListFailureIsUnavailableNotNotFound) { + auto transport = transport_holding("SUB", {"/x/detector/diag"}); + transport->failing_owner = "/x/detector/diag"; + FaultManager mgr(transport); + + auto result = handlers::detail::resolve_scoped_rosbag(mgr, {"/x/detector"}, "fault_SUB_1", "detector"); + ASSERT_FALSE(result.has_value()); + EXPECT_EQ(result.error().http_status, 503); + EXPECT_EQ(result.error().code, ERR_SERVICE_UNAVAILABLE); +} + +TEST(ResolveScopedRosbagTest, AnotherOwnerHoldingTheRecordingOutweighsAFailedList) { + // Two owners in scope hold the recording; one listing failing cannot change the answer. + auto transport = transport_holding("SUB", {"/x/detector/a", "/x/detector/b"}); + FaultManager mgr(transport); + + for (const auto & failing : {"/x/detector/a", "/x/detector/b"}) { + transport->failing_owner = failing; + auto result = handlers::detail::resolve_scoped_rosbag(mgr, {"/x/detector"}, "fault_SUB_1", "detector"); + EXPECT_TRUE(result.has_value()) << "failing " << failing << ": " << result.error().message; + } +} + +TEST(ResolveScopedRosbagTest, GetRosbagWithoutAnAnswerIsUnavailable) { + auto transport = transport_holding("SUB", {"/x/detector/diag"}); + transport->served = nullptr; + transport->rosbag_failure = FaultFailure::Unavailable; + FaultManager mgr(transport); + + auto result = handlers::detail::resolve_scoped_rosbag(mgr, {"/x/detector"}, "fault_SUB_1", "detector"); + ASSERT_FALSE(result.has_value()); + EXPECT_EQ(result.error().http_status, 503); + EXPECT_EQ(result.error().code, ERR_SERVICE_UNAVAILABLE); +} + +TEST(ResolveScopedRosbagTest, GetRosbagDeclinedIsNotFound) { + auto transport = transport_holding("SUB", {"/x/detector/diag"}); + transport->served = nullptr; + transport->rosbag_failure = FaultFailure::Declined; + FaultManager mgr(transport); + + auto result = handlers::detail::resolve_scoped_rosbag(mgr, {"/x/detector"}, "fault_SUB_1", "detector"); + ASSERT_FALSE(result.has_value()); + EXPECT_EQ(result.error().http_status, 404); +} + +TEST(ResolveScopedRosbagTest, AFailedListBesideAnOwnerWithoutTheRecordingIsUnavailable) { + // Both owners have a record of the code; /x/detector/b answers without the + // recording, so only the failed listing of /x/detector/a could hold it. + auto transport = transport_holding("SUB", {"/x/detector/a"}); + transport->faults.push_back(listing_record("SUB", "/x/detector/b")); + transport->failing_owner = "/x/detector/a"; + FaultManager mgr(transport); + + auto result = handlers::detail::resolve_scoped_rosbag(mgr, {"/x/detector"}, "fault_SUB_1", "detector"); + ASSERT_FALSE(result.has_value()); + EXPECT_EQ(result.error().http_status, 503); + EXPECT_EQ(transport->list_rosbags_calls, (std::map{{"/x/detector/a", 1}, {"/x/detector/b", 1}})); +} diff --git a/src/ros2_medkit_gateway/test/test_entity_freeze_frame_capture.cpp b/src/ros2_medkit_gateway/test/test_entity_freeze_frame_capture.cpp index e94e657de..101815046 100644 --- a/src/ros2_medkit_gateway/test/test_entity_freeze_frame_capture.cpp +++ b/src/ros2_medkit_gateway/test/test_entity_freeze_frame_capture.cpp @@ -17,11 +17,14 @@ #include #include #include +#include #include #include +#include #include #include #include +#include #include #include #include @@ -38,6 +41,7 @@ using ros2_medkit_gateway::DataProvider; using ros2_medkit_gateway::DataProviderError; using ros2_medkit_gateway::DataProviderErrorInfo; using ros2_medkit_gateway::EntityFreezeFrameCapture; +using ros2_medkit_gateway::FreezeFrameCapture; using ros2_medkit_gateway::handlers::FaultHandlers; using ros2_medkit_gateway::ros2_common::Ros2SubscriptionExecutor; using ros2_medkit_msgs::msg::Fault; @@ -114,6 +118,69 @@ class StaticContentDataProvider : public DataProvider { json content_; }; +/// The capture EntityFreezeFrameCapture builds, with a second subscriber on its +/// feed. The feed calls its subscribers in order on one thread and the capture +/// subscribes first, so the probe sees a confirm once the capture has queued it. +/// The standing-fault lister waits for that before it answers. +class ObservedCapture { + public: + ObservedCapture(rclcpp::Node * node, Ros2SubscriptionExecutor & exec, + EntityFreezeFrameCapture::RouteDataFetcher route_fetcher, std::string queued_code, + std::vector standing) + : events_(node, exec) + , capture_(std::make_unique( + events_, ros2_medkit_gateway::FreezeFrameCaptureConfig{true, 256}, + FreezeFrameCapture::Hooks{ros2_medkit_gateway::PluginDataReaders{[](const std::string &) -> DataProvider * { + return nullptr; + }, + std::move(route_fetcher)}, + [this, code = std::move(queued_code), + standing = std::move(standing)](const std::function & should_abort) { + wait_queued(code, should_abort); + return standing; + }, + [this](const std::function & should_abort) { + events_.wait_for_publisher(should_abort); + }})) + , probe_(events_.subscribe([this](const ros2_medkit::fault_detection::FaultEventRecord & event) { + if (event.event_type != ros2_medkit::fault_detection::fault_event_type::kConfirmed) { + return; + } + { + std::lock_guard lock(mutex_); + queued_.insert(event.fault.fault_code); + } + cv_.notify_all(); + })) { + } + + ~ObservedCapture() = default; + ObservedCapture(const ObservedCapture &) = delete; + ObservedCapture & operator=(const ObservedCapture &) = delete; + ObservedCapture(ObservedCapture &&) = delete; + ObservedCapture & operator=(ObservedCapture &&) = delete; + + std::vector frames_for(const std::string & fault_code, const std::string & owner) const { + return capture_->frames_for(fault_code, owner); + } + + private: + void wait_queued(const std::string & code, const std::function & should_abort) { + const auto deadline = std::chrono::steady_clock::now() + 5s; + std::unique_lock lock(mutex_); + while (queued_.count(code) == 0 && !should_abort() && std::chrono::steady_clock::now() < deadline) { + cv_.wait_for(lock, 10ms); + } + } + + ros2_medkit_gateway::ros2_capture_detail::SlotFaultEvents events_; + std::mutex mutex_; + std::condition_variable cv_; + std::set queued_; + std::unique_ptr capture_; + ros2_medkit_gateway::FaultEventSubscription probe_; +}; + FaultEvent make_confirmed_event(const std::string & fault_code, const std::vector & sources) { FaultEvent event; event.event_type = FaultEvent::EVENT_CONFIRMED; @@ -185,7 +252,7 @@ class EntityFreezeFrameCaptureTest : public ::testing::Test { const auto deadline = std::chrono::steady_clock::now() + 5s; while (std::chrono::steady_clock::now() < deadline) { publisher_->publish(event); - if (!capture.frames_for(event.fault.fault_code).empty()) { + if (!capture.frames_for(event.fault.fault_code, event.fault.reporting_sources.front()).empty()) { return true; } std::this_thread::sleep_for(20ms); @@ -212,7 +279,7 @@ TEST_F(EntityFreezeFrameCaptureTest, ConfirmedPluginFaultCapturesEntityValues) { ASSERT_TRUE(publish_and_wait(capture, make_confirmed_event("PLC_OVERPRESSURE", {"plc_app"}))); - auto frames = capture.frames_for("PLC_OVERPRESSURE"); + auto frames = capture.frames_for("PLC_OVERPRESSURE", "plc_app"); ASSERT_EQ(frames.size(), 1u); EXPECT_EQ(frames[0].entity_id, "plc_app"); EXPECT_DOUBLE_EQ(frames[0].values.value("temperature", 0.0), 42.5); @@ -234,7 +301,7 @@ TEST_F(EntityFreezeFrameCaptureTest, NonPluginSourceCapturesNothing) { publisher_->publish(event); std::this_thread::sleep_for(10ms); } - EXPECT_TRUE(capture.frames_for("ROS_FAULT").empty()); + EXPECT_TRUE(capture.frames_for("ROS_FAULT", "/powertrain/engine/temp_sensor").empty()); } /// @verifies REQ_INTEROP_088 @@ -250,7 +317,7 @@ TEST_F(EntityFreezeFrameCaptureTest, NonConfirmedEventsAreIgnored) { publisher_->publish(event); std::this_thread::sleep_for(10ms); } - EXPECT_TRUE(capture.frames_for("PLC_UPDATED_ONLY").empty()); + EXPECT_TRUE(capture.frames_for("PLC_UPDATED_ONLY", "plc_app").empty()); } /// @verifies REQ_INTEROP_088 @@ -272,10 +339,10 @@ TEST_F(EntityFreezeFrameCaptureTest, StandingFaultsAreFramedWithoutAnyEvent) { // No publish at all - the frame must appear from the catch-up alone. const auto deadline = std::chrono::steady_clock::now() + 15s; - while (capture.frames_for("PLC_STANDING").empty() && std::chrono::steady_clock::now() < deadline) { + while (capture.frames_for("PLC_STANDING", "route_plc_app").empty() && std::chrono::steady_clock::now() < deadline) { std::this_thread::sleep_for(20ms); } - const auto frames = capture.frames_for("PLC_STANDING"); + const auto frames = capture.frames_for("PLC_STANDING", "route_plc_app"); ASSERT_EQ(frames.size(), 1u); EXPECT_EQ(frames[0].entity_id, "route_plc_app"); EXPECT_DOUBLE_EQ(frames[0].values.value("level", 0.0), 7.0); @@ -283,66 +350,124 @@ TEST_F(EntityFreezeFrameCaptureTest, StandingFaultsAreFramedWithoutAnyEvent) { } /// @verifies REQ_INTEROP_088 -TEST_F(EntityFreezeFrameCaptureTest, QueuedLiveConfirmDedupesStandingCatchUp) { - // Both paths carry the same fault code and the confirm is already queued - // when the catch-up snapshots the queue: the catch-up must skip that code - // entirely (one plugin read per confirm, no double capture) and the drain - // loop's live frame is the one that lands. - std::atomic published{false}; - std::atomic standing_sampled{false}; - EntityFreezeFrameCapture capture( - node_.get(), *sub_exec_, - [](const std::string &) -> DataProvider * { - return nullptr; - }, - [&standing_sampled](const std::string & entity_id) -> std::optional { - double level = 0.0; - if (entity_id == "route_live_app") { - level = 99.0; - } else if (entity_id == "route_standing_app") { - level = 1.0; - standing_sampled.store(true); - } else { - return std::nullopt; - } - return json{{"connected", true}, {"items", json::array({{{"name", "level"}, {"value", level}}})}}; - }, - 256, - [&published](const std::function &) -> std::vector { - // Hold the catch-up until the live confirm has reached the queue so - // the standing entry and the queued event genuinely overlap. - const auto deadline = std::chrono::steady_clock::now() + 5s; - while (!published.load() && std::chrono::steady_clock::now() < deadline) { - std::this_thread::sleep_for(10ms); - } - std::this_thread::sleep_for(500ms); // let the event reach the queue - return {{"PLC_BOTH_PATHS", {"route_standing_app"}}}; - }); +TEST_F(EntityFreezeFrameCaptureTest, QueuedLiveConfirmDedupesTheSameStandingRecord) { + // The same record - one code, one source - arrives on both paths, and the + // confirm is already queued when the catch-up snapshots the queue: the + // catch-up must skip that record (one plugin read per confirm, no double + // capture) and the drain loop's live frame is the one that lands. + std::atomic route_reads{0}; + ObservedCapture capture(node_.get(), *sub_exec_, + [&route_reads](const std::string & entity_id) -> std::optional { + if (entity_id != "route_live_app") { + return std::nullopt; + } + route_reads.fetch_add(1); + return std::optional{json{ + {"connected", true}, {"items", json::array({{{"name", "level"}, {"value", 99.0}}})}}}; + }, + "PLC_BOTH_PATHS", {{"PLC_BOTH_PATHS", {"route_live_app"}}}); ASSERT_TRUE(wait_for_match()); - const auto live = make_confirmed_event("PLC_BOTH_PATHS", {"route_live_app"}); - publisher_->publish(live); - published.store(true); + publisher_->publish(make_confirmed_event("PLC_BOTH_PATHS", {"route_live_app"})); + // The catch-up runs before the drain loop, so once the live frame is there a + // second read by the catch-up would already have happened. const auto deadline = std::chrono::steady_clock::now() + 10s; std::vector frames; while (std::chrono::steady_clock::now() < deadline) { - frames = capture.frames_for("PLC_BOTH_PATHS"); - if (!frames.empty()) { + frames = capture.frames_for("PLC_BOTH_PATHS", "route_live_app"); + if (!frames.empty() && !frames[0].startup_catchup) { break; } std::this_thread::sleep_for(20ms); } - // The queued confirm dedupes the catch-up: the standing entity was never - // sampled and the only frame is the live one. - EXPECT_FALSE(standing_sampled.load()); + EXPECT_EQ(route_reads.load(), 1); ASSERT_EQ(frames.size(), 1u); EXPECT_EQ(frames[0].entity_id, "route_live_app"); EXPECT_DOUBLE_EQ(frames[0].values.value("level", 0.0), 99.0); EXPECT_FALSE(frames[0].startup_catchup); } +/// A queued confirm dedupes its own record, not the code. Another source +/// standing on the same code is a different record with its own frame, and +/// dropping it on the code alone left that record's detail page blank. +/// @verifies REQ_INTEROP_088 +TEST_F(EntityFreezeFrameCaptureTest, QueuedConfirmDoesNotDedupeAnotherOwnerOfTheSameCode) { + ObservedCapture capture(node_.get(), *sub_exec_, + [](const std::string & entity_id) -> std::optional { + double level = 0.0; + if (entity_id == "route_live_app") { + level = 99.0; + } else if (entity_id == "route_standing_app") { + level = 1.0; + } else { + return std::nullopt; + } + return std::optional{json{ + {"connected", true}, {"items", json::array({{{"name", "level"}, {"value", level}}})}}}; + }, + "PLC_TWO_OWNERS", {{"PLC_TWO_OWNERS", {"route_standing_app"}}}); + + ASSERT_TRUE(wait_for_match()); + publisher_->publish(make_confirmed_event("PLC_TWO_OWNERS", {"route_live_app"})); + + const auto deadline = std::chrono::steady_clock::now() + 10s; + std::vector live_frames; + std::vector standing_frames; + while (std::chrono::steady_clock::now() < deadline) { + live_frames = capture.frames_for("PLC_TWO_OWNERS", "route_live_app"); + standing_frames = capture.frames_for("PLC_TWO_OWNERS", "route_standing_app"); + if (!live_frames.empty() && !standing_frames.empty()) { + break; + } + std::this_thread::sleep_for(20ms); + } + + ASSERT_EQ(live_frames.size(), 1u) << "the live record lost its frame"; + EXPECT_DOUBLE_EQ(live_frames[0].values.value("level", 0.0), 99.0); + ASSERT_EQ(standing_frames.size(), 1u) << "the standing record of the same code was dropped as a duplicate"; + EXPECT_DOUBLE_EQ(standing_frames[0].values.value("level", 0.0), 1.0); + EXPECT_TRUE(standing_frames[0].startup_catchup); +} + +/// Two sources confirming one code are two records, each keeping its own +/// values. Under a code-only key the second confirmation overwrote the first +/// source's frames and served them on the first source's detail page. +/// @verifies REQ_INTEROP_088 +TEST_F(EntityFreezeFrameCaptureTest, TwoOwnersOfOneCodeKeepSeparateFrames) { + EntityFreezeFrameCapture capture( + node_.get(), *sub_exec_, + [](const std::string &) -> DataProvider * { + return nullptr; + }, + [](const std::string & entity_id) -> std::optional { + double level = 0.0; + if (entity_id == "line_a") { + level = 11.0; + } else if (entity_id == "line_b") { + level = 22.0; + } else { + return std::nullopt; + } + return std::optional{ + json{{"connected", true}, {"items", json::array({{{"name", "level"}, {"value", level}}})}}}; + }); + + ASSERT_TRUE(publish_and_wait(capture, make_confirmed_event("SHARED_CODE", {"line_a"}))); + ASSERT_TRUE(publish_and_wait(capture, make_confirmed_event("SHARED_CODE", {"line_b"}))); + + const auto a = capture.frames_for("SHARED_CODE", "line_a"); + const auto b = capture.frames_for("SHARED_CODE", "line_b"); + + ASSERT_EQ(a.size(), 1u) << "line_a's record lost its frame to line_b's confirmation"; + EXPECT_EQ(a[0].entity_id, "line_a"); + EXPECT_DOUBLE_EQ(a[0].values.value("level", 0.0), 11.0); + ASSERT_EQ(b.size(), 1u); + EXPECT_EQ(b[0].entity_id, "line_b"); + EXPECT_DOUBLE_EQ(b[0].values.value("level", 0.0), 22.0); +} + /// @verifies REQ_INTEROP_088 TEST_F(EntityFreezeFrameCaptureTest, ThrowingListerDoesNotKillCaptureThread) { // The lister runs on the capture thread's entry path: an escaping exception @@ -359,7 +484,7 @@ TEST_F(EntityFreezeFrameCaptureTest, ThrowingListerDoesNotKillCaptureThread) { }); ASSERT_TRUE(publish_and_wait(capture, make_confirmed_event("PLC_AFTER_THROW", {"plc_app"}))); - EXPECT_FALSE(capture.frames_for("PLC_AFTER_THROW").empty()); + EXPECT_FALSE(capture.frames_for("PLC_AFTER_THROW", "plc_app").empty()); } /// @verifies REQ_INTEROP_088 @@ -456,14 +581,14 @@ TEST_F(EntityFreezeFrameCaptureTest, CatchUpCapsStoredFramesAtMaxFaults) { }); const auto deadline = std::chrono::steady_clock::now() + 15s; - while (capture.frames_for("PLC_CAP_B").empty() && std::chrono::steady_clock::now() < deadline) { + while (capture.frames_for("PLC_CAP_B", "route_b").empty() && std::chrono::steady_clock::now() < deadline) { std::this_thread::sleep_for(20ms); } std::this_thread::sleep_for(300ms); // would be enough for an uncapped C capture - EXPECT_TRUE(capture.frames_for("PLC_CAP_NO_DATA").empty()); - EXPECT_FALSE(capture.frames_for("PLC_CAP_A").empty()); // not evicted by C - EXPECT_FALSE(capture.frames_for("PLC_CAP_B").empty()); - EXPECT_TRUE(capture.frames_for("PLC_CAP_C").empty()); // past the cap + EXPECT_TRUE(capture.frames_for("PLC_CAP_NO_DATA", "route_no_data_app").empty()); + EXPECT_FALSE(capture.frames_for("PLC_CAP_A", "route_a").empty()); // not evicted by C + EXPECT_FALSE(capture.frames_for("PLC_CAP_B", "route_b").empty()); + EXPECT_TRUE(capture.frames_for("PLC_CAP_C", "route_c").empty()); // past the cap } /// @verifies REQ_INTEROP_088 @@ -473,7 +598,7 @@ TEST_F(EntityFreezeFrameCaptureTest, FramesRetainedAcrossClearAndOverwrittenOnRe }); ASSERT_TRUE(publish_and_wait(capture, make_confirmed_event("PLC_CYCLING", {"plc_app"}))); - ASSERT_DOUBLE_EQ(capture.frames_for("PLC_CYCLING")[0].values.value("temperature", 0.0), 42.5); + ASSERT_DOUBLE_EQ(capture.frames_for("PLC_CYCLING", "plc_app")[0].values.value("temperature", 0.0), 42.5); // Clearing retains the confirmed-state record, mirroring the fault_manager's // freeze-frame retention across clear_fault. @@ -483,7 +608,7 @@ TEST_F(EntityFreezeFrameCaptureTest, FramesRetainedAcrossClearAndOverwrittenOnRe publisher_->publish(cleared); std::this_thread::sleep_for(10ms); } - auto retained = capture.frames_for("PLC_CYCLING"); + auto retained = capture.frames_for("PLC_CYCLING", "plc_app"); ASSERT_EQ(retained.size(), 1u); EXPECT_DOUBLE_EQ(retained[0].values.value("temperature", 0.0), 42.5); @@ -496,7 +621,7 @@ TEST_F(EntityFreezeFrameCaptureTest, FramesRetainedAcrossClearAndOverwrittenOnRe while (std::chrono::steady_clock::now() < deadline && !overwritten) { publisher_->publish(reconfirm); std::this_thread::sleep_for(20ms); - auto latest = capture.frames_for("PLC_CYCLING"); + auto latest = capture.frames_for("PLC_CYCLING", "plc_app"); overwritten = !latest.empty() && std::abs(latest[0].values.value("temperature", 0.0) - 99.0) < 1e-9; } EXPECT_TRUE(overwritten); @@ -522,7 +647,7 @@ TEST_F(EntityFreezeFrameCaptureTest, RouteFallbackCapturesWhenPluginHasNoDataPro ASSERT_TRUE(publish_and_wait(capture, make_confirmed_event("PLC_ROUTE_LEVEL_HIGH", {"route_plc_app"}))); - auto frames = capture.frames_for("PLC_ROUTE_LEVEL_HIGH"); + auto frames = capture.frames_for("PLC_ROUTE_LEVEL_HIGH", "route_plc_app"); ASSERT_EQ(frames.size(), 1u); EXPECT_EQ(frames[0].entity_id, "route_plc_app"); EXPECT_DOUBLE_EQ(frames[0].values.value("level", 0.0), 87.5); @@ -550,7 +675,7 @@ TEST_F(EntityFreezeFrameCaptureTest, RouteFallbackSkipsDisconnectedPlcWithNoValu publisher_->publish(event); std::this_thread::sleep_for(10ms); } - EXPECT_TRUE(capture.frames_for("PLC_DISCONNECTED").empty()); + EXPECT_TRUE(capture.frames_for("PLC_DISCONNECTED", "route_plc_app").empty()); } /// @verifies REQ_INTEROP_088 @@ -571,7 +696,7 @@ TEST_F(EntityFreezeFrameCaptureTest, DisconnectedEntityWithLastKnownValuesIsCapt ASSERT_TRUE(publish_and_wait(capture, make_confirmed_event("PLC_COMMS_LOST", {"route_plc_app"}))); - const auto frames = capture.frames_for("PLC_COMMS_LOST"); + const auto frames = capture.frames_for("PLC_COMMS_LOST", "route_plc_app"); ASSERT_EQ(frames.size(), 1u); EXPECT_EQ(frames[0].values["level"], 42.0); ASSERT_TRUE(frames[0].connected.has_value()); @@ -603,7 +728,7 @@ TEST_F(EntityFreezeFrameCaptureTest, DataProviderWinsOverRouteFallback) { ASSERT_TRUE(publish_and_wait(capture, make_confirmed_event("PLC_PROVIDER_FIRST", {"plc_app"}))); - auto frames = capture.frames_for("PLC_PROVIDER_FIRST"); + auto frames = capture.frames_for("PLC_PROVIDER_FIRST", "plc_app"); ASSERT_EQ(frames.size(), 1u); EXPECT_DOUBLE_EQ(frames[0].values.value("temperature", 0.0), 42.5); // provider values, not route EXPECT_FALSE(fetcher_called.load()); @@ -633,7 +758,7 @@ TEST_F(EntityFreezeFrameCaptureTest, DataProviderWithoutLiveValuesCapturesNothin publisher_->publish(event); std::this_thread::sleep_for(10ms); } - EXPECT_TRUE(capture.frames_for("PLC_NO_LIVE_DATA").empty()); + EXPECT_TRUE(capture.frames_for("PLC_NO_LIVE_DATA", "cold_app").empty()); } /// @verifies REQ_INTEROP_088 @@ -649,7 +774,7 @@ TEST_F(EntityFreezeFrameCaptureTest, DisconnectedDataProviderWithLastKnownValues ASSERT_TRUE(publish_and_wait(capture, make_confirmed_event("PLC_PROVIDER_COMMS_LOST", {"down_app"}))); - const auto frames = capture.frames_for("PLC_PROVIDER_COMMS_LOST"); + const auto frames = capture.frames_for("PLC_PROVIDER_COMMS_LOST", "down_app"); ASSERT_EQ(frames.size(), 1u); EXPECT_EQ(frames[0].values["level"], 5.5); ASSERT_TRUE(frames[0].connected.has_value()); @@ -675,9 +800,9 @@ TEST_F(EntityFreezeFrameCaptureTest, OldestFaultEvictedPastMaxFaults) { ASSERT_TRUE(publish_and_wait(capture, make_confirmed_event("PLC_EVICT_B", {"plc_app"}))); ASSERT_TRUE(publish_and_wait(capture, make_confirmed_event("PLC_EVICT_C", {"plc_app"}))); - EXPECT_TRUE(capture.frames_for("PLC_EVICT_A").empty()); // FIFO-evicted - EXPECT_FALSE(capture.frames_for("PLC_EVICT_B").empty()); - EXPECT_FALSE(capture.frames_for("PLC_EVICT_C").empty()); + EXPECT_TRUE(capture.frames_for("PLC_EVICT_A", "plc_app").empty()); // FIFO-evicted + EXPECT_FALSE(capture.frames_for("PLC_EVICT_B", "plc_app").empty()); + EXPECT_FALSE(capture.frames_for("PLC_EVICT_C", "plc_app").empty()); } TEST(ContentHasLiveData, GatesOnItemsNotOnTheLinkFlag) { diff --git a/src/ros2_medkit_gateway/test/test_fault_handlers.cpp b/src/ros2_medkit_gateway/test/test_fault_handlers.cpp index 29b53ef22..8ffd3e5fc 100644 --- a/src/ros2_medkit_gateway/test/test_fault_handlers.cpp +++ b/src/ros2_medkit_gateway/test/test_fault_handlers.cpp @@ -16,6 +16,7 @@ #include +#include "ros2_medkit_gateway/core/http/error_codes.hpp" #include "ros2_medkit_gateway/dto/faults.hpp" #include "ros2_medkit_gateway/dto/json_reader.hpp" #include "ros2_medkit_gateway/dto/json_writer.hpp" @@ -41,8 +42,9 @@ namespace dto = ros2_medkit_gateway::dto; class FaultHandlersTest : public ::testing::Test { protected: + /// The item the detail route builds a single record's response from. static json fault_json(const ros2_medkit_msgs::msg::Fault & f) { - return conversions::fault_to_json(f); + return ros2_medkit_gateway::faults::record_item(conversions::fault_to_json(f), nullptr); } static json env_json(const ros2_medkit_msgs::msg::EnvironmentData & e) { return conversions::environment_data_to_json(e); @@ -429,11 +431,17 @@ TEST_F(FaultHandlersTest, BuildSovdFaultResponsePrimaryValueExtraction) { } } +// A record is (fault_code, owner), so the detail names exactly one reporting +// source and repeats it as x-medkit.owner, which is the value the per-record +// routes address the record by. This test used to pin one record +// carrying three sources. Three sources reporting one code are three records +// now, and each detail response describes one of them. // @verifies REQ_INTEROP_013 -TEST_F(FaultHandlersTest, BuildSovdFaultResponseMultipleSources) { +TEST_F(FaultHandlersTest, BuildSovdFaultResponseNamesTheOwningSource) { ros2_medkit_msgs::msg::Fault fault; - fault.fault_code = "MULTI_SOURCE_FAULT"; - fault.reporting_sources = {"/perception/lidar", "/perception/camera", "/control/motor"}; + fault.fault_code = "SHARED_CODE"; + fault.reporting_sources = {"/perception/lidar"}; + fault.owner = "/perception/lidar"; ros2_medkit_msgs::msg::EnvironmentData env_data; @@ -441,10 +449,45 @@ TEST_F(FaultHandlersTest, BuildSovdFaultResponseMultipleSources) { to_json(FaultHandlers::build_sovd_fault_response(fault_json(fault), env_json(env_data), "/apps/test")); auto sources = response["x-medkit"]["reporting_sources"]; - ASSERT_EQ(sources.size(), 3); + ASSERT_EQ(sources.size(), 1); EXPECT_EQ(sources[0], "/perception/lidar"); - EXPECT_EQ(sources[1], "/perception/camera"); - EXPECT_EQ(sources[2], "/control/motor"); + ASSERT_TRUE(response["x-medkit"].contains("owner")); + EXPECT_EQ(response["x-medkit"]["owner"], "/perception/lidar"); + // Named `owner`, never `source_id`: a fault LIST's x-medkit.source_id is the + // addressed entity's namespace path, so one key with two meanings inside one + // API is the thing this avoids. + EXPECT_FALSE(response["x-medkit"].contains("source_id")) + << "the detail must not reuse the list's source_id key for the record owner"; + + // A shared record names no owner, as before records had owners. + fault.owner = "*"; + auto shared = to_json(FaultHandlers::build_sovd_fault_response(fault_json(fault), env_json(env_data), "/apps/test")); + EXPECT_FALSE(shared["x-medkit"].contains("owner")); + EXPECT_FALSE(shared["x-medkit"].contains("records")); + EXPECT_EQ(shared["x-medkit"]["reporting_sources"], json::array({"/perception/lidar"})); +} + +// The other owner of the same code is a different record, and its detail says +// so: same code, different source_id. +// @verifies REQ_INTEROP_013 +TEST_F(FaultHandlersTest, BuildSovdFaultResponseSeparatesOwnersOfOneCode) { + ros2_medkit_msgs::msg::Fault first; + first.fault_code = "SHARED_CODE"; + first.reporting_sources = {"/perception/lidar"}; + first.owner = "/perception/lidar"; + ros2_medkit_msgs::msg::Fault second; + second.fault_code = "SHARED_CODE"; + second.reporting_sources = {"/control/motor"}; + second.owner = "/control/motor"; + + ros2_medkit_msgs::msg::EnvironmentData env_data; + + auto a = to_json(FaultHandlers::build_sovd_fault_response(fault_json(first), env_json(env_data), "/apps/lidar")); + auto b = to_json(FaultHandlers::build_sovd_fault_response(fault_json(second), env_json(env_data), "/apps/motor")); + + EXPECT_EQ(a["item"]["code"], b["item"]["code"]); + EXPECT_EQ(a["x-medkit"]["owner"], "/perception/lidar"); + EXPECT_EQ(b["x-medkit"]["owner"], "/control/motor"); } // @verifies REQ_INTEROP_013 @@ -603,9 +646,10 @@ TEST(FaultInSourceScopeTest, BareEntityIdScopeRejectsOtherEntityFault) { } TEST(FaultInSourceScopeTest, BareEntityIdScopeRejectsMixedSourceFault) { - // Cross-entity clear/disclosure stays blocked under the all-sources rule: a - // fault co-reported by "process" and another entity is out of scope for - // "process" alone. + // Cross-entity clear/disclosure stays blocked under the all-sources rule. A + // record names one source, but one relayed from a peer or read from an older + // store may carry two, and one that names "process" and another entity is + // out of scope for "process" alone. EXPECT_FALSE(FaultHandlers::fault_in_source_scope(make_fault({"process", "s7_status"}), {"process"})); } @@ -620,6 +664,174 @@ TEST(FaultInSourceScopeTest, BareEntityIdScopeRejectsPrefixCollision) { // This pins the shape build_sovd_fault_response produces for the PLC case: // reporting_sources=[bare id] + a freeze-frame snapshot under a plugin entity // path. +// ============================================================================= +// records_of_code_in_scope / addressable_records - which record a recording +// URL carrying a fault code resolves to +// ============================================================================= +namespace { + +json record(const std::string & code, const std::string & owner) { + return json{{"fault_code", code}, {"status", "CONFIRMED"}, {"reporting_sources", json::array({owner})}}; +} + +std::vector in_scope(const json & faults, const std::string & code, + const std::set & scope) { + return ros2_medkit_gateway::faults::records_of_code_in_scope(faults, code, scope); +} + +} // namespace + +// The records of ONE code only: a second code the entity also owns is a +// different resource and must not make its sibling ambiguous. +TEST(RecordsOfCodeInScopeTest, OtherCodesAreNotCollected) { + const json faults = json::array({record("SHARED_CODE", "app_a"), record("OTHER_CODE", "app_b")}); + + const auto records = in_scope(faults, "SHARED_CODE", {"app_a", "app_b"}); + + ASSERT_EQ(records.size(), 1u); + EXPECT_EQ(records[0].owner, "app_a"); +} + +// The rosbag download reads the same records to find the owners it asks +// whether they hold the recording, and a code addressing several of them is not +// an ambiguity there: every owner in scope is asked. Reading the code unscoped +// instead answered "ambiguous" and the entity was refused a recording it owns. +TEST(RecordsOfCodeInScopeTest, EveryOwnerInScopeIsARecordingDownloadCandidate) { + const json faults = json::array({record("SHARED_CODE", "app_a"), record("SHARED_CODE", "app_b")}); + + EXPECT_FALSE(in_scope(faults, "SHARED_CODE", {"app_a"}).empty()) << "app_a owns a record of this code"; + EXPECT_EQ(in_scope(faults, "SHARED_CODE", {"app_a", "app_b"}).size(), 2u) + << "a component hosting both owners asks both of them"; + EXPECT_TRUE(in_scope(faults, "SHARED_CODE", {"unrelated_app"}).empty()) + << "an entity owning no record of this code has no owner to ask"; +} + +// The rosbag download resolves the SAME way the fault routes do when the URL +// carries a bare fault code: exactly one record in scope names a recording, and +// several name none of them. Serving the lowest-sorting owner's bag would hand a +// caller another owner's bytes under a URL that never said whose they were. The +// recording-id form addresses the bag directly and is unaffected. +TEST(RecordsOfCodeInScopeTest, SeveralOwnersMakeABareCodeUrlAmbiguous) { + const json faults = json::array({record("SHARED_CODE", "app_b"), record("SHARED_CODE", "app_a")}); + + const auto both = in_scope(faults, "SHARED_CODE", {"app_a", "app_b"}); + ASSERT_EQ(both.size(), 2u) << "a component hosting both owners resolves two records"; + EXPECT_EQ(both[0].owner, "app_a"); + EXPECT_EQ(both[1].owner, "app_b"); + + // One owner in scope is still one record, so that URL keeps working. + const auto single = in_scope(faults, "SHARED_CODE", {"app_a"}); + ASSERT_EQ(single.size(), 1u); + EXPECT_EQ(single[0].owner, "app_a"); +} + +// A muted_faults entry names one record by its code AND its owner. An entry for +// another owner of the same code, or for the same owner under another code, +// hides nothing here, so the one record the entity's list shows is the only +// candidate and the code is not ambiguous. +TEST(AddressableRecordsTest, AMutedEntryHidesOnlyTheRecordItNames) { + const json listing{ + {"faults", json::array({record("SHARED_CODE", "tank"), record("SHARED_CODE", "pump")})}, + {"muted_faults", json::array({json{{"fault_code", "SHARED_CODE"}, {"source_id", "pump"}}, + json{{"fault_code", "OTHER_CODE"}, {"source_id", "tank"}}})}, + }; + + const auto records = ros2_medkit_gateway::faults::addressable_records(listing, "SHARED_CODE", {"tank", "pump"}); + + ASSERT_EQ(records.size(), 1u) << "only pump's record of this code is muted"; + EXPECT_EQ(records[0].owner, "tank"); +} + +// With no shown record of the code the muted ones are the candidates, every one +// of them, so two of them still make the code ambiguous rather than picking one. +TEST(AddressableRecordsTest, MutedRecordsAnswerOnlyWhenNoneIsShown) { + const json listing{ + {"faults", json::array({record("SHARED_CODE", "tank"), record("SHARED_CODE", "pump")})}, + {"muted_faults", json::array({json{{"fault_code", "SHARED_CODE"}, {"source_id", "pump"}}, + json{{"fault_code", "SHARED_CODE"}, {"source_id", "tank"}}})}, + }; + + const auto records = ros2_medkit_gateway::faults::addressable_records(listing, "SHARED_CODE", {"tank", "pump"}); + + ASSERT_EQ(records.size(), 2u); + EXPECT_EQ(records[0].owner, "pump"); + EXPECT_EQ(records[1].owner, "tank"); +} + +namespace { + +json record_in(const std::string & status, const std::string & code, const std::string & owner) { + json r = record(code, owner); + r["status"] = status; + return r; +} + +} // namespace + +// The entity's default fault list shows PREFAILED and CONFIRMED records. A +// record the list only shows under status=cleared or status=healed (CLEARED, +// HEALED, PREPASSED) yields to one it shows by default, so once one of two +// sources has cleared its record the code names the other one again instead of +// answering 409 for good. +TEST(AddressableRecordsTest, ARecordTheListShowsOutranksOneItShowsOnlyOnRequest) { + for (const std::string inactive : {"CLEARED", "HEALED", "PREPASSED"}) { + for (const std::string active : {"CONFIRMED", "PREFAILED"}) { + const json listing{ + {"faults", + json::array({record_in(inactive, "SHARED_CODE", "tank"), record_in(active, "SHARED_CODE", "pump")})}, + }; + + const auto records = ros2_medkit_gateway::faults::addressable_records(listing, "SHARED_CODE", {"tank", "pump"}); + + ASSERT_EQ(records.size(), 1u) << inactive << " beside " << active; + EXPECT_EQ(records[0].owner, "pump") << inactive << " beside " << active; + } + } +} + +// A muted record is hidden from the list only because a root cause muted it. +// It still outranks a record the list hides for its status. +TEST(AddressableRecordsTest, AMutedActiveRecordOutranksAClearedOne) { + const json listing{ + {"faults", + json::array({record_in("CONFIRMED", "SHARED_CODE", "tank"), record_in("CLEARED", "SHARED_CODE", "pump")})}, + {"muted_faults", json::array({json{{"fault_code", "SHARED_CODE"}, {"source_id", "tank"}}})}, + }; + + const auto records = ros2_medkit_gateway::faults::addressable_records(listing, "SHARED_CODE", {"tank", "pump"}); + + ASSERT_EQ(records.size(), 1u); + EXPECT_EQ(records[0].owner, "tank"); +} + +// Nothing the list shows and nothing muted: the cleared and healed records are +// the candidates, all of them. One alone is addressable, two make the code +// ambiguous. A muted record the list would not show anyway (here HEALED) is in +// that last tier too, because its status hides it from the list whether it is +// muted or not. +TEST(AddressableRecordsTest, ClearedAndHealedRecordsAnswerOnlyWhenNothingElseDoes) { + const json one{{"faults", json::array({record_in("CLEARED", "SHARED_CODE", "tank")})}}; + const auto single = ros2_medkit_gateway::faults::addressable_records(one, "SHARED_CODE", {"tank", "pump"}); + ASSERT_EQ(single.size(), 1u) << "one cleared record alone is still the record the code names"; + EXPECT_EQ(single[0].owner, "tank"); + + const json two{ + {"faults", + json::array({record_in("CLEARED", "SHARED_CODE", "tank"), record_in("HEALED", "SHARED_CODE", "pump")})}, + {"muted_faults", json::array({json{{"fault_code", "SHARED_CODE"}, {"source_id", "pump"}}})}, + }; + const auto both = ros2_medkit_gateway::faults::addressable_records(two, "SHARED_CODE", {"tank", "pump"}); + ASSERT_EQ(both.size(), 2u) << "two inactive records name neither"; + EXPECT_EQ(both[0].owner, "pump"); + EXPECT_EQ(both[1].owner, "tank"); +} + +TEST(RecordsOfCodeInScopeTest, RecordOwnerIsTheSingleReportingSource) { + EXPECT_EQ(ros2_medkit_gateway::faults::record_owner(record("C", "app_a")), "app_a"); + EXPECT_EQ(ros2_medkit_gateway::faults::record_owner(json{{"fault_code", "C"}}), ""); + EXPECT_EQ(ros2_medkit_gateway::faults::record_owner(json{{"reporting_sources", json::array()}}), ""); +} + TEST_F(FaultHandlersTest, BuildSovdFaultResponseExternalEntityFreezeFrame) { ros2_medkit_msgs::msg::Fault fault; fault.fault_code = "PLC_LEVEL_HIGH"; @@ -648,14 +860,13 @@ TEST_F(FaultHandlersTest, BuildSovdFaultResponseExternalEntityFreezeFrame) { } // ============================================================================= -// FaultListItem schema <-> fault_to_json producer contract. +// FaultListItem schema <-> producer contract. // -// The fault list endpoints emit items verbatim from fault_to_json inside the -// opaque FaultListResult envelope (never through JsonWriter), so -// the published FaultListItem schema is otherwise an unverified claim about that -// wire. These tests pin the schema to its real in-tree producer: any field key, -// type, or enum drift in fault_to_json now fails here instead of silently -// diverging the spec from the wire (the drift the DTO contract exists to kill). +// The fault list endpoints emit faults::record_item(fault_to_json(...)) items +// verbatim inside the opaque FaultListResult envelope (never through +// JsonWriter), so the published FaultListItem schema is otherwise +// an unverified claim about that wire. These tests pin the schema to that +// producer: any field key, type, or enum drift now fails here. // ============================================================================= TEST(FaultListItemSchema, FaultToJsonConformsAndRoundTrips) { ros2_medkit_msgs::msg::Fault fault; @@ -664,16 +875,52 @@ TEST(FaultListItemSchema, FaultToJsonConformsAndRoundTrips) { fault.description = "Brake pressure below threshold"; fault.occurrence_count = 3; fault.status = "active"; - fault.reporting_sources = {"brake_ecu", "abs_node"}; + fault.reporting_sources = {"brake_ecu"}; fault.last_passed.sec = 1200; // absent-when-zero covered separately below - const json wire = conversions::fault_to_json(fault); + // A per_source record, then a shared one. + for (const std::string owner : {"brake_ecu", "*"}) { + fault.owner = owner; + const json wire = ros2_medkit_gateway::faults::record_item(conversions::fault_to_json(fault), nullptr); - // The published FaultListItem schema must accept the verbatim wire ... - const auto parsed = dto::JsonReader::read(wire); - ASSERT_TRUE(parsed.has_value()) << "fault_to_json output does not conform to FaultListItem"; - // ... and round-trip back to identical wire (no field added or dropped). - EXPECT_EQ(dto::JsonWriter::write(parsed.value()), wire); + // The published FaultListItem schema must accept the verbatim wire ... + const auto parsed = dto::JsonReader::read(wire); + ASSERT_TRUE(parsed.has_value()) << owner << ": the item does not conform to FaultListItem"; + // ... and round-trip back to identical wire (no field added or dropped). + EXPECT_EQ(dto::JsonWriter::write(parsed.value()), wire) << owner; + } +} + +TEST(FaultListItemSchema, FlatItemNamesTheOwningSource) { + // A per_source record's item names its source inside x-medkit only: SOVD + // keeps vendor fields under x-medkit, so the item has no top-level source_id. + ros2_medkit_msgs::msg::Fault fault; + fault.fault_code = "SHARED_CODE"; + fault.status = "CONFIRMED"; + fault.reporting_sources = {"app_a"}; + fault.owner = "app_a"; + + const json wire = ros2_medkit_gateway::faults::record_item(conversions::fault_to_json(fault), nullptr); + + EXPECT_EQ(wire["x-medkit"]["records"], json::array({json{{"source_id", "app_a"}, {"status", "CONFIRMED"}}})); + EXPECT_FALSE(wire.contains("source_id")); + EXPECT_FALSE(wire.contains("owner")) << "the internal owner key must not reach the wire"; +} + +TEST(FaultListItemSchema, FlatItemOmitsSourceIdWithoutAnOwner) { + // A shared record names no owner: no source_id, no x-medkit, as before records + // had owners. + ros2_medkit_msgs::msg::Fault fault; + fault.fault_code = "SHARED"; + fault.status = "CONFIRMED"; + fault.reporting_sources = {"app_a", "app_b"}; + fault.owner = "*"; + + const json wire = ros2_medkit_gateway::faults::record_item(conversions::fault_to_json(fault), nullptr); + + EXPECT_FALSE(wire.contains("source_id")); + EXPECT_FALSE(wire.contains("owner")); + EXPECT_FALSE(wire.contains("x-medkit")); } TEST(FaultListItemSchema, LastPassedOmittedWhenNeverPassed) { diff --git a/src/ros2_medkit_gateway/test/test_fault_handlers_plugin_clear.cpp b/src/ros2_medkit_gateway/test/test_fault_handlers_plugin_clear.cpp new file mode 100644 index 000000000..3c81ea270 --- /dev/null +++ b/src/ros2_medkit_gateway/test/test_fault_handlers_plugin_clear.cpp @@ -0,0 +1,1188 @@ +// Copyright 2026 bburda +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +// The per-record routes, driven through the handlers with a real GatewayNode: +// the fault detail and clear (native and plugin provider) and the recording +// download by fault code. +// +// The other fault-handler suites test pure helpers. These need the handler +// itself, because what they pin is which record the handler resolves and which +// owner it hands on: a record is (fault_code, owner), the owner is what the +// gateway resolved in the entity's fault scope, and for a COMPONENT that owner +// is the hosted APP, not the component in the URL. Handing the entity id down +// instead addresses a record nobody owns, the fault manager declines it, and +// the route still answers 2xx with the record untouched. +// +// The fault manager is a stub service on a second node, the same shape +// test_fault_manager.cpp uses. It answers ListFaults the way the real one +// does: a muted record is left out unless the listing asks for muted records, +// and then it is also named in `muted_faults`. The plugin is a mock +// FaultProvider added to the node's own PluginManager. + +#include + +#include +#include +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "ros2_medkit_gateway/core/discovery/models/app.hpp" +#include "ros2_medkit_gateway/core/discovery/models/component.hpp" +#include "ros2_medkit_gateway/core/http/error_codes.hpp" +#include "ros2_medkit_gateway/core/http/handlers/bulkdata_handlers.hpp" +#include "ros2_medkit_gateway/core/models/thread_safe_entity_cache.hpp" +#include "ros2_medkit_gateway/core/plugins/plugin_manager.hpp" +#include "ros2_medkit_gateway/core/providers/fault_provider.hpp" +#include "ros2_medkit_gateway/gateway_node.hpp" +#include "ros2_medkit_gateway/http/handlers/fault_handlers.hpp" +#include "ros2_medkit_gateway/http/handlers/handler_context.hpp" +#include "ros2_medkit_msgs/msg/muted_fault_info.hpp" +#include "ros2_medkit_msgs/srv/clear_fault.hpp" +#include "ros2_medkit_msgs/srv/get_fault.hpp" +#include "ros2_medkit_msgs/srv/get_rosbag.hpp" +#include "ros2_medkit_msgs/srv/list_faults.hpp" +#include "ros2_medkit_msgs/srv/list_rosbags.hpp" + +using json = nlohmann::json; +using namespace std::chrono_literals; +using ros2_medkit_gateway::App; +using ros2_medkit_gateway::AuthConfig; +using ros2_medkit_gateway::Component; +using ros2_medkit_gateway::CorsConfig; +using ros2_medkit_gateway::FaultProvider; +using ros2_medkit_gateway::FaultProviderErrorInfo; +using ros2_medkit_gateway::GatewayNode; +using ros2_medkit_gateway::GatewayPlugin; +using ros2_medkit_gateway::PluginManager; +using ros2_medkit_gateway::ThreadSafeEntityCache; +using ros2_medkit_gateway::TlsConfig; +using ros2_medkit_gateway::dto::FaultClearResult; +using ros2_medkit_gateway::dto::FaultDetailResult; +using ros2_medkit_gateway::dto::FaultListResult; +using ros2_medkit_gateway::handlers::BulkDataHandlers; +using ros2_medkit_gateway::handlers::FaultHandlers; +using ros2_medkit_gateway::handlers::HandlerContext; +using ros2_medkit_msgs::srv::GetFault; +using ros2_medkit_msgs::srv::GetRosbag; +using ros2_medkit_msgs::srv::ListFaults; +using ros2_medkit_msgs::srv::ListRosbags; + +namespace { + +constexpr const char * kCode = "SHARED_CODE"; +constexpr const char * kComponent = "device_hub"; +constexpr const char * kHostedApp = "tank"; +constexpr const char * kOtherApp = "pump"; + +/// Records the (entity_id, fault_code, owner) triple of every clear it is asked +/// for, which is the whole point of the fixture. +class RecordingFaultPlugin : public GatewayPlugin, public FaultProvider { + public: + struct ClearCall { + std::string entity_id; + std::string fault_code; + std::string owner; + }; + + std::string name() const override { + return "recording_fault_plugin"; + } + void configure(const json & /*config*/) override { + } + void shutdown() override { + } + + tl::expected list_faults(const std::string & /*entity_id*/) override { + return FaultListResult{json{{"items", listed_items_}}}; + } + tl::expected get_fault(const std::string & /*entity_id*/, + const std::string & code) override { + return FaultDetailResult{json{{"code", code}}}; + } + // The gateway clears through clear_fault_record, so a call landing here means + // it went back to the code-only clear and dropped the owner it resolved. + tl::expected clear_fault(const std::string & /*entity_id*/, + const std::string & code) override { + ADD_FAILURE() << "the gateway cleared '" << code << "' by code alone instead of calling clear_fault_record"; + return FaultClearResult{json{{"code", code}, {"cleared", true}}}; + } + tl::expected + clear_fault_record(const std::string & entity_id, const std::string & code, const std::string & owner) override { + if (fail_if_called) { + ADD_FAILURE() << "the provider was asked to clear '" << code << "' with owner '" << owner + << "', which the gateway never resolved"; + } + clears.push_back(ClearCall{entity_id, code, owner}); + return FaultClearResult{json{{"code", code}, {"cleared", true}}}; + } + + std::vector clears; + json listed_items_ = json::array(); + /// Set by a test in which any clear reaching the provider is itself the defect. + bool fail_if_called = false; +}; + +/// A provider written against the two-argument clear contract: it overrides +/// clear_fault(entity_id, fault_code) and nothing else of the clear. It has to +/// keep compiling against the current headers and keep receiving the clear. +class TwoArgumentClearPlugin : public GatewayPlugin, public FaultProvider { + public: + struct ClearCall { + std::string entity_id; + std::string fault_code; + }; + + std::string name() const override { + return "two_argument_clear_plugin"; + } + void configure(const json & /*config*/) override { + } + void shutdown() override { + } + + tl::expected list_faults(const std::string & /*entity_id*/) override { + return FaultListResult{json{{"items", json::array()}}}; + } + tl::expected get_fault(const std::string & /*entity_id*/, + const std::string & code) override { + return FaultDetailResult{json{{"code", code}}}; + } + tl::expected clear_fault(const std::string & entity_id, + const std::string & code) override { + clears.push_back(ClearCall{entity_id, code}); + return FaultClearResult{json{{"code", code}, {"cleared", true}}}; + } + + std::vector clears; +}; + +/// A TypedRequest carrying the two positional captures the fault routes read. +/// The smatch holds iterators into `path`, so both outlive the request. +class RoutedRequest { + public: + RoutedRequest(std::string path, const std::string & pattern, const std::string & method = "DELETE") + : path_(std::move(path)) { + std::regex re(pattern); + matched_ = std::regex_match(path_, raw_.matches, re); + raw_.path = path_; + raw_.method = method; + } + + bool matched() const { + return matched_; + } + const httplib::Request & raw() const { + return raw_; + } + + private: + std::string path_; + httplib::Request raw_; + bool matched_ = false; +}; + +} // namespace + +class PluginClearOwnerTest : public ::testing::Test { + protected: + /// A recording the stub fault manager holds for one record. + struct Recording { + std::string id; + std::string fault_code; + std::string owner; + std::string path; + }; + + static void SetUpTestSuite() { + rclcpp::init(0, nullptr); + } + static void TearDownTestSuite() { + rclcpp::shutdown(); + } + + void SetUp() override { + const int test_id = test_counter_++; + ns_ = "pco" + std::to_string(test_id); + bag_dir_ = std::filesystem::temp_directory_path() / ("pco_bags_" + std::to_string(::getpid()) + "_" + ns_); + std::filesystem::create_directories(bag_dir_); + + auto options = rclcpp::NodeOptions{}.automatically_declare_parameters_from_overrides(false).parameter_overrides({ + {"server.port", 0}, + {"fault_manager.namespace", ns_}, + {"fault_manager.service_timeout_sec", 5.0}, + }); + node_ = std::make_shared(options); + // The injected cache is the single source of truth for these tests. The + // graph-event refresh would reconcile it back to the live ROS graph. + node_->stop_discovery_refresh_for_testing(); + + store_ = std::make_shared("pco_store_" + std::to_string(test_id)); + const std::string base = "/" + ns_ + "/fault_manager/"; + list_srv_ = store_->create_service( + base + "list_faults", + [this](const std::shared_ptr & req, const std::shared_ptr & res) { + std::lock_guard lock(store_mutex_); + last_include_muted_.store(req->include_muted); + res->faults = stored_faults_; + res->muted_count = static_cast(muted_faults_.size()); + // The real fault manager's contract: a muted record is one the + // correlation engine hid from the default listing, so it is served + // only to a listing that asks, and then named in muted_faults by its + // code and owner. + if (req->include_muted) { + for (const auto & muted : muted_faults_) { + res->faults.push_back(muted); + ros2_medkit_msgs::msg::MutedFaultInfo info; + info.fault_code = muted.fault_code; + info.source_id = muted.reporting_sources.front(); + info.root_cause_code = "ROOT_CAUSE"; + info.rule_id = "rule"; + res->muted_faults.push_back(info); + } + } + }); + get_srv_ = + store_->create_service(base + "get_fault", [this](const std::shared_ptr & req, + const std::shared_ptr & res) { + std::lock_guard lock(store_mutex_); + get_requests_.emplace_back(req->fault_code, req->source_id); + for (const auto * list : {&stored_faults_, &muted_faults_}) { + for (const auto & fault : *list) { + if (fault.fault_code == req->fault_code && fault.reporting_sources.front() == req->source_id) { + res->success = true; + res->fault = fault; + return; + } + } + } + res->success = false; + res->error_message = "Fault not found: " + req->fault_code; + }); + clear_srv_ = store_->create_service( + base + "clear_fault", [this](const std::shared_ptr & req, + const std::shared_ptr & res) { + { + std::lock_guard lock(cleared_mutex_); + cleared_.push_back({req->fault_code, req->source_id}); + } + // The record the request names moves to CLEARED, so a test can read + // the outcome off the store rather than off the request log alone. + std::lock_guard lock(store_mutex_); + for (auto * list : {&stored_faults_, &muted_faults_}) { + for (auto & fault : *list) { + if (fault.fault_code == req->fault_code && fault.reporting_sources.front() == req->source_id) { + fault.status = ros2_medkit_msgs::msg::Fault::STATUS_CLEARED; + } + } + } + res->success = true; + res->message = "cleared"; + }); + // The fault manager's lookup order: the recording id first, then the fault + // code scoped to the owner the gateway sent. An unscoped code with more than + // one owner holding recordings answers nothing. + rosbag_srv_ = store_->create_service( + base + "get_rosbag", + [this](const std::shared_ptr & req, const std::shared_ptr & res) { + std::lock_guard lock(store_mutex_); + rosbag_requests_.push_back(req->source_id); + const Recording * found = nullptr; + for (const auto & rec : recordings_) { + if (rec.id == req->recording_id) { + found = &rec; + } + } + if (found == nullptr) { + std::vector by_code; + for (const auto & rec : recordings_) { + if (rec.fault_code == req->fault_code && (req->source_id.empty() || rec.owner == req->source_id)) { + by_code.push_back(&rec); + } + } + if (by_code.size() == 1) { + found = by_code.front(); + } + } + if (found == nullptr) { + res->success = false; + res->error_message = "No recording for " + req->recording_id; + return; + } + res->success = true; + res->file_path = found->path; + res->recording_id = found->id; + res->fault_codes = {found->fault_code}; + res->format = "mcap"; + res->size_bytes = std::filesystem::file_size(found->path); + }); + list_rosbags_srv_ = store_->create_service( + base + "list_rosbags", + [this](const std::shared_ptr & req, const std::shared_ptr & res) { + std::lock_guard lock(store_mutex_); + res->success = true; + for (const auto & rec : recordings_) { + if (rec.owner != req->entity_fqn) { + continue; + } + res->fault_codes.push_back(rec.fault_code); + res->recording_ids.push_back(rec.id); + res->file_paths.push_back(rec.path); + res->formats.push_back("mcap"); + res->durations_sec.push_back(1.0); + res->sizes_bytes.push_back(std::filesystem::file_size(rec.path)); + res->created_at_ns.push_back(1'700'000'000'000'000'000); + } + }); + executor_ = std::make_unique(); + executor_->add_node(store_); + spin_ = std::thread([this] { + executor_->spin(); + }); + + ctx_ = std::make_unique(*node_, cors_, auth_, tls_, nullptr); + handlers_ = std::make_unique(*ctx_); + bulk_handlers_ = std::make_unique(*ctx_); + } + + void TearDown() override { + if (spin_.joinable()) { + executor_->cancel(); + spin_.join(); + } + executor_.reset(); + store_.reset(); + bulk_handlers_.reset(); + handlers_.reset(); + ctx_.reset(); + node_.reset(); + std::error_code ec; + std::filesystem::remove_all(bag_dir_, ec); + } + + static ros2_medkit_msgs::msg::Fault make_record(const std::string & code, const std::string & owner) { + ros2_medkit_msgs::msg::Fault fault; + fault.fault_code = code; + fault.status = ros2_medkit_msgs::msg::Fault::STATUS_CONFIRMED; + fault.severity = ros2_medkit_msgs::msg::Fault::SEVERITY_ERROR; + fault.reporting_sources = {owner}; + fault.owner = owner; + return fault; + } + + /// One stored record, owned by `owner`. + void store_record(const std::string & code, const std::string & owner) { + std::lock_guard lock(store_mutex_); + stored_faults_.push_back(make_record(code, owner)); + } + + /// One muted record, owned by `owner`. Served only to a listing that asks. + void store_muted_record(const std::string & code, const std::string & owner) { + std::lock_guard lock(store_mutex_); + muted_faults_.push_back(make_record(code, owner)); + } + + /// Move the stored record (code, owner) to `status`, muted or not. + void set_status(const std::string & code, const std::string & owner, const std::string & status) { + std::lock_guard lock(store_mutex_); + for (auto * list : {&stored_faults_, &muted_faults_}) { + for (auto & fault : *list) { + if (fault.fault_code == code && fault.reporting_sources.front() == owner) { + fault.status = status; + } + } + } + } + + /// The HTTP status a recording download answers: 200 when it serves bytes, + /// otherwise the status of the error it returns. + int download_status(const std::string & path, const std::string & pattern) { + RoutedRequest req(path, pattern, "GET"); + EXPECT_TRUE(req.matched()) << path; + auto result = bulk_handlers_->download(ros2_medkit_gateway::http::TypedRequest(req.raw())); + return result.has_value() ? 200 : result.error().http_status; + } + + static std::string app_bag_path(const std::string & app, const std::string & id) { + return std::string("/api/v1/apps/") + app + "/bulk-data/rosbags/" + id; + } + static std::string component_bag_path(const std::string & id) { + return std::string("/api/v1/components/") + kComponent + "/bulk-data/rosbags/" + id; + } + + /// A recording of the record (code, owner), with real bytes behind it. + void store_recording(const std::string & id, const std::string & code, const std::string & owner) { + const auto path = bag_dir_ / (id + ".mcap"); + { + std::ofstream out(path, std::ios::binary); + out << "bag of " << owner; + } + std::lock_guard lock(store_mutex_); + recordings_.push_back(Recording{id, code, owner, path.string()}); + } + + /// A component hosting two external apps, all three owned by the plugin, so + /// the fault routes take the plugin branch on every one of them. + RecordingFaultPlugin * seed_topology() { + return seed_plugin_topology(); + } + + /// The same topology owned by a plugin of type P. + template + P * seed_plugin_topology() { + seed_entities(); + auto plugin = std::make_unique

(); + auto * raw = plugin.get(); + const std::string plugin_name = raw->name(); + auto * pmgr = node_->get_plugin_manager(); + pmgr->add_plugin(std::move(plugin)); + pmgr->register_entity_ownership(plugin_name, {kComponent, kHostedApp, kOtherApp}); + return raw; + } + + /// The same component and apps, owned by NOBODY, so the fault routes take the + /// native fault-manager path and resolve through resolve_scoped_fault. + void seed_native_topology() { + seed_entities(); + } + + bool wait_for_store(std::chrono::milliseconds timeout = 5s) { + auto probe = store_->create_client("/" + ns_ + "/fault_manager/list_faults"); + const auto deadline = std::chrono::steady_clock::now() + timeout; + while (std::chrono::steady_clock::now() < deadline) { + if (probe->service_is_ready()) { + return true; + } + std::this_thread::sleep_for(20ms); + } + return false; + } + + std::vector> cleared_snapshot() { + std::lock_guard lock(cleared_mutex_); + return cleared_; + } + + /// The stored status of the record (code, owner), muted or not. + std::string status_of(const std::string & code, const std::string & owner) { + std::lock_guard lock(store_mutex_); + for (const auto * list : {&stored_faults_, &muted_faults_}) { + for (const auto & fault : *list) { + if (fault.fault_code == code && fault.reporting_sources.front() == owner) { + return fault.status; + } + } + } + return ""; + } + + std::vector> get_requests_snapshot() { + std::lock_guard lock(store_mutex_); + return get_requests_; + } + + std::vector rosbag_requests_snapshot() { + std::lock_guard lock(store_mutex_); + return rosbag_requests_; + } + + static std::string component_fault_path() { + return std::string("/api/v1/components/") + kComponent + "/faults/" + kCode; + } + static constexpr const char * kComponentFaultPattern = R"(/api/v1/components/([^/]+)/faults/([^/]+))"; + static constexpr const char * kComponentBagPattern = R"(/api/v1/components/([^/]+)/bulk-data/([^/]+)/([^/]+))"; + static constexpr const char * kAppBagPattern = R"(/api/v1/apps/([^/]+)/bulk-data/([^/]+)/([^/]+))"; + + static inline int test_counter_ = 0; + CorsConfig cors_{}; + AuthConfig auth_{}; + TlsConfig tls_{}; + std::string ns_; + std::filesystem::path bag_dir_; + std::shared_ptr node_; + std::shared_ptr store_; + rclcpp::Service::SharedPtr list_srv_; + rclcpp::Service::SharedPtr get_srv_; + rclcpp::Service::SharedPtr clear_srv_; + rclcpp::Service::SharedPtr rosbag_srv_; + rclcpp::Service::SharedPtr list_rosbags_srv_; + std::unique_ptr executor_; + std::thread spin_; + + std::mutex store_mutex_; + std::vector stored_faults_; + std::vector muted_faults_; + std::vector recordings_; + std::vector> get_requests_; + std::vector rosbag_requests_; + std::atomic last_include_muted_{false}; + + std::mutex cleared_mutex_; + std::vector> cleared_; + std::unique_ptr ctx_; + std::unique_ptr handlers_; + std::unique_ptr bulk_handlers_; + + private: + void seed_entities() { + App a; + a.id = kHostedApp; + a.component_id = kComponent; + a.external = true; + App b; + b.id = kOtherApp; + b.component_id = kComponent; + b.external = true; + Component c; + c.id = kComponent; + c.external = true; + auto & cache = const_cast(node_->get_thread_safe_cache()); + cache.update_all({}, {c}, {a, b}, {}); + } +}; + +// A component owns the records its hosted apps reported. The clear the plugin +// receives has to name that app, because that is the record the gateway +// resolved. Handing it the component id addresses a record no source owns. +// @verifies REQ_INTEROP_015 +TEST_F(PluginClearOwnerTest, ComponentClearHandsTheHostedAppsOwnerToThePlugin) { + auto * plugin = seed_topology(); + store_record(kCode, kHostedApp); + ASSERT_TRUE(wait_for_store()); + + RoutedRequest req(std::string("/api/v1/components/") + kComponent + "/faults/" + kCode, + R"(/api/v1/components/([^/]+)/faults/([^/]+))"); + ASSERT_TRUE(req.matched()); + ros2_medkit_gateway::http::TypedRequest typed(req.raw()); + + auto result = handlers_->clear_fault(typed); + + ASSERT_TRUE(result.has_value()) << result.error().message; + ASSERT_EQ(plugin->clears.size(), 1u); + EXPECT_EQ(plugin->clears[0].entity_id, kComponent); + EXPECT_EQ(plugin->clears[0].fault_code, kCode); + EXPECT_EQ(plugin->clears[0].owner, kHostedApp) << "the clear must name the record's owner, not the entity in the URL"; +} + +// The app route resolves to its own record, so owner and entity coincide there. +// @verifies REQ_INTEROP_015 +TEST_F(PluginClearOwnerTest, AppClearHandsItsOwnOwnerToThePlugin) { + auto * plugin = seed_topology(); + store_record(kCode, kHostedApp); + ASSERT_TRUE(wait_for_store()); + + RoutedRequest req(std::string("/api/v1/apps/") + kHostedApp + "/faults/" + kCode, + R"(/api/v1/apps/([^/]+)/faults/([^/]+))"); + ASSERT_TRUE(req.matched()); + ros2_medkit_gateway::http::TypedRequest typed(req.raw()); + + auto result = handlers_->clear_fault(typed); + + ASSERT_TRUE(result.has_value()) << result.error().message; + ASSERT_EQ(plugin->clears.size(), 1u); + EXPECT_EQ(plugin->clears[0].owner, kHostedApp); +} + +// Two hosted apps report the code, so the component's URL covers both records +// and the plugin clears each with its own owner. +// @verifies REQ_INTEROP_015 +TEST_F(PluginClearOwnerTest, ComponentClearHandsEachRecordsOwnerToThePlugin) { + auto * plugin = seed_topology(); + store_record(kCode, kHostedApp); + store_record(kCode, kOtherApp); + ASSERT_TRUE(wait_for_store()); + + RoutedRequest req(std::string("/api/v1/components/") + kComponent + "/faults/" + kCode, + R"(/api/v1/components/([^/]+)/faults/([^/]+))"); + ASSERT_TRUE(req.matched()); + ros2_medkit_gateway::http::TypedRequest typed(req.raw()); + + auto result = handlers_->clear_fault(typed); + + ASSERT_TRUE(result.has_value()) << result.error().message; + ASSERT_EQ(plugin->clears.size(), 2u); + // Ordered by owner. + EXPECT_EQ(plugin->clears[0].owner, std::min(kHostedApp, kOtherApp)); + EXPECT_EQ(plugin->clears[1].owner, std::max(kHostedApp, kOtherApp)); +} + +// The bulk clear walks the plugin's own listing, and each item names its own +// record. Sending the entity id for all of them clears one owner's record at +// most and silently misses the rest. +// @verifies REQ_INTEROP_014 +TEST_F(PluginClearOwnerTest, ComponentBulkClearSendsEachRecordsOwnOwner) { + auto * plugin = seed_topology(); + plugin->listed_items_ = json::array({ + json{{"code", kCode}, {"source_id", kHostedApp}}, + json{{"code", kCode}, {"source_id", kOtherApp}}, + }); + ASSERT_TRUE(wait_for_store()); + + RoutedRequest req(std::string("/api/v1/components/") + kComponent + "/faults", + R"(/api/v1/components/([^/]+)/faults)"); + ASSERT_TRUE(req.matched()); + ros2_medkit_gateway::http::TypedRequest typed(req.raw()); + + auto result = handlers_->clear_all_faults(typed); + + ASSERT_TRUE(result.has_value()) << result.error().message; + ASSERT_EQ(plugin->clears.size(), 2u); + EXPECT_EQ(plugin->clears[0].owner, kHostedApp); + EXPECT_EQ(plugin->clears[1].owner, kOtherApp) << "each item's own source_id must travel with its clear"; +} + +// The correlation engine mutes a symptom to keep it out of the default listing. +// That is a display decision, not an unaddressing: the record still exists and +// its entity could read and clear it before. resolve_scoped_fault resolving out +// of a listing that excludes muted records 404s it instead. +// +// The topology here is deliberately NOT plugin-owned, so the route takes the +// native fault-manager path and resolve_scoped_fault is the function under test. +// @verifies REQ_INTEROP_015 +TEST_F(PluginClearOwnerTest, ResolutionKeepsAMutedRecordAddressable) { + seed_native_topology(); + store_muted_record(kCode, kHostedApp); + ASSERT_TRUE(wait_for_store()); + + RoutedRequest req(std::string("/api/v1/apps/") + kHostedApp + "/faults/" + kCode, + R"(/api/v1/apps/([^/]+)/faults/([^/]+))"); + ASSERT_TRUE(req.matched()); + ros2_medkit_gateway::http::TypedRequest typed(req.raw()); + + auto result = handlers_->clear_fault(typed); + + ASSERT_TRUE(result.has_value()) << "a muted record must stay addressable: " << result.error().message; + const auto cleared = cleared_snapshot(); + ASSERT_EQ(cleared.size(), 1u) << "the muted record was never cleared"; + EXPECT_EQ(cleared[0].first, kCode); + EXPECT_EQ(cleared[0].second, kHostedApp) << "the clear must carry the muted record's owner"; +} + +// The same route on an unmuted record, so the muted case above is not simply +// passing because the harness serves everything regardless of the flag. +// @verifies REQ_INTEROP_015 +TEST_F(PluginClearOwnerTest, ResolutionClearsAnUnmutedRecordWithItsOwner) { + seed_native_topology(); + store_record(kCode, kHostedApp); + ASSERT_TRUE(wait_for_store()); + + RoutedRequest req(std::string("/api/v1/apps/") + kHostedApp + "/faults/" + kCode, + R"(/api/v1/apps/([^/]+)/faults/([^/]+))"); + ASSERT_TRUE(req.matched()); + ros2_medkit_gateway::http::TypedRequest typed(req.raw()); + + auto result = handlers_->clear_fault(typed); + + ASSERT_TRUE(result.has_value()) << result.error().message; + const auto cleared = cleared_snapshot(); + ASSERT_EQ(cleared.size(), 1u); + EXPECT_EQ(cleared[0].second, kHostedApp); +} + +// The plugin clear reads the fault manager to learn which record the code +// names. When that read fails the route cannot tell a record the fault manager +// holds from a plugin-internal one, so it must not guess: calling the provider +// with an empty owner answered 2xx while the record stayed untouched. +// @verifies REQ_INTEROP_015 +TEST_F(PluginClearOwnerTest, PluginClearAnswers503WhenTheFaultManagerCannotBeRead) { + auto * plugin = seed_topology(); + plugin->fail_if_called = true; + store_record(kCode, kHostedApp); + ASSERT_TRUE(wait_for_store()); + // Take ListFaults away and wait until the graph agrees it is gone, so the + // call below fails on the read and not on a race with the service teardown. + list_srv_.reset(); + auto probe = store_->create_client("/" + ns_ + "/fault_manager/list_faults"); + const auto deadline = std::chrono::steady_clock::now() + 5s; + while (probe->service_is_ready() && std::chrono::steady_clock::now() < deadline) { + std::this_thread::sleep_for(20ms); + } + ASSERT_FALSE(probe->service_is_ready()) << "ListFaults is still discoverable"; + + RoutedRequest req(component_fault_path(), kComponentFaultPattern); + ASSERT_TRUE(req.matched()); + ros2_medkit_gateway::http::TypedRequest typed(req.raw()); + + auto result = handlers_->clear_fault(typed); + + ASSERT_FALSE(result.has_value()) << "a clear the gateway could not resolve must not report success"; + EXPECT_EQ(result.error().http_status, 503); + EXPECT_EQ(result.error().code, ros2_medkit_gateway::ERR_SERVICE_UNAVAILABLE); + EXPECT_TRUE(plugin->clears.empty()) << "the provider was called without a resolved owner"; +} + +// The positive control for the case above, on the same fixture and the same +// stub: with ListFaults answering, the same request reaches the provider with +// the owner the gateway resolved. So the silence above is the route refusing, +// not a harness that never reaches the provider. +// @verifies REQ_INTEROP_015 +TEST_F(PluginClearOwnerTest, PluginClearReachesTheProviderWhenTheFaultManagerAnswers) { + auto * plugin = seed_topology(); + store_record(kCode, kHostedApp); + ASSERT_TRUE(wait_for_store()); + + RoutedRequest req(component_fault_path(), kComponentFaultPattern); + ASSERT_TRUE(req.matched()); + ros2_medkit_gateway::http::TypedRequest typed(req.raw()); + + auto result = handlers_->clear_fault(typed); + + ASSERT_TRUE(result.has_value()) << result.error().message; + ASSERT_EQ(plugin->clears.size(), 1u); + EXPECT_EQ(plugin->clears[0].owner, kHostedApp); +} + +// A provider that predates the owner-aware clear overrides only the +// two-argument clear_fault. It must still compile against these headers and +// still receive the clear the gateway resolved, through the default +// clear_fault_record, which hands it the entity and the code as before. +// @verifies REQ_INTEROP_015 +TEST_F(PluginClearOwnerTest, ATwoArgumentClearProviderStillReceivesTheClear) { + auto * plugin = seed_plugin_topology(); + store_record(kCode, kHostedApp); + ASSERT_TRUE(wait_for_store()); + + RoutedRequest req(component_fault_path(), kComponentFaultPattern); + ASSERT_TRUE(req.matched()); + ros2_medkit_gateway::http::TypedRequest typed(req.raw()); + + auto result = handlers_->clear_fault(typed); + + ASSERT_TRUE(result.has_value()) << result.error().message; + ASSERT_TRUE(std::holds_alternative(*result)) << "the plugin's acknowledgement is the answer"; + ASSERT_EQ(plugin->clears.size(), 1u) << "the two-argument clear_fault never received the clear"; + EXPECT_EQ(plugin->clears[0].entity_id, kComponent); + EXPECT_EQ(plugin->clears[0].fault_code, kCode); +} + +// --------------------------------------------------------------------------- +// Muted records under a code in the URL. +// +// Every per-entity fault list leaves muted records out. The per-code fault +// routes act on every record of the code, muted or not. A recording URL +// carrying a code resolves it over the records the list shows first, then the +// muted ones. Each route gets its own case, because each reads the fault +// manager on its own. +// --------------------------------------------------------------------------- + +// The component's list shows tank's record only, because pump's record of the +// same code is muted. The per-code DELETE still clears every record of the code +// under the component, the muted one too. +// @verifies REQ_INTEROP_015 +TEST_F(PluginClearOwnerTest, NativeComponentClearTakesTheMutedRecordToo) { + seed_native_topology(); + store_record(kCode, kHostedApp); + store_muted_record(kCode, kOtherApp); + ASSERT_TRUE(wait_for_store()); + + RoutedRequest req(component_fault_path(), kComponentFaultPattern); + ASSERT_TRUE(req.matched()); + ros2_medkit_gateway::http::TypedRequest typed(req.raw()); + + auto result = handlers_->clear_fault(typed); + + ASSERT_TRUE(result.has_value()) << result.error().message; + EXPECT_TRUE(std::holds_alternative(*result)); + auto cleared = cleared_snapshot(); + std::sort(cleared.begin(), cleared.end()); + ASSERT_EQ(cleared.size(), 2u) << "both records of the code are cleared"; + EXPECT_EQ(cleared[0], std::make_pair(std::string(kCode), std::min(kHostedApp, kOtherApp))); + EXPECT_EQ(cleared[1], std::make_pair(std::string(kCode), std::max(kHostedApp, kOtherApp))); +} + +// The same scenario on a plugin-owned component: the plugin clear reads the +// fault manager on its own, so it needs its own case. +// @verifies REQ_INTEROP_015 +TEST_F(PluginClearOwnerTest, PluginComponentClearTakesTheMutedRecordToo) { + auto * plugin = seed_topology(); + store_record(kCode, kHostedApp); + store_muted_record(kCode, kOtherApp); + ASSERT_TRUE(wait_for_store()); + + RoutedRequest req(component_fault_path(), kComponentFaultPattern); + ASSERT_TRUE(req.matched()); + ros2_medkit_gateway::http::TypedRequest typed(req.raw()); + + auto result = handlers_->clear_fault(typed); + + ASSERT_TRUE(result.has_value()) << result.error().message; + ASSERT_EQ(plugin->clears.size(), 2u) << "the plugin must be handed each record's owner"; + EXPECT_EQ(plugin->clears[0].owner, std::min(kHostedApp, kOtherApp)); + EXPECT_EQ(plugin->clears[1].owner, std::max(kHostedApp, kOtherApp)); +} + +// A muted record with no shown record of its code is still the record the code +// names, and the detail route serves it with its owner. +// @verifies REQ_INTEROP_013 +TEST_F(PluginClearOwnerTest, AMutedRecordAloneIsServedOnTheDetailRoute) { + seed_native_topology(); + store_muted_record(kCode, kOtherApp); + ASSERT_TRUE(wait_for_store()); + + RoutedRequest req(component_fault_path(), kComponentFaultPattern, "GET"); + ASSERT_TRUE(req.matched()); + ros2_medkit_gateway::http::TypedRequest typed(req.raw()); + + auto result = handlers_->get_fault(typed); + + ASSERT_TRUE(result.has_value()) << "a muted record must stay addressable: " << result.error().message; + EXPECT_EQ(result->content["x-medkit"]["owner"], kOtherApp); + const auto reads = get_requests_snapshot(); + ASSERT_EQ(reads.size(), 1u); + EXPECT_EQ(reads[0].second, kOtherApp) << "the enriched read must be addressed to the muted record's owner"; +} + +// @verifies REQ_INTEROP_015 +TEST_F(PluginClearOwnerTest, AMutedRecordAloneIsClearedOnTheNativeRoute) { + seed_native_topology(); + store_muted_record(kCode, kOtherApp); + ASSERT_TRUE(wait_for_store()); + + RoutedRequest req(component_fault_path(), kComponentFaultPattern); + ASSERT_TRUE(req.matched()); + ros2_medkit_gateway::http::TypedRequest typed(req.raw()); + + auto result = handlers_->clear_fault(typed); + + ASSERT_TRUE(result.has_value()) << "a muted record must stay addressable: " << result.error().message; + const auto cleared = cleared_snapshot(); + ASSERT_EQ(cleared.size(), 1u); + EXPECT_EQ(cleared[0].second, kOtherApp); +} + +// @verifies REQ_INTEROP_015 +TEST_F(PluginClearOwnerTest, AMutedRecordAloneReachesThePluginWithItsOwner) { + auto * plugin = seed_topology(); + store_muted_record(kCode, kOtherApp); + ASSERT_TRUE(wait_for_store()); + + RoutedRequest req(component_fault_path(), kComponentFaultPattern); + ASSERT_TRUE(req.matched()); + ros2_medkit_gateway::http::TypedRequest typed(req.raw()); + + auto result = handlers_->clear_fault(typed); + + ASSERT_TRUE(result.has_value()) << result.error().message; + ASSERT_EQ(plugin->clears.size(), 1u); + EXPECT_EQ(plugin->clears[0].owner, kOtherApp) << "the plugin must be handed the muted record's owner"; +} + +// A recording URL carrying a fault code resolves that code to a record first. +// A muted record keeps its recordings, so its code still serves its bytes. +// @verifies REQ_INTEROP_072 +TEST_F(PluginClearOwnerTest, AMutedRecordAloneServesItsRecordingByCode) { + seed_native_topology(); + store_muted_record(kCode, kOtherApp); + store_recording("rec_pump", kCode, kOtherApp); + ASSERT_TRUE(wait_for_store()); + + RoutedRequest req(std::string("/api/v1/components/") + kComponent + "/bulk-data/rosbags/" + kCode, + kComponentBagPattern, "GET"); + ASSERT_TRUE(req.matched()); + ros2_medkit_gateway::http::TypedRequest typed(req.raw()); + + auto result = bulk_handlers_->download(typed); + + ASSERT_TRUE(result.has_value()) << "a muted record's recording must stay reachable: " << result.error().message; + EXPECT_EQ(result->filename.value_or(""), "rec_pump.mcap"); + const auto asked = rosbag_requests_snapshot(); + ASSERT_EQ(asked.size(), 1u); + EXPECT_EQ(asked[0], kOtherApp) << "the recording must be asked for under the muted record's owner"; +} + +// The recording route resolves a code the way the fault routes do: tank's +// record is the one the component's list shows, so the code names tank's +// recording and pump's muted record does not make it ambiguous. +// @verifies REQ_INTEROP_072 +TEST_F(PluginClearOwnerTest, ARecordingCodeTakesTheShownRecordOverAMutedOne) { + seed_native_topology(); + store_record(kCode, kHostedApp); + store_muted_record(kCode, kOtherApp); + store_recording("rec_tank", kCode, kHostedApp); + store_recording("rec_pump", kCode, kOtherApp); + ASSERT_TRUE(wait_for_store()); + + RoutedRequest req(std::string("/api/v1/components/") + kComponent + "/bulk-data/rosbags/" + kCode, + kComponentBagPattern, "GET"); + ASSERT_TRUE(req.matched()); + ros2_medkit_gateway::http::TypedRequest typed(req.raw()); + + auto result = bulk_handlers_->download(typed); + + ASSERT_TRUE(result.has_value()) << "the shown record is the one the code names: " << result.error().message; + EXPECT_EQ(result->filename.value_or(""), "rec_tank.mcap"); + const auto asked = rosbag_requests_snapshot(); + ASSERT_EQ(asked.size(), 1u); + EXPECT_EQ(asked[0], kHostedApp); +} + +// Two muted records and no shown one: the per-code DELETE clears both. +// @verifies REQ_INTEROP_015 +TEST_F(PluginClearOwnerTest, TwoMutedRecordsAloneAreBothCleared) { + seed_native_topology(); + store_muted_record(kCode, kHostedApp); + store_muted_record(kCode, kOtherApp); + ASSERT_TRUE(wait_for_store()); + + RoutedRequest req(component_fault_path(), kComponentFaultPattern); + ASSERT_TRUE(req.matched()); + ros2_medkit_gateway::http::TypedRequest typed(req.raw()); + + auto result = handlers_->clear_fault(typed); + + ASSERT_TRUE(result.has_value()) << result.error().message; + EXPECT_EQ(cleared_snapshot().size(), 2u) << "each muted record of the code is cleared"; +} + +// @verifies REQ_INTEROP_072 +TEST_F(PluginClearOwnerTest, TwoMutedRecordsAloneMakeTheRecordingCodeAmbiguous) { + seed_native_topology(); + store_muted_record(kCode, kHostedApp); + store_muted_record(kCode, kOtherApp); + store_recording("rec_tank", kCode, kHostedApp); + store_recording("rec_pump", kCode, kOtherApp); + ASSERT_TRUE(wait_for_store()); + + RoutedRequest req(std::string("/api/v1/components/") + kComponent + "/bulk-data/rosbags/" + kCode, + kComponentBagPattern, "GET"); + ASSERT_TRUE(req.matched()); + ros2_medkit_gateway::http::TypedRequest typed(req.raw()); + + auto result = bulk_handlers_->download(typed); + + ASSERT_FALSE(result.has_value()); + EXPECT_EQ(result.error().http_status, 409); + EXPECT_EQ(result.error().params["owners"], json::array({kOtherApp, kHostedApp})); + EXPECT_TRUE(rosbag_requests_snapshot().empty()) << "no recording is looked up for an ambiguous code"; +} + +// The bulk clear walks the entity's fault list, which leaves muted records out, +// and it always has. It clears what the list shows and leaves a muted record +// CONFIRMED. That record is not stranded: its own per-code DELETE clears it. +// @verifies REQ_INTEROP_014 +TEST_F(PluginClearOwnerTest, BulkClearLeavesAMutedRecordToItsPerCodeClear) { + seed_native_topology(); + store_record(kCode, kHostedApp); + store_muted_record(kCode, kOtherApp); + ASSERT_TRUE(wait_for_store()); + + RoutedRequest bulk(std::string("/api/v1/components/") + kComponent + "/faults", + R"(/api/v1/components/([^/]+)/faults)"); + ASSERT_TRUE(bulk.matched()); + auto bulk_result = handlers_->clear_all_faults(ros2_medkit_gateway::http::TypedRequest(bulk.raw())); + + ASSERT_TRUE(bulk_result.has_value()) << bulk_result.error().message; + EXPECT_EQ(status_of(kCode, kHostedApp), "CLEARED") << "the bulk clear must clear the record the list shows"; + EXPECT_EQ(status_of(kCode, kOtherApp), "CONFIRMED") << "the bulk clear must leave the muted record alone"; + + RoutedRequest per_code(std::string("/api/v1/apps/") + kOtherApp + "/faults/" + kCode, + R"(/api/v1/apps/([^/]+)/faults/([^/]+))"); + ASSERT_TRUE(per_code.matched()); + auto per_code_result = handlers_->clear_fault(ros2_medkit_gateway::http::TypedRequest(per_code.raw())); + + ASSERT_TRUE(per_code_result.has_value()) << per_code_result.error().message; + EXPECT_EQ(status_of(kCode, kOtherApp), "CLEARED") << "the muted record's own per-code DELETE must clear it"; +} + +// The recording route's 409 has to send the client somewhere that works. It +// names the rosbags listing and the fault detail's bulk_data_uri, and following +// it does work: the listing carries one descriptor per recording, and each +// descriptor id downloads that owner's bytes. +// @verifies REQ_INTEROP_072 +TEST_F(PluginClearOwnerTest, TheAmbiguousRecordingCodeNamesWhereEachRecordingIsListed) { + seed_native_topology(); + store_record(kCode, kHostedApp); + store_record(kCode, kOtherApp); + store_recording("rec_tank", kCode, kHostedApp); + store_recording("rec_pump", kCode, kOtherApp); + ASSERT_TRUE(wait_for_store()); + const std::string rosbags = std::string("/api/v1/components/") + kComponent + "/bulk-data/rosbags"; + + RoutedRequest by_code(rosbags + "/" + kCode, kComponentBagPattern, "GET"); + ASSERT_TRUE(by_code.matched()); + auto refused = bulk_handlers_->download(ros2_medkit_gateway::http::TypedRequest(by_code.raw())); + ASSERT_FALSE(refused.has_value()); + ASSERT_EQ(refused.error().http_status, 409); + const std::string details = refused.error().params.value("details", ""); + EXPECT_NE(details.find("GET .../bulk-data/rosbags"), std::string::npos) << details; + EXPECT_NE(details.find("descriptor's id"), std::string::npos) << details; + EXPECT_NE(details.find("environment_data.snapshots[].bulk_data_uri"), std::string::npos) << details; + EXPECT_EQ(details.find("x-medkit.recording_id"), std::string::npos) + << "the fault listing carries no recording id, so the message must not send the client there: " << details; + + // Follow the message: list the category, then download each descriptor id. + RoutedRequest listing(rosbags, R"(/api/v1/components/([^/]+)/bulk-data/([^/]+))", "GET"); + ASSERT_TRUE(listing.matched()); + auto listed = bulk_handlers_->list_descriptors(ros2_medkit_gateway::http::TypedRequest(listing.raw())); + ASSERT_TRUE(listed.has_value()) << listed.error().message; + std::vector ids; + for (const auto & descriptor : listed->items) { + ids.push_back(descriptor.id); + } + std::sort(ids.begin(), ids.end()); + ASSERT_EQ(ids, (std::vector{"rec_pump", "rec_tank"})); + + for (const auto & id : ids) { + RoutedRequest by_id((rosbags + "/").append(id), kComponentBagPattern, "GET"); + ASSERT_TRUE(by_id.matched()); + auto served = bulk_handlers_->download(ros2_medkit_gateway::http::TypedRequest(by_id.raw())); + ASSERT_TRUE(served.has_value()) << id << ": " << served.error().message; + EXPECT_EQ(served->filename.value_or(""), id + ".mcap"); + } +} + +// Two sources report one code and one of them clears its record. The +// component's detail merges both, the active one deciding the status and the +// environment data, its DELETE clears both, and the bag URL by code resolves to +// the active record's recording. +// @verifies REQ_INTEROP_013 +TEST_F(PluginClearOwnerTest, AClearedRecordLeavesTheComponentRoutesToTheActiveOne) { + seed_native_topology(); + store_record(kCode, kHostedApp); + store_record(kCode, kOtherApp); + store_recording("rec_tank", kCode, kHostedApp); + store_recording("rec_pump", kCode, kOtherApp); + set_status(kCode, kOtherApp, ros2_medkit_msgs::msg::Fault::STATUS_CLEARED); + ASSERT_TRUE(wait_for_store()); + + // Each route is checked on its own, so one that fails does not hide the others. + RoutedRequest get_req(component_fault_path(), kComponentFaultPattern, "GET"); + ASSERT_TRUE(get_req.matched()); + auto got = handlers_->get_fault(ros2_medkit_gateway::http::TypedRequest(get_req.raw())); + EXPECT_TRUE(got.has_value()) << "GET: " << got.error().message; + if (got.has_value()) { + EXPECT_FALSE(got->content["x-medkit"].contains("owner")) << "a merge of two records has no single owner"; + EXPECT_EQ(got->content["x-medkit"]["status_raw"], "CONFIRMED"); + EXPECT_EQ(got->content["x-medkit"]["records"].size(), 2u); + const auto reads = get_requests_snapshot(); + ASSERT_EQ(reads.size(), 1u); + EXPECT_EQ(reads[0].second, kHostedApp) << "the environment data comes from the active record"; + } + + RoutedRequest bag_req(component_bag_path(kCode), kComponentBagPattern, "GET"); + ASSERT_TRUE(bag_req.matched()); + auto served = bulk_handlers_->download(ros2_medkit_gateway::http::TypedRequest(bag_req.raw())); + EXPECT_TRUE(served.has_value()) << "bag URL: the active record's recording is the one the code names: " + << served.error().message; + if (served.has_value()) { + EXPECT_EQ(served->filename.value_or(""), "rec_tank.mcap"); + } + + RoutedRequest del_req(component_fault_path(), kComponentFaultPattern); + ASSERT_TRUE(del_req.matched()); + auto cleared = handlers_->clear_fault(ros2_medkit_gateway::http::TypedRequest(del_req.raw())); + EXPECT_TRUE(cleared.has_value()) << "DELETE: " << cleared.error().message; + EXPECT_EQ(cleared_snapshot().size(), cleared.has_value() ? 2u : 0u) << "the DELETE clears both records of the code"; + EXPECT_EQ(status_of(kCode, kHostedApp), "CLEARED"); +} + +// With every record of the code cleared, the detail is their merge, still +// cleared, and its environment data comes from one of them. +// @verifies REQ_INTEROP_013 +TEST_F(PluginClearOwnerTest, TwoClearedRecordsAloneAreMerged) { + seed_native_topology(); + store_record(kCode, kHostedApp); + store_record(kCode, kOtherApp); + set_status(kCode, kHostedApp, ros2_medkit_msgs::msg::Fault::STATUS_CLEARED); + set_status(kCode, kOtherApp, ros2_medkit_msgs::msg::Fault::STATUS_CLEARED); + ASSERT_TRUE(wait_for_store()); + + RoutedRequest req(component_fault_path(), kComponentFaultPattern, "GET"); + ASSERT_TRUE(req.matched()); + auto result = handlers_->get_fault(ros2_medkit_gateway::http::TypedRequest(req.raw())); + + ASSERT_TRUE(result.has_value()) << result.error().message; + EXPECT_EQ(result->content["x-medkit"]["status_raw"], "CLEARED"); + EXPECT_EQ(result->content["x-medkit"]["records"].size(), 2u); + EXPECT_EQ(get_requests_snapshot().size(), 1u) << "one record's environment data is read"; +} + +// One cleared record and nothing else: the code still names it, and the detail +// route serves it the way it always served an acknowledged fault. +// @verifies REQ_INTEROP_013 +TEST_F(PluginClearOwnerTest, OneClearedRecordAloneIsServedOnTheDetailRoute) { + seed_native_topology(); + store_record(kCode, kHostedApp); + set_status(kCode, kHostedApp, ros2_medkit_msgs::msg::Fault::STATUS_CLEARED); + ASSERT_TRUE(wait_for_store()); + + RoutedRequest req(component_fault_path(), kComponentFaultPattern, "GET"); + ASSERT_TRUE(req.matched()); + auto result = handlers_->get_fault(ros2_medkit_gateway::http::TypedRequest(req.raw())); + + ASSERT_TRUE(result.has_value()) << "a cleared record alone stays addressable: " << result.error().message; + EXPECT_EQ(result->content["x-medkit"]["owner"], kHostedApp); +} + +// A recording belongs to the (fault code, owner) records it is attached to. Two +// owners of one code each have their own recording: each app downloads its own +// by id and gets 404 on the other's, while the component hosting both owns both. +// The 200s are the positive control for the 404s on the same harness. +// @verifies REQ_INTEROP_072 +TEST_F(PluginClearOwnerTest, EachOwnerDownloadsOnlyItsOwnRecordingById) { + seed_native_topology(); + store_record(kCode, kHostedApp); + store_record(kCode, kOtherApp); + store_recording("rec_tank", kCode, kHostedApp); + store_recording("rec_pump", kCode, kOtherApp); + ASSERT_TRUE(wait_for_store()); + + EXPECT_EQ(download_status(app_bag_path(kHostedApp, "rec_tank"), kAppBagPattern), 200); + EXPECT_EQ(download_status(app_bag_path(kHostedApp, "rec_pump"), kAppBagPattern), 404) + << "tank must not download pump's recording of the same code"; + EXPECT_EQ(download_status(app_bag_path(kOtherApp, "rec_pump"), kAppBagPattern), 200); + EXPECT_EQ(download_status(app_bag_path(kOtherApp, "rec_tank"), kAppBagPattern), 404) + << "pump must not download tank's recording of the same code"; + EXPECT_EQ(download_status(component_bag_path("rec_tank"), kComponentBagPattern), 200); + EXPECT_EQ(download_status(component_bag_path("rec_pump"), kComponentBagPattern), 200); +} + +// Clearing a record keeps its owner in the entity's scope, but it does not make +// that owner the owner of another record's recording. tank cleared its record +// and still gets 404 on pump's recording, while its own recording stays served. +// @verifies REQ_INTEROP_072 +TEST_F(PluginClearOwnerTest, AClearedRecordDoesNotOpenAnotherOwnersRecording) { + seed_native_topology(); + store_record(kCode, kHostedApp); + store_record(kCode, kOtherApp); + store_recording("rec_tank", kCode, kHostedApp); + store_recording("rec_pump", kCode, kOtherApp); + set_status(kCode, kHostedApp, ros2_medkit_msgs::msg::Fault::STATUS_CLEARED); + ASSERT_TRUE(wait_for_store()); + + EXPECT_EQ(download_status(app_bag_path(kHostedApp, "rec_pump"), kAppBagPattern), 404) + << "tank's cleared record must not authorize pump's recording"; + EXPECT_EQ(download_status(app_bag_path(kHostedApp, "rec_tank"), kAppBagPattern), 200) + << "a cleared record keeps serving its own recording"; + EXPECT_EQ(download_status(app_bag_path(kOtherApp, "rec_tank"), kAppBagPattern), 404); + EXPECT_EQ(download_status(app_bag_path(kOtherApp, "rec_pump"), kAppBagPattern), 200); +} + +int main(int argc, char ** argv) { + ::testing::InitGoogleTest(&argc, argv); + return RUN_ALL_TESTS(); +} diff --git a/src/ros2_medkit_gateway/test/test_fault_manager.cpp b/src/ros2_medkit_gateway/test/test_fault_manager.cpp index c7f1cd29a..0ab8b82b8 100644 --- a/src/ros2_medkit_gateway/test/test_fault_manager.cpp +++ b/src/ros2_medkit_gateway/test/test_fault_manager.cpp @@ -18,8 +18,11 @@ #include #include #include +#include #include #include +#include +#include #include #include @@ -30,9 +33,11 @@ #include "ros2_medkit_gateway/ros2/transports/ros2_fault_service_transport.hpp" #include "ros2_medkit_gateway/trigger_fault_subscriber.hpp" #include "ros2_medkit_msgs/msg/fault_event.hpp" +#include "ros2_medkit_msgs/srv/clear_fault.hpp" #include "ros2_medkit_msgs/srv/get_fault.hpp" #include "ros2_medkit_msgs/srv/get_rosbag.hpp" #include "ros2_medkit_msgs/srv/get_snapshots.hpp" +#include "ros2_medkit_msgs/srv/list_faults.hpp" using namespace std::chrono_literals; using ros2_medkit_gateway::FaultFailure; @@ -40,9 +45,11 @@ using ros2_medkit_gateway::FaultManager; using ros2_medkit_gateway::ResourceChange; using ros2_medkit_gateway::ResourceChangeNotifier; using ros2_medkit_gateway::TriggerFaultSubscriber; +using ros2_medkit_msgs::srv::ClearFault; using ros2_medkit_msgs::srv::GetFault; using ros2_medkit_msgs::srv::GetRosbag; using ros2_medkit_msgs::srv::GetSnapshots; +using ros2_medkit_msgs::srv::ListFaults; class FaultManagerTest : public ::testing::Test { protected: @@ -136,7 +143,7 @@ TEST_F(FaultManagerTest, GetSnapshotsServiceNotAvailable) { FaultManager fault_manager(std::make_shared(node_.get())); // Don't create a service, so it will timeout - auto result = fault_manager.get_snapshots("TEST_FAULT"); + auto result = fault_manager.get_snapshots("TEST_FAULT", "/cell_a/sensor"); EXPECT_FALSE(result.success); // On Humble, wait_for_service may report ready before DDS confirms absence, @@ -164,7 +171,7 @@ TEST_F(FaultManagerTest, GetSnapshotsSuccessWithValidJson) { start_spinning(); FaultManager fault_manager(std::make_shared(node_.get())); - auto result = fault_manager.get_snapshots("MOTOR_OVERHEAT"); + auto result = fault_manager.get_snapshots("MOTOR_OVERHEAT", "/cell_a/sensor"); stop_spinning(); EXPECT_TRUE(result.success); @@ -203,7 +210,7 @@ TEST_F(FaultManagerTest, GetSnapshotsDrivesPrivateClientWithoutSpinningHostNode) // node_ is deliberately never spun by this test. FaultManager fault_manager(std::make_shared(node_.get())); - auto result = fault_manager.get_snapshots("SELF_DRIVEN_FAULT"); + auto result = fault_manager.get_snapshots("SELF_DRIVEN_FAULT", "/cell_a/sensor"); service_executor.cancel(); service_thread.join(); @@ -236,7 +243,7 @@ TEST_F(FaultManagerTest, GetSnapshotsUsesConfiguredFaultManagerNamespace) { start_spinning(); FaultManager fault_manager(std::make_shared(node_.get())); - auto result = fault_manager.get_snapshots("NAMESPACED_FAULT"); + auto result = fault_manager.get_snapshots("NAMESPACED_FAULT", "/cell_a/sensor"); stop_spinning(); EXPECT_TRUE(result.success); @@ -266,7 +273,7 @@ TEST_F(FaultManagerTest, InvalidFaultManagerNamespaceFallsBackToRootServicePath) start_spinning(); FaultManager fault_manager(std::make_shared(node_.get())); - auto result = fault_manager.get_snapshots("INVALID_NAMESPACE_FAULT"); + auto result = fault_manager.get_snapshots("INVALID_NAMESPACE_FAULT", "/cell_a/sensor"); stop_spinning(); EXPECT_TRUE(result.success); @@ -288,7 +295,7 @@ TEST_F(FaultManagerTest, GetSnapshotsSuccessWithTopicFilter) { start_spinning(); FaultManager fault_manager(std::make_shared(node_.get())); - fault_manager.get_snapshots("TEST_FAULT", "/specific_topic"); + fault_manager.get_snapshots("TEST_FAULT", "/cell_a/sensor", "/specific_topic"); stop_spinning(); EXPECT_EQ(received_topic, "/specific_topic"); @@ -306,7 +313,7 @@ TEST_F(FaultManagerTest, GetSnapshotsErrorResponse) { start_spinning(); FaultManager fault_manager(std::make_shared(node_.get())); - auto result = fault_manager.get_snapshots("NONEXISTENT_FAULT"); + auto result = fault_manager.get_snapshots("NONEXISTENT_FAULT", "/cell_a/sensor"); stop_spinning(); EXPECT_FALSE(result.success); @@ -325,7 +332,7 @@ TEST_F(FaultManagerTest, GetSnapshotsInvalidJsonFallback) { start_spinning(); FaultManager fault_manager(std::make_shared(node_.get())); - auto result = fault_manager.get_snapshots("TEST_FAULT"); + auto result = fault_manager.get_snapshots("TEST_FAULT", "/cell_a/sensor"); stop_spinning(); EXPECT_TRUE(result.success); @@ -346,7 +353,7 @@ TEST_F(FaultManagerTest, GetSnapshotsEmptyResponse) { start_spinning(); FaultManager fault_manager(std::make_shared(node_.get())); - auto result = fault_manager.get_snapshots("TEST_FAULT"); + auto result = fault_manager.get_snapshots("TEST_FAULT", "/cell_a/sensor"); stop_spinning(); EXPECT_TRUE(result.success); @@ -361,7 +368,7 @@ TEST_F(FaultManagerTest, GetRosbagServiceNotAvailable) { FaultManager fault_manager(std::make_shared(node_.get())); // Don't create a service, so it will timeout - auto result = fault_manager.get_rosbag("TEST_FAULT"); + auto result = fault_manager.get_rosbag("TEST_FAULT", "/cell_a/sensor"); EXPECT_FALSE(result.success); // On Humble, wait_for_service may report ready before DDS confirms absence, @@ -385,7 +392,7 @@ TEST_F(FaultManagerTest, GetRosbagSuccess) { start_spinning(); FaultManager fault_manager(std::make_shared(node_.get())); - auto result = fault_manager.get_rosbag("TEST_ROSBAG_FAULT"); + auto result = fault_manager.get_rosbag("TEST_ROSBAG_FAULT", "/cell_a/sensor"); stop_spinning(); EXPECT_TRUE(result.success); @@ -419,7 +426,7 @@ TEST_F(FaultManagerTest, GetRosbagUsesConfiguredFaultManagerNamespace) { start_spinning(); FaultManager fault_manager(std::make_shared(node_.get())); - auto result = fault_manager.get_rosbag("NAMESPACED_ROSBAG"); + auto result = fault_manager.get_rosbag("NAMESPACED_ROSBAG", "/cell_a/sensor"); stop_spinning(); EXPECT_TRUE(result.success); @@ -462,7 +469,7 @@ TEST_F(FaultManagerTest, TriggerFaultSubscriberForwardsNamespacedFaultEvents) { std::promise change_promise; auto change_future = change_promise.get_future(); auto subscription_id = - notifier.subscribe({"faults", "camera_node", "/FAULT_1"}, [&change_promise](const ResourceChange & change) { + notifier.subscribe({"faults", "", "/FAULT_1"}, [&change_promise](const ResourceChange & change) { change_promise.set_value(change); }); @@ -481,8 +488,10 @@ TEST_F(FaultManagerTest, TriggerFaultSubscriberForwardsNamespacedFaultEvents) { auto change = change_future.get(); EXPECT_EQ(change.collection, "faults"); - EXPECT_EQ(change.entity_id, "camera_node"); + // No owner resolver: the entity id stays empty, never the source's last segment. + EXPECT_EQ(change.entity_id, ""); EXPECT_EQ(change.resource_path, "/FAULT_1"); + EXPECT_EQ(change.value["reporting_sources"], nlohmann::json::array({"/pipeline/camera_node"})); EXPECT_EQ(change.change_type, ros2_medkit_gateway::ChangeType::CREATED); EXPECT_EQ(change.value["fault_code"], "FAULT_1"); EXPECT_EQ(change.value["event_type"], "fault_confirmed"); @@ -491,9 +500,9 @@ TEST_F(FaultManagerTest, TriggerFaultSubscriberForwardsNamespacedFaultEvents) { notifier.unsubscribe(subscription_id); } -TEST_F(FaultManagerTest, TriggerFaultSubscriberResolverCalledAndEntityIdUsed) { - // Resolver is called with the raw FQN and the returned entity_id is used in the notification. - node_ = std::make_shared("test_trigger_resolver_called", +TEST_F(FaultManagerTest, TriggerFaultSubscriberOwnerResolverGetsTheRecordOwner) { + // The owner resolver sees the record's owner and its answer is the change's entity id. + node_ = std::make_shared("test_trigger_owner_resolver", rclcpp::NodeOptions().automatically_declare_parameters_from_overrides(true)); executor_ = std::make_unique(); executor_->add_node(node_); @@ -501,54 +510,113 @@ TEST_F(FaultManagerTest, TriggerFaultSubscriberResolverCalledAndEntityIdUsed) { ResourceChangeNotifier notifier; TriggerFaultSubscriber subscriber(node_.get(), notifier); - std::string received_fqn; - subscriber.set_node_to_entity_resolver([&received_fqn](const std::string & fqn) -> std::string { - received_fqn = fqn; - return "manifest_entity_id"; + std::promise owner_promise; + auto owner_future = owner_promise.get_future(); + subscriber.set_fault_owner_resolver([&owner_promise](const std::string & owner) -> std::string { + owner_promise.set_value(owner); + return "detector-app"; }); auto publisher = node_->create_publisher("/fault_manager/events", rclcpp::QoS(10)); std::promise change_promise; auto change_future = change_promise.get_future(); - auto subscription_id = notifier.subscribe({"faults", "manifest_entity_id", "/RESOLVER_FAULT"}, - [&change_promise](const ResourceChange & change) { - change_promise.set_value(change); - }); + auto subscription_id = + notifier.subscribe({"faults", "", "/OWNED_FAULT"}, [&change_promise](const ResourceChange & c) { + change_promise.set_value(c); + }); ASSERT_TRUE(wait_for_subscription_count("/fault_manager/events", 1u)); start_spinning(); ros2_medkit_msgs::msg::FaultEvent event; - event.event_type = "fault_confirmed"; - event.fault.fault_code = "RESOLVER_FAULT"; - event.fault.reporting_sources.push_back("/pipeline/some_node"); + event.event_type = "fault_updated"; + event.fault.fault_code = "OWNED_FAULT"; + event.fault.reporting_sources = {"/x/detector/goal_status"}; event.fault.severity = ros2_medkit_msgs::msg::Fault::SEVERITY_ERROR; publisher->publish(event); - auto status = change_future.wait_for(2s); - ASSERT_EQ(status, std::future_status::ready); - + ASSERT_EQ(change_future.wait_for(2s), std::future_status::ready); auto change = change_future.get(); - EXPECT_EQ(change.entity_id, "manifest_entity_id"); - EXPECT_EQ(received_fqn, "/pipeline/some_node"); + EXPECT_EQ(change.entity_id, "detector-app"); + EXPECT_EQ(change.change_type, ros2_medkit_gateway::ChangeType::UPDATED); + ASSERT_EQ(owner_future.wait_for(0s), std::future_status::ready); + EXPECT_EQ(owner_future.get(), "/x/detector/goal_status"); stop_spinning(); notifier.unsubscribe(subscription_id); } -TEST_F(FaultManagerTest, TriggerFaultSubscriberResolverEmptyFallsBackToLastSegment) { - // When resolver returns empty string, last-segment extraction is used. - node_ = std::make_shared("test_trigger_resolver_empty_fallback", +TEST_F(FaultManagerTest, TriggerFaultSubscriberForwardsClearAndHealAsDeleted) { + // A clear and a heal both reach the scope check, owner resolved, as DELETED. + node_ = std::make_shared("test_trigger_cleared", rclcpp::NodeOptions().automatically_declare_parameters_from_overrides(true)); executor_ = std::make_unique(); executor_->add_node(node_); ResourceChangeNotifier notifier; TriggerFaultSubscriber subscriber(node_.get(), notifier); + subscriber.set_fault_owner_resolver([](const std::string & /*owner*/) -> std::string { + return "detector-app"; + }); + + auto publisher = node_->create_publisher("/fault_manager/events", rclcpp::QoS(10)); + + std::mutex changes_mutex; + std::vector changes; + std::promise both_promise; + auto both_future = both_promise.get_future(); + auto subscription_id = + notifier.subscribe({"faults", "", ""}, [&changes_mutex, &changes, &both_promise](const ResourceChange & c) { + std::lock_guard lock(changes_mutex); + changes.push_back(c); + if (changes.size() == 2) { + both_promise.set_value(); + } + }); + + ASSERT_TRUE(wait_for_subscription_count("/fault_manager/events", 1u)); + start_spinning(); + + for (const auto & [code, status] : {std::pair{"CLEARED_FAULT", "CLEARED"}, + std::pair{"HEALED_FAULT", "HEALED"}}) { + ros2_medkit_msgs::msg::FaultEvent event; + event.event_type = "fault_cleared"; + event.fault.fault_code = code; + event.fault.status = status; + event.fault.reporting_sources = {"/x/detector/diag"}; + event.fault.severity = ros2_medkit_msgs::msg::Fault::SEVERITY_ERROR; + publisher->publish(event); + } + + ASSERT_EQ(both_future.wait_for(2s), std::future_status::ready); + std::lock_guard lock(changes_mutex); + ASSERT_EQ(changes.size(), 2u); + EXPECT_EQ(changes[0].resource_path, "/CLEARED_FAULT"); + EXPECT_EQ(changes[0].value.value("status", ""), "CLEARED"); + EXPECT_EQ(changes[1].resource_path, "/HEALED_FAULT"); + EXPECT_EQ(changes[1].value.value("status", ""), "HEALED"); + for (const auto & change : changes) { + EXPECT_EQ(change.entity_id, "detector-app"); + EXPECT_EQ(change.change_type, ros2_medkit_gateway::ChangeType::DELETED); + EXPECT_EQ(change.value.value("event_type", ""), "fault_cleared"); + } + + stop_spinning(); + notifier.unsubscribe(subscription_id); +} + +TEST_F(FaultManagerTest, TriggerFaultSubscriberForwardsFaultNoAppOwns) { + // A record no app owns can still be listed by an external component, so it + // is forwarded with an empty entity id instead of being dropped. + node_ = std::make_shared("test_trigger_no_owner", + rclcpp::NodeOptions().automatically_declare_parameters_from_overrides(true)); + executor_ = std::make_unique(); + executor_->add_node(node_); - // Resolver always returns empty -> fallback to last segment - subscriber.set_node_to_entity_resolver([](const std::string & /*fqn*/) -> std::string { + ResourceChangeNotifier notifier; + TriggerFaultSubscriber subscriber(node_.get(), notifier); + subscriber.set_fault_owner_resolver([](const std::string & /*owner*/) -> std::string { return ""; }); @@ -556,66 +624,74 @@ TEST_F(FaultManagerTest, TriggerFaultSubscriberResolverEmptyFallsBackToLastSegme std::promise change_promise; auto change_future = change_promise.get_future(); - auto subscription_id = notifier.subscribe({"faults", "fallback_node", "/FALLBACK_FAULT"}, - [&change_promise](const ResourceChange & change) { - change_promise.set_value(change); - }); + auto subscription_id = notifier.subscribe({"faults", "", "/RACK_FAULT"}, [&change_promise](const ResourceChange & c) { + change_promise.set_value(c); + }); ASSERT_TRUE(wait_for_subscription_count("/fault_manager/events", 1u)); start_spinning(); ros2_medkit_msgs::msg::FaultEvent event; event.event_type = "fault_confirmed"; - event.fault.fault_code = "FALLBACK_FAULT"; - event.fault.reporting_sources.push_back("/pipeline/fallback_node"); + event.fault.fault_code = "RACK_FAULT"; + event.fault.reporting_sources = {"plc-rack/slot_1"}; event.fault.severity = ros2_medkit_msgs::msg::Fault::SEVERITY_ERROR; publisher->publish(event); - auto status = change_future.wait_for(2s); - ASSERT_EQ(status, std::future_status::ready); - + ASSERT_EQ(change_future.wait_for(2s), std::future_status::ready); auto change = change_future.get(); - EXPECT_EQ(change.entity_id, "fallback_node"); + EXPECT_EQ(change.entity_id, ""); + EXPECT_EQ(change.change_type, ros2_medkit_gateway::ChangeType::CREATED); stop_spinning(); notifier.unsubscribe(subscription_id); } -TEST_F(FaultManagerTest, TriggerFaultSubscriberNoResolverUsesLastSegment) { - // When no resolver is set, last-segment extraction works as before. - node_ = std::make_shared("test_trigger_no_resolver", +TEST_F(FaultManagerTest, TriggerFaultSubscriberSkipsFaultWithoutReportingSources) { + // No reporting source means no entity lists the fault. + node_ = std::make_shared("test_trigger_no_sources", rclcpp::NodeOptions().automatically_declare_parameters_from_overrides(true)); executor_ = std::make_unique(); executor_->add_node(node_); ResourceChangeNotifier notifier; TriggerFaultSubscriber subscriber(node_.get(), notifier); - // No resolver set auto publisher = node_->create_publisher("/fault_manager/events", rclcpp::QoS(10)); - std::promise change_promise; - auto change_future = change_promise.get_future(); - auto subscription_id = notifier.subscribe({"faults", "camera_proc", "/NO_RESOLVER_FAULT"}, - [&change_promise](const ResourceChange & change) { - change_promise.set_value(change); - }); + std::mutex codes_mutex; + std::vector codes; + std::promise probe_promise; + auto probe_future = probe_promise.get_future(); + auto subscription_id = + notifier.subscribe({"faults", "", ""}, [&codes_mutex, &codes, &probe_promise](const ResourceChange & c) { + std::lock_guard lock(codes_mutex); + codes.push_back(c.resource_path); + if (c.resource_path == "/PROBE") { + probe_promise.set_value(); + } + }); ASSERT_TRUE(wait_for_subscription_count("/fault_manager/events", 1u)); start_spinning(); - ros2_medkit_msgs::msg::FaultEvent event; - event.event_type = "fault_confirmed"; - event.fault.fault_code = "NO_RESOLVER_FAULT"; - event.fault.reporting_sources.push_back("/sensors/camera_proc"); - event.fault.severity = ros2_medkit_msgs::msg::Fault::SEVERITY_ERROR; - publisher->publish(event); - - auto status = change_future.wait_for(2s); - ASSERT_EQ(status, std::future_status::ready); - - auto change = change_future.get(); - EXPECT_EQ(change.entity_id, "camera_proc"); + ros2_medkit_msgs::msg::FaultEvent orphan; + orphan.event_type = "fault_confirmed"; + orphan.fault.fault_code = "ORPHAN"; + publisher->publish(orphan); + + // Delivered in order after the orphan, so its arrival means the orphan was judged. + ros2_medkit_msgs::msg::FaultEvent probe; + probe.event_type = "fault_confirmed"; + probe.fault.fault_code = "PROBE"; + probe.fault.reporting_sources = {"/sensors/camera_proc"}; + publisher->publish(probe); + + ASSERT_EQ(probe_future.wait_for(2s), std::future_status::ready); + { + std::lock_guard lock(codes_mutex); + EXPECT_EQ(codes, std::vector{"/PROBE"}); + } stop_spinning(); notifier.unsubscribe(subscription_id); @@ -633,7 +709,7 @@ TEST_F(FaultManagerTest, GetRosbagNotFound) { start_spinning(); FaultManager fault_manager(std::make_shared(node_.get())); - auto result = fault_manager.get_rosbag("NONEXISTENT_FAULT"); + auto result = fault_manager.get_rosbag("NONEXISTENT_FAULT", "/cell_a/sensor"); stop_spinning(); EXPECT_FALSE(result.success); @@ -688,6 +764,141 @@ TEST_F(FaultManagerTest, GetFaultRefusedByTheStoreReportsDeclined) { EXPECT_EQ(result.error_message, "fault_code contains invalid character '~'"); } +// ============================================================================ +// The owning source reaches the wire +// +// A fault record is the pair (fault_code, owning reporting source), so every +// service that acts on one record carries the owner in its request. The +// gateway used to filter the answer instead, which meant asking the fault +// manager for an arbitrary record of that code and hoping it was the right +// one. These tests read the field off the request the service actually +// received, which is the only place that claim can be checked. +// ============================================================================ + +// @verifies REQ_INTEROP_013 +TEST_F(FaultManagerTest, GetFaultSendsTheOwningSourceInTheRequest) { + std::string received_source; + auto service = node_->create_service( + "/fault_manager/get_fault", [&received_source](const std::shared_ptr & request, + const std::shared_ptr & response) { + received_source = request->source_id; + response->success = true; + response->fault.fault_code = request->fault_code; + response->fault.reporting_sources = {request->source_id}; + }); + + start_spinning(); + FaultManager fault_manager(std::make_shared(node_.get())); + + auto result = fault_manager.get_fault_with_env("SHARED_CODE", "app_b"); + stop_spinning(); + + EXPECT_EQ(received_source, "app_b"); + ASSERT_TRUE(result.success) << result.error_message; + EXPECT_EQ(result.data["fault"]["source_id"], "app_b"); +} + +// @verifies REQ_INTEROP_015 +TEST_F(FaultManagerTest, ClearFaultSendsTheOwningSourceInTheRequest) { + std::string received_source; + std::string received_code; + auto service = node_->create_service( + "/fault_manager/clear_fault", + [&received_source, &received_code](const std::shared_ptr & request, + const std::shared_ptr & response) { + received_code = request->fault_code; + received_source = request->source_id; + response->success = true; + response->message = "cleared"; + }); + + start_spinning(); + FaultManager fault_manager(std::make_shared(node_.get())); + + auto result = fault_manager.clear_fault("SHARED_CODE", "app_a", /*skip_correlation_auto_clear=*/true); + stop_spinning(); + + EXPECT_TRUE(result.success) << result.error_message; + EXPECT_EQ(received_code, "SHARED_CODE"); + EXPECT_EQ(received_source, "app_a"); +} + +// @verifies REQ_INTEROP_088 +TEST_F(FaultManagerTest, GetSnapshotsSendsTheOwningSourceInTheRequest) { + std::string received_source; + auto service = node_->create_service( + "/fault_manager/get_snapshots", [&received_source](const std::shared_ptr & request, + const std::shared_ptr & response) { + received_source = request->source_id; + response->success = true; + response->data = "{}"; + }); + + start_spinning(); + FaultManager fault_manager(std::make_shared(node_.get())); + + fault_manager.get_snapshots("SHARED_CODE", "app_b", "/joint_states"); + stop_spinning(); + + EXPECT_EQ(received_source, "app_b"); +} + +// @verifies REQ_INTEROP_088 +TEST_F(FaultManagerTest, GetRosbagSendsTheOwningSourceInTheRequest) { + std::string received_source; + auto service = node_->create_service( + "/fault_manager/get_rosbag", [&received_source](const std::shared_ptr & request, + const std::shared_ptr & response) { + received_source = request->source_id; + response->success = true; + response->file_path = "/tmp/bag"; + }); + + start_spinning(); + FaultManager fault_manager(std::make_shared(node_.get())); + + fault_manager.get_rosbag("SHARED_CODE", "app_b"); + stop_spinning(); + + EXPECT_EQ(received_source, "app_b"); +} + +// A muted entry describes one record, so it has to name the owner. Without it +// the correlation view hands a client a bare fault_code, which addresses as many +// records as there are sources reporting it - the ambiguity the per-record +// identity exists to remove, reintroduced one layer up. +TEST_F(FaultManagerTest, MutedFaultEntriesNameTheRecordTheyDescribe) { + auto service = node_->create_service( + "/fault_manager/list_faults", + [](const std::shared_ptr & /*request*/, const std::shared_ptr & res) { + ros2_medkit_msgs::msg::MutedFaultInfo muted; + muted.fault_code = "SHARED_SYMPTOM"; + muted.root_cause_code = "ROOT"; + muted.rule_id = "cascade"; + muted.delay_ms = 150; + muted.source_id = "app_b"; + res->muted_faults.push_back(muted); + res->muted_count = 1; + }); + + start_spinning(); + FaultManager fault_manager(std::make_shared(node_.get())); + + auto result = fault_manager.list_faults("", /*include_prefailed=*/true, /*include_confirmed=*/true, + /*include_cleared=*/false, /*include_healed=*/false, + /*include_muted=*/true, /*include_clusters=*/false); + stop_spinning(); + + ASSERT_TRUE(result.success) << result.error_message; + ASSERT_TRUE(result.data.contains("muted_faults")); + ASSERT_EQ(result.data["muted_faults"].size(), 1u); + const auto & entry = result.data["muted_faults"][0]; + EXPECT_EQ(entry["fault_code"], "SHARED_SYMPTOM"); + EXPECT_EQ(entry["root_cause_code"], "ROOT"); + ASSERT_TRUE(entry.contains("source_id")) << "a muted entry must name the record's owner"; + EXPECT_EQ(entry["source_id"], "app_b"); +} + int main(int argc, char ** argv) { ::testing::InitGoogleTest(&argc, argv); return RUN_ALL_TESTS(); diff --git a/src/ros2_medkit_gateway/test/test_fault_manager_routing.cpp b/src/ros2_medkit_gateway/test/test_fault_manager_routing.cpp index ff001a933..7c0d4dc56 100644 --- a/src/ros2_medkit_gateway/test/test_fault_manager_routing.cpp +++ b/src/ros2_medkit_gateway/test/test_fault_manager_routing.cpp @@ -88,8 +88,10 @@ class MockFaultServiceTransport : public FaultServiceTransport { return r; } - FaultResult clear_fault(const std::string & fault_code, bool skip_correlation_auto_clear) override { + FaultResult clear_fault(const std::string & fault_code, const std::string & source_id, + bool skip_correlation_auto_clear) override { last_clear_code_ = fault_code; + last_clear_source_ = source_id; last_clear_skip_correlation_ = skip_correlation_auto_clear; ++clear_calls_; FaultResult r; @@ -99,8 +101,10 @@ class MockFaultServiceTransport : public FaultServiceTransport { return r; } - FaultResult get_snapshots(const std::string & fault_code, const std::string & topic) override { + FaultResult get_snapshots(const std::string & fault_code, const std::string & source_id, + const std::string & topic) override { last_snapshots_code_ = fault_code; + last_snapshots_source_ = source_id; last_snapshots_topic_ = topic; ++snapshots_calls_; FaultResult r; @@ -110,8 +114,9 @@ class MockFaultServiceTransport : public FaultServiceTransport { return r; } - FaultResult get_rosbag(const std::string & fault_code) override { - last_rosbag_code_ = fault_code; + FaultResult get_rosbag(const std::string & id, const std::string & source_id) override { + last_rosbag_code_ = id; + last_rosbag_source_ = source_id; ++rosbag_calls_; FaultResult r; r.success = rosbag_success_; @@ -181,6 +186,7 @@ class MockFaultServiceTransport : public FaultServiceTransport { json clear_data_ = json{{"success", true}, {"message", "ok"}}; std::string clear_error_; std::string last_clear_code_; + std::string last_clear_source_; bool last_clear_skip_correlation_ = false; int clear_calls_ = 0; @@ -188,6 +194,7 @@ class MockFaultServiceTransport : public FaultServiceTransport { json snapshots_data_ = json::object(); std::string snapshots_error_; std::string last_snapshots_code_; + std::string last_snapshots_source_; std::string last_snapshots_topic_; int snapshots_calls_ = 0; @@ -195,6 +202,7 @@ class MockFaultServiceTransport : public FaultServiceTransport { json rosbag_data_ = json::object(); std::string rosbag_error_; std::string last_rosbag_code_; + std::string last_rosbag_source_; int rosbag_calls_ = 0; bool list_rosbags_success_ = true; @@ -343,7 +351,7 @@ TEST(FaultManagerRoutingTest, ClearFaultDelegatesAndReturnsAutoClearedCodes) { mock->clear_data_ = {{"success", true}, {"message", "ok"}, {"auto_cleared_codes", json::array({"S1", "S2"})}}; FaultManager mgr(mock); - auto r = mgr.clear_fault("ROOT"); + auto r = mgr.clear_fault("ROOT", "/cell_a/root_node"); EXPECT_TRUE(r.success); EXPECT_EQ(mock->last_clear_code_, "ROOT"); @@ -351,6 +359,19 @@ TEST(FaultManagerRoutingTest, ClearFaultDelegatesAndReturnsAutoClearedCodes) { EXPECT_EQ(r.data["auto_cleared_codes"].size(), 2u); } +TEST(FaultManagerRoutingTest, ClearFaultCarriesTheOwningSourceToTheTransport) { + // A clear addresses one record, so the owner travels with the code. Without + // it the fault manager would have to guess which owner's record to clear + // whenever two sources report the same code. + auto mock = std::make_shared(); + FaultManager mgr(mock); + + mgr.clear_fault("SHARED_CODE", "app_b", /*skip_correlation_auto_clear=*/true); + + EXPECT_EQ(mock->last_clear_code_, "SHARED_CODE"); + EXPECT_EQ(mock->last_clear_source_, "app_b"); +} + TEST(FaultManagerRoutingTest, ClearFaultDefaultsToCorrelationAutoClear) { // Global `DELETE /api/v1/faults` and any unscoped caller should preserve // the legacy auto-clear-with-root behaviour (skip flag defaults to false). @@ -358,21 +379,21 @@ TEST(FaultManagerRoutingTest, ClearFaultDefaultsToCorrelationAutoClear) { mock->clear_success_ = true; FaultManager mgr(mock); - mgr.clear_fault("ROOT"); + mgr.clear_fault("ROOT", "/cell_a/root_node"); EXPECT_EQ(mock->last_clear_code_, "ROOT"); EXPECT_FALSE(mock->last_clear_skip_correlation_); } TEST(FaultManagerRoutingTest, ClearFaultForwardsSkipCorrelationFlag) { - // Per-entity DELETE routes call `clear_fault(code, /*skip=*/true)` so the - // fault manager does NOT cascade-clear correlated symptoms reported by + // Per-entity DELETE routes call `clear_fault(code, owner, /*skip=*/true)` so + // the fault manager does NOT cascade-clear correlated symptoms reported by // apps outside the addressed entity. Pin the routing here. auto mock = std::make_shared(); mock->clear_success_ = true; FaultManager mgr(mock); - mgr.clear_fault("ROOT", /*skip_correlation_auto_clear=*/true); + mgr.clear_fault("ROOT", "/cell_a/root_node", /*skip_correlation_auto_clear=*/true); EXPECT_EQ(mock->last_clear_code_, "ROOT"); EXPECT_TRUE(mock->last_clear_skip_correlation_); @@ -383,13 +404,26 @@ TEST(FaultManagerRoutingTest, GetSnapshotsRoutesTopicFilter) { mock->snapshots_data_ = {{"topics", json::object()}}; FaultManager mgr(mock); - auto r = mgr.get_snapshots("F1", "/joint_states"); + auto r = mgr.get_snapshots("F1", "/cell_a/sensor", "/joint_states"); EXPECT_TRUE(r.success); EXPECT_EQ(mock->last_snapshots_code_, "F1"); + EXPECT_EQ(mock->last_snapshots_source_, "/cell_a/sensor"); EXPECT_EQ(mock->last_snapshots_topic_, "/joint_states"); } +TEST(FaultManagerRoutingTest, GetRosbagCarriesTheOwningSourceToTheTransport) { + // Snapshots and recordings belong to one record. The owner scopes the + // fault-code lookup so a shared code does not serve another owner's bytes. + auto mock = std::make_shared(); + FaultManager mgr(mock); + + mgr.get_rosbag("SHARED_CODE", "app_b"); + + EXPECT_EQ(mock->last_rosbag_code_, "SHARED_CODE"); + EXPECT_EQ(mock->last_rosbag_source_, "app_b"); +} + TEST(FaultManagerRoutingTest, GetRosbagPropagatesErrorMessage) { auto mock = std::make_shared(); mock->rosbag_success_ = false; diff --git a/src/ros2_medkit_gateway/test/test_fault_trigger_engine.cpp b/src/ros2_medkit_gateway/test/test_fault_trigger_engine.cpp index e54415a9d..905a4912d 100644 --- a/src/ros2_medkit_gateway/test/test_fault_trigger_engine.cpp +++ b/src/ros2_medkit_gateway/test/test_fault_trigger_engine.cpp @@ -145,6 +145,25 @@ TEST(FaultTriggerEngineTest, CreateWithoutEnumeratorSkipsExistenceCheck) { EXPECT_TRUE(engine.create("tank", make_body("whatever", ">", 1.0, "F", "ERROR"))); } +// A code another app's rule holds is refused, and the refusal states the +// engine's own rule. The fault manager would keep the two apps' records apart, +// so a message blaming the fault store sends the operator after the wrong cause. +TEST(FaultTriggerEngineTest, ADuplicateCodeIsRefusedWithTheEnginesOwnRule) { + FaultTriggerEngine engine("", nullptr, nullptr, nullptr, nullptr); + auto first = engine.create("app_a", make_body("x", ">", 1.0, "SHARED", "ERROR")); + ASSERT_TRUE(first); + + auto second = engine.create("app_b", make_body("y", ">", 1.0, "SHARED", "ERROR")); + + ASSERT_FALSE(second); + EXPECT_EQ(second.error().first, 409); + const auto & msg = second.error().second; + EXPECT_NE(msg.find("rule '" + first->id + "' on app 'app_a'"), std::string::npos) << msg; + EXPECT_NE(msg.find("The trigger engine keeps one rule per fault code across every app"), std::string::npos) << msg; + EXPECT_EQ(msg.find("fault store"), std::string::npos) << "the refusal is the engine's rule, not the store's: " << msg; + EXPECT_TRUE(engine.list("app_b").empty()); +} + TEST(FaultTriggerEngineTest, ListIsScopedPerApp) { FaultTriggerEngine engine("", nullptr, nullptr, nullptr, nullptr); ASSERT_TRUE(engine.create("app_a", make_body("x", ">", 1.0, "FA", "ERROR"))); diff --git a/src/ros2_medkit_gateway/test/test_handler_context.cpp b/src/ros2_medkit_gateway/test/test_handler_context.cpp index 3cfb1f660..dcaf21887 100644 --- a/src/ros2_medkit_gateway/test/test_handler_context.cpp +++ b/src/ros2_medkit_gateway/test/test_handler_context.cpp @@ -38,10 +38,12 @@ #include "ros2_medkit_gateway/core/discovery/models/area.hpp" #include "ros2_medkit_gateway/core/discovery/models/component.hpp" #include "ros2_medkit_gateway/core/discovery/models/function.hpp" +#include "ros2_medkit_gateway/core/faults/fault_scope.hpp" #include "ros2_medkit_gateway/core/http/error_codes.hpp" #include "ros2_medkit_gateway/core/models/error_info.hpp" #include "ros2_medkit_gateway/core/models/thread_safe_entity_cache.hpp" #include "ros2_medkit_gateway/gateway_node.hpp" +#include "ros2_medkit_gateway/http/handlers/fault_handlers.hpp" #include "ros2_medkit_gateway/http/handlers/handler_context.hpp" #include "ros2_medkit_gateway/http/typed_router.hpp" #include "ros2_medkit_gateway/test_support/refused_port.hpp" @@ -2055,6 +2057,89 @@ TEST(ResolveEntitySourceFqnsTest, ExternalAppWithStrayRosBindingOwnsFaultsByBare EXPECT_EQ(fqns, std::set{"process"}); } +// ============================================================================= +// build_source_entity_map - whose lock the global fault clear consults +// +// `DELETE /api/v1/faults` skips a record whose owning entity is locked by +// another client. It finds that entity by looking the record's reporting source +// up in this map, so a source the map cannot attribute is a lock the route +// silently walks through. +// ============================================================================= +TEST(BuildSourceEntityMapTest, BoundAppIsFoundByItsFqn) { + ThreadSafeEntityCache cache; + cache.update_apps({make_app_with_binding("temp_sensor", "temp_sensor", "/powertrain/engine")}); + + const auto map = handlers::FaultHandlers::build_source_entity_map(cache); + + ASSERT_EQ(map.count("/powertrain/engine/temp_sensor"), 1u); + EXPECT_EQ(map.at("/powertrain/engine/temp_sensor"), std::set{"temp_sensor"}); +} + +// Two apps can bind one node path (a namespace spelled with and without a +// trailing slash). Keeping one of them let a lock on the other go unhonoured. +TEST(BuildSourceEntityMapTest, EveryAppBoundToOneNodePathIsFound) { + ThreadSafeEntityCache cache; + cache.update_apps( + {make_app_with_binding("twin-a", "twin", "/scope/t"), make_app_with_binding("twin-b", "twin", "/scope/t/")}); + + const auto map = handlers::FaultHandlers::build_source_entity_map(cache); + + ASSERT_EQ(map.count("/scope/t/twin"), 1u); + EXPECT_EQ(map.at("/scope/t/twin"), (std::set{"twin-a", "twin-b"})); +} + +// An external app has no ROS binding, so its effective_fqn() is empty and the +// old map held nothing for it. It reports under its bare id, which is the value +// that arrives on the wire, so the lock of every plugin-owned app went +// unhonoured on this route. +TEST(BuildSourceEntityMapTest, ExternalAppIsFoundByItsBareId) { + ThreadSafeEntityCache cache; + cache.update_apps({make_external_app("plc_line1", "plc_hw")}); + + const auto map = handlers::FaultHandlers::build_source_entity_map(cache); + + ASSERT_EQ(map.count("plc_line1"), 1u) << "an external app's bare id must resolve to its entity"; + EXPECT_EQ(map.at("plc_line1"), std::set{"plc_line1"}); +} + +// A protocol bridge raises its link faults under the component's own id, and +// the map was built from apps only, so it never held a component at all. +TEST(BuildSourceEntityMapTest, ExternalComponentIsFoundByItsBareId) { + ThreadSafeEntityCache cache; + Component device; + device.id = "line_controller"; + device.external = true; + cache.update_components({device}); + + const auto map = handlers::FaultHandlers::build_source_entity_map(cache); + + ASSERT_EQ(map.count("line_controller"), 1u) << "an external component's bare id must resolve to its entity"; + EXPECT_EQ(map.at("line_controller"), std::set{"line_controller"}); +} + +// The #516 guardrail, restated for the lock map: a runtime host component never +// claims its bare id, so attributing a record to it would let an unrelated +// lock block a clear. +TEST(BuildSourceEntityMapTest, InternalComponentClaimsNoBareId) { + ThreadSafeEntityCache cache; + Component host; + host.id = "runtime_host"; + cache.update_components({host}); + + EXPECT_EQ(handlers::FaultHandlers::build_source_entity_map(cache).count("runtime_host"), 0u); +} + +// An unbound non-external app owns no reporting source at all: granting it its +// bare id would let it claim records it never reported. +TEST(BuildSourceEntityMapTest, UnboundInternalAppContributesNothing) { + ThreadSafeEntityCache cache; + App unbound; + unbound.id = "manifest_only"; + cache.update_apps({unbound}); + + EXPECT_TRUE(handlers::FaultHandlers::build_source_entity_map(cache).empty()); +} + TEST(ResolveEntitySourceFqnsTest, ExternalComponentWithNoAppsOwnsFaultsUnderItsOwnId) { // The #516 case: a PLC modeled as an external Component with no child App. // It reports faults under its own component id; the scope must be exactly @@ -2133,6 +2218,95 @@ TEST(ResolveEntitySourceFqnsTest, AreaHostingExternalComponentOwnsItsFaults) { EXPECT_EQ(fqns, std::set{"s7_1500"}); } +// ============================================================================= +// AppSourceIndex: the app whose GET /apps/{id}/faults lists a record. It names +// the owner in the fault stream's x-medkit and the entity of a fault trigger +// notification, so it must follow the scope rule and never the last path +// segment of the owner. +// ============================================================================= + +TEST(AppSourceIndexTest, SubPathSourceResolvesToOwnerNotNamesake) { + ThreadSafeEntityCache cache; + cache.update_apps({ + make_app_with_binding("detector-app", "detector", "/x"), + make_app_with_binding("goal_status", "goal_status", "/z"), + }); + faults::AppSourceIndex index; + EXPECT_EQ(index.find_owner(cache, "/x/detector/goal_status"), "detector-app"); + EXPECT_EQ(index.find_owner(cache, "/x/detector/diag/goal_status"), "detector-app"); + EXPECT_EQ(index.find_owner(cache, "/x/detector"), "detector-app"); + EXPECT_EQ(index.find_owner(cache, "/z/goal_status"), "goal_status"); +} + +TEST(AppSourceIndexTest, SourceOwnedByNoAppHasNoOwner) { + ThreadSafeEntityCache cache; + cache.update_apps({make_app_with_binding("goal_status", "goal_status", "/z")}); + faults::AppSourceIndex index; + // The last segment names an app, the path does not. + EXPECT_EQ(index.find_owner(cache, "/w/solo/goal_status"), ""); + EXPECT_EQ(index.find_owner(cache, ""), ""); +} + +TEST(AppSourceIndexTest, PrefixCollidingNodeIsNotTheOwner) { + ThreadSafeEntityCache cache; + cache.update_apps({make_app_with_binding("node", "node", "/ns")}); + faults::AppSourceIndex index; + EXPECT_EQ(index.find_owner(cache, "/ns/node_extra"), ""); +} + +TEST(AppSourceIndexTest, MostSpecificOwnerWinsForNestedNodes) { + // Node /a/b and node /a/b/c both own /a/b/c/x; the nearer one names it. + ThreadSafeEntityCache cache; + cache.update_apps({ + make_app_with_binding("outer", "b", "/a"), + make_app_with_binding("inner", "c", "/a/b"), + }); + faults::AppSourceIndex index; + EXPECT_EQ(index.find_owner(cache, "/a/b/c/x"), "inner"); + EXPECT_EQ(index.find_owner(cache, "/a/b/other"), "outer"); +} + +TEST(AppSourceIndexTest, EqualOwnersResolveToSmallestId) { + // Two apps bound to one node both list its faults; the pick is stable. + ThreadSafeEntityCache cache; + cache.update_apps({ + make_app_with_binding("manifest-bridge", "diagnostic_bridge", "/bridge"), + make_app_with_binding("diagnostic_bridge", "diagnostic_bridge", "/bridge"), + }); + faults::AppSourceIndex index; + EXPECT_EQ(index.find_owner(cache, "/bridge/diagnostic_bridge"), "diagnostic_bridge"); +} + +TEST(AppSourceIndexTest, ExternalAppOwnsItsBareId) { + ThreadSafeEntityCache cache; + cache.update_apps({make_external_app("plc-app", "plc-unit")}); + faults::AppSourceIndex index; + EXPECT_EQ(index.find_owner(cache, "plc-app"), "plc-app"); + EXPECT_EQ(index.find_owner(cache, "plc-app/channel_1"), "plc-app"); + EXPECT_EQ(index.find_owner(cache, "plc-app-2"), ""); +} + +TEST(AppSourceIndexTest, FollowsCacheChanges) { + ThreadSafeEntityCache cache; + cache.update_apps({make_app_with_binding("first", "detector", "/x")}); + faults::AppSourceIndex index; + EXPECT_EQ(index.find_owner(cache, "/x/detector/sub"), "first"); + + cache.update_apps({make_app_with_binding("second", "detector", "/x")}); + EXPECT_EQ(index.find_owner(cache, "/x/detector/sub"), "second"); + + cache.update_apps({}); + EXPECT_EQ(index.find_owner(cache, "/x/detector/sub"), ""); +} + +TEST(SourceInScopeTest, ExactOrPathChildOnly) { + EXPECT_TRUE(faults::source_in_scope("/ns/node", "/ns/node")); + EXPECT_TRUE(faults::source_in_scope("/ns/node/sub", "/ns/node")); + EXPECT_FALSE(faults::source_in_scope("/ns/node_extra", "/ns/node")); + EXPECT_FALSE(faults::source_in_scope("/ns", "/ns/node")); + EXPECT_FALSE(faults::source_in_scope("/ns/node", "")); +} + int main(int argc, char ** argv) { testing::InitGoogleTest(&argc, argv); return RUN_ALL_TESTS(); diff --git a/src/ros2_medkit_gateway/test/test_log_change_owner.cpp b/src/ros2_medkit_gateway/test/test_log_change_owner.cpp new file mode 100644 index 000000000..d2bb9ef96 --- /dev/null +++ b/src/ros2_medkit_gateway/test/test_log_change_owner.cpp @@ -0,0 +1,446 @@ +// Copyright 2026 bburda +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +// "logs" changes as a plugin sees them through PluginContext: a real +// GatewayNode in hybrid discovery, real bound nodes and real /rosout entries. + +#include +#include +#include +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "ros2_medkit_gateway/core/managers/trigger_manager.hpp" +#include "ros2_medkit_gateway/core/resource_change_notifier.hpp" +#include "ros2_medkit_gateway/gateway_node.hpp" +#include "ros2_medkit_gateway/plugins/ros_plugin_context.hpp" + +using namespace std::chrono_literals; +using ros2_medkit_gateway::NotifierFilter; +using ros2_medkit_gateway::NotifierSubscriptionId; +using ros2_medkit_gateway::ResourceChange; +using ros2_medkit_gateway::ResourceChangeNotifier; +using ros2_medkit_gateway::TriggerCreateRequest; + +namespace { + +int reserve_local_port() { + int sock = socket(AF_INET, SOCK_STREAM, 0); + if (sock < 0) { + ADD_FAILURE() << "socket: " << std::strerror(errno); + return 0; + } + sockaddr_in addr{}; + addr.sin_family = AF_INET; + addr.sin_addr.s_addr = htonl(INADDR_LOOPBACK); + addr.sin_port = 0; + if (bind(sock, reinterpret_cast(&addr), sizeof(addr)) != 0) { + ADD_FAILURE() << "bind: " << std::strerror(errno); + close(sock); + return 0; + } + socklen_t addr_len = sizeof(addr); + if (getsockname(sock, reinterpret_cast(&addr), &addr_len) != 0) { + ADD_FAILURE() << "getsockname: " << std::strerror(errno); + close(sock); + return 0; + } + int port = ntohs(addr.sin_port); + close(sock); + return port; +} + +// engine-temp-app is bound to a root node, rpm-app to a namespaced one. The +// app ids differ from the node names, so an id taken from a logger name never +// equals them. gauge is a root node whose app id is its node name. +constexpr const char * kManifest = R"( +manifest_version: "1.0" +metadata: + name: log-change-owner + version: "0.0.1" +config: + unmanifested_nodes: ignore +areas: + - id: vehicle + name: Vehicle +components: + - id: ecu + name: ECU + area: vehicle +apps: + - id: engine-temp-app + name: Engine temperature + is_located_on: ecu + ros_binding: + node_name: engine_temp + namespace: / + - id: rpm-app + name: Engine RPM + is_located_on: ecu + ros_binding: + node_name: rpm + namespace: /powertrain/engine + - id: gauge + name: Gauge + is_located_on: ecu + ros_binding: + node_name: gauge + namespace: / +)"; + +constexpr auto kDeliveryTimeout = 20s; + +/// Records the "logs" changes one notifier subscription receives. +class Recorder { + public: + Recorder(ResourceChangeNotifier & notifier, NotifierFilter filter) : notifier_(notifier) { + id_ = notifier_.subscribe(std::move(filter), [this](const ResourceChange & change) { + std::lock_guard lock(mutex_); + changes_.push_back(change); + }); + } + ~Recorder() { + notifier_.unsubscribe(id_); + } + Recorder(const Recorder &) = delete; + Recorder & operator=(const Recorder &) = delete; + Recorder(Recorder &&) = delete; + Recorder & operator=(Recorder &&) = delete; + + /// The change whose entry message is `message`, if one arrived. + std::optional find(const std::string & message) const { + std::lock_guard lock(mutex_); + for (const auto & change : changes_) { + if (change.value.value("message", std::string{}) == message) { + return change; + } + } + return std::nullopt; + } + + /// The changes whose entry message starts with `prefix`. + std::vector with_prefix(const std::string & prefix) const { + std::lock_guard lock(mutex_); + std::vector out; + for (const auto & change : changes_) { + if (change.value.value("message", std::string{}).rfind(prefix, 0) == 0) { + out.push_back(change); + } + } + return out; + } + + private: + ResourceChangeNotifier & notifier_; + NotifierSubscriptionId id_{0}; + mutable std::mutex mutex_; + std::vector changes_; +}; + +class LogChangeOwnerTest : public ::testing::Test { + protected: + static void SetUpTestSuite() { + if (!rclcpp::ok()) { + rclcpp::init(0, nullptr); + } + } + + static void TearDownTestSuite() { + if (rclcpp::ok()) { + rclcpp::shutdown(); + } + } + + void SetUp() override { + const auto * info = ::testing::UnitTest::GetInstance()->current_test_info(); + const std::string name = info ? std::string(info->name()) : std::string("unknown"); + work_dir_ = + std::filesystem::temp_directory_path() / ("medkit-log-change-owner-" + std::to_string(::getpid()) + "-" + name); + std::error_code ec; + std::filesystem::remove_all(work_dir_, ec); + std::filesystem::create_directories(work_dir_); + const auto manifest_path = work_dir_ / "manifest.yaml"; + std::ofstream(manifest_path) << kManifest; + + const int port = reserve_local_port(); + ASSERT_GT(port, 0); + rclcpp::NodeOptions opts; + opts.parameter_overrides({ + {"discovery.mode", "hybrid"}, + {"discovery.manifest_path", manifest_path.string()}, + {"discovery.manifest_strict_validation", false}, + {"discovery.merge_pipeline.gap_fill.allow_heuristic_apps", false}, + {"refresh_interval_ms", 1000}, + {"server.host", std::string("127.0.0.1")}, + {"server.port", port}, + }); + gateway_ = std::make_shared(opts); + root_node_ = std::make_shared("engine_temp", "/"); + ns_node_ = std::make_shared("rpm", "/powertrain/engine"); + gauge_node_ = std::make_shared("gauge", "/"); + probe_ = std::make_shared("log_owner_probe", "/probe"); + rosout_ = probe_->create_publisher("/rosout", rclcpp::QoS(1000).reliable()); + + ctx_ = ros2_medkit_gateway::make_gateway_plugin_context(gateway_.get(), gateway_->get_fault_manager()); + notifier_ = ctx_->get_resource_change_notifier(); + ASSERT_NE(notifier_, nullptr); + + executor_ = std::make_unique(); + executor_->add_node(gateway_); + executor_->add_node(root_node_); + executor_->add_node(ns_node_); + executor_->add_node(gauge_node_); + executor_->add_node(probe_); + } + + void TearDown() override { + recorders_.clear(); + if (executor_) { + executor_->cancel(); + } + executor_.reset(); + ctx_.reset(); + rosout_.reset(); + probe_.reset(); + gauge_node_.reset(); + ns_node_.reset(); + root_node_.reset(); + gateway_.reset(); + std::error_code ec; + std::filesystem::remove_all(work_dir_, ec); + } + + Recorder & record(NotifierFilter filter) { + recorders_.push_back(std::make_unique(*notifier_, std::move(filter))); + return *recorders_.back(); + } + + void publish(const std::string & logger, const std::string & message) { + rcl_interfaces::msg::Log msg; + msg.stamp = probe_->now(); + msg.level = rcl_interfaces::msg::Log::INFO; + msg.name = logger; + msg.msg = message; + msg.file = "test_log_change_owner.cpp"; + msg.function = "publish"; + msg.line = 1; + rosout_->publish(msg); + } + + /// A multishot logs trigger on an app, with log_settings. + std::string create_logs_trigger(const std::string & app_id, const std::string & marker) { + TriggerCreateRequest req; + req.entity_id = app_id; + req.entity_type = "apps"; + req.resource_uri = "/api/v1/apps/" + app_id + "/logs"; + req.collection = "logs"; + req.condition_type = "OnChange"; + req.condition_params = nlohmann::json::object(); + req.multishot = true; + req.lifetime_sec = 120; + req.log_settings = nlohmann::json{{"severity", "warning"}, {"marker", marker}}; + auto created = gateway_->get_trigger_manager()->create(req); + EXPECT_TRUE(created.has_value()) << (created ? "" : created.error().message); + return created ? created->id : std::string{}; + } + + bool spin_until(const std::function & done, std::chrono::nanoseconds timeout) { + const auto deadline = std::chrono::steady_clock::now() + timeout; + while (std::chrono::steady_clock::now() < deadline) { + executor_->spin_some(50ms); + if (done()) { + return true; + } + std::this_thread::sleep_for(10ms); + } + return done(); + } + + /// Publish numbered entries on `logger` until `recorder` gets one; discovery + /// and linking take a few refresh cycles. Returns the change received. + std::optional publish_until_received(Recorder & recorder, const std::string & logger, + const std::string & tag) { + std::optional got; + int attempt = 0; + auto next_publish = std::chrono::steady_clock::now(); + spin_until( + [&]() { + for (int i = 1; i <= attempt && !got; ++i) { + got = recorder.find(tag + "-" + std::to_string(i)); + } + if (got) { + return true; + } + if (std::chrono::steady_clock::now() >= next_publish) { + publish(logger, tag + "-" + std::to_string(++attempt)); + next_publish = std::chrono::steady_clock::now() + 500ms; + } + return false; + }, + kDeliveryTimeout); + return got; + } + + std::filesystem::path work_dir_; + std::shared_ptr gateway_; + std::shared_ptr root_node_; + std::shared_ptr ns_node_; + std::shared_ptr gauge_node_; + std::shared_ptr probe_; + rclcpp::Publisher::SharedPtr rosout_; + std::unique_ptr ctx_; + ResourceChangeNotifier * notifier_{nullptr}; + std::unique_ptr executor_; + std::vector> recorders_; +}; + +} // namespace + +TEST_F(LogChangeOwnerTest, RootNodeEntriesReachTheBoundAppId) { + auto & app = record(NotifierFilter{"logs", "engine-temp-app", ""}); + + auto got = publish_until_received(app, "engine_temp", "root"); + + ASSERT_TRUE(got.has_value()) << "a plugin filtering on engine-temp-app got no entry of logger engine_temp"; + EXPECT_EQ(got->entity_id, "engine-temp-app"); + EXPECT_EQ(got->value["context"]["node"], "engine_temp"); +} + +TEST_F(LogChangeOwnerTest, SlashFormEntriesReachTheBoundAppId) { + auto & app = record(NotifierFilter{"logs", "rpm-app", ""}); + + auto got = publish_until_received(app, "powertrain/engine/rpm", "slash"); + + ASSERT_TRUE(got.has_value()) << "a plugin filtering on rpm-app got no entry of logger powertrain/engine/rpm"; + EXPECT_EQ(got->entity_id, "rpm-app"); + EXPECT_EQ(got->value["context"]["node"], "powertrain/engine/rpm"); +} + +TEST_F(LogChangeOwnerTest, DotFormEntriesReachTheBoundAppId) { + auto & app = record(NotifierFilter{"logs", "rpm-app", ""}); + + auto got = publish_until_received(app, "powertrain.engine.rpm", "dot"); + + ASSERT_TRUE(got.has_value()) << "a plugin filtering on rpm-app got no entry of logger powertrain.engine.rpm"; + EXPECT_EQ(got->entity_id, "rpm-app"); + EXPECT_EQ(got->value["context"]["node"], "powertrain.engine.rpm"); +} + +TEST_F(LogChangeOwnerTest, LoggerNoAppOwnsCarriesNoEntityId) { + auto & app = record(NotifierFilter{"logs", "rpm-app", ""}); + auto & all = record(NotifierFilter{"logs", "", ""}); + // Owned by rpm-app; its arrival proves the app-filtered stream is live. + ASSERT_TRUE(publish_until_received(app, "powertrain.engine.rpm", "live").has_value()); + + // A child logger and a logger of no node: GET /apps/rpm-app/logs lists neither. + publish("powertrain.engine.rpm.worker", "child"); + publish("stranger", "stranger"); + publish("powertrain.engine.rpm", "sentinel"); + ASSERT_TRUE(spin_until( + [&]() { + return app.find("sentinel").has_value() && all.find("child").has_value() && all.find("stranger").has_value(); + }, + kDeliveryTimeout)); + + EXPECT_FALSE(app.find("child").has_value()); + EXPECT_FALSE(app.find("stranger").has_value()); + EXPECT_EQ(all.find("child")->entity_id, ""); + EXPECT_EQ(all.find("child")->value["context"]["node"], "powertrain.engine.rpm.worker"); + EXPECT_EQ(all.find("stranger")->entity_id, ""); + EXPECT_EQ(all.find("stranger")->value["context"]["node"], "stranger"); +} + +// A marker a trigger's log_settings writes reaches plugins as one "logs" +// CREATED change: entity_id of the app that owns its logger name (the +// trigger's entity id), or "". +TEST_F(LogChangeOwnerTest, TriggerMarkerReachesPluginsAsOneLogsChange) { + auto & all = record(NotifierFilter{"logs", "", ""}); + auto & app = record(NotifierFilter{"logs", "engine-temp-app", ""}); + ASSERT_TRUE(publish_until_received(app, "engine_temp", "live").has_value()); + + const auto trigger_id = create_logs_trigger("engine-temp-app", "pin-marker"); + ASSERT_FALSE(trigger_id.empty()); + publish("engine_temp", "pin-real"); + ASSERT_TRUE(spin_until( + [&]() { + return !all.with_prefix("pin-marker ").empty(); + }, + kDeliveryTimeout)) + << "no marker change reached the plugin"; + publish("engine_temp", "pin-sentinel"); + ASSERT_TRUE(spin_until( + [&]() { + return all.find("pin-sentinel").has_value(); + }, + kDeliveryTimeout)); + spin_until( + [&]() { + return all.with_prefix("pin-marker ").size() >= 2; + }, + 2s); + + const auto markers = all.with_prefix("pin-marker "); + ASSERT_EQ(markers.size(), 2u) << "one change per firing: pin-real and pin-sentinel"; + const auto & change = markers.front(); + EXPECT_EQ(change.collection, "logs"); + EXPECT_EQ(change.entity_id, ""); + EXPECT_EQ(change.resource_path, ""); + EXPECT_EQ(change.change_type, ros2_medkit_gateway::ChangeType::CREATED); + EXPECT_EQ(change.value["severity"], "warning"); + EXPECT_EQ(change.value["context"], nlohmann::json({{"node", "engine-temp-app"}})); + const nlohmann::json metadata{ + {"trigger_id", trigger_id}, {"condition_type", "OnChange"}, {"resource", "/api/v1/apps/engine-temp-app/logs"}}; + EXPECT_EQ(change.value["message"], "pin-marker " + metadata.dump()); + EXPECT_EQ(change.value["id"].get().rfind("log_", 0), 0u); + EXPECT_TRUE(app.with_prefix("pin-marker ").empty()) << "no app owns the logger engine-temp-app"; + gateway_->get_trigger_manager()->remove(trigger_id); +} + +TEST_F(LogChangeOwnerTest, TriggerMarkerUnderAnAppLoggerReachesThatApp) { + auto & gauge = record(NotifierFilter{"logs", "gauge", ""}); + ASSERT_TRUE(publish_until_received(gauge, "gauge", "live").has_value()); + + const auto trigger_id = create_logs_trigger("gauge", "gauge-marker"); + ASSERT_FALSE(trigger_id.empty()); + publish("gauge", "gauge-real"); + ASSERT_TRUE(spin_until( + [&]() { + return !gauge.with_prefix("gauge-marker ").empty(); + }, + kDeliveryTimeout)) + << "a plugin filtering on gauge got no marker of the gauge trigger"; + gateway_->get_trigger_manager()->remove(trigger_id); + + const auto change = gauge.with_prefix("gauge-marker ").front(); + EXPECT_EQ(change.entity_id, "gauge"); + EXPECT_EQ(change.change_type, ros2_medkit_gateway::ChangeType::CREATED); + EXPECT_EQ(change.value["context"], nlohmann::json({{"node", "gauge"}})); + EXPECT_EQ(change.value["severity"], "warning"); +} diff --git a/src/ros2_medkit_gateway/test/test_log_manager.cpp b/src/ros2_medkit_gateway/test/test_log_manager.cpp index 83a2f5573..da027be83 100644 --- a/src/ros2_medkit_gateway/test/test_log_manager.cpp +++ b/src/ros2_medkit_gateway/test/test_log_manager.cpp @@ -17,9 +17,14 @@ #include #include +#include #include #include +#include +#include #include +#include +#include #include "ros2_medkit_gateway/core/managers/log_manager.hpp" #include "ros2_medkit_gateway/core/plugins/gateway_plugin.hpp" @@ -744,25 +749,30 @@ TEST_F(LogManagerIngestionTest, PluginGetConfigThrowReturnsError) { } // ============================================================ -// Resolver notification tests — verify on_rosout() resolves -// node names to entity IDs and passes them to the notifier +// Notification tests - a /rosout entry reaches the notifier with +// its logger name, whatever the form // ============================================================ -class LogManagerResolverTest : public ::testing::Test { +class LogManagerNotifyTest : public ::testing::Test { protected: void SetUp() override { rclcpp::init(0, nullptr); - node_ = std::make_shared("test_log_resolver"); + node_ = std::make_shared("test_log_notify"); notifier_ = std::make_unique(); mgr_ = std::make_unique(std::make_shared(node_.get()), nullptr, 10); + mgr_->set_logger_owner_fn([](const std::string & logger_name) { + return logger_name == "powertrain/engine/temp_sensor" || logger_name == "powertrain.engine.temp_sensor" + ? std::string("engine-temp") + : std::string(); + }); mgr_->set_notifier(notifier_.get()); - // Subscribe to notifier to capture entity IDs + // Capture (entity_id, logger name) of every logs change notifier_->subscribe(ros2_medkit_gateway::NotifierFilter{"logs", "", ""}, [this](const ros2_medkit_gateway::ResourceChange & change) { std::lock_guard lock(captured_mutex_); - captured_entity_ids_.push_back(change.entity_id); + captured_.emplace_back(change.entity_id, change.value["context"].value("node", "")); }); // Create /rosout publisher for sending test log messages @@ -789,7 +799,7 @@ class LogManagerResolverTest : public ::testing::Test { std::this_thread::sleep_for(std::chrono::milliseconds(50)); // Clear whatever was captured during startup std::lock_guard lock(captured_mutex_); - captured_entity_ids_.clear(); + captured_.clear(); } void publish_and_spin(const std::string & logger_name) { @@ -798,25 +808,31 @@ class LogManagerResolverTest : public ::testing::Test { msg.name = logger_name; msg.msg = "test message"; rosout_pub_->publish(msg); - // Spin to process the subscription callback + // Spin until the notifier worker has seen this logger name. auto start = std::chrono::steady_clock::now(); - while (std::chrono::steady_clock::now() - start < std::chrono::seconds(2)) { + while (std::chrono::steady_clock::now() - start < std::chrono::seconds(5)) { rclcpp::spin_some(node_); { std::lock_guard lock(captured_mutex_); - if (!captured_entity_ids_.empty()) { - break; + if (std::any_of(captured_.begin(), captured_.end(), [&](const auto & c) { + return c.second == logger_name; + })) { + return; } } std::this_thread::sleep_for(std::chrono::milliseconds(10)); } - // Allow notifier worker thread to process - std::this_thread::sleep_for(std::chrono::milliseconds(100)); } - std::vector get_captured() { + /// Entity id of the first change for `logger_name`, nullopt when none came. + std::optional entity_id_for(const std::string & logger_name) { std::lock_guard lock(captured_mutex_); - return captured_entity_ids_; + for (const auto & [entity_id, logger] : captured_) { + if (logger == logger_name) { + return entity_id; + } + } + return std::nullopt; } std::shared_ptr node_; @@ -825,64 +841,20 @@ class LogManagerResolverTest : public ::testing::Test { rclcpp::Publisher::SharedPtr rosout_pub_; std::mutex captured_mutex_; - std::vector captured_entity_ids_; + std::vector> captured_; }; -// @verifies REQ_INTEROP_061 -TEST_F(LogManagerResolverTest, ResolverMatchesWithSlashPrefixReturnsEntityId) { - // Resolver returns "temp_sensor_app" when called with "/powertrain/temp_sensor" - mgr_->set_node_to_entity_resolver([](const std::string & fqn) -> std::string { - if (fqn == "/powertrain/temp_sensor") { - return "temp_sensor_app"; - } - return ""; - }); - - // Drain startup log messages that arrive on /rosout before our test message +TEST_F(LogManagerNotifyTest, RosoutEntryCarriesItsOwnerAndLoggerName) { drain_initial_messages(); // /rosout msg.name has no leading slash per rcl convention - publish_and_spin("powertrain/temp_sensor"); - - auto captured = get_captured(); - ASSERT_FALSE(captured.empty()) << "Notifier should have received at least one event"; - EXPECT_EQ(captured[0], "temp_sensor_app"); -} - -// @verifies REQ_INTEROP_061 -TEST_F(LogManagerResolverTest, ResolverMatchesBareNameReturnsEntityId) { - // Resolver returns "my_app" when called with bare name (no leading slash) - mgr_->set_node_to_entity_resolver([](const std::string & fqn) -> std::string { - if (fqn == "simple_node") { - return "my_app"; - } - return ""; - }); - - drain_initial_messages(); - - publish_and_spin("simple_node"); - - auto captured = get_captured(); - ASSERT_FALSE(captured.empty()) << "Notifier should have received at least one event"; - EXPECT_EQ(captured[0], "my_app"); -} - -// @verifies REQ_INTEROP_061 -TEST_F(LogManagerResolverTest, ResolverReturnsEmptyFallsBackToLastSegment) { - // Resolver always returns empty -> on_rosout falls back to last path segment - mgr_->set_node_to_entity_resolver([](const std::string & /*fqn*/) -> std::string { - return ""; - }); - - drain_initial_messages(); - publish_and_spin("powertrain/engine/temp_sensor"); + publish_and_spin("powertrain.engine.temp_sensor"); + publish_and_spin("temp_sensor"); - auto captured = get_captured(); - ASSERT_FALSE(captured.empty()) << "Notifier should have received at least one event"; - // Fallback: last segment of "powertrain/engine/temp_sensor" is "temp_sensor" - EXPECT_EQ(captured[0], "temp_sensor"); + EXPECT_EQ(entity_id_for("powertrain/engine/temp_sensor"), "engine-temp"); + EXPECT_EQ(entity_id_for("powertrain.engine.temp_sensor"), "engine-temp"); + EXPECT_EQ(entity_id_for("temp_sensor"), "") << "a logger no app owns names no entity"; } // ============================================================ diff --git a/src/ros2_medkit_gateway/test/test_log_manager_routing.cpp b/src/ros2_medkit_gateway/test/test_log_manager_routing.cpp index be1a55cff..332cbbd24 100644 --- a/src/ros2_medkit_gateway/test/test_log_manager_routing.cpp +++ b/src/ros2_medkit_gateway/test/test_log_manager_routing.cpp @@ -21,12 +21,18 @@ #include +#include #include +#include +#include +#include #include +#include #include #include #include #include +#include #include #include #include @@ -35,6 +41,7 @@ #include "ros2_medkit_gateway/core/managers/log_manager.hpp" #include "ros2_medkit_gateway/core/providers/log_provider.hpp" #include "ros2_medkit_gateway/core/providers/log_provider_registry.hpp" +#include "ros2_medkit_gateway/core/resource_change_notifier.hpp" #include "ros2_medkit_gateway/core/transports/log_source.hpp" namespace ros2_medkit_gateway { @@ -489,4 +496,156 @@ TEST(LogManagerRouting, NullLogSinkIsSafe) { ASSERT_TRUE(source->emit(make_entry("my_node", /*level=*/20, "hi"))); } +// ============================================================ +// Trigger notifications +// ============================================================ + +/// Records (entity_id, logger name) of every "logs" change, in delivery order. +class LogChangeRecorder { + public: + using Change = std::pair; + + explicit LogChangeRecorder(ResourceChangeNotifier & notifier) { + notifier.subscribe(NotifierFilter{"logs", "", ""}, [this](const ResourceChange & change) { + std::lock_guard lock(mutex_); + changes_.emplace_back(change.entity_id, change.value["context"].value("node", "")); + cv_.notify_all(); + }); + } + + /// Wait until `logger` was recorded `count` times; return everything + /// recorded so far. + std::vector wait_for(const std::string & logger, std::ptrdiff_t count = 1) { + std::unique_lock lock(mutex_); + cv_.wait_for(lock, std::chrono::seconds(5), [&] { + return std::count_if(changes_.begin(), changes_.end(), [&](const Change & c) { + return c.second == logger; + }) >= count; + }); + return changes_; + } + + private: + std::mutex mutex_; + std::condition_variable cv_; + std::vector changes_; +}; + +TEST(LogManagerRouting, ChangeCarriesTheOwnerAndTheLoggerName) { + ResourceChangeNotifier notifier; + LogChangeRecorder recorder(notifier); + auto source = std::make_shared(); + LogManager manager(source, /*plugin_mgr=*/nullptr, /*max_buffer_size=*/10); + manager.set_logger_owner_fn([](const std::string & logger) { + return logger == "powertrain.engine.temp_sensor" ? std::string("engine-temp") : std::string(); + }); + manager.set_notifier(¬ifier); + + ASSERT_TRUE(source->emit(make_entry("powertrain.engine.temp_sensor", /*level=*/20, "owned"))); + ASSERT_TRUE(source->emit(make_entry("powertrain/engine", /*level=*/20, "unowned"))); + manager.add_log_entry("powertrain.engine.temp_sensor", "warning", "programmatic", json::object()); + + auto seen = recorder.wait_for("powertrain.engine.temp_sensor", 2); + EXPECT_EQ(seen, (std::vector{{"engine-temp", "powertrain.engine.temp_sensor"}, + {"", "powertrain/engine"}, + {"engine-temp", "powertrain.engine.temp_sensor"}})); + notifier.shutdown(); +} + +TEST(LogManagerRouting, ChangeNamesNoEntityWithoutAnOwnerFunction) { + ResourceChangeNotifier notifier; + LogChangeRecorder recorder(notifier); + auto source = std::make_shared(); + LogManager manager(source, /*plugin_mgr=*/nullptr, /*max_buffer_size=*/10); + manager.set_notifier(¬ifier); + + ASSERT_TRUE(source->emit(make_entry("temp_sensor", /*level=*/20, "root"))); + + auto seen = recorder.wait_for("temp_sensor"); + EXPECT_EQ(seen, (std::vector{{"", "temp_sensor"}})); + notifier.shutdown(); +} + +TEST(LogManagerRouting, EntryLevelIsExactForLevelsWithoutASeverityName) { + ResourceChangeNotifier notifier; + std::mutex mutex; + std::vector values; + notifier.subscribe(NotifierFilter{"logs", "", ""}, [&](const ResourceChange & change) { + std::lock_guard lock(mutex); + values.push_back(change.value); + }); + auto source = std::make_shared(); + LogManager manager(source, /*plugin_mgr=*/nullptr, /*max_buffer_size=*/10); + manager.set_notifier(¬ifier); + + const std::vector levels = {25, 0, 99, 9, 20, 50, 255}; + for (auto level : levels) { + ASSERT_TRUE(source->emit(make_entry("node", level, "level " + std::to_string(level)))); + } + const auto deadline = std::chrono::steady_clock::now() + std::chrono::seconds(5); + while (std::chrono::steady_clock::now() < deadline) { + { + std::lock_guard lock(mutex); + if (values.size() == levels.size()) { + break; + } + } + std::this_thread::sleep_for(std::chrono::milliseconds(10)); + } + notifier.shutdown(); + ASSERT_EQ(values.size(), levels.size()); + for (std::size_t i = 0; i < levels.size(); ++i) { + EXPECT_EQ(manager.entry_level(values[i]), levels[i]) << values[i].dump(); + } +} + +TEST(LogManagerRouting, EntryLevelFallsBackToTheSeverityNamePastWhatTheNotifierHolds) { + ResourceChangeNotifier notifier; + notifier.set_max_queue_size(5); + ASSERT_EQ(notifier.max_in_flight(), 10u); + auto source = std::make_shared(); + LogManager manager(source, /*plugin_mgr=*/nullptr, /*max_buffer_size=*/10); + manager.set_notifier(¬ifier); + + for (std::size_t i = 0; i <= notifier.max_in_flight(); ++i) { + ASSERT_TRUE(source->emit(make_entry("node", /*level=*/25, "unnamed"))); + } + notifier.shutdown(); + // Ids start at 1: the first entry is past what the notifier holds, the second is not. + EXPECT_EQ(manager.entry_level(json{{"id", "log_1"}, {"severity", "debug"}}), 10); + EXPECT_EQ(manager.entry_level(json{{"id", "log_2"}, {"severity", "debug"}}), 25); + // Not an entry id: the severity name decides. + EXPECT_EQ(manager.entry_level(json{{"id", "log_2x"}, {"severity", "error"}}), 40); + EXPECT_EQ(manager.entry_level(json{{"severity", "fatal"}}), 50); + EXPECT_EQ(manager.entry_level(json(7)), 0); +} + +TEST(LogManagerRouting, EntryDroppedByTheCapEmitsNoChange) { + ResourceChangeNotifier notifier; + LogChangeRecorder recorder(notifier); + auto source = std::make_shared(); + // Cap = 1 x 10 distinct logger names. + LogManager manager(source, /*plugin_mgr=*/nullptr, /*max_buffer_size=*/1); + manager.set_notifier(¬ifier); + + for (int i = 0; i < 10; ++i) { + ASSERT_TRUE(source->emit(make_entry("node_" + std::to_string(i), /*level=*/20, "in-cap"))); + } + ASSERT_TRUE(source->emit(make_entry("over_cap_node", /*level=*/20, "dropped"))); + ASSERT_TRUE(source->emit(make_entry("node_0", /*level=*/20, "kept"))); + ASSERT_EQ(manager.dropped_entries_count(), 1u); + + // The worker delivers in order, so node_0's second change comes after + // anything the dropped entry could have emitted. + auto seen = recorder.wait_for("node_0", 2); + auto count = [&](const std::string & logger) { + return std::count_if(seen.begin(), seen.end(), [&](const LogChangeRecorder::Change & c) { + return c.second == logger; + }); + }; + EXPECT_EQ(count("node_0"), 2); + EXPECT_EQ(count("over_cap_node"), 0); + notifier.shutdown(); +} + } // namespace ros2_medkit_gateway diff --git a/src/ros2_medkit_gateway/test/test_log_scope.cpp b/src/ros2_medkit_gateway/test/test_log_scope.cpp index 7367d4d07..ec6bf5fd1 100644 --- a/src/ros2_medkit_gateway/test/test_log_scope.cpp +++ b/src/ros2_medkit_gateway/test/test_log_scope.cpp @@ -14,18 +14,33 @@ #include +#include +#include +#include +#include #include +#include #include +#include #include +#include #include +#include +#include +#include "ros2_medkit_gateway/core/condition_evaluator.hpp" #include "ros2_medkit_gateway/core/discovery/models/app.hpp" #include "ros2_medkit_gateway/core/discovery/models/area.hpp" #include "ros2_medkit_gateway/core/discovery/models/component.hpp" #include "ros2_medkit_gateway/core/discovery/models/function.hpp" #include "ros2_medkit_gateway/core/logs/log_scope.hpp" #include "ros2_medkit_gateway/core/managers/log_manager.hpp" +#include "ros2_medkit_gateway/core/managers/trigger_manager.hpp" #include "ros2_medkit_gateway/core/models/thread_safe_entity_cache.hpp" +#include "ros2_medkit_gateway/core/providers/log_provider.hpp" +#include "ros2_medkit_gateway/core/providers/log_provider_registry.hpp" +#include "ros2_medkit_gateway/core/resource_change_notifier.hpp" +#include "ros2_medkit_gateway/core/sqlite_trigger_store.hpp" #include "ros2_medkit_gateway/core/transports/log_source.hpp" using json = nlohmann::json; @@ -35,12 +50,31 @@ namespace { class NullLogSource : public LogSource { public: - void start(EntryCallback) override { + void start(EntryCallback /*callback*/) override { } void stop() override { } }; +/// Source that hands each emitted entry to the LogManager, as /rosout does. +class EmittingLogSource : public LogSource { + public: + void start(EntryCallback callback) override { + callback_ = std::move(callback); + } + void stop() override { + callback_ = nullptr; + } + void emit(const LogEntry & entry) { + if (callback_) { + callback_(entry); + } + } + + private: + EntryCallback callback_; +}; + std::set messages_of(const json & items) { std::set out; for (const auto & item : items) { @@ -211,3 +245,377 @@ TEST_F(LogScopeTest, UnknownTypeReturnsEmpty) { ASSERT_TRUE(result.has_value()); EXPECT_TRUE(result->empty()); } + +// ============================================================================= +// logs::entity_lists_log_entry +// +// The trigger-side predicate must agree with the query for every entity type +// and logger-name form: a logger is in scope exactly when query_entity_logs +// lists its entries. +// ============================================================================= + +TEST_F(LogScopeTest, PredicateAgreesWithQueryForEveryLoggerForm) { + const std::vector loggers = { + "plant.temp", "plant.temp.child", "plant/temp/child", "plant.node2", "plantx/node", "plant", + "temp", "ext-plc.child", "plc-box/sub", "other.node", "plant_cell/child", "plant_cell"}; + for (const auto & name : loggers) { + mgr_.add_log_entry(name, "info", "entry of " + name, {}); + } + + const std::vector> entities = {{SovdEntityType::APP, "temp"}, + {SovdEntityType::APP, "ext-plc"}, + {SovdEntityType::COMPONENT, "plant_cell"}, + {SovdEntityType::COMPONENT, "plc-box"}, + {SovdEntityType::AREA, "cell_area"}, + {SovdEntityType::FUNCTION, "line_func"}}; + + std::vector all_loggers = loggers; + for (const auto & name : {"ext-plc", "plant/node1", "plant/temp", "plc-box", "other/node"}) { + all_loggers.emplace_back(name); + } + + for (const auto & [type, id] : entities) { + auto result = logs::query_entity_logs(mgr_, cache_, type, id, logs::entity_log_fqn(cache_, type, id), "", ""); + ASSERT_TRUE(result.has_value()); + std::set listed; + for (const auto & item : *result) { + listed.insert(item["context"].value("node", "")); + } + for (const auto & name : all_loggers) { + EXPECT_EQ(logs::entity_lists_log_entry(mgr_, cache_, type, id, name, 20), listed.count(name) == 1) + << "entity " << id << ", logger " << name; + } + } +} + +TEST_F(LogScopeTest, PredicateHonoursTheEntitySeverityFilter) { + ASSERT_EQ(mgr_.update_config("temp", std::string("warning"), std::nullopt), ""); + EXPECT_FALSE(logs::entity_lists_log_entry(mgr_, cache_, SovdEntityType::APP, "temp", "plant/temp", 20)); + EXPECT_TRUE(logs::entity_lists_log_entry(mgr_, cache_, SovdEntityType::APP, "temp", "plant/temp", 30)); + // The filter belongs to the entity: the component still lists info entries. + EXPECT_TRUE(logs::entity_lists_log_entry(mgr_, cache_, SovdEntityType::COMPONENT, "plant_cell", "plant/temp", 20)); +} + +TEST_F(LogScopeTest, PredicateNeverUsesTheLastPathSegment) { + // "other/node/temp" ends in the id of app temp; the app owns only its FQN. + EXPECT_FALSE(logs::entity_lists_log_entry(mgr_, cache_, SovdEntityType::APP, "temp", "other/node/temp", 20)); + EXPECT_FALSE(logs::entity_lists_log_entry(mgr_, cache_, SovdEntityType::APP, "temp", "temp", 20)); +} + +TEST_F(LogScopeTest, ChangePredicateReadsLoggerAndSeverityFromTheEntry) { + const json entry = {{"id", "log_1"}, {"severity", "info"}, {"context", {{"node", "plant.temp"}}}}; + EXPECT_TRUE(logs::entity_lists_log_change(mgr_, cache_, SovdEntityType::APP, "temp", entry)); + EXPECT_FALSE(logs::entity_lists_log_change(mgr_, cache_, SovdEntityType::APP, "ext-plc", entry)); + // Values a plugin may notify under "logs" without an entry's shape. + for (const json & odd : {json(42), json::object(), json{{"context", "plant.temp"}}, + json{{"context", {{"node", 7}}}, {"severity", "info"}}}) { + EXPECT_FALSE(logs::entity_lists_log_change(mgr_, cache_, SovdEntityType::APP, "temp", odd)) << odd.dump(); + } +} + +// ============================================================================= +// logs::LoggerOwnerIndex +// ============================================================================= + +TEST_F(LogScopeTest, OwnerIsTheOneAppWhoseListingHoldsTheLogger) { + const std::vector loggers = { + "plant.temp", "plant/temp", "plant.temp.child", "temp", "ext-plc", + "ext-plc.child", "plant", "plc-box", "other/node", "plant/node1"}; + for (const auto & name : loggers) { + mgr_.add_log_entry(name, "info", "entry of " + name, {}); + } + logs::LoggerOwnerIndex index; + for (const auto & name : loggers) { + std::vector listing_apps; + for (const auto & app : cache_.get_apps()) { + auto result = logs::query_entity_logs(mgr_, cache_, SovdEntityType::APP, app.id, + logs::entity_log_fqn(cache_, SovdEntityType::APP, app.id), "", ""); + ASSERT_TRUE(result.has_value()); + for (const auto & item : *result) { + if (item["context"].value("node", "") == name) { + listing_apps.push_back(app.id); + break; + } + } + } + const std::string expected = listing_apps.size() == 1 ? listing_apps.front() : ""; + EXPECT_EQ(index.find_owner(cache_, name), expected) << "logger " << name; + } +} + +TEST_F(LogScopeTest, OwnerIsEmptyWhenTwoAppsHoldTheLogger) { + logs::LoggerOwnerIndex index; + ASSERT_EQ(index.find_owner(cache_, "plant.temp"), "temp"); + + // A second app bound to the same node, published by a later discovery pass. + auto apps = cache_.get_apps(); + App twin = apps.front().id == "temp" ? apps.front() : apps.back(); + twin.id = "temp-twin"; + apps.push_back(twin); + cache_.update_all(cache_.get_areas(), cache_.get_components(), apps, cache_.get_functions()); + + EXPECT_EQ(index.find_owner(cache_, "plant.temp"), ""); + EXPECT_EQ(index.find_owner(cache_, "plant/temp"), ""); + EXPECT_EQ(index.find_owner(cache_, "ext-plc"), "ext-plc"); +} + +// ============================================================================= +// Exact levels of changes waiting for dispatch +// ============================================================================= + +// The notifier's worker drains its whole queue as one batch and the queue +// refills while the batch is dispatched, so a batch and a full queue can wait +// at once. Every one of them must still be judged on its exact level. +TEST_F(LogScopeTest, LevelsOfEveryWaitingChangeReachTheTriggerDecision) { + const std::size_t queue_size = ResourceChangeNotifier::kDefaultMaxQueueSize; + auto source = std::make_shared(); + LogManager mgr(source, nullptr, /*max_buffer_size=*/2 * queue_size + 1); + ASSERT_EQ(mgr.update_config("temp", std::string("info"), 4 * queue_size), ""); + + // The first change of each of the first two batches blocks the worker until + // the test releases it. + std::mutex mutex; + std::condition_variable cv; + int calls = 0; + int entered = 0; + int released = 0; + std::set admitted; + std::size_t decided = 0; + ResourceChangeNotifier notifier; + mgr.set_notifier(¬ifier); + // Unblocks the worker on every exit path, before the notifier joins it. + // Not an aggregate under C++20: the deleted copy and move make the constructor necessary. + struct Unblock { + Unblock(std::mutex & mutex_ref, std::condition_variable & cv_ref, int & released_ref) + : m(mutex_ref), c(cv_ref), r(released_ref) { + } + std::mutex & m; + std::condition_variable & c; + int & r; + ~Unblock() { + std::lock_guard lock(m); + r = 2; + c.notify_all(); + } + Unblock(const Unblock &) = delete; + Unblock & operator=(const Unblock &) = delete; + Unblock(Unblock &&) = delete; + Unblock & operator=(Unblock &&) = delete; + } unblock{mutex, cv, released}; + notifier.subscribe(NotifierFilter{"logs", "", ""}, [&](const ResourceChange & change) { + std::unique_lock lock(mutex); + const int call = calls++; + if (call < 2) { + entered = call + 1; + cv.notify_all(); + cv.wait(lock, [&] { + return released > call; + }); + } + lock.unlock(); + const bool lists = logs::entity_lists_log_change(mgr, cache_, SovdEntityType::APP, "temp", change.value); + lock.lock(); + if (lists) { + admitted.insert(change.value.value("id", "")); + } + ++decided; + cv.notify_all(); + }); + + std::size_t emitted = 0; + auto emit = [&]() { + LogEntry entry{}; + entry.stamp_sec = 1; + // 25 and 15 have no severity name; GET under "info" lists only 25. + entry.level = emitted % 2 == 0 ? 25 : 15; + entry.name = "plant.temp"; + entry.msg = "m" + std::to_string(emitted); + source->emit(entry); + ++emitted; + }; + auto wait_for = [&](const std::function & done) { + std::unique_lock lock(mutex); + return cv.wait_for(lock, std::chrono::seconds(120), done); + }; + auto release = [&](int n) { + std::lock_guard lock(mutex); + released = n; + cv.notify_all(); + }; + + emit(); + ASSERT_TRUE(wait_for([&] { + return entered == 1; + })); + for (std::size_t i = 0; i < queue_size; ++i) { + emit(); + } + release(1); + ASSERT_TRUE(wait_for([&] { + return entered == 2; + })); + for (std::size_t i = 0; i < queue_size; ++i) { + emit(); + } + release(2); + ASSERT_TRUE(wait_for([&] { + return decided == emitted; + })) << "decided " + << decided << " of " << emitted; + notifier.shutdown(); + + auto listed = logs::query_entity_logs(mgr, cache_, SovdEntityType::APP, "temp", + logs::entity_log_fqn(cache_, SovdEntityType::APP, "temp"), "", ""); + ASSERT_TRUE(listed.has_value()); + std::set listed_ids; + for (const auto & item : *listed) { + listed_ids.insert(item.value("id", "")); + } + EXPECT_EQ(listed_ids.size(), (emitted + 1) / 2); + EXPECT_EQ(admitted.size(), listed_ids.size()); + EXPECT_TRUE(admitted == listed_ids) << "trigger decisions differ from the GET listing"; +} + +// ============================================================================= +// logs::restore_replay_entries +// ============================================================================= + +namespace { + +/// Primary LogProvider whose first get_logs call throws. +class FailOnceLogProvider : public LogProvider { + public: + explicit FailOnceLogProvider(std::vector entries) : entries_(std::move(entries)) { + } + + std::vector get_logs(const std::vector & /*node_fqns*/, bool /*prefix_match*/, + const std::string & /*min_severity*/, const std::string & /*context_filter*/, + const std::string & /*entity_id*/) override { + if (calls_++ == 0) { + throw std::runtime_error("log backend unavailable"); + } + return entries_; + } + + LogConfig get_config(const std::string & /*entity_id*/) const override { + return LogConfig{}; + } + + std::string update_config(const std::string & /*entity_id*/, const std::optional & /*severity_filter*/, + const std::optional & /*max_entries*/) override { + return ""; + } + + int calls() const { + return calls_; + } + + private: + std::vector entries_; + int calls_{0}; +}; + +class SingleProviderRegistry : public LogProviderRegistry { + public: + explicit SingleProviderRegistry(LogProvider * provider) : provider_(provider) { + } + LogProvider * primary_log_provider() const override { + return provider_; + } + std::vector log_observers() const override { + return {provider_}; + } + + private: + LogProvider * provider_; +}; + +LogEntry provider_entry(int64_t id) { + LogEntry entry{}; + entry.id = id; + entry.stamp_sec = 1; + entry.level = 20; + entry.name = "plant.temp"; + entry.msg = "provider " + std::to_string(id); + return entry; +} + +} // namespace + +// The fixture's add_log_entry() entries are trigger markers: listed, not replayed. +TEST_F(LogScopeTest, ReplayListingWaitsForAnEntityOfItsTypeAndLeavesOutMarkers) { + LogEntry real{}; + real.id = 1000; + real.stamp_sec = 1; + real.level = 20; + real.name = "plant/temp"; + real.msg = "real temp entry"; + mgr_.inject_entry_for_testing(real); + + EXPECT_FALSE(logs::restore_replay_entries(mgr_, cache_, SovdEntityType::APP, "plant_cell").has_value()) + << "a Component with the id is not the App"; + auto entries = logs::restore_replay_entries(mgr_, cache_, SovdEntityType::APP, "temp"); + ASSERT_TRUE(entries.has_value()); + ASSERT_EQ(entries->size(), 1u); + EXPECT_EQ((*entries)[0]["message"], "real temp entry"); + auto listed = logs::query_entity_logs(mgr_, cache_, SovdEntityType::APP, "temp", "/plant/temp", "", ""); + ASSERT_TRUE(listed.has_value()); + EXPECT_EQ(listed->size(), 2u) << "GET lists the marker too"; +} + +// A failed listing leaves the replay pending; the next sweep delivers each +// entry once. +TEST_F(LogScopeTest, FailedReplayListingIsRetried) { + FailOnceLogProvider provider({provider_entry(1), provider_entry(2)}); + SingleProviderRegistry registry(&provider); + LogManager mgr(std::make_shared(), ®istry); + + TriggerInfo info; + info.id = "trig_retry"; + info.entity_id = "temp"; + info.entity_type = "apps"; + info.resource_uri = "/api/v1/apps/temp/logs"; + info.collection = "logs"; + info.condition_type = "OnChange"; + info.condition_params = json::object(); + info.protocol = "sse"; + info.multishot = true; + info.persistent = true; + info.status = TriggerStatus::ACTIVE; + info.created_at = std::chrono::system_clock::now(); + SqliteTriggerStore store(":memory:"); + ASSERT_TRUE(store.save(info).has_value()); + + ResourceChangeNotifier notifier; + ConditionRegistry conditions; + conditions.register_condition("OnChange", std::make_shared()); + TriggerConfig config; + config.max_triggers = 10; + config.on_restart_behavior = "restore"; + TriggerManager triggers(notifier, conditions, store, config); + triggers.set_restore_replay_fn("logs", [&](const std::string & /*entity_type*/, const std::string & entity_id) { + return logs::restore_replay_entries(mgr, cache_, SovdEntityType::APP, entity_id); + }); + triggers.set_entity_exists_fn([&](const std::string & entity_id, const std::string & /*entity_type*/) { + return cache_.has_app(entity_id); + }); + triggers.load_persistent_triggers(); + + auto drain = [&] { + std::vector ids; + while (auto event = triggers.consume_pending_event("trig_retry")) { + ids.push_back((*event)["payload"].value("id", "")); + } + return ids; + }; + triggers.sweep_orphaned_triggers(); + EXPECT_TRUE(drain().empty()) << "a failed listing delivered events"; + triggers.sweep_orphaned_triggers(); + EXPECT_EQ(drain(), (std::vector{"log_1", "log_2"})); + triggers.sweep_orphaned_triggers(); + EXPECT_TRUE(drain().empty()) << "replayed twice"; + EXPECT_EQ(provider.calls(), 2); + + triggers.shutdown(); + notifier.shutdown(); +} diff --git a/src/ros2_medkit_gateway/test/test_log_trigger_restore.cpp b/src/ros2_medkit_gateway/test/test_log_trigger_restore.cpp new file mode 100644 index 000000000..96badbdbc --- /dev/null +++ b/src/ros2_medkit_gateway/test/test_log_trigger_restore.cpp @@ -0,0 +1,491 @@ +// Copyright 2026 bburda +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +// A persistent logs trigger restored by a real GatewayNode from a real trigger +// store, with real discovery and real /rosout entries. + +#include +#include +#include +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "ros2_medkit_gateway/core/managers/trigger_manager.hpp" +#include "ros2_medkit_gateway/core/models/thread_safe_entity_cache.hpp" +#include "ros2_medkit_gateway/core/sqlite_trigger_store.hpp" +#include "ros2_medkit_gateway/gateway_node.hpp" + +using namespace std::chrono_literals; +using ros2_medkit_gateway::SqliteTriggerStore; +using ros2_medkit_gateway::TriggerInfo; +using ros2_medkit_gateway::TriggerStatus; + +namespace { + +int reserve_local_port() { + int sock = socket(AF_INET, SOCK_STREAM, 0); + if (sock < 0) { + ADD_FAILURE() << "socket: " << std::strerror(errno); + return 0; + } + sockaddr_in addr{}; + addr.sin_family = AF_INET; + addr.sin_addr.s_addr = htonl(INADDR_LOOPBACK); + addr.sin_port = 0; + if (bind(sock, reinterpret_cast(&addr), sizeof(addr)) != 0) { + ADD_FAILURE() << "bind: " << std::strerror(errno); + close(sock); + return 0; + } + socklen_t addr_len = sizeof(addr); + if (getsockname(sock, reinterpret_cast(&addr), &addr_len) != 0) { + ADD_FAILURE() << "getsockname: " << std::strerror(errno); + close(sock); + return 0; + } + int port = ntohs(addr.sin_port); + close(sock); + return port; +} + +// A manifest Component with the id the runtime App of node /dup_node gets. +constexpr const char * kManifest = R"( +manifest_version: "1.0" +metadata: + name: log-trigger-restore + version: "0.0.1" +areas: + - id: vehicle + name: Vehicle +components: + - id: dup_node + name: Component sharing an app id + area: vehicle +)"; + +constexpr const char * kTriggerId = "trig_dup"; + +TriggerInfo persistent_logs_trigger(const std::string & app_id, const std::string & id = kTriggerId) { + TriggerInfo info; + info.id = id; + info.entity_id = app_id; + info.entity_type = "apps"; + info.resource_uri = "/api/v1/apps/" + app_id + "/logs"; + info.collection = "logs"; + info.condition_type = "OnChange"; + info.condition_params = nlohmann::json::object(); + info.protocol = "sse"; + info.multishot = true; + info.persistent = true; + info.status = TriggerStatus::ACTIVE; + info.created_at = std::chrono::system_clock::now(); + return info; +} + +class LogTriggerRestoreTest : public ::testing::Test { + protected: + static void SetUpTestSuite() { + if (!rclcpp::ok()) { + rclcpp::init(0, nullptr); + } + } + + static void TearDownTestSuite() { + if (rclcpp::ok()) { + rclcpp::shutdown(); + } + } + + void SetUp() override { + const auto * info = ::testing::UnitTest::GetInstance()->current_test_info(); + const std::string name = info ? std::string(info->name()) : std::string("unknown"); + work_dir_ = std::filesystem::temp_directory_path() / + ("medkit-log-trigger-restore-" + std::to_string(::getpid()) + "-" + name); + std::error_code ec; + std::filesystem::remove_all(work_dir_, ec); + std::filesystem::create_directories(work_dir_); + const auto manifest_path = work_dir_ / "manifest.yaml"; + std::ofstream(manifest_path) << kManifest; + db_path_ = work_dir_ / "triggers.db"; + const auto & db_path = db_path_; + { + SqliteTriggerStore store(db_path.string()); + for (const auto & trigger : stored_triggers()) { + ASSERT_TRUE(store.save(trigger).has_value()); + } + } + + const int port = reserve_local_port(); + ASSERT_GT(port, 0); + rclcpp::NodeOptions opts; + opts.parameter_overrides({ + {"discovery.mode", "hybrid"}, + {"discovery.manifest_path", manifest_path.string()}, + {"discovery.manifest_strict_validation", false}, + {"refresh_interval_ms", 500}, + {"triggers.storage.path", db_path.string()}, + {"triggers.on_restart_behavior", std::string("restore")}, + {"server.host", std::string("127.0.0.1")}, + {"server.port", port}, + }); + gateway_ = std::make_shared(opts); + probe_ = std::make_shared("log_restore_probe", "/probe"); + rosout_ = probe_->create_publisher("/rosout", rclcpp::QoS(100).reliable()); + executor_ = std::make_unique(); + executor_->add_node(gateway_); + executor_->add_node(probe_); + } + + void TearDown() override { + if (executor_) { + executor_->cancel(); + } + executor_.reset(); + app_node_.reset(); + rosout_.reset(); + probe_.reset(); + gateway_.reset(); + std::error_code ec; + std::filesystem::remove_all(work_dir_, ec); + } + + /// Triggers in the store before the gateway starts. + virtual std::vector stored_triggers() const { + return {persistent_logs_trigger("dup_node")}; + } + + void publish(const std::string & logger, const std::string & message) { + rcl_interfaces::msg::Log msg; + msg.stamp = probe_->now(); + msg.level = rcl_interfaces::msg::Log::INFO; + msg.name = logger; + msg.msg = message; + rosout_->publish(msg); + } + + bool spin_until(const std::function & done, std::chrono::nanoseconds timeout) { + const auto deadline = std::chrono::steady_clock::now() + timeout; + while (std::chrono::steady_clock::now() < deadline) { + executor_->spin_some(50ms); + if (done()) { + return true; + } + std::this_thread::sleep_for(10ms); + } + return done(); + } + + /// The orphan sweep and restore replay the gateway's refresh timer runs, run here. + void sweep() { + gateway_->get_trigger_manager()->sweep_orphaned_triggers(); + } + + /// Entries the gateway holds under logger dup_node, markers included. + size_t buffered() { + auto items = gateway_->get_log_manager()->get_logs({"/dup_node"}, false, "", "", "dup_node"); + return items ? items->size() : 0; + } + + /// Moves the pending events of @p trigger_id into events_. + void drain(const std::string & trigger_id) { + while (auto event = gateway_->get_trigger_manager()->consume_pending_event(trigger_id)) { + events_[trigger_id].push_back((*event)["payload"].value("message", "")); + } + } + + /// Spins until every trigger in @p trigger_ids has delivered @p message. Events reach a + /// trigger in log order, so nothing logged before @p message can arrive after it. + bool spin_until_delivered(const std::vector & trigger_ids, const std::string & message) { + return spin_until( + [&] { + bool all = true; + for (const auto & id : trigger_ids) { + drain(id); + const auto & got = events_[id]; + all = all && std::find(got.begin(), got.end(), message) != got.end(); + } + return all; + }, + 20s); + } + + std::map> events_; + + std::filesystem::path work_dir_; + std::filesystem::path db_path_; + std::shared_ptr gateway_; + std::shared_ptr probe_; + std::shared_ptr app_node_; + rclcpp::Publisher::SharedPtr rosout_; + std::unique_ptr executor_; +}; + +constexpr const char * kMarkerTriggerA = "trig_mark_a"; +constexpr const char * kMarkerTriggerB = "trig_mark_b"; + +/// Two restored triggers with log_settings on the root app dup_node. Their +/// markers are logged under "dup_node", which the app's GET lists. +class LogTriggerRestoreMarkerTest : public LogTriggerRestoreTest { + protected: + std::vector stored_triggers() const override { + auto a = persistent_logs_trigger("dup_node", kMarkerTriggerA); + a.log_settings = nlohmann::json{{"severity", "info"}, {"marker", "mark-a"}}; + auto b = persistent_logs_trigger("dup_node", kMarkerTriggerB); + b.log_settings = nlohmann::json{{"severity", "info"}, {"marker", "mark-b"}}; + return {a, b}; + } + + /// Messages GET /apps/dup_node/logs lists. + std::vector listed() { + std::vector out; + auto items = gateway_->get_log_manager()->get_logs({"/dup_node"}, false, "", "", "dup_node"); + if (items) { + for (const auto & item : *items) { + out.push_back(item.value("message", "")); + } + } + return out; + } +}; + +size_t count_prefix(const std::vector & messages, const std::string & prefix) { + return static_cast(std::count_if(messages.begin(), messages.end(), [&](const std::string & m) { + return m.rfind(prefix, 0) == 0; + })); +} + +} // namespace + +TEST_F(LogTriggerRestoreTest, AppReplayWaitsForTheAppNotForAComponentWithItsId) { + const auto & cache = gateway_->get_thread_safe_cache(); + auto * triggers = gateway_->get_trigger_manager(); + ASSERT_NE(triggers, nullptr); + ASSERT_TRUE(triggers->get(kTriggerId).has_value()) << "the trigger was not restored"; + ASSERT_TRUE(spin_until( + [&] { + return cache.has_component("dup_node") && rosout_->get_subscription_count() >= 1; + }, + 20s)); + ASSERT_FALSE(cache.has_app("dup_node")); + + // Logged while only the Component carries the id. Sweeps run, and the trigger waits. + publish("dup_node", "early"); + ASSERT_TRUE(spin_until( + [&] { + return buffered() == 1; + }, + 10s)) + << "the gateway did not take the early entry"; + sweep(); + sweep(); + ASSERT_FALSE(cache.has_app("dup_node")); + drain(kTriggerId); + EXPECT_TRUE(events_[kTriggerId].empty()) << "replayed for the Component with the app's id"; + + app_node_ = std::make_shared("dup_node", "/"); + executor_->add_node(app_node_); + ASSERT_TRUE(spin_until( + [&] { + return cache.has_app("dup_node"); + }, + 20s)); + + sweep(); + ASSERT_TRUE(spin_until_delivered({kTriggerId}, "early")) << "the early entry never reached the restored trigger"; + // A sweep after a live entry replays nothing again: an OnChange trigger would fire on a + // repeated "early", which differs from "later". A last entry closes the check. + publish("dup_node", "later"); + ASSERT_TRUE(spin_until_delivered({kTriggerId}, "later")); + sweep(); + publish("dup_node", "last"); + ASSERT_TRUE(spin_until_delivered({kTriggerId}, "last")); + EXPECT_EQ(events_[kTriggerId], (std::vector{"early", "later", "last"})); +} + +TEST_F(LogTriggerRestoreTest, TriggerIsAnOrphanOnceItsAppIsGoneAlthoughAComponentKeepsTheId) { + const auto & cache = gateway_->get_thread_safe_cache(); + auto * triggers = gateway_->get_trigger_manager(); + ASSERT_NE(triggers, nullptr); + ASSERT_TRUE(triggers->get(kTriggerId).has_value()) << "the trigger was not restored"; + + app_node_ = std::make_shared("dup_node", "/"); + executor_->add_node(app_node_); + ASSERT_TRUE(spin_until( + [&] { + return cache.has_app("dup_node") && cache.has_component("dup_node"); + }, + 20s)); + // A sweep while the App is there: the trigger has seen its entity and stays. + sweep(); + ASSERT_TRUE(triggers->get(kTriggerId).has_value()) << "swept while its App was there"; + + executor_->remove_node(app_node_); + app_node_.reset(); + ASSERT_TRUE(spin_until( + [&] { + return !cache.has_app("dup_node"); + }, + 20s)); + ASSERT_TRUE(cache.has_component("dup_node")); + + EXPECT_TRUE(spin_until( + [&] { + return !triggers->get(kTriggerId).has_value(); + }, + 10s)) + << "the App is gone; the Component with its id must not keep the trigger"; + SqliteTriggerStore store(db_path_.string()); + auto rows = store.load_all(); + ASSERT_TRUE(rows.has_value()); + EXPECT_TRUE(std::none_of(rows->begin(), rows->end(), [](const TriggerInfo & row) { + return row.id == kTriggerId; + })) << "the store row outlived the orphan"; +} + +TEST_F(LogTriggerRestoreMarkerTest, MarkersWrittenByTheReplayFireNoTrigger) { + const auto & cache = gateway_->get_thread_safe_cache(); + auto * triggers = gateway_->get_trigger_manager(); + ASSERT_NE(triggers, nullptr); + ASSERT_TRUE(triggers->get(kMarkerTriggerA).has_value()) << "trigger a was not restored"; + ASSERT_TRUE(triggers->get(kMarkerTriggerB).has_value()) << "trigger b was not restored"; + ASSERT_TRUE(spin_until( + [&] { + return cache.has_component("dup_node") && rosout_->get_subscription_count() >= 1; + }, + 20s)); + publish("dup_node", "early"); + ASSERT_TRUE(spin_until( + [&] { + return listed().size() == 1; + }, + 10s)) + << "the gateway did not take the early entry"; + sweep(); + ASSERT_FALSE(cache.has_app("dup_node")); + + app_node_ = std::make_shared("dup_node", "/"); + executor_->add_node(app_node_); + ASSERT_TRUE(spin_until( + [&] { + return cache.has_app("dup_node"); + }, + 20s)); + // The replay writes one marker per trigger. Each marker reaches the notifier like an + // entry, before the entries logged after it. + sweep(); + ASSERT_TRUE(spin_until( + [&] { + const auto messages = listed(); + return count_prefix(messages, "mark-a ") >= 1 && count_prefix(messages, "mark-b ") >= 1; + }, + 20s)) + << "the replays wrote no markers"; + + // A live entry fires each trigger once and writes one marker each; a last entry is + // delivered after anything the markers could fire. + publish("dup_node", "late"); + ASSERT_TRUE(spin_until_delivered({kMarkerTriggerA, kMarkerTriggerB}, "late")) << "the live entry fired no trigger"; + publish("dup_node", "done"); + ASSERT_TRUE(spin_until_delivered({kMarkerTriggerA, kMarkerTriggerB}, "done")); + ASSERT_TRUE(spin_until( + [&] { + const auto messages = listed(); + return count_prefix(messages, "mark-a ") >= 3 && count_prefix(messages, "mark-b ") >= 3; + }, + 20s)) + << "a firing wrote no marker"; + const std::vector entries{"early", "late", "done"}; + EXPECT_EQ(events_[kMarkerTriggerA], entries); + EXPECT_EQ(events_[kMarkerTriggerB], entries); + const auto messages = listed(); + EXPECT_EQ(count_prefix(messages, "mark-a "), 3u); + EXPECT_EQ(count_prefix(messages, "mark-b "), 3u); + EXPECT_EQ(messages.size(), 9u) << "three entries plus one marker per firing"; +} + +// A real entry logged before discovery with the text of trigger a's marker, +// its trigger_id included, under the marker's logger name. The replay takes +// markers by how they were written, so it delivers this one. +TEST_F(LogTriggerRestoreMarkerTest, ReplayDeliversARealEntryThatLooksLikeAMarker) { + const std::string lookalike = "mark-a " + nlohmann::json{{"condition_type", "OnChange"}, + {"resource", "/api/v1/apps/dup_node/logs"}, + {"trigger_id", kMarkerTriggerA}} + .dump(); + const auto & cache = gateway_->get_thread_safe_cache(); + auto * triggers = gateway_->get_trigger_manager(); + ASSERT_NE(triggers, nullptr); + ASSERT_TRUE(triggers->get(kMarkerTriggerA).has_value()) << "trigger a was not restored"; + ASSERT_TRUE(triggers->get(kMarkerTriggerB).has_value()) << "trigger b was not restored"; + ASSERT_TRUE(spin_until( + [&] { + return cache.has_component("dup_node") && rosout_->get_subscription_count() >= 1; + }, + 20s)); + publish("dup_node", "early"); + publish("dup_node", lookalike); + ASSERT_TRUE(spin_until( + [&] { + return listed().size() == 2; + }, + 10s)) + << "the gateway did not take both early entries"; + ASSERT_FALSE(cache.has_app("dup_node")); + + app_node_ = std::make_shared("dup_node", "/"); + executor_->add_node(app_node_); + ASSERT_TRUE(spin_until( + [&] { + return cache.has_app("dup_node"); + }, + 20s)); + + sweep(); + ASSERT_TRUE(spin_until( + [&] { + return count_prefix(listed(), "mark-b ") >= 2; + }, + 20s)) + << "the replays wrote no markers"; + // A last entry is delivered after anything the replay's markers could fire. + publish("dup_node", "done"); + ASSERT_TRUE(spin_until_delivered({kMarkerTriggerA, kMarkerTriggerB}, "done")); + ASSERT_TRUE(spin_until( + [&] { + return count_prefix(listed(), "mark-b ") >= 3; + }, + 20s)) + << "a firing wrote no marker"; + const std::vector entries{"early", lookalike, "done"}; + EXPECT_EQ(events_[kMarkerTriggerA], entries); + EXPECT_EQ(events_[kMarkerTriggerB], entries); + const auto messages = listed(); + EXPECT_EQ(count_prefix(messages, "mark-b "), 3u); + EXPECT_EQ(count_prefix(messages, "mark-a "), 4u) << "the look-alike plus one marker per firing of a"; + EXPECT_EQ(messages.size(), 9u); +} diff --git a/src/ros2_medkit_gateway/test/test_plugin_context_aggregation.cpp b/src/ros2_medkit_gateway/test/test_plugin_context_aggregation.cpp index 9119cd2f1..f9622fd5e 100644 --- a/src/ros2_medkit_gateway/test/test_plugin_context_aggregation.cpp +++ b/src/ros2_medkit_gateway/test/test_plugin_context_aggregation.cpp @@ -179,15 +179,15 @@ class FakeFaultTransport : public ros2_medkit_gateway::FaultServiceTransport { const std::string & /*source_id*/) override { return {false, json::object(), "not implemented"}; } - ros2_medkit_gateway::FaultResult clear_fault(const std::string & /*fault_code*/, + ros2_medkit_gateway::FaultResult clear_fault(const std::string & /*fault_code*/, const std::string & /*source_id*/, bool /*skip_correlation_auto_clear*/) override { return {false, json::object(), "not implemented"}; } - ros2_medkit_gateway::FaultResult get_snapshots(const std::string & /*fault_code*/, + ros2_medkit_gateway::FaultResult get_snapshots(const std::string & /*fault_code*/, const std::string & /*source_id*/, const std::string & /*topic*/) override { return {false, json::object(), "not implemented"}; } - ros2_medkit_gateway::FaultResult get_rosbag(const std::string & /*fault_code*/) override { + ros2_medkit_gateway::FaultResult get_rosbag(const std::string & /*id*/, const std::string & /*source_id*/) override { return {false, json::object(), "not implemented"}; } ros2_medkit_gateway::FaultResult list_rosbags(const std::string & /*entity_fqn*/) override { diff --git a/src/ros2_medkit_gateway/test/test_plugin_entity_routing.cpp b/src/ros2_medkit_gateway/test/test_plugin_entity_routing.cpp index 260fad967..20e8b286a 100644 --- a/src/ros2_medkit_gateway/test/test_plugin_entity_routing.cpp +++ b/src/ros2_medkit_gateway/test/test_plugin_entity_routing.cpp @@ -14,6 +14,10 @@ #include +#include +#include + +#include "ros2_medkit_gateway/core/plugins/plugin_http_types.hpp" #include "ros2_medkit_gateway/core/plugins/plugin_manager.hpp" #include "ros2_medkit_gateway/core/providers/data_provider.hpp" #include "ros2_medkit_gateway/core/providers/fault_provider.hpp" @@ -21,6 +25,7 @@ #include "ros2_medkit_gateway/dto/data.hpp" #include "ros2_medkit_gateway/dto/faults.hpp" #include "ros2_medkit_gateway/dto/operations.hpp" +#include "ros2_medkit_gateway/entity_freeze_frame_capture.hpp" using namespace ros2_medkit_gateway; // json alias already available via ros2_medkit_gateway namespace headers @@ -129,6 +134,237 @@ class MockMixedFitnessPlugin : public GatewayPlugin, public DataProvider, public } }; +// A plugin that serves /data ONLY through its DataProvider: no vendor route, +// which is what an in-tree plugin looks like. +class ProviderOnlyPlugin : public GatewayPlugin, public DataProvider { + public: + std::string name() const override { + return "provider_only"; + } + void configure(const json & /*config*/) override { + } + void shutdown() override { + } + + tl::expected list_data(const std::string & entity_id) override { + if (throws_) { + throw std::runtime_error("provider exploded"); + } + return dto::DataListResult{ + json{{"connected", true}, {"items", json::array({{{"id", "level"}, {"value", 7.5}, {"entity", entity_id}}})}}}; + } + tl::expected read_data(const std::string & /*entity_id*/, + const std::string & resource) override { + return dto::DataValue{json{{"value", resource}}}; + } + tl::expected + write_data(const std::string & /*entity_id*/, const std::string & /*resource*/, const json & /*payload*/) override { + return dto::DataWriteResult{json{{"status", "ok"}}}; + } + + bool throws_ = false; +}; + +// A plugin that serves the SAME entity through BOTH a DataProvider and its own +// data route, with different values in each. Without this fixture the +// provider-first order is unpinned: every provider-only and route-only case +// passes under either order, so restoring route-first breaks nothing and is +// caught by nothing. +class ProviderAndRoutePlugin : public GatewayPlugin, public DataProvider { + public: + static constexpr double kProviderLevel = 11.0; + static constexpr double kRouteLevel = 99.0; + + std::string name() const override { + return "provider_and_route"; + } + void configure(const json & /*config*/) override { + } + void shutdown() override { + } + + std::vector get_routes() override { + return { + {"GET", R"(apps/([^/]+)/x-plc-data)", + [this](const PluginRequest & /*req*/, PluginResponse & res) { + ++route_calls; + res.send_json(json{{"connected", route_connected}, + {"items", json::array({{{"id", "level"}, {"value", kRouteLevel}}})}}); + }}, + }; + } + + tl::expected list_data(const std::string & /*entity_id*/) override { + ++provider_calls; + return dto::DataListResult{json{{"connected", provider_connected}, + {"items", json::array({{{"id", "level"}, {"value", kProviderLevel}}})}}}; + } + tl::expected read_data(const std::string & /*entity_id*/, + const std::string & resource) override { + return dto::DataValue{json{{"value", resource}}}; + } + tl::expected + write_data(const std::string & /*entity_id*/, const std::string & /*resource*/, const json & /*payload*/) override { + return dto::DataWriteResult{json{{"status", "ok"}}}; + } + + int provider_calls = 0; + int route_calls = 0; + bool provider_connected = true; + bool route_connected = true; +}; + +// The same both-sources shape, with a provider that throws. Used to show the +// route really is reachable on this fixture, so the order test above is about +// the order and not about an unreachable route. +class ThrowingProviderAndRoutePlugin : public ProviderAndRoutePlugin { + public: + std::string name() const override { + return "throwing_provider_and_route"; + } + tl::expected list_data(const std::string & /*entity_id*/) override { + ++provider_calls; + throw std::runtime_error("provider exploded"); + } +}; + +// A plugin that exposes no DataProvider at all and registers no route either, +// which is every grouping-only plugin. +class NoDataPlugin : public GatewayPlugin { + public: + std::string name() const override { + return "no_data"; + } + void configure(const json & /*config*/) override { + } + void shutdown() override { + } +}; + +// ============================================================================= +// fetch_entity_data_content - which source an entity's /data comes from +// +// Provider first, the plugin's own vendor route only when it exposes no +// provider. The fault-trigger engine reads rule values through this and +// enumerates a rule's data points through it, so reading the route first meant +// a rule on a provider-served app evaluated to nothing every tick and sat +// there silently. +// ============================================================================= + +TEST(PluginEntityDataContent, ProviderServesAnEntityWithNoVendorRoute) { + PluginManager mgr; + auto plugin = std::make_unique(); + mgr.add_plugin(std::move(plugin)); + mgr.register_entity_ownership("provider_only", {"tank"}); + + auto content = mgr.fetch_entity_data_content("tank"); + + ASSERT_TRUE(content.has_value()) << "an entity served by a DataProvider must not read as having no data"; + ASSERT_TRUE(content->contains("items")); + EXPECT_EQ((*content)["items"][0]["id"], "level"); + EXPECT_DOUBLE_EQ((*content)["items"][0]["value"].get(), 7.5); +} + +TEST(PluginEntityDataContent, AThrowingProviderFallsThroughInsteadOfEscaping) { + // The fault-trigger engine calls this on its own evaluation loop, and a + // plugin exception must not leave that loop. + PluginManager mgr; + auto plugin = std::make_unique(); + plugin->throws_ = true; + mgr.add_plugin(std::move(plugin)); + mgr.register_entity_ownership("provider_only", {"tank"}); + + std::optional content; + EXPECT_NO_THROW(content = mgr.fetch_entity_data_content("tank")); + // No vendor route behind it, so there is nothing left to answer with. + EXPECT_FALSE(content.has_value()); +} + +TEST(PluginEntityDataContent, NeitherProviderNorRouteIsNullopt) { + PluginManager mgr; + mgr.add_plugin(std::make_unique()); + mgr.register_entity_ownership("no_data", {"grouping"}); + + EXPECT_FALSE(mgr.fetch_entity_data_content("grouping").has_value()); +} + +TEST(PluginEntityDataContent, AnUnownedEntityIsNullopt) { + PluginManager mgr; + EXPECT_FALSE(mgr.fetch_entity_data_content("nobody").has_value()); +} + +// One entity, both sources, different values in each. This is the only fixture +// in which the ORDER is observable at all. +TEST(PluginEntityDataContent, TheProviderIsReadBeforeTheVendorRoute) { + PluginManager mgr; + auto plugin = std::make_unique(); + auto * raw = plugin.get(); + mgr.add_plugin(std::move(plugin)); + mgr.register_entity_ownership("provider_and_route", {"tank"}); + + auto content = mgr.fetch_entity_data_content("tank"); + + ASSERT_TRUE(content.has_value()); + EXPECT_DOUBLE_EQ((*content)["items"][0]["value"].get(), ProviderAndRoutePlugin::kProviderLevel) + << "the route answered first, so a provider-served entity reads its vendor route instead"; + EXPECT_EQ(raw->provider_calls, 1); + EXPECT_EQ(raw->route_calls, 0) << "the route must not be dispatched when the provider answered"; +} + +// The route is the fallback, not the second opinion. On the same both-sources +// fixture, a provider that throws hands over and the value then comes from the +// route, which is what makes the previous test about ORDER rather than about the +// route being unreachable. +TEST(PluginEntityDataContent, TheVendorRouteAnswersWhenTheProviderThrows) { + PluginManager mgr; + auto plugin = std::make_unique(); + auto * raw = plugin.get(); + mgr.add_plugin(std::move(plugin)); + mgr.register_entity_ownership("throwing_provider_and_route", {"tank"}); + + auto content = mgr.fetch_entity_data_content("tank"); + + ASSERT_TRUE(content.has_value()); + EXPECT_DOUBLE_EQ((*content)["items"][0]["value"].get(), ProviderAndRoutePlugin::kRouteLevel); + EXPECT_EQ(raw->route_calls, 1) << "the route is the fallback and must have been dispatched"; +} + +// The trigger fetcher's link-down guard reads `connected` off this content. A +// provider envelope without it left the guard unable to fire for every +// provider-served entity, so a rule evaluated on frozen last-known values for a +// whole outage. +TEST(PluginEntityDataContent, ADisconnectedProviderEnvelopeReportsDisconnected) { + PluginManager mgr; + auto plugin = std::make_unique(); + auto * raw = plugin.get(); + mgr.add_plugin(std::move(plugin)); + mgr.register_entity_ownership("provider_and_route", {"tank"}); + raw->provider_connected = false; + + auto content = mgr.fetch_entity_data_content("tank"); + + ASSERT_TRUE(content.has_value()); + EXPECT_TRUE(EntityFreezeFrameCapture::content_has_live_data(*content)); + EXPECT_TRUE(EntityFreezeFrameCapture::content_reports_disconnected(*content)) + << "a provider envelope reporting a down link must read as disconnected, " + "or the trigger fetcher evaluates on frozen values"; +} + +// The control on the same harness: a connected provider envelope must NOT read +// as disconnected, so the assertion above is about the flag and not about the +// guard answering true for everything. +TEST(PluginEntityDataContent, AConnectedProviderEnvelopeReportsConnected) { + PluginManager mgr; + auto plugin = std::make_unique(); + mgr.add_plugin(std::move(plugin)); + mgr.register_entity_ownership("provider_and_route", {"tank"}); + + auto content = mgr.fetch_entity_data_content("tank"); + + ASSERT_TRUE(content.has_value()); + EXPECT_FALSE(EntityFreezeFrameCapture::content_reports_disconnected(*content)); +} + // ============================================================================= // Entity Ownership Tests // ============================================================================= diff --git a/src/ros2_medkit_gateway/test/test_resource_change_notifier.cpp b/src/ros2_medkit_gateway/test/test_resource_change_notifier.cpp index b29cf144b..e3d084a58 100644 --- a/src/ros2_medkit_gateway/test/test_resource_change_notifier.cpp +++ b/src/ros2_medkit_gateway/test/test_resource_change_notifier.cpp @@ -19,6 +19,7 @@ #include #include #include +#include #include #include #include @@ -424,6 +425,36 @@ TEST(ResourceChangeNotifier, UnsubscribeInvalidIdIsSafe) { EXPECT_NO_THROW(notifier.unsubscribe(999)); } +namespace { + +/// Releases a blocked worker on every exit path. A failed assert otherwise +/// leaves the worker waiting, and ~ResourceChangeNotifier joins it forever. +class BlockerRelease { + public: + explicit BlockerRelease(std::promise & unblock) : unblock_(&unblock) { + } + ~BlockerRelease() { + release(); + } + BlockerRelease(const BlockerRelease &) = delete; + BlockerRelease & operator=(const BlockerRelease &) = delete; + BlockerRelease(BlockerRelease &&) = delete; + BlockerRelease & operator=(BlockerRelease &&) = delete; + + void release() { + if (!released_) { + released_ = true; + unblock_->set_value(); + } + } + + private: + std::promise * unblock_; + bool released_{false}; +}; + +} // namespace + // --- Bounded queue: overflow drops oldest notifications --- TEST(ResourceChangeNotifier, QueueOverflow_DropsOldest) { @@ -440,6 +471,8 @@ TEST(ResourceChangeNotifier, QueueOverflow_DropsOldest) { ResourceChangeNotifier notifier; notifier.set_max_queue_size(5); + // Declared after the notifier, so it releases the worker before the notifier joins it. + BlockerRelease release_blocker(unblock); // Subscribe with a blocking callback to hold the worker while we fill the queue. notifier.subscribe({"data", "blocker", ""}, [&](const ResourceChange & /*change*/) { @@ -471,7 +504,7 @@ TEST(ResourceChangeNotifier, QueueOverflow_DropsOldest) { }); // Unblock the worker thread so it processes the queued notifications. - unblock.set_value(); + release_blocker.release(); auto status = done_future.wait_for(std::chrono::seconds(5)); ASSERT_EQ(status, std::future_status::ready); @@ -618,3 +651,148 @@ TEST(ResourceChangeNotifier, UnsubscribeOfIdZeroIsNoOp) { EXPECT_TRUE(unsubscribe_zero_returns(false)) << "unsubscribe(0) on a fresh notifier did not return"; EXPECT_TRUE(unsubscribe_zero_returns(true)) << "unsubscribe(0) after shutdown() did not return"; } + +TEST(ResourceChangeNotifier, QueueOverflow_DefaultCapacityKeepsNewest10000) { + // The documented bound: 10000 pending changes, the oldest dropped past it. + constexpr int kDocumentedBound = 10000; + constexpr int kSent = kDocumentedBound + 500; + constexpr int kSentinel = -1; + + // Shared state is declared before the notifier so it outlives the worker thread. + std::promise unblock; + auto unblock_future = unblock.get_future(); + std::promise worker_blocked; + auto worker_blocked_future = worker_blocked.get_future(); + std::mutex recv_mutex; + std::vector received; + std::promise bound_delivered; + auto bound_future = bound_delivered.get_future(); + std::promise sentinel_delivered; + auto sentinel_future = sentinel_delivered.get_future(); + std::atomic logged_drops{0}; + + ResourceChangeNotifier notifier([&logged_drops](const std::string & message) { + if (message.find("queue overflow") != std::string::npos) { + logged_drops.fetch_add(1); + } + }); + // Declared after the notifier, so it releases the worker before the notifier joins it. + BlockerRelease release_blocker(unblock); + + notifier.subscribe({"data", "blocker", ""}, [&](const ResourceChange & /*change*/) { + worker_blocked.set_value(); + unblock_future.wait(); + }); + notifier.notify("data", "blocker", "block", {}, ChangeType::UPDATED); + ASSERT_EQ(worker_blocked_future.wait_for(std::chrono::seconds(2)), std::future_status::ready); + + for (int i = 0; i < kSent; ++i) { + notifier.notify("data", "sensor", "temp", {{"index", i}}, ChangeType::UPDATED); + } + + notifier.subscribe({"data", "sensor", ""}, [&](const ResourceChange & change) { + const int index = change.value.at("index").get(); + std::lock_guard lk(recv_mutex); + if (index == kSentinel) { + sentinel_delivered.set_value(); + return; + } + received.push_back(index); + if (received.size() == static_cast(kDocumentedBound)) { + bound_delivered.set_value(); + } + }); + release_blocker.release(); + + ASSERT_EQ(bound_future.wait_for(std::chrono::seconds(30)), std::future_status::ready); + // Queued after the drain, so everything still pending arrives before it. + notifier.notify("data", "sensor", "temp", {{"index", kSentinel}}, ChangeType::UPDATED); + ASSERT_EQ(sentinel_future.wait_for(std::chrono::seconds(5)), std::future_status::ready); + + std::lock_guard lk(recv_mutex); + ASSERT_EQ(received.size(), static_cast(kDocumentedBound)); + EXPECT_EQ(received.front(), kSent - kDocumentedBound); + EXPECT_EQ(received.back(), kSent - 1); + EXPECT_TRUE(std::is_sorted(received.begin(), received.end())); + EXPECT_EQ(std::adjacent_find(received.begin(), received.end()), received.end()); + EXPECT_EQ(logged_drops.load(), static_cast(kSent - kDocumentedBound)); +} + +// --- notify_except --- + +TEST(ResourceChangeNotifier, NotifyExceptSkipsOnlyTheNamedSubscription) { + std::mutex mutex; + std::vector skipped_got; + std::vector other_got; + + ResourceChangeNotifier notifier; + + const auto skipped = notifier.subscribe({"logs", "", ""}, [&](const ResourceChange & change) { + std::lock_guard lock(mutex); + skipped_got.push_back(change.resource_path); + }); + notifier.subscribe({"logs", "", ""}, [&](const ResourceChange & change) { + std::lock_guard lock(mutex); + other_got.push_back(change.resource_path); + }); + + notifier.notify_except(skipped, "logs", "", "marker", nlohmann::json::object(), ChangeType::CREATED); + notifier.notify_except(0, "logs", "", "all", nlohmann::json::object(), ChangeType::CREATED); + notifier.notify("logs", "", "plain", nlohmann::json::object(), ChangeType::CREATED); + // Dispatches what is queued, then joins the worker. + notifier.shutdown(); + + std::lock_guard lock(mutex); + EXPECT_EQ(skipped_got, (std::vector{"all", "plain"})); + EXPECT_EQ(other_got, (std::vector{"marker", "all", "plain"})); +} + +// Past the queue bound, each surviving change keeps the subscription it skips. +TEST(ResourceChangeNotifier, NotifyExceptKeepsItsSkipThroughQueueOverflow) { + std::promise unblock; + auto unblock_future = unblock.get_future(); + std::promise worker_blocked; + auto worker_blocked_future = worker_blocked.get_future(); + std::mutex mutex; + std::map> got; + + ResourceChangeNotifier notifier; + notifier.set_max_queue_size(5); + // Declared after the notifier, so it releases the worker before the notifier joins it. + BlockerRelease release_blocker(unblock); + + notifier.subscribe({"gate", "", ""}, [&](const ResourceChange & /*change*/) { + worker_blocked.set_value(); + unblock_future.wait(); + }); + auto record = [&](char name) { + return [&, name](const ResourceChange & change) { + std::lock_guard lock(mutex); + got[name].push_back(change.value.at("index").get()); + }; + }; + const auto a = notifier.subscribe({"logs", "", ""}, record('a')); + const auto c = notifier.subscribe({"logs", "", ""}, record('c')); + notifier.subscribe({"logs", "", ""}, record('b')); + + notifier.notify("gate", "", "", {}, ChangeType::UPDATED); + ASSERT_EQ(worker_blocked_future.wait_for(std::chrono::seconds(2)), std::future_status::ready); + + // Twelve changes into a queue of five: 0..6 are dropped, 7..11 survive. + const std::vector skips{a, c, 0, a, 0, c, 0, 0, a, c, a, 0}; + for (int i = 0; i < static_cast(skips.size()); ++i) { + if (skips[static_cast(i)] == 0) { + notifier.notify("logs", "", "", {{"index", i}}, ChangeType::CREATED); + } else { + notifier.notify_except(skips[static_cast(i)], "logs", "", "", {{"index", i}}, ChangeType::CREATED); + } + } + release_blocker.release(); + // Dispatches what is queued, then joins the worker. + notifier.shutdown(); + + std::lock_guard lock(mutex); + EXPECT_EQ(got['b'], (std::vector{7, 8, 9, 10, 11})) << "the overflow did not drop 0..6"; + EXPECT_EQ(got['a'], (std::vector{7, 9, 11})); + EXPECT_EQ(got['c'], (std::vector{7, 8, 10, 11})); +} diff --git a/src/ros2_medkit_gateway/test/test_rosbag_recording.cpp b/src/ros2_medkit_gateway/test/test_rosbag_recording.cpp index 6f80efc06..6999a13fb 100644 --- a/src/ros2_medkit_gateway/test/test_rosbag_recording.cpp +++ b/src/ros2_medkit_gateway/test/test_rosbag_recording.cpp @@ -61,12 +61,13 @@ class FakeFaultTransport : public FaultServiceTransport { /// Stands for a store that could not be read: an answer, but not about the recording. bool get_rosbag_declines_everything{false}; bool get_fault_unavailable{false}; + bool list_faults_fails{false}; bool list_rosbags_fails{false}; /// Filters whose listing fails while the others answer. std::set list_rosbags_fails_for; int list_rosbags_calls{0}; - FaultResult get_rosbag(const std::string & id) override { + FaultResult get_rosbag(const std::string & id, const std::string & /*source_id*/) override { FaultResult r; r.success = false; if (get_rosbag_unavailable) { @@ -140,19 +141,33 @@ class FakeFaultTransport : public FaultServiceTransport { const std::string & /*description*/, const std::string & /*source_id*/) override { return {}; } + /// One confirmed record per fault, owned by its first reporting source. FaultResult list_faults(const std::string & /*source_id*/, bool /*include_prefailed*/, bool /*include_confirmed*/, bool /*include_cleared*/, bool /*include_healed*/, bool /*include_muted*/, bool /*include_clusters*/) override { - return {}; + FaultResult r; + r.success = !list_faults_fails; + if (!r.success) { + r.failure = FaultFailure::Unavailable; + return r; + } + json items = json::array(); + for (const auto & [code, sources] : faults) { + items.push_back({{"fault_code", code}, {"status", "CONFIRMED"}, {"reporting_sources", sources}}); + } + r.data = {{"faults", items}}; + return r; } FaultWithEnvJsonResult get_fault_with_env(const std::string & /*fault_code*/, const std::string & /*source_id*/) override { return {}; } - FaultResult clear_fault(const std::string & /*fault_code*/, bool /*skip_correlation_auto_clear*/) override { + FaultResult clear_fault(const std::string & /*fault_code*/, const std::string & /*source_id*/, + bool /*skip_correlation_auto_clear*/) override { return {}; } - FaultResult get_snapshots(const std::string & /*fault_code*/, const std::string & /*topic*/) override { + FaultResult get_snapshots(const std::string & /*fault_code*/, const std::string & /*source_id*/, + const std::string & /*topic*/) override { return {}; } bool wait_for_services(std::chrono::duration /*timeout*/) override { @@ -750,6 +765,13 @@ TEST_F(RecordingHandlersTest, AListingThatCouldNotReadEverySourceIs503) { const auto down = handlers_.list_descriptors(http::TypedRequest(single)); ASSERT_FALSE(down.has_value()); EXPECT_EQ(down.error().http_status, 503); + + // The fault list says which owners are in scope, so without it nothing is listed. + transport_->list_rosbags_fails = false; + transport_->list_faults_fails = true; + const auto no_records = handlers_.list_descriptors(http::TypedRequest(single)); + ASSERT_FALSE(no_records.has_value()); + EXPECT_EQ(no_records.error().http_status, 503); } } // namespace diff --git a/src/ros2_medkit_gateway/test/test_sse_fault_handler.cpp b/src/ros2_medkit_gateway/test/test_sse_fault_handler.cpp index 21d439833..e8ec84f56 100644 --- a/src/ros2_medkit_gateway/test/test_sse_fault_handler.cpp +++ b/src/ros2_medkit_gateway/test/test_sse_fault_handler.cpp @@ -29,6 +29,7 @@ #include "ros2_medkit_gateway/core/config.hpp" #include "ros2_medkit_gateway/core/discovery/models/app.hpp" +#include "ros2_medkit_gateway/core/discovery/models/component.hpp" #include "ros2_medkit_gateway/core/http/sse_client_tracker.hpp" #include "ros2_medkit_gateway/core/models/thread_safe_entity_cache.hpp" #include "ros2_medkit_gateway/fault_manager_paths.hpp" @@ -496,6 +497,45 @@ TEST_F(SSEFaultHandlerTest, CoalescedReplayKeepsTransitionsAndEndsOnCurrentState release_stream(res); } +TEST_F(SSEFaultHandlerTest, AnotherOwnersEventsDoNotSupersedeARecordsLastUpdate) { + // A fault code names as many records as there are sources reporting it. A + // newer event of one owner's record says nothing about another owner's record + // of the same code, so it must not coalesce that record's last update away: + // a lagging client would never learn that record's current state, and the + // loss would not even be counted. + auto req = make_stream_request("127.0.0.1"); + httplib::Response res; + handler_->handle_stream(req, res); // cursor open, never drained while the buffer fills + + auto owned_by = [](FaultEvent event, const std::string & owner) { + event.fault.reporting_sources = {owner}; + return event; + }; + enqueue_event(owned_by(make_fault_event(FaultEvent::EVENT_CONFIRMED, "SHARED", 1), "/owner_a")); + enqueue_event(owned_by(make_fault_event(FaultEvent::EVENT_UPDATED, "SHARED", 2), "/owner_a")); + enqueue_event(owned_by(make_fault_event(FaultEvent::EVENT_CONFIRMED, "SHARED", 3), "/owner_b")); + for (int i = 1; i <= 148; ++i) { + enqueue_event(owned_by(make_fault_event(FaultEvent::EVENT_UPDATED, "SHARED", 100 + i), "/owner_b")); + } + + EXPECT_GT(handler_->coalesced_events(), 0u) << "owner_b's own superseded updates are the ones to coalesce"; + EXPECT_EQ(handler_->dropped_events(), 0u); + + auto output = read_stream_once(res, 100); + int owner_a_updates = 0; + for (auto pos = output.find("event: "); pos != std::string::npos; pos = output.find("event: ", pos + 1)) { + auto payload = parse_sse_payload(output.substr(pos)); + if (payload["event_type"] == "fault_updated" && + payload["fault"]["reporting_sources"] == json::array({"/owner_a"})) { + ++owner_a_updates; + EXPECT_DOUBLE_EQ(payload["timestamp"].get(), 2.0); + } + } + EXPECT_EQ(owner_a_updates, 1) << "owner_a's last update was coalesced away by owner_b's events of the same code"; + + release_stream(res); +} + TEST_F(SSEFaultHandlerTest, OwedDistinctEventsLostUnderPressureAreCountedAndLogged) { // The counter this PR is about: a live client is owed events, the buffer // overflows with distinct fault codes (nothing to coalesce), so events are @@ -601,9 +641,8 @@ TEST_F(SSEFaultHandlerTest, NonPositiveKeepaliveOverrideLogsWarning) { } TEST_F(SSEFaultHandlerTest, StreamOmitsXMedkitWhenNoMatchingApp) { - // Empty cache: reporting source ("/apps/temp_sensor") has no manifest mapping - // and no App with id "temp_sensor" exists in runtime cache, so the - // ``x-medkit`` payload extension is not emitted (consumer falls back to + // Empty cache: no app owns the reporting source ("/apps/temp_sensor"), so + // the ``x-medkit`` payload extension is not emitted (consumer falls back to // discovery). enqueue_event(make_fault_event(FaultEvent::EVENT_CONFIRMED, "NO_OWNER", 10)); @@ -620,9 +659,8 @@ TEST_F(SSEFaultHandlerTest, StreamOmitsXMedkitWhenNoMatchingApp) { } TEST_F(SSEFaultHandlerTest, StreamEmitsXMedkitForRuntimeApp) { - // Runtime_only fallback: synthetic App with id matching the reporting - // source's last segment exists in the cache, so the ``x-medkit`` payload - // extension points consumers straight at + // A runtime app bound to the reporting node owns the fault, so the + // ``x-medkit`` payload extension points consumers straight at // /apps/temp_sensor/bulk-data/rosbags/. App app; app.id = "temp_sensor"; @@ -649,19 +687,17 @@ TEST_F(SSEFaultHandlerTest, StreamEmitsXMedkitForRuntimeApp) { } TEST_F(SSEFaultHandlerTest, StreamEmitsXMedkitForManifestApp) { - // Manifest/hybrid mode: linking populates node_to_app with the ROS FQN -> - // manifest app id mapping. The fault's reporting source uses the ROS FQN - // form (with leading slash), but the manifest app id can be arbitrary (here - // "diagnostic-bridge" with a hyphen, to model the FQN/id naming mismatch - // called out in #380). + // A manifest app id can differ from its node name (here "diagnostic-bridge" + // with a hyphen, the FQN/id naming mismatch called out in #380). Ownership + // follows the bound FQN, so no node -> app index is needed: manifest_only + // discovery builds none. App app; app.id = "diagnostic-bridge"; app.name = "diagnostic_bridge"; app.source = "manifest"; app.bound_fqn = "/bridge/diagnostic_bridge"; - std::unordered_map node_to_app{{"/bridge/diagnostic_bridge", "diagnostic-bridge"}}; auto & cache = const_cast(node_->get_thread_safe_cache()); - cache.update_all({}, {}, {app}, {}, node_to_app); + cache.update_apps({app}); auto event = make_fault_event(FaultEvent::EVENT_CONFIRMED, "BRIDGE_FAULT", 30); event.fault.reporting_sources = {"/bridge/diagnostic_bridge"}; @@ -681,31 +717,25 @@ TEST_F(SSEFaultHandlerTest, StreamEmitsXMedkitForManifestApp) { release_stream(res); } -TEST_F(SSEFaultHandlerTest, StreamPrefersManifestMatchOverRuntimeFallback) { - // Manifest match must win over the runtime heuristic when both could apply - // to the same FQN. Seed both: (1) a manifest node_to_app entry mapping - // "/bridge/diagnostic_bridge" -> "manifest-bridge", and (2) a runtime-style - // App whose id matches the FQN's last segment ("diagnostic_bridge"). The - // resolver tries manifest first - if anyone ever flips the ordering, the - // x-medkit emits the runtime id and this test catches it. - App manifest_app; - manifest_app.id = "manifest-bridge"; - manifest_app.name = "diagnostic_bridge"; - manifest_app.source = "manifest"; - manifest_app.bound_fqn = "/bridge/diagnostic_bridge"; - - App runtime_app; - runtime_app.id = "diagnostic_bridge"; - runtime_app.name = "diagnostic_bridge"; - runtime_app.source = "heuristic"; - runtime_app.bound_fqn = "/bridge/diagnostic_bridge"; - - std::unordered_map node_to_app{{"/bridge/diagnostic_bridge", "manifest-bridge"}}; +TEST_F(SSEFaultHandlerTest, StreamNamesOwnerOfSubPathSourceNotNamesake) { + // "/x/detector/goal_status" is a path child of the detector node, so the + // detector app lists the fault. The app named "goal_status" does not, and + // its rosbag address would answer 404. + App detector; + detector.id = "detector-app"; + detector.name = "detector"; + detector.source = "manifest"; + detector.bound_fqn = "/x/detector"; + App namesake; + namesake.id = "goal_status"; + namesake.name = "goal_status"; + namesake.source = "heuristic"; + namesake.bound_fqn = "/z/goal_status"; auto & cache = const_cast(node_->get_thread_safe_cache()); - cache.update_all({}, {}, {manifest_app, runtime_app}, {}, node_to_app); + cache.update_apps({detector, namesake}); - auto event = make_fault_event(FaultEvent::EVENT_CONFIRMED, "PRECEDENCE", 35); - event.fault.reporting_sources = {"/bridge/diagnostic_bridge"}; + auto event = make_fault_event(FaultEvent::EVENT_CONFIRMED, "SUB_PATH", 35); + event.fault.reporting_sources = {"/x/detector/goal_status"}; enqueue_event(event); auto req = make_stream_request("127.0.0.1"); @@ -716,7 +746,8 @@ TEST_F(SSEFaultHandlerTest, StreamPrefersManifestMatchOverRuntimeFallback) { auto payload = parse_sse_payload(output); ASSERT_TRUE(payload.contains("x-medkit")) << payload.dump(); - EXPECT_EQ(payload["x-medkit"]["entity_id"], "manifest-bridge"); + EXPECT_EQ(payload["x-medkit"]["entity_type"], "apps"); + EXPECT_EQ(payload["x-medkit"]["entity_id"], "detector-app"); release_stream(res); } @@ -755,23 +786,23 @@ TEST_F(SSEFaultHandlerTest, StreamSnapshotsEntityContextAtEnqueue) { release_stream(res); } -TEST_F(SSEFaultHandlerTest, StreamResolvesManifestFqnWithoutLeadingSlash) { - // gateway_node's node_resolver tries both FQN forms; the linking layer may - // store keys without the leading '/' (e.g. when the manifest declares - // namespace-only FQNs). The handler must try the slash-stripped form so - // /bridge/diagnostic_bridge resolves against a node_to_app key of - // "bridge/diagnostic_bridge". - App app; - app.id = "diagnostic-bridge"; - app.name = "diagnostic_bridge"; - app.source = "manifest"; - app.bound_fqn = "/bridge/diagnostic_bridge"; - std::unordered_map node_to_app{{"bridge/diagnostic_bridge", "diagnostic-bridge"}}; +TEST_F(SSEFaultHandlerTest, StreamOmitsXMedkitForAnOwnerNoAppHolds) { + // "/z/goal_status_x" shares a name prefix with "/z/goal_status" but is not + // a path child of it, so no app lists the record. Naming one would send the + // consumer to a rosbag address that answers 404. + App detector; + detector.id = "detector-app"; + detector.name = "detector"; + detector.bound_fqn = "/x/detector"; + App goal; + goal.id = "goal_status"; + goal.name = "goal_status"; + goal.bound_fqn = "/z/goal_status"; auto & cache = const_cast(node_->get_thread_safe_cache()); - cache.update_all({}, {}, {app}, {}, node_to_app); + cache.update_apps({detector, goal}); - auto event = make_fault_event(FaultEvent::EVENT_CONFIRMED, "SLASH_STRIP", 60); - event.fault.reporting_sources = {"/bridge/diagnostic_bridge"}; + auto event = make_fault_event(FaultEvent::EVENT_UPDATED, "PREFIX_ONLY", 60); + event.fault.reporting_sources = {"/z/goal_status_x"}; enqueue_event(event); auto req = make_stream_request("127.0.0.1"); @@ -781,16 +812,15 @@ TEST_F(SSEFaultHandlerTest, StreamResolvesManifestFqnWithoutLeadingSlash) { auto output = read_stream_once(res, 1); auto payload = parse_sse_payload(output); - ASSERT_TRUE(payload.contains("x-medkit")) << payload.dump(); - EXPECT_EQ(payload["x-medkit"]["entity_id"], "diagnostic-bridge"); + EXPECT_FALSE(payload.contains("x-medkit")) << payload.dump(); release_stream(res); } TEST_F(SSEFaultHandlerTest, StreamResolvesRuntimeCollisionRenamedApp) { // Runtime discovery renames collision-prone nodes to _ - // (see ros2_runtime_introspection.cpp). The fallback must also try this - // form so faults from those apps still get a usable x-medkit context. + // (see ros2_runtime_introspection.cpp). The owner is found by its bound + // FQN, so the renamed id needs no rule of its own. App app; app.id = "bridge_diagnostic_bridge"; app.name = "diagnostic_bridge"; @@ -816,6 +846,117 @@ TEST_F(SSEFaultHandlerTest, StreamResolvesRuntimeCollisionRenamedApp) { release_stream(res); } +// An external app owns its record under its bare SOVD id, which is not a ROS +// node FQN. That id is the app's owned source, so the app is named. +TEST_F(SSEFaultHandlerTest, StreamResolvesABareExternalAppId) { + App app; + app.id = "plc_process"; + app.name = "process"; + app.source = "plugin"; + app.external = true; + auto & cache = const_cast(node_->get_thread_safe_cache()); + cache.update_apps({app}); + + auto event = make_fault_event(FaultEvent::EVENT_CONFIRMED, "PROCESS_LEVEL_HIGH", 72); + event.fault.reporting_sources = {"plc_process"}; + enqueue_event(event); + + auto req = make_stream_request("127.0.0.1"); + httplib::Response res; + handler_->handle_stream(req, res); + + auto payload = parse_sse_payload(read_stream_once(res, 1)); + + ASSERT_TRUE(payload.contains("x-medkit")) << payload.dump(); + EXPECT_EQ(payload["x-medkit"]["entity_type"], "apps"); + EXPECT_EQ(payload["x-medkit"]["entity_id"], "plc_process"); + + release_stream(res); +} + +// A protocol bridge raises its link faults under the component's own id. The +// hint must name the component, because that is the entity whose fault routes +// serve the record. +TEST_F(SSEFaultHandlerTest, StreamResolvesABareExternalComponentId) { + ros2_medkit_gateway::Component component; + component.id = "line_controller"; + component.name = "Line Controller"; + component.source = "plugin"; + component.external = true; + auto & cache = const_cast(node_->get_thread_safe_cache()); + cache.update_all({}, {component}, {}, {}); + + auto event = make_fault_event(FaultEvent::EVENT_CONFIRMED, "DEVICE_COMMS_LOST", 73); + event.fault.reporting_sources = {"line_controller"}; + enqueue_event(event); + + auto req = make_stream_request("127.0.0.1"); + httplib::Response res; + handler_->handle_stream(req, res); + + auto payload = parse_sse_payload(read_stream_once(res, 1)); + + ASSERT_TRUE(payload.contains("x-medkit")) << payload.dump(); + EXPECT_EQ(payload["x-medkit"]["entity_type"], "components"); + EXPECT_EQ(payload["x-medkit"]["entity_id"], "line_controller"); + + release_stream(res); +} + +// A path below an external component's id belongs to the component too, the +// rule its fault routes list the record by. +TEST_F(SSEFaultHandlerTest, StreamResolvesASubPathOfAnExternalComponent) { + ros2_medkit_gateway::Component component; + component.id = "line_controller"; + component.name = "Line Controller"; + component.source = "plugin"; + component.external = true; + auto & cache = const_cast(node_->get_thread_safe_cache()); + cache.update_all({}, {component}, {}, {}); + + auto event = make_fault_event(FaultEvent::EVENT_CONFIRMED, "SLOT_FAULT", 75); + event.fault.reporting_sources = {"line_controller/slot_1"}; + enqueue_event(event); + + auto req = make_stream_request("127.0.0.1"); + httplib::Response res; + handler_->handle_stream(req, res); + + auto payload = parse_sse_payload(read_stream_once(res, 1)); + + ASSERT_TRUE(payload.contains("x-medkit")) << payload.dump(); + EXPECT_EQ(payload["x-medkit"]["entity_type"], "components"); + EXPECT_EQ(payload["x-medkit"]["entity_id"], "line_controller"); + + release_stream(res); +} + +// Only an external component claims its bare id as a reporting source. A +// runtime host component never does, and naming it would point the consumer at +// an entity whose own fault routes drop the record. +TEST_F(SSEFaultHandlerTest, StreamDoesNotNameANonExternalComponent) { + ros2_medkit_gateway::Component component; + component.id = "runtime_host"; + component.name = "runtime host"; + component.source = "heuristic"; + auto & cache = const_cast(node_->get_thread_safe_cache()); + cache.update_all({}, {component}, {}, {}); + + auto event = make_fault_event(FaultEvent::EVENT_CONFIRMED, "HOST_FAULT", 74); + event.fault.reporting_sources = {"runtime_host"}; + enqueue_event(event); + + auto req = make_stream_request("127.0.0.1"); + httplib::Response res; + handler_->handle_stream(req, res); + + auto payload = parse_sse_payload(read_stream_once(res, 1)); + + EXPECT_FALSE(payload.contains("x-medkit")) << payload.dump(); + + release_stream(res); +} + TEST_F(SSEFaultHandlerTest, StreamOmitsXMedkitWhenReportingSourcesEmpty) { auto event = make_fault_event(FaultEvent::EVENT_CONFIRMED, "ORPHAN", 40); event.fault.reporting_sources.clear(); diff --git a/src/ros2_medkit_gateway/test/test_trigger_manager.cpp b/src/ros2_medkit_gateway/test/test_trigger_manager.cpp index 5485f942f..0c3cc01bf 100644 --- a/src/ros2_medkit_gateway/test/test_trigger_manager.cpp +++ b/src/ros2_medkit_gateway/test/test_trigger_manager.cpp @@ -14,8 +14,12 @@ #include +#include #include #include +#include +#include +#include #include #include #include @@ -440,6 +444,75 @@ TEST_F(TriggerManagerTest, Hierarchy_NoMatchForUnrelatedEntity) { EXPECT_FALSE(event.has_value()); } +// =========================================================================== +// Collection scope function +// =========================================================================== + +namespace { + +TriggerCreateRequest make_fault_trigger(const std::string & entity_type, const std::string & entity_id) { + TriggerCreateRequest req; + req.entity_id = entity_id; + req.entity_type = entity_type; + req.resource_uri = "/api/v1/" + entity_type + "/" + entity_id + "/faults"; + req.collection = "faults"; + req.condition_type = "OnChange"; + req.protocol = "sse"; + req.multishot = true; + return req; +} + +} // namespace + +TEST_F(TriggerManagerTest, CollectionScope_DecidesFromPayloadNotFromEntityId) { + // The hierarchy would route a change for "goal_status" to "plant" and to the + // app itself; with a scope function only the payload decides. + manager_->set_entity_children_fn( + [](const std::string & entity_id, const std::string & /*entity_type*/) -> std::vector { + return entity_id == "plant" ? std::vector{"goal_status"} : std::vector{}; + }); + manager_->set_collection_scope_fn( + "faults", [](const std::string & entity_type, const std::string & entity_id, const json & value) { + return value.value("owner_type", "") == entity_type && value.value("owner_id", "") == entity_id; + }); + + auto area = manager_->create(make_fault_trigger("areas", "plant")); + auto app = manager_->create(make_fault_trigger("apps", "goal_status")); + ASSERT_TRUE(area.has_value()); + ASSERT_TRUE(app.has_value()); + + notifier_.notify("faults", "goal_status", "/F1", json{{"owner_type", "areas"}, {"owner_id", "plant"}}); + // A later change for the app proves the earlier one was already judged: + // the notifier delivers in order on one worker. + notifier_.notify("faults", "", "/F2", json{{"owner_type", "apps"}, {"owner_id", "goal_status"}}); + + ASSERT_TRUE(manager_->wait_for_event(app->id, std::chrono::milliseconds(2000))); + auto app_event = manager_->consume_pending_event(app->id); + ASSERT_TRUE(app_event.has_value()); + EXPECT_EQ((*app_event)["payload"]["owner_id"], "goal_status"); + EXPECT_FALSE(manager_->consume_pending_event(app->id).has_value()); + + auto area_event = manager_->consume_pending_event(area->id); + ASSERT_TRUE(area_event.has_value()); + EXPECT_EQ((*area_event)["payload"]["owner_id"], "plant"); + EXPECT_FALSE(manager_->consume_pending_event(area->id).has_value()); +} + +TEST_F(TriggerManagerTest, CollectionScope_LeavesOtherCollectionsOnIdMatching) { + manager_->set_collection_scope_fn("faults", [](const std::string &, const std::string &, const json &) { + return false; + }); + + auto req = make_request("sensor", "OnChange"); + req.multishot = true; + auto created = manager_->create(req); + ASSERT_TRUE(created.has_value()); + + notifier_.notify("data", "sensor", "/temperature", json(42.0)); + ASSERT_TRUE(manager_->wait_for_event(created->id, std::chrono::milliseconds(2000))); + EXPECT_TRUE(manager_->consume_pending_event(created->id).has_value()); +} + // =========================================================================== // Callback Tests // =========================================================================== @@ -1179,6 +1252,264 @@ TEST(LoadPersistentTriggers, SweepRemovesRestoredTriggerAfterItsEntityIsDiscover notifier.shutdown(); } +// =========================================================================== +// Restore replay +// =========================================================================== + +namespace { + +TriggerInfo make_persistent_logs_trigger(const std::string & id, bool multishot = true) { + auto info = make_persistent_trigger(id, "late_app"); + info.resource_uri = "/api/v1/apps/late_app/logs"; + info.collection = "logs"; + info.resource_path = ""; + info.resolved_topic_name = ""; + info.multishot = multishot; + return info; +} + +json log_item(int n) { + return json{{"id", "log_" + std::to_string(n)}, {"message", "m" + std::to_string(n)}}; +} + +std::vector drain_event_ids(TriggerManager & manager, const std::string & trigger_id) { + std::vector ids; + while (auto event = manager.consume_pending_event(trigger_id)) { + ids.push_back((*event)["payload"].value("id", "")); + } + return ids; +} + +/// OnChange that records the id of every value it evaluates. +class RecordingOnChange : public OnChangeEvaluator { + public: + bool evaluate(const std::optional & previous, const json & current, const json & params) const override { + { + std::lock_guard lock(mutex_); + seen_.push_back(current.is_object() ? current.value("id", "") : std::string{}); + } + return OnChangeEvaluator::evaluate(previous, current, params); + } + + std::vector seen() const { + std::lock_guard lock(mutex_); + return seen_; + } + + private: + mutable std::mutex mutex_; + mutable std::vector seen_; +}; + +/// Restored logs trigger whose entity the test "discovers" by flipping a flag. +struct RestoredLogsTrigger { + explicit RestoredLogsTrigger(bool multishot = true, bool with_gone_trigger = false) : store(":memory:") { + registry.register_condition("OnChange", evaluator); + EXPECT_TRUE(store.save(make_persistent_logs_trigger("trig_logs", multishot)).has_value()); + if (with_gone_trigger) { + EXPECT_TRUE(store.save(make_persistent_trigger("trig_gone", "gone_app")).has_value()); + } + TriggerConfig config; + config.max_triggers = 100; + config.on_restart_behavior = "restore"; + manager = std::make_unique(notifier, registry, store, config); + manager->set_collection_scope_fn("logs", [this](const std::string &, const std::string &, const json &) { + return discovered.load(); + }); + manager->set_restore_replay_fn( + "logs", + [this](const std::string & entity_type, const std::string & entity_id) -> std::optional> { + ++replay_calls; + if (!discovered.load()) { + return std::nullopt; + } + if (failing_listings.load() > 0) { + --failing_listings; + return std::nullopt; + } + EXPECT_EQ(entity_type, "apps"); + EXPECT_EQ(entity_id, "late_app"); + return listed; + }); + manager->set_entity_exists_fn([this](const std::string & id, const std::string &) { + return id == "gone_app" ? gone_present.load() : discovered.load(); + }); + manager->load_persistent_triggers(); + } + + ~RestoredLogsTrigger() { + manager->shutdown(); + notifier.shutdown(); + } + RestoredLogsTrigger(const RestoredLogsTrigger &) = delete; + RestoredLogsTrigger & operator=(const RestoredLogsTrigger &) = delete; + RestoredLogsTrigger(RestoredLogsTrigger &&) = delete; + RestoredLogsTrigger & operator=(RestoredLogsTrigger &&) = delete; + + /// Event ids drained until the one for `last_id`. The notifier delivers in + /// order, so every earlier change was judged by then. + std::vector events_through(const std::string & last_id) { + std::vector ids; + const auto deadline = std::chrono::steady_clock::now() + std::chrono::seconds(2); + while (std::find(ids.begin(), ids.end(), last_id) == ids.end() && std::chrono::steady_clock::now() < deadline) { + manager->wait_for_event("trig_logs", std::chrono::milliseconds(100)); + auto more = drain_event_ids(*manager, "trig_logs"); + ids.insert(ids.end(), more.begin(), more.end()); + } + return ids; + } + + ResourceChangeNotifier notifier; + std::shared_ptr evaluator = std::make_shared(); + ConditionRegistry registry; + SqliteTriggerStore store; + std::unique_ptr manager; + std::atomic discovered{false}; + std::atomic gone_present{true}; + std::atomic replay_calls{0}; + /// Replay listings that fail before one succeeds. + std::atomic failing_listings{0}; + std::vector listed; +}; + +} // namespace + +TEST(RestoreReplay, SweepReplaysWhatTheEntityListsOnceItIsDiscovered) { + RestoredLogsTrigger t; + t.listed = {log_item(1), log_item(2)}; + + t.manager->sweep_orphaned_triggers(); + EXPECT_TRUE(drain_event_ids(*t.manager, "trig_logs").empty()) << "nothing to replay before discovery"; + + t.discovered = true; + t.manager->sweep_orphaned_triggers(); + EXPECT_EQ(drain_event_ids(*t.manager, "trig_logs"), (std::vector{"log_1", "log_2"})); + + // Replays once; a replayed entry arriving live is not delivered again. + t.manager->sweep_orphaned_triggers(); + t.notifier.notify("logs", "", "", log_item(2), ChangeType::CREATED); + t.notifier.notify("logs", "", "", log_item(3), ChangeType::CREATED); + EXPECT_EQ(t.events_through("log_3"), (std::vector{"log_3"})); + EXPECT_EQ(t.replay_calls.load(), 1) << "the sweep asks only once the entity exists"; +} + +TEST(RestoreReplay, FirstLiveChangeReplaysBeforeItIsEvaluated) { + RestoredLogsTrigger t; + t.listed = {log_item(1), log_item(2)}; + t.discovered = true; + + // No sweep yet: the live change for entry 2 makes the trigger replay first. + t.notifier.notify("logs", "", "", log_item(2), ChangeType::CREATED); + t.notifier.notify("logs", "", "", log_item(3), ChangeType::CREATED); + EXPECT_EQ(t.events_through("log_3"), (std::vector{"log_1", "log_2", "log_3"})); + + t.manager->sweep_orphaned_triggers(); + EXPECT_TRUE(drain_event_ids(*t.manager, "trig_logs").empty()); +} + +TEST(RestoreReplay, LiveReplayBetweenSweepPhasesIsNotRepeated) { + // trig_gone's entity goes away, so the sweep calls the warn function + // between collecting trig_logs and replaying it. + RestoredLogsTrigger t(/*multishot=*/true, /*with_gone_trigger=*/true); + t.listed = {log_item(1), log_item(2)}; + t.manager->sweep_orphaned_triggers(); // sees gone_app once + + std::vector during; + t.manager->set_warn_log_fn([&](const std::string &) { + // The worker replays for the live change before the sweep's own replay. + t.notifier.notify("logs", "", "", log_item(2), ChangeType::CREATED); + during = t.events_through("log_2"); + }); + t.discovered = true; + t.gone_present = false; + t.manager->sweep_orphaned_triggers(); + + EXPECT_EQ(during, (std::vector{"log_1", "log_2"})); + EXPECT_TRUE(drain_event_ids(*t.manager, "trig_logs").empty()) << "the sweep replayed a second time"; +} + +TEST(RestoreReplay, LiveReplayMarksTheEntitySeenForTheOrphanSweep) { + RestoredLogsTrigger t; + t.listed = {log_item(1)}; + t.discovered = true; + + // A live change finds the entity and replays; no sweep runs while it exists. + t.notifier.notify("logs", "", "", log_item(2), ChangeType::CREATED); + EXPECT_EQ(t.events_through("log_2"), (std::vector{"log_1", "log_2"})); + + t.discovered = false; + t.manager->sweep_orphaned_triggers(); + EXPECT_FALSE(t.manager->get("trig_logs").has_value()) << "an entity that was found and then lost is an orphan"; + auto loaded = t.store.load_all(); + ASSERT_TRUE(loaded.has_value()); + EXPECT_TRUE(loaded->empty()) << "sweeping an orphan drops its store row as well"; +} + +// GET lists an entry before it is dispatched, so a replay after a failed one +// covers the live entry the failed attempt was triggered by. +TEST(RestoreReplay, LiveChangeAfterAFailedReplayWaitsForTheReplay) { + RestoredLogsTrigger t; + t.discovered = true; + t.failing_listings = 1; + t.listed = {log_item(1), log_item(2), log_item(3)}; + + t.notifier.notify("logs", "", "", log_item(2), ChangeType::CREATED); + t.notifier.notify("logs", "", "", log_item(3), ChangeType::CREATED); + + EXPECT_EQ(t.events_through("log_3"), (std::vector{"log_1", "log_2", "log_3"})); + EXPECT_EQ(t.evaluator->seen(), (std::vector{"log_1", "log_2", "log_3"})) + << "each entry is judged once, in log order"; +} + +TEST(RestoreReplay, OneShotTriggerAfterAFailedReplayFiresOnTheEarliestEntry) { + RestoredLogsTrigger t(/*multishot=*/false); + t.discovered = true; + t.failing_listings = 1; + t.listed = {log_item(1), log_item(2), log_item(3)}; + + t.notifier.notify("logs", "", "", log_item(2), ChangeType::CREATED); + t.notifier.notify("logs", "", "", log_item(3), ChangeType::CREATED); + + const auto deadline = std::chrono::steady_clock::now() + std::chrono::seconds(2); + while (t.manager->get("trig_logs").has_value() && std::chrono::steady_clock::now() < deadline) { + std::this_thread::sleep_for(std::chrono::milliseconds(10)); + } + EXPECT_FALSE(t.manager->get("trig_logs").has_value()) << "a one-shot trigger ends when it fires"; + EXPECT_EQ(t.evaluator->seen(), (std::vector{"log_1"})) << "it fires on the earliest entry"; +} + +TEST(RestoreReplay, OneShotTriggerEndsAtTheFirstReplayedEvent) { + RestoredLogsTrigger t(/*multishot=*/false); + t.listed = {log_item(1), log_item(2)}; + t.discovered = true; + + t.manager->sweep_orphaned_triggers(); + EXPECT_FALSE(t.manager->get("trig_logs").has_value()) << "a one-shot trigger ends when it fires"; + auto loaded = t.store.load_all(); + ASSERT_TRUE(loaded.has_value()); + EXPECT_TRUE(loaded->empty()); +} + +TEST(RestoreReplay, TriggerCreatedThroughTheApiDoesNotReplay) { + RestoredLogsTrigger t; + t.listed = {log_item(1)}; + t.discovered = true; + TriggerCreateRequest req; + req.entity_id = "late_app"; + req.entity_type = "apps"; + req.resource_uri = "/api/v1/apps/late_app/logs"; + req.collection = "logs"; + req.condition_type = "OnChange"; + req.protocol = "sse"; + req.multishot = true; + auto created = t.manager->create(req); + ASSERT_TRUE(created.has_value()); + + t.notifier.notify("logs", "", "", log_item(2), ChangeType::CREATED); + ASSERT_TRUE(t.manager->wait_for_event(created->id, std::chrono::milliseconds(2000))); + EXPECT_EQ(drain_event_ids(*t.manager, created->id), (std::vector{"log_2"})); +} + // =========================================================================== // Deferred topic resolution tests // =========================================================================== diff --git a/src/ros2_medkit_integration_tests/CMakeLists.txt b/src/ros2_medkit_integration_tests/CMakeLists.txt index b1b21bb2f..4e28d1c20 100644 --- a/src/ros2_medkit_integration_tests/CMakeLists.txt +++ b/src/ros2_medkit_integration_tests/CMakeLists.txt @@ -292,6 +292,7 @@ if(BUILD_TESTING) test_aggregated_function_hosts test_peer_recovery test_triggers_restore_before_discovery + test_triggers_logs_restore_before_discovery test_aggregator_fault_stream test_aggregated_colliding_node_names) set(_TWO_GATEWAY_DOMAINS 2) @@ -376,6 +377,9 @@ if(BUILD_TESTING) test_triggers_restore_before_discovery 300 # Seven cases share one gateway, and their scaled give-up bounds add up past 120s. test_graph_leftover_nodes 300 + # Thirteen cases, most logging for at least five seconds and then settling, + # after starting two gateways and a fault manager. + test_triggers_logs_markers 300 # Three discovery budgets in setUpClass before the first case runs, and a # broken merge spends all of them. The glob default cuts that short and # reports a timeout with no test name, which is the one answer that says diff --git a/src/ros2_medkit_integration_tests/test/features/refusing_fault_manager.py b/src/ros2_medkit_integration_tests/test/features/refusing_fault_manager.py new file mode 100644 index 000000000..55976b5a2 --- /dev/null +++ b/src/ros2_medkit_integration_tests/test/features/refusing_fault_manager.py @@ -0,0 +1,108 @@ +#!/usr/bin/env python3 +# Copyright 2026 bburda +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""A fault manager that refuses to clear one of its records. + +Companion process for test_faults_entity_clear_failure.test.py. It serves the +fault manager's services under /fault_manager. ListFaults answers the records it +holds. ClearFault refuses REFUSED_CODE with the answer the fault manager gives +for a record it does not hold, and clears any other record it holds. +""" + +import rclpy +from rclpy.executors import ExternalShutdownException +from rclpy.node import Node +from ros2_medkit_msgs.msg import Fault +from ros2_medkit_msgs.srv import ClearFault, GetFault, ListFaults, ReportFault + +ACCEPTED_CODE = 'CLEAR_ACCEPTED' +REFUSED_CODE = 'CLEAR_REFUSED' +OWNER_A = 'owner-a' +OWNER_B = 'owner-b' + + +def _record(node, code, owner): + fault = Fault() + fault.fault_code = code + fault.severity = Fault.SEVERITY_ERROR + fault.description = f'{code} of {owner}' + fault.first_occurred = node.get_clock().now().to_msg() + fault.last_occurred = fault.first_occurred + fault.occurrence_count = 1 + fault.status = Fault.STATUS_CONFIRMED + fault.reporting_sources = [owner] + fault.owner = owner + return fault + + +class RefusingFaultManager(Node): + + def __init__(self): + super().__init__('fault_manager') + self.records = { + (ACCEPTED_CODE, OWNER_A): _record(self, ACCEPTED_CODE, OWNER_A), + (REFUSED_CODE, OWNER_B): _record(self, REFUSED_CODE, OWNER_B), + } + self.create_service(ListFaults, '/fault_manager/list_faults', self.on_list) + self.create_service(ClearFault, '/fault_manager/clear_fault', self.on_clear) + self.create_service(GetFault, '/fault_manager/get_fault', self.on_get) + self.create_service(ReportFault, '/fault_manager/report_fault', self.on_report) + + def on_list(self, request, response): + response.faults = [record for record in self.records.values() + if not request.statuses or record.status in request.statuses] + return response + + def on_clear(self, request, response): + key = (request.fault_code, request.source_id) + if request.fault_code == REFUSED_CODE or key not in self.records: + response.success = False + response.message = (f'No fault record for code {request.fault_code} ' + f'and source {request.source_id}') + return response + self.records[key].status = Fault.STATUS_CLEARED + response.success = True + response.message = 'Fault cleared' + return response + + def on_get(self, request, response): + record = self.records.get((request.fault_code, request.source_id)) + if record is None: + response.success = False + response.error_message = f'Fault not found: {request.fault_code}' + return response + response.success = True + response.fault = record + return response + + def on_report(self, request, response): + response.accepted = False + return response + + +def main(): + rclpy.init() + node = RefusingFaultManager() + try: + rclpy.spin(node) + except (KeyboardInterrupt, ExternalShutdownException): + pass + finally: + node.destroy_node() + rclpy.try_shutdown() + + +if __name__ == '__main__': + main() diff --git a/src/ros2_medkit_integration_tests/test/features/test_bulk_data_api.test.py b/src/ros2_medkit_integration_tests/test/features/test_bulk_data_api.test.py index 6c8cc58cc..4034dc8b8 100644 --- a/src/ros2_medkit_integration_tests/test/features/test_bulk_data_api.test.py +++ b/src/ros2_medkit_integration_tests/test/features/test_bulk_data_api.test.py @@ -26,27 +26,49 @@ import atexit import os import shutil +import signal import tempfile +import time import unittest import launch_testing import launch_testing.actions +import rclpy +from rclpy.node import Node import requests +from ros2_medkit_msgs.msg import Fault +from ros2_medkit_msgs.srv import ReportFault -from ros2_medkit_test_utils.constants import ALLOWED_EXIT_CODES +from ros2_medkit_test_utils.constants import ALLOWED_EXIT_CODES, get_time_scale, ROSBAG_TIMEOUT from ros2_medkit_test_utils.gateway_test_case import GatewayTestCase from ros2_medkit_test_utils.launch_helpers import create_test_launch # The fault manager writes bag directories here, so a test can reach the files of a recording. ROSBAG_DIR = tempfile.mkdtemp(prefix='medkit_bulk_data_rosbags_') atexit.register(shutil.rmtree, ROSBAG_DIR, True) +CONFIG_DIR = tempfile.mkdtemp(prefix='medkit_bulk_data_config_') +atexit.register(shutil.rmtree, CONFIG_DIR, True) +THRESHOLDS_PATH = os.path.join(CONFIG_DIR, 'fault_thresholds.yaml') + +LIDAR_FQN = '/perception/lidar/lidar_sensor' +REPORTER_NODE = 'bulk_data_reporter' +# Reported from two sources and read back as two records, so one record per source. +PER_SOURCE_CODES = ('BULK_TWICE',) def generate_test_description(): + with open(THRESHOLDS_PATH, 'w') as f: + for code in PER_SOURCE_CODES: + f.write(f'{code}:\n per_source: true\n') return create_test_launch( demo_nodes=['lidar_sensor'], fault_manager=True, - fault_manager_params={'snapshots.rosbag.storage_path': ROSBAG_DIR}, + # The stalled fault manager case waits this long on every call it makes. + gateway_params={'fault_manager.service_timeout_sec': 1.0 * get_time_scale()}, + fault_manager_params={ + 'snapshots.rosbag.storage_path': ROSBAG_DIR, + 'fault_thresholds.config_file': THRESHOLDS_PATH, + }, lidar_faulty=True, ) @@ -57,6 +79,59 @@ class TestBulkDataApi(GatewayTestCase): MIN_EXPECTED_APPS = 1 REQUIRED_APPS = {'lidar_sensor'} + @classmethod + def setUpClass(cls): + rclpy.init() + # Runtime discovery makes this node an app, so it owns records that the + # host component lists and lidar_sensor does not. + cls._reporter = Node(REPORTER_NODE) + cls._report_client = cls._reporter.create_client( + ReportFault, '/fault_manager/report_fault') + super().setUpClass() + assert cls._report_client.wait_for_service(timeout_sec=15.0), \ + 'report_fault service not available' + + @classmethod + def tearDownClass(cls): + cls._reporter.destroy_node() + rclpy.shutdown() + + def _report(self, fault_code, source): + """Report one FAILED event; resend while the reply is lost. + + A reply can be lost while the fault manager is still matching this + client, even though the request arrived; resending is harmless. + """ + req = ReportFault.Request() + req.fault_code = fault_code + req.event_type = ReportFault.Request.EVENT_FAILED + req.severity = Fault.SEVERITY_ERROR + req.description = 'bulk-data scope test fault' + req.source_id = source + for _ in range(6): + future = self._report_client.call_async(req) + deadline = time.monotonic() + 5.0 + while not future.done() and time.monotonic() < deadline: + rclpy.spin_once(self._reporter, timeout_sec=0.1) + if future.done(): + self.assertTrue(future.result().accepted) + return + self.fail(f'report_fault {fault_code} never answered') + + def _bag_fault_codes(self, endpoint, wanted): + """Fault codes on the rosbag descriptors of ``endpoint`` once ``wanted`` are all listed.""" + deadline = time.monotonic() + ROSBAG_TIMEOUT + codes = set() + while time.monotonic() < deadline: + data = self.get_json(endpoint) + codes = {code for item in data.get('items', []) + for code in item.get('x-medkit', {}).get('fault_codes', [])} + if wanted <= codes: + return codes + time.sleep(0.5) + self.fail(f'{endpoint} never listed bags of {sorted(wanted - codes)}; ' + f'listed {sorted(codes)}') + def test_bulk_data_list_categories_success(self): """GET /apps/{app}/bulk-data returns categories. @@ -394,6 +469,135 @@ def test_bulk_data_nested_entity_path(self): self.assertIn('items', data) self.assertIsInstance(data['items'], list) + def test_bulk_data_rosbag_list_follows_fault_scope(self): + """An entity lists the bag of a record when the record's owner is in its scope. + + ``BULK_SUBPATH`` and ``BULK_SUBPATH2`` are owned by path children of the + lidar node FQN, one and two levels down, which the lidar app owns. + ``BULK_FOREIGN`` is owned by another app, so it belongs to the host + component and not to the lidar app. Each listing must agree with the + download of the same entity, which applies that rule. + + @verifies REQ_INTEROP_072 + """ + self.poll_endpoint(f'/apps/{REPORTER_NODE}', timeout=30.0) + comp_id = next( + (c['id'] for c in self.get_json('/components').get('items', []) + if any(h.get('id') == 'lidar_sensor' + for h in self.get_json(f"/components/{c['id']}/hosts").get('items', []))), + None) + self.assertIsNotNone(comp_id, 'No component hosts lidar_sensor') + + self._report('BULK_SUBPATH', f'{LIDAR_FQN}/diag') + self._report('BULK_SUBPATH2', f'{LIDAR_FQN}/diag/deep') + self._report('BULK_FOREIGN', f'/{REPORTER_NODE}') + + owned = {'BULK_SUBPATH', 'BULK_SUBPATH2'} + component_bags = f'/components/{comp_id}/bulk-data/rosbags' + self._bag_fault_codes(component_bags, owned | {'BULK_FOREIGN'}) + app_codes = self._bag_fault_codes('/apps/lidar_sensor/bulk-data/rosbags', owned) + self.assertNotIn( + 'BULK_FOREIGN', app_codes, + 'lidar_sensor lists the bag of a record it does not own') + + # The listings agree with the downloads (fault code resolves to the + # record's newest recording). + downloads = { + '/apps/lidar_sensor/bulk-data/rosbags/BULK_SUBPATH': 200, + '/apps/lidar_sensor/bulk-data/rosbags/BULK_SUBPATH2': 200, + '/apps/lidar_sensor/bulk-data/rosbags/BULK_FOREIGN': 404, + f'{component_bags}/BULK_SUBPATH': 200, + f'{component_bags}/BULK_SUBPATH2': 200, + f'{component_bags}/BULK_FOREIGN': 200, + } + for endpoint, status in downloads.items(): + with self.subTest(endpoint=endpoint): + response = requests.get(f'{self.BASE_URL}{endpoint}', timeout=10) + self.assertEqual(response.status_code, status, response.text[:200]) + + def test_bulk_data_code_url_names_no_record_when_two_owners_sit_below_one_app(self): + """Two path children of one node own records of one code: the code URL answers 409. + + The recording-id form from the listing still reaches the recording. + + @verifies REQ_INTEROP_073 + """ + owners = [f'{LIDAR_FQN}/diag', f'{LIDAR_FQN}/diag/deep'] + for owner in owners: + self._report('BULK_TWICE', owner) + endpoint = '/apps/lidar_sensor/bulk-data/rosbags' + self._bag_fault_codes(endpoint, {'BULK_TWICE'}) + + response = requests.get(f'{self.BASE_URL}{endpoint}/BULK_TWICE', timeout=10) + self.assertEqual(response.status_code, 409, response.text[:300]) + body = response.json() + self.assertEqual(body.get('vendor_code', body.get('error_code')), + 'x-medkit-ambiguous-fault', body) + self.assertEqual(sorted(body.get('parameters', body).get('owners', [])), owners, body) + + ids = [item['id'] for item in self.get_json(endpoint).get('items', []) + if 'BULK_TWICE' in item.get('x-medkit', {}).get('fault_codes', [])] + self.assertTrue(ids, 'no descriptor carries BULK_TWICE') + for recording_id in ids: + with self.subTest(recording_id=recording_id): + download = requests.get(f'{self.BASE_URL}{endpoint}/{recording_id}', timeout=30) + self.assertEqual(download.status_code, 200, download.text[:200]) + + def test_rosbag_routes_declare_their_503(self): + """The listing and download routes answer 503 when the fault manager is silent, and say so. + + @verifies REQ_INTEROP_072 + """ + spec = self.poll_endpoint_until('/docs', lambda d: d if 'openapi' in d else None) + + routes = { + path: item['get'] for path, item in spec['paths'].items() + if path.endswith(('/bulk-data/{category_id}', '/bulk-data/{category_id}/{file_id}')) + and 'get' in item + } + # A list and a download per entity type that serves bulk data, so the + # loop below cannot pass by matching nothing. + self.assertEqual(len(routes), 12, sorted(routes)) + for path, operation in sorted(routes.items()): + self.assertIn('503', operation['responses'], + f'GET {path} answers 503 when the fault manager fails') + + def test_zz_rosbag_routes_answer_503_while_the_fault_manager_is_stalled( + self, proc_info, fault_manager_node): + """The rosbag listing and download report an unanswering fault manager. + + SIGSTOP keeps the fault manager's services on the graph while they + never answer. A 200 with no items would tell the client there are no + recordings, and a 404 that the recording does not exist. Named to run + last; the process resumes on every exit path. + + @verifies REQ_INTEROP_072 + @verifies REQ_INTEROP_073 + """ + self._report('BULK_STALL', f'{LIDAR_FQN}/diag') + listing = '/apps/lidar_sensor/bulk-data/rosbags' + self._bag_fault_codes(listing, {'BULK_STALL'}) + recording_id = next( + item['id'] for item in self.get_json(listing).get('items', []) + if 'BULK_STALL' in item.get('x-medkit', {}).get('fault_codes', [])) + download = f'{self.BASE_URL}{listing}/{recording_id}' + self.assertEqual(requests.get(download, timeout=30).status_code, 200) + + pid = proc_info[fault_manager_node].pid + self.addCleanup(os.kill, pid, signal.SIGCONT) + os.kill(pid, signal.SIGSTOP) + try: + listed = requests.get(f'{self.BASE_URL}{listing}', timeout=60) + downloaded = requests.get(download, timeout=60) + finally: + os.kill(pid, signal.SIGCONT) + + self.assertEqual(listed.status_code, 503, listed.text[:300]) + self.assertEqual(downloaded.status_code, 503, downloaded.text[:300]) + # Both routes recover once the fault manager answers again. + self.poll_endpoint(listing, timeout=30.0) + self.assertEqual(requests.get(download, timeout=30).status_code, 200) + @launch_testing.post_shutdown_test() class TestShutdown(unittest.TestCase): diff --git a/src/ros2_medkit_integration_tests/test/features/test_fault_scope_notifications.test.py b/src/ros2_medkit_integration_tests/test/features/test_fault_scope_notifications.test.py new file mode 100644 index 000000000..91d612457 --- /dev/null +++ b/src/ros2_medkit_integration_tests/test/features/test_fault_scope_notifications.test.py @@ -0,0 +1,963 @@ +#!/usr/bin/env python3 +# Copyright 2026 bburda +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Fault notifications reach exactly the entities whose source scope holds the record. + +A fault record has one owner, the reporting source that raised it. The record +belongs to an entity when that owner equals one of the entity's owned sources +or is a path child of one. ``GET /{entity}/faults`` uses that rule. This file +checks that the two notification paths use it too, on the record as the event +carries it and whatever its status (what ``GET /{entity}/faults?status=all`` +lists after the event): a trigger on ``/{entity}/faults`` gets the event iff +the entity's scope holds the record, and ``x-medkit`` in ``GET /faults/stream`` +names the app whose scope holds it, never a namesake. + +One fault manager feeds three gateways, one per discovery mode, so every report +is judged in hybrid, runtime_only and manifest_only at once. The ROS nodes are +created by this process: + +- ``/scope/x/detector`` and ``/scope/y/detector`` share a name, so runtime + discovery calls them ``scope_x_detector`` and ``scope_y_detector``. +- ``/scope/z/goal_status`` is an app named ``goal_status``: the last segment of + every sub-path source below. A notification keyed on that segment lands on it. +- ``/scope/w/solo`` has a unique name, so its runtime id is ``solo``. +- ``/scope/n/arm`` and ``/scope/n/arm/wrist`` nest, so a source below the + wrist node is owned by both apps. The inner one has the larger id in every + mode, so "most specific owner" and "smallest id" name different apps. + +The manifest (hybrid and manifest_only) gives the detector an id that is not +its node name, puts it in a subarea, hosts it in a Function through its +Component, declares an external app that reports under its bare id, and binds +two apps to one node path (spelled two ways) that has no node. + +A negative check is only as good as the stream it reads, so every trigger +stream ends with a probe fault that must reach it. Events on one gateway are +delivered in order, so a probe that arrived means every earlier event has +been judged. +""" + +import json +import os +import tempfile +import threading +import time +import unittest + +from launch import LaunchDescription +from launch.actions import TimerAction +import launch_testing +import launch_testing.actions +import rclpy +from rclpy.executors import SingleThreadedExecutor +from rclpy.node import Node +import requests +from ros2_medkit_msgs.msg import Fault +from ros2_medkit_msgs.srv import ClearFault, ReportFault + +from ros2_medkit_test_utils.constants import ( + ALLOWED_EXIT_CODES, + API_BASE_PATH, + DISCOVERY_TIMEOUT, + FAULT_TIMEOUT, + get_test_port, + ROSBAG_TIMEOUT, +) +from ros2_medkit_test_utils.gateway_test_case import GatewayTestCase +from ros2_medkit_test_utils.launch_helpers import ( + create_fault_manager_node, + create_gateway_node, +) +from ros2_medkit_test_utils.service_round_trip import answer_once_each + +from std_msgs.msg import Float32 + + +HYBRID = 'hybrid' +RUNTIME = 'runtime_only' +MANIFEST = 'manifest_only' +PORTS = {HYBRID: get_test_port(0), RUNTIME: get_test_port(1), MANIFEST: get_test_port(2)} + +DETECTOR_X = '/scope/x/detector' +DETECTOR_Y = '/scope/y/detector' +GOAL_STATUS = '/scope/z/goal_status' +SOLO = '/scope/w/solo' +EXTERNAL_APP = 'plc-app' +SIGNAL_TOPIC = '/scope/signal' + +MANIFEST_YAML = """\ +manifest_version: "1.0" +metadata: + name: fault-scope-notifications + version: "1.0.0" +config: + unmanifested_nodes: ignore +areas: + - id: plant + name: Plant + namespace: /scope + subareas: + - id: cell + name: Cell + namespace: /scope/x + - id: aux + name: Aux + namespace: /aux + - id: yard + name: Yard + namespace: /yard + - id: nest + name: Nest + namespace: /nest +components: + - id: detector-unit + name: Detector unit + area: cell + - id: aux-unit + name: Aux unit + area: aux + - id: plc-unit + name: PLC unit + area: yard + - id: nest-unit + name: Nest unit + area: nest +apps: + - id: detector-app + name: Detector + is_located_on: detector-unit + ros_binding: + node_name: detector + namespace: /scope/x + - id: goal_status + name: Goal status + is_located_on: aux-unit + ros_binding: + node_name: goal_status + namespace: /scope/z + - id: plc-app + name: PLC app + external: true + is_located_on: plc-unit + - id: arm-app + name: Arm + is_located_on: nest-unit + ros_binding: + node_name: arm + namespace: /scope/n + - id: wrist-app + name: Wrist + is_located_on: nest-unit + ros_binding: + node_name: wrist + namespace: /scope/n/arm + - id: twin-a + name: Twin A + is_located_on: nest-unit + ros_binding: + node_name: twin + namespace: /scope/t + # The same node path spelled with a trailing slash: an identical binding + # would be rejected as a duplicate (R010). + - id: twin-b + name: Twin B + is_located_on: nest-unit + ros_binding: + node_name: twin + namespace: /scope/t/ +functions: + - id: detection + name: Detection + hosted_by: + - detector-unit + - id: site-watch + name: Site watch + hosted_by: + - detector-app + - goal_status +""" + + +def _write_manifest(): + fd, path = tempfile.mkstemp(prefix='medkit_fault_scope_', suffix='.yaml') + os.write(fd, MANIFEST_YAML.encode('utf-8')) + os.close(fd) + return path + + +MANIFEST_PATH = _write_manifest() + +# Every stream a case opens stays open until its trigger is deleted; the +# default two SSE clients and six HTTP workers would turn later streams +# into 503s, which read as "no event". +STREAM_CAPACITY = {'sse.max_clients': 40, 'server.http_thread_pool_size': 48} + + +def generate_test_description(): + manifest_params = { + 'discovery.manifest_path': MANIFEST_PATH, + 'discovery.manifest_strict_validation': False, + } + gateways = [ + create_gateway_node( + port=PORTS[HYBRID], name='gateway_hybrid', + extra_params={**STREAM_CAPACITY, **manifest_params, 'discovery.mode': 'hybrid'}), + create_gateway_node( + port=PORTS[RUNTIME], name='gateway_runtime', + extra_params={**STREAM_CAPACITY, 'discovery.mode': 'runtime_only'}), + create_gateway_node( + port=PORTS[MANIFEST], name='gateway_manifest', + extra_params={**STREAM_CAPACITY, **manifest_params, + 'discovery.mode': 'manifest_only'}), + ] + fault_manager = create_fault_manager_node( + rosbag_topics=[SIGNAL_TOPIC], + snapshot_topics=[SIGNAL_TOPIC], + # One FAILED report confirms; a later report from another source is + # a fault_updated event on the confirmed fault. Two PASSED reports heal. + extra_params={'confirmation_threshold': -1, 'healing_enabled': True, + 'healing_threshold': 1}, + ) + return ( + LaunchDescription([ + *gateways, + TimerAction(period=2.0, actions=[fault_manager]), + launch_testing.actions.ReadyToTest(), + ]), + {'gateway_node': gateways[0]}, + ) + + +def _url(mode, path): + return f'http://localhost:{PORTS[mode]}{API_BASE_PATH}{path}' + + +class SseReader: + """Collects the JSON data frames of one SSE stream on a background thread. + + The thread owns the connection. A trigger stream sends its headers with + the first event or keepalive, up to 15 s later, and the trigger buffers + events until then, so the constructor does not wait for it. Closing a + streamed response from another thread blocks until the next frame + arrives, so close() only asks the thread to stop at its next frame; a + deleted trigger ends its stream at once. + """ + + def __init__(self, url, label): + self.label = label + self.payloads = [] + self.error = None + self._url = url + self._lock = threading.Lock() + self._closed = threading.Event() + self._thread = threading.Thread(target=self._pump, daemon=True) + self._thread.start() + + def _pump(self): + try: + response = requests.get(self._url, stream=True, timeout=(5, 300)) + except requests.exceptions.RequestException as e: + self.error = f'{self._url}: {e}' + return + try: + if response.status_code != 200: + self.error = f'{self._url} answered {response.status_code}: {response.text}' + return + data = None + for line in response.iter_lines(decode_unicode=True): + if self._closed.is_set(): + return + if line is None: + continue + if line == '': + if data is not None: + with self._lock: + self.payloads.append(json.loads(data)) + data = None + continue + if line.startswith('data:'): + data = line[len('data:'):].strip() + except Exception: # noqa: BLE001 - the stream ends when its trigger goes + pass + finally: + response.close() + + def snapshot(self): + with self._lock: + return list(self.payloads) + + def close(self): + self._closed.set() + + +def _trigger_events(reader): + """(fault_code, event_type) of every fault payload a trigger stream delivered.""" + events = [] + for envelope in reader.snapshot(): + fault = envelope.get('payload', {}) + events.append((fault.get('fault_code'), fault.get('event_type'))) + return events + + +def _trigger_details(reader): + """(fault_code, event_type, status, source count) of every delivered fault payload.""" + return [(f.get('fault_code'), f.get('event_type'), f.get('status'), + len(f.get('reporting_sources', []))) + for f in (envelope.get('payload', {}) for envelope in reader.snapshot())] + + +def _stream_events(reader): + """(fault_code, event_type, source count) -> x-medkit of every /faults/stream event.""" + return {(p['fault']['fault_code'], p['event_type'], len(p['fault']['reporting_sources'])): + p.get('x-medkit') for p in reader.snapshot() if 'fault' in p} + + +class TestFaultScopeNotifications(GatewayTestCase): + """Triggers and the fault stream follow the fault-listing scope rule.""" + + BASE_URL = f'http://localhost:{PORTS[HYBRID]}{API_BASE_PATH}' + MIN_EXPECTED_APPS = 3 + REQUIRED_APPS = {'detector-app', 'goal_status', EXTERNAL_APP} + + @classmethod + def setUpClass(cls): + rclpy.init() + cls._nodes = [ + Node('detector', namespace='/scope/x'), + Node('detector', namespace='/scope/y'), + Node('goal_status', namespace='/scope/z'), + Node('solo', namespace='/scope/w'), + Node('arm', namespace='/scope/n'), + Node('wrist', namespace='/scope/n/arm'), + ] + cls._reporter = Node('fault_scope_reporter') + cls._report_client = cls._reporter.create_client( + ReportFault, '/fault_manager/report_fault') + cls._clear_client = cls._reporter.create_client( + ClearFault, '/fault_manager/clear_fault') + # The rosbag ring buffer needs data on the captured topic before a + # fault confirms, and the fault manager resolves an explicit topic's + # type only in its first seconds, so the publisher starts first. + signal = cls._reporter.create_publisher(Float32, SIGNAL_TOPIC, 10) + cls._reporter.create_timer(0.05, lambda: signal.publish(Float32(data=1.0))) + cls._executor = SingleThreadedExecutor() + for node in [*cls._nodes, cls._reporter]: + cls._executor.add_node(node) + cls._spin_thread = threading.Thread(target=cls._executor.spin, daemon=True) + cls._spin_thread.start() + + super().setUpClass() + assert cls._report_client.wait_for_service(timeout_sec=30.0), \ + 'report_fault service not available' + answer_once_each(cls._reporter, (cls._report_client, cls._clear_client), 30.0, spin=False) + cls._wait_for_entities() + cls._host_component = cls._runtime_host_component() + cls._report_counter = 0 + cls._prime_fault_streams() + + @classmethod + def tearDownClass(cls): + cls._executor.shutdown() + cls._spin_thread.join(timeout=5) + for node in [*cls._nodes, cls._reporter]: + node.destroy_node() + rclpy.shutdown() + + # ------------------------------------------------------------------ + # Fixture helpers + # ------------------------------------------------------------------ + + @classmethod + def _wait_until(cls, what, condition, timeout=DISCOVERY_TIMEOUT): + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + try: + if condition(): + return + except requests.exceptions.RequestException: + pass + time.sleep(0.5) + raise AssertionError(f'timed out after {timeout}s waiting for {what}') + + @classmethod + def _app_online(cls, mode, app_id): + response = requests.get(_url(mode, f'/apps/{app_id}'), timeout=5) + if response.status_code != 200: + return False + return response.json().get('x-medkit', {}).get('is_online') is True + + @classmethod + def _entity_exists(cls, mode, path): + return requests.get(_url(mode, path), timeout=5).status_code == 200 + + @classmethod + def _wait_for_entities(cls): + # Hybrid links a manifest app to its node on a refresh; until then the + # node -> app map is empty and the test would measure the unlinked case. + for app_id in ('detector-app', 'goal_status', 'arm-app', 'wrist-app'): + cls._wait_until(f'hybrid app {app_id} linked', + lambda a=app_id: cls._app_online(HYBRID, a)) + for app_id in ('scope_x_detector', 'scope_y_detector', 'goal_status', 'solo', + 'arm', 'wrist'): + cls._wait_until(f'runtime app {app_id}', + lambda a=app_id: cls._entity_exists(RUNTIME, f'/apps/{a}')) + for path in ('/apps/detector-app', '/apps/goal_status', f'/apps/{EXTERNAL_APP}', + '/apps/arm-app', '/apps/wrist-app', '/apps/twin-a', '/apps/twin-b', + '/areas/plant', '/functions/detection', '/functions/site-watch'): + cls._wait_until(f'manifest_only {path}', + lambda p=path: cls._entity_exists(MANIFEST, p)) + for path in ('/apps/twin-a', '/apps/twin-b'): + cls._wait_until(f'hybrid {path}', lambda p=path: cls._entity_exists(HYBRID, p)) + + @classmethod + def _runtime_host_component(cls): + items = requests.get(_url(RUNTIME, '/components'), timeout=5).json().get('items', []) + assert items, 'runtime_only gateway has no host component' + return items[0]['id'] + + @classmethod + def _next_code(cls, stem): + cls._report_counter += 1 + return f'{stem}_{cls._report_counter}' + + @classmethod + def _send_report(cls, code, source, answer_within, passed=False): + """Send one FAILED (or PASSED) report; True once the fault manager accepted it in time.""" + request = ReportFault.Request() + request.fault_code = code + request.event_type = (ReportFault.Request.EVENT_PASSED if passed + else ReportFault.Request.EVENT_FAILED) + request.severity = Fault.SEVERITY_ERROR + request.description = 'fault scope notification test' + request.source_id = source + future = cls._report_client.call_async(request) + deadline = time.monotonic() + answer_within + while not future.done() and time.monotonic() < deadline: + time.sleep(0.02) + return future.done() and future.result().accepted + + @classmethod + def _report(cls, code, source): + """Report one FAILED event and wait for the fault manager to accept it. + + Waiting keeps the reports ordered: the fault manager publishes the + event before it answers, so the next report's event comes after it. + """ + assert cls._send_report(code, source, answer_within=10.0), \ + f'report_fault {code} from {source} got no accepted answer' + + @classmethod + def _report_passed(cls, code, source): + assert cls._send_report(code, source, answer_within=10.0, passed=True), \ + f'PASSED report {code} from {source} got no accepted answer' + + @classmethod + def _clear(cls, code, source): + """Clear the record of ``code`` owned by ``source`` and wait for the answer.""" + request = ClearFault.Request() + request.fault_code = code + request.source_id = source + future = cls._clear_client.call_async(request) + deadline = time.monotonic() + 10.0 + while not future.done() and time.monotonic() < deadline: + time.sleep(0.02) + assert future.done() and future.result().success, f'clear_fault {code} failed' + + @classmethod + def _prime_fault_streams(cls): + """Prove each gateway receives fault events before any case measures them. + + A service reply can be lost while the fault manager is still matching + this client, even though the request arrived, so priming does not wait + for answers. It ends with a round trip that must be answered. + """ + readers = {mode: SseReader(_url(mode, '/faults/stream'), f'{mode} prime') + for mode in PORTS} + try: + deadline = time.monotonic() + FAULT_TIMEOUT + pending = set(readers) + while pending and time.monotonic() < deadline: + code = cls._next_code('SCOPE_PRIME') + cls._send_report(code, '/scope/prime', answer_within=0.0) + settle = time.monotonic() + 1.0 + while pending and time.monotonic() < settle: + for mode in list(pending): + codes = {p.get('fault', {}).get('fault_code') + for p in readers[mode].snapshot()} + if code in codes: + pending.discard(mode) + time.sleep(0.1) + assert not pending, \ + f'no fault event reached {[(m, readers[m].error) for m in sorted(pending)]}' + while not cls._send_report(cls._next_code('SCOPE_PRIME'), '/scope/prime', + answer_within=2.0): + assert time.monotonic() < deadline, 'report_fault never answered' + finally: + for reader in readers.values(): + reader.close() + + def _watch(self, mode, entity_path): + """Create a multishot OnChange trigger on ``{entity_path}/faults`` and stream it.""" + body = { + 'resource': f'{API_BASE_PATH}{entity_path}/faults', + 'trigger_condition': {'condition_type': 'OnChange'}, + 'multishot': True, + 'lifetime': 600, + } + response = requests.post(_url(mode, f'{entity_path}/triggers'), json=body, timeout=10) + self.assertEqual(response.status_code, 201, + f'{mode} trigger on {entity_path}: {response.text}') + trigger = response.json() + delete_url = _url(mode, f'{entity_path}/triggers/{trigger["id"]}') + self.addCleanup(lambda: requests.delete(delete_url, timeout=10)) + event_source = trigger['event_source'].removeprefix(API_BASE_PATH) + reader = SseReader(_url(mode, event_source), f'{mode} {entity_path}') + self.addCleanup(reader.close) + return reader + + def _probe(self, targets): + """Report one probe per source and wait until it reaches every listed reader. + + ``targets`` maps a reporting source to the readers whose entity owns it. + Returns the failures, one line per reader the probe did not reach. + """ + probes = [] + for source, readers in targets.items(): + code = self._next_code('SCOPE_PROBE') + self._report(code, source) + probes.extend((reader, code) for reader in readers) + deadline = time.monotonic() + FAULT_TIMEOUT + missing = list(probes) + while missing and time.monotonic() < deadline: + missing = [(reader, code) for reader, code in missing + if code not in {c for c, _ in _trigger_events(reader)}] + time.sleep(0.2) + return [f'{reader.label}: probe {code} never arrived; stream is not live ' + f'(error: {reader.error}, events: {_trigger_events(reader)})' + for reader, code in missing] + + def _listed_codes(self, mode, entity_path): + response = requests.get(_url(mode, f'{entity_path}/faults'), timeout=10) + self.assertEqual(response.status_code, 200, f'{mode} {entity_path}: {response.text}') + return {item['fault_code'] for item in response.json().get('items', [])} + + def _check(self, failures, reader, expected, unexpected): + """Record missing expected (code, event_type) pairs and forbidden ones that arrived.""" + events = _trigger_events(reader) + for pair in expected: + if pair not in events: + failures.append(f'{reader.label}: missing {pair}; got {events}') + for pair in unexpected: + if pair in events: + failures.append(f'{reader.label}: must not receive {pair}; got {events}') + + def _check_listing(self, failures, mode, entity_path, listed, not_listed): + codes = self._listed_codes(mode, entity_path) + for code in listed: + if code not in codes: + failures.append(f'{mode} GET {entity_path}/faults does not list {code}: {codes}') + for code in not_listed: + if code in codes: + failures.append(f'{mode} GET {entity_path}/faults lists {code}: {codes}') + + # ------------------------------------------------------------------ + # Triggers + # ------------------------------------------------------------------ + + # @verifies REQ_INTEROP_097 + def test_01_subpath_source_fires_on_every_owner(self): + """A sub-path source fires the triggers of its owning app and every ancestor. + + Depth 1 and depth 2 below the node FQN, in all three discovery modes; + on a Component, on an Area that holds the app only through a subarea, + and on a Function that hosts the app through its Component. + """ + watched = {} + for mode in (HYBRID, MANIFEST): + for path in ('/apps/detector-app', '/components/detector-unit', + '/areas/plant', '/functions/detection'): + watched[(mode, path)] = self._watch(mode, path) + host = f'/components/{self._host_component}' + for path in ('/apps/scope_x_detector', '/apps/solo', host): + watched[(RUNTIME, path)] = self._watch(RUNTIME, path) + + d1 = self._next_code('SUB_D1') + d2 = self._next_code('SUB_D2') + s1 = self._next_code('SUB_S1') + s2 = self._next_code('SUB_S2') + self._report(d1, f'{DETECTOR_X}/goal_status') + self._report(d2, f'{DETECTOR_X}/diag/goal_status') + self._report(s1, f'{SOLO}/goal_status') + self._report(s2, f'{SOLO}/diag/goal_status') + + detector_readers = [r for (mode, path), r in watched.items() + if path not in ('/apps/solo',)] + failures = self._probe({ + DETECTOR_X: detector_readers, + SOLO: [watched[(RUNTIME, '/apps/solo')], watched[(RUNTIME, host)]], + }) + + confirmed = 'fault_confirmed' + for (mode, path), reader in watched.items(): + if path == '/apps/solo': + codes = (s1, s2) + elif path == host: + codes = (d1, d2, s1, s2) + else: + codes = (d1, d2) + self._check(failures, reader, [(c, confirmed) for c in codes], []) + self._check_listing(failures, mode, path, codes, []) + if failures: + self.fail('\n'.join(failures)) + + # @verifies REQ_INTEROP_097 + def test_02_namesake_of_last_segment_is_not_notified(self): + """The app named like a sub-path's last segment neither lists nor hears the fault.""" + namesakes = {mode: self._watch(mode, '/apps/goal_status') for mode in PORTS} + + d1 = self._next_code('NAME_D1') + d2 = self._next_code('NAME_D2') + s1 = self._next_code('NAME_S1') + self._report(d1, f'{DETECTOR_X}/goal_status') + self._report(d2, f'{DETECTOR_X}/diag/goal_status') + self._report(s1, f'{SOLO}/goal_status') + + failures = self._probe({GOAL_STATUS: list(namesakes.values())}) + for mode, reader in namesakes.items(): + forbidden = [(code, 'fault_confirmed') for code in (d1, d2, s1)] + self._check(failures, reader, [], forbidden) + self._check_listing(failures, mode, '/apps/goal_status', [], [d1, d2, s1]) + if failures: + self.fail('\n'.join(failures)) + + # @verifies REQ_INTEROP_097 + def test_03_exact_fqn_fires_in_manifest_only_and_on_a_name_collision(self): + """An exact node FQN fires its app's trigger whatever the app id looks like.""" + manifest_detector = self._watch(MANIFEST, '/apps/detector-app') + hybrid_detector = self._watch(HYBRID, '/apps/detector-app') + runtime_x = self._watch(RUNTIME, '/apps/scope_x_detector') + runtime_y = self._watch(RUNTIME, '/apps/scope_y_detector') + + exact_x = self._next_code('EXACT_X') + exact_y = self._next_code('EXACT_Y') + self._report(exact_x, DETECTOR_X) + self._report(exact_y, DETECTOR_Y) + + failures = self._probe({ + DETECTOR_X: [manifest_detector, hybrid_detector, runtime_x], + DETECTOR_Y: [runtime_y], + }) + confirmed = 'fault_confirmed' + self._check(failures, manifest_detector, [(exact_x, confirmed)], []) + self._check(failures, hybrid_detector, [(exact_x, confirmed)], []) + self._check(failures, runtime_x, [(exact_x, confirmed)], [(exact_y, confirmed)]) + self._check(failures, runtime_y, [(exact_y, confirmed)], [(exact_x, confirmed)]) + if failures: + self.fail('\n'.join(failures)) + + # @verifies REQ_INTEROP_097 + def test_04_record_outside_the_scope_fires_no_component_area_or_function(self): + """A Component, Area or Function trigger stays silent for a record it does not own. + + The detector's record is owned by detector-unit, the plant area and the + detection function; goal_status's record by aux-unit and the aux area. + Each entity is checked against the other's record. + """ + watched = {} + for mode in (HYBRID, MANIFEST): + for path in ('/components/detector-unit', '/areas/plant', '/areas/aux', + '/functions/detection'): + watched[(mode, path)] = self._watch(mode, path) + + detector_code = self._next_code('OUT_DETECTOR') + goal_code = self._next_code('OUT_GOAL') + self._report(detector_code, f'{DETECTOR_X}/diag') + self._report(goal_code, f'{GOAL_STATUS}/diag') + + aux = [r for (_, path), r in watched.items() if path == '/areas/aux'] + failures = self._probe({ + DETECTOR_X: [r for (_, path), r in watched.items() if path != '/areas/aux'], + GOAL_STATUS: aux, + }) + confirmed = 'fault_confirmed' + for (mode, path), reader in watched.items(): + owned, foreign = ((goal_code, detector_code) if path == '/areas/aux' + else (detector_code, goal_code)) + self._check(failures, reader, [(owned, confirmed)], [(foreign, confirmed)]) + self._check_listing(failures, mode, path, [owned], [foreign]) + if failures: + self.fail('\n'.join(failures)) + + # @verifies REQ_INTEROP_097 + def test_05_external_app_bare_id_fires_its_app_and_component(self): + """A source equal to an external app's id fires that app and its Component.""" + watched = [self._watch(mode, path) + for mode in (HYBRID, MANIFEST) + for path in (f'/apps/{EXTERNAL_APP}', '/components/plc-unit')] + code = self._next_code('EXTERNAL') + self._report(code, EXTERNAL_APP) + failures = self._probe({EXTERNAL_APP: watched}) + for reader in watched: + self._check(failures, reader, [(code, 'fault_confirmed')], []) + if failures: + self.fail('\n'.join(failures)) + + # ------------------------------------------------------------------ + # Global fault stream + # ------------------------------------------------------------------ + + def test_06_fault_stream_names_the_owner_and_its_bag_downloads(self): + """``x-medkit`` names the app that lists the record, or is absent. + + The documented use of ``x-medkit`` is the rosbag address + ``/{entity_type}/{entity_id}/bulk-data/rosbags/{fault_code}``, so a named + entity must answer 200 there. A record no app owns carries no + ``x-medkit``, and the namesake of a sub-path's last segment is never + named. The owner's rosbag listing includes the sub-path bag and leaves + out the bag of a record it does not list. + """ + streams = {mode: SseReader(_url(mode, '/faults/stream'), f'{mode} /faults/stream') + for mode in PORTS} + for reader in streams.values(): + self.addCleanup(reader.close) + + detector_sub = self._next_code('STREAM_DETECTOR_SUB') + solo_sub = self._next_code('STREAM_SOLO_SUB') + unowned = self._next_code('STREAM_UNOWNED') + self._report(detector_sub, f'{DETECTOR_X}/goal_status') + self._report(solo_sub, f'{SOLO}/goal_status') + self._report(unowned, '/scope/a/unowned') + + def events(reader): + return {(p['fault']['fault_code'], p['event_type']): p.get('x-medkit') + for p in reader.snapshot() if 'fault' in p} + + last = (unowned, 'fault_confirmed') + deadline = time.monotonic() + FAULT_TIMEOUT + while time.monotonic() < deadline: + if all(last in events(r) for r in streams.values()): + break + time.sleep(0.2) + + owner = { + HYBRID: {detector_sub: 'detector-app', solo_sub: None}, + MANIFEST: {detector_sub: 'detector-app', solo_sub: None}, + RUNTIME: {detector_sub: 'scope_x_detector', solo_sub: 'solo'}, + } + failures = [] + named = [] + for mode, reader in streams.items(): + seen = events(reader) + if last not in seen: + failures.append(f'{mode}: {last} never arrived on /faults/stream ' + f'(error: {reader.error}): {seen}') + continue + if seen[last] is not None: + failures.append(f'{mode}: {unowned} is owned by no app, ' + f'x-medkit must be absent: {seen[last]}') + for code, app_id in owner[mode].items(): + got = seen.get((code, 'fault_confirmed'), 'missing') + want = None if app_id is None else {'entity_type': 'apps', 'entity_id': app_id} + if got != want: + failures.append(f'{mode}: x-medkit for {code} is {got}, want {want}') + elif app_id is not None: + named.append((mode, app_id, code)) + + for mode, app_id, code in named: + status = self._wait_for_status( + _url(mode, f'/apps/{app_id}/bulk-data/rosbags/{code}'), 200) + if status != 200: + failures.append(f'{mode}: rosbag of {code} at the x-medkit address of ' + f'{app_id} answered {status}') + + listed = self._listed_bag_codes(HYBRID, '/apps/detector-app', detector_sub) + if detector_sub not in listed: + failures.append(f'hybrid /apps/detector-app/bulk-data/rosbags does not list the ' + f'bag of {detector_sub}: {sorted(listed)}') + if unowned in listed: + failures.append(f'hybrid /apps/detector-app/bulk-data/rosbags lists the bag of ' + f'{unowned}, which the app does not list') + if failures: + self.fail('\n'.join(failures)) + + # ------------------------------------------------------------------ + # Record end and nested owners + # ------------------------------------------------------------------ + + def _open_fault_streams(self): + streams = {mode: SseReader(_url(mode, '/faults/stream'), f'{mode} /faults/stream') + for mode in PORTS} + for reader in streams.values(): + self.addCleanup(reader.close) + return streams + + def _await_stream_event(self, streams, key): + """Wait until every fault stream carries ``key``; returns the failures.""" + deadline = time.monotonic() + FAULT_TIMEOUT + while time.monotonic() < deadline: + if all(key in _stream_events(r) for r in streams.values()): + return [] + time.sleep(0.2) + return [f'{mode}: {key} never arrived on /faults/stream (error: {r.error}): ' + f'{sorted(_stream_events(r))}' for mode, r in streams.items() + if key not in _stream_events(r)] + + @staticmethod + def _check_x_medkit(failures, streams, key, wanted): + """``wanted`` maps each mode to the app id x-medkit must name, or None for absent.""" + for mode, reader in streams.items(): + got = _stream_events(reader).get(key, 'missing') + want = (None if wanted[mode] is None + else {'entity_type': 'apps', 'entity_id': wanted[mode]}) + if got != want: + failures.append(f'{mode}: x-medkit for {key} is {got}, want {want}') + + def _listed_all(self, mode, entity_path): + """Fault code -> status of every fault ``GET {entity_path}/faults?status=all`` lists.""" + response = requests.get(_url(mode, f'{entity_path}/faults'), + params={'status': 'all'}, timeout=10) + self.assertEqual(response.status_code, 200, f'{mode} {entity_path}: {response.text}') + return {item['fault_code']: item.get('status') + for item in response.json().get('items', [])} + + # @verifies REQ_INTEROP_097 + def test_07_clear_and_heal_reach_exactly_the_entities_that_own_the_fault(self): + """A record's end reaches the entities whose scope holds it, and no other app. + + The default listing drops a cleared or healed record, so the + fault_cleared event is how a subscriber learns the record ended. Its + payload status tells a clear (CLEARED) from a heal (HEALED). + """ + owner = {HYBRID: '/apps/detector-app', MANIFEST: '/apps/detector-app', + RUNTIME: '/apps/scope_x_detector'} + host = f'/components/{self._host_component}' + group_path = {HYBRID: '/functions/site-watch', MANIFEST: '/functions/site-watch', + RUNTIME: host} + detectors = {mode: self._watch(mode, owner[mode]) for mode in PORTS} + goals = {mode: self._watch(mode, '/apps/goal_status') for mode in PORTS} + groups = {mode: self._watch(mode, group_path[mode]) for mode in PORTS} + streams = self._open_fault_streams() + + cleared = self._next_code('END_CLEARED_SUB') + healed = self._next_code('END_HEALED_SUB') + self._report(cleared, f'{DETECTOR_X}/goal_status') + self._report(healed, f'{DETECTOR_X}/diag') + self._clear(cleared, f'{DETECTOR_X}/goal_status') + self._report_passed(healed, f'{DETECTOR_X}/diag') + self._report_passed(healed, f'{DETECTOR_X}/diag') + + failures = self._probe({ + DETECTOR_X: [*detectors.values(), *groups.values()], + GOAL_STATUS: list(goals.values()), + }) + end = 'fault_cleared' + for mode in PORTS: + ends = {(c, st) for c, e, st, _ in _trigger_details(detectors[mode]) if e == end} + for pair in ((cleared, 'CLEARED'), (healed, 'HEALED')): + if pair not in ends: + failures.append(f'{mode} {owner[mode]}: missing end {pair}; got {ends}') + goal_ends = {c for c, e, _, _ in _trigger_details(goals[mode]) if e == end} + if goal_ends & {cleared, healed}: + failures.append(f'{mode} /apps/goal_status: must not receive ends of ' + f'{sorted(goal_ends & {cleared, healed})}') + group_ends = {c for c, e, _, _ in _trigger_details(groups[mode]) if e == end} + if not {cleared, healed} <= group_ends: + failures.append(f'{mode} {group_path[mode]}: missing ends of ' + f'{sorted({cleared, healed} - group_ends)}') + + listed = self._listed_all(mode, owner[mode]) + if listed.get(cleared) != 'CLEARED' or listed.get(healed) != 'HEALED': + failures.append(f'{mode} GET {owner[mode]}/faults?status=all: {listed}') + + failures += self._await_stream_event(streams, (healed, end, 1)) + owner_app = {HYBRID: 'detector-app', MANIFEST: 'detector-app', + RUNTIME: 'scope_x_detector'} + self._check_x_medkit(failures, streams, (cleared, end, 1), owner_app) + self._check_x_medkit(failures, streams, (healed, end, 1), owner_app) + if failures: + self.fail('\n'.join(failures)) + + def test_09_fault_stream_names_the_nearest_owner_and_breaks_ties_by_id(self): + """Nested node paths: x-medkit names the app owning the longest source. + + Both the arm and the wrist app list a fault from below the wrist node, + and the wrist app, the more specific owner, has the larger id. Two + manifest apps bound to one node path list the same faults; the + smaller id is named. + """ + streams = self._open_fault_streams() + inner = self._next_code('NEST_INNER') + outer = self._next_code('NEST_OUTER') + tie = self._next_code('NEST_TIE') + self._report(inner, '/scope/n/arm/wrist/diag') + self._report(outer, '/scope/n/arm/diag') + self._report(tie, '/scope/t/twin') + + c = 'fault_confirmed' + failures = self._await_stream_event(streams, (tie, c, 1)) + self._check_x_medkit(failures, streams, (inner, c, 1), + {HYBRID: 'wrist-app', MANIFEST: 'wrist-app', RUNTIME: 'wrist'}) + self._check_x_medkit(failures, streams, (outer, c, 1), + {HYBRID: 'arm-app', MANIFEST: 'arm-app', RUNTIME: 'arm'}) + self._check_x_medkit(failures, streams, (tie, c, 1), + {HYBRID: 'twin-a', MANIFEST: 'twin-a', RUNTIME: None}) + # Preconditions: every app the rules choose between does list the fault. + both = {HYBRID: ('/apps/arm-app', '/apps/wrist-app'), + MANIFEST: ('/apps/arm-app', '/apps/wrist-app'), + RUNTIME: ('/apps/arm', '/apps/wrist')} + for mode, paths in both.items(): + for path in paths: + self._check_listing(failures, mode, path, [inner], []) + for mode in (HYBRID, MANIFEST): + for path in ('/apps/twin-a', '/apps/twin-b'): + self._check_listing(failures, mode, path, [tie], []) + if failures: + self.fail('\n'.join(failures)) + + def _wait_for_status(self, url, wanted): + """Poll a download URL until it answers ``wanted``; a bag finalizes after the fault.""" + deadline = time.monotonic() + ROSBAG_TIMEOUT + status = None + while time.monotonic() < deadline: + status = requests.get(url, timeout=10).status_code + if status == wanted: + return status + time.sleep(0.5) + return status + + def _listed_bag_codes(self, mode, entity_path, awaited): + """Fault codes on the entity's rosbag descriptors, once ``awaited`` shows or time is up.""" + deadline = time.monotonic() + ROSBAG_TIMEOUT + codes = set() + while time.monotonic() < deadline: + response = requests.get(_url(mode, f'{entity_path}/bulk-data/rosbags'), timeout=10) + self.assertEqual(response.status_code, 200, response.text) + codes = {code for item in response.json().get('items', []) + for code in item.get('x-medkit', {}).get('fault_codes', [])} + if awaited in codes: + return codes + time.sleep(0.5) + return codes + + +@launch_testing.post_shutdown_test() +class TestShutdown(unittest.TestCase): + + def test_exit_codes(self, proc_info): + """Check all processes exited cleanly (SIGTERM allowed).""" + for info in proc_info: + self.assertIn( + info.returncode, ALLOWED_EXIT_CODES, + f'{info.process_name} exited with code {info.returncode}' + ) diff --git a/src/ros2_medkit_integration_tests/test/features/test_faults_entity_clear_failure.test.py b/src/ros2_medkit_integration_tests/test/features/test_faults_entity_clear_failure.test.py new file mode 100644 index 000000000..493acfe0d --- /dev/null +++ b/src/ros2_medkit_integration_tests/test/features/test_faults_entity_clear_failure.test.py @@ -0,0 +1,179 @@ +# Copyright 2026 bburda +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""A per-entity clear that cannot clear every record says so. + +DELETE /{entity}/faults lists the records under the entity and sends one +ClearFault per record. While no fault manager answers, nothing can be listed +and the route answers 503. When the fault manager refuses one record, the route +answers 500 and names its code in ``failed_codes``; the records it could clear +stay cleared. + +The fault manager clears a record it holds whatever its status, so it refuses +only a record that is gone between the list and the clear. No test can hold +that window open, so refusing_fault_manager.py stands in for it: it lists two +records and refuses the clear of one. + +@verifies REQ_INTEROP_015 +""" + +import os +import sys +import tempfile +import time +import unittest + +from ament_index_python.packages import get_package_share_directory +from launch import LaunchDescription +from launch.actions import ExecuteProcess, RegisterEventHandler +from launch.event_handlers import OnProcessExit +import launch_testing +import launch_testing.actions +import requests + +from ros2_medkit_test_utils.constants import ( + ALLOWED_EXIT_CODES, + API_BASE_PATH, + get_test_port, + get_time_scale, +) +from ros2_medkit_test_utils.launch_helpers import create_gateway_node + +PORT = get_test_port(0) +BASE_URL = f'http://localhost:{PORT}{API_BASE_PATH}' + +TIME_SCALE = get_time_scale() +HTTP_TIMEOUT_SEC = 30.0 * TIME_SCALE +DISCOVERY_TIMEOUT_SEC = 60.0 * TIME_SCALE + +OWNER_A = 'owner-a' +OWNER_B = 'owner-b' +HOST_COMPONENT = 'shared-code-hub' +ACCEPTED_CODE = 'CLEAR_ACCEPTED' +REFUSED_CODE = 'CLEAR_REFUSED' + +FAULT_MANAGER_GATE = os.path.join( + tempfile.gettempdir(), f'test_faults_entity_clear_failure_gate_{os.getpid()}') + + +def generate_test_description(): + manifest_path = os.path.join( + get_package_share_directory('ros2_medkit_gateway'), + 'config', 'examples', 'fault_owner_identity_manifest.yaml', + ) + gateway = create_gateway_node( + port=PORT, + extra_params={ + 'discovery.mode': 'hybrid', + 'discovery.manifest_path': manifest_path, + 'discovery.manifest_strict_validation': False, + 'fault_manager.service_timeout_sec': 2.0, + }, + ) + gate = ExecuteProcess( + cmd=['sh', '-c', f'while [ ! -e "{FAULT_MANAGER_GATE}" ]; do sleep 0.2; done'], + name='fault_manager_gate', + output='screen', + ) + fault_manager = ExecuteProcess( + cmd=[ + sys.executable, + os.path.join(os.path.dirname(os.path.abspath(__file__)), + 'refusing_fault_manager.py'), + ], + name='refusing_fault_manager', + output='screen', + ) + return ( + LaunchDescription([ + gateway, + gate, + RegisterEventHandler(OnProcessExit(target_action=gate, on_exit=[fault_manager])), + launch_testing.actions.ReadyToTest(), + ]), + {'gateway_node': gateway}, + ) + + +def _poll(predicate, *, what, timeout=DISCOVERY_TIMEOUT_SEC): + deadline = time.monotonic() + timeout + last = None + while time.monotonic() < deadline: + try: + last = predicate() + if last: + return last + except requests.exceptions.RequestException as exc: + last = exc + time.sleep(0.3) + raise AssertionError(f'timed out after {timeout}s waiting for {what}; last={last!r}') + + +def _get(path): + return requests.get(f'{BASE_URL}{path}', timeout=HTTP_TIMEOUT_SEC) + + +def _delete(path): + return requests.delete(f'{BASE_URL}{path}', timeout=HTTP_TIMEOUT_SEC) + + +def _statuses(path): + """Code -> status of every item GET {path}?status=all lists.""" + response = _get(f'{path}?status=all') + response.raise_for_status() + return {item['fault_code']: item['status'] for item in response.json().get('items', [])} + + +class TestEntityClearFailure(unittest.TestCase): + """The bulk clear answers 503 before the fault manager and 500 for a refused record.""" + + @classmethod + def setUpClass(cls): + cls.addClassCleanup(lambda: os.path.exists(FAULT_MANAGER_GATE) + and os.unlink(FAULT_MANAGER_GATE)) + for path in (f'/components/{HOST_COMPONENT}', f'/apps/{OWNER_A}', f'/apps/{OWNER_B}'): + _poll(lambda p=path: _get(p).status_code == 200, what=f'{path} to be discovered') + + def test_01_without_a_fault_manager_the_bulk_clear_is_503(self): + response = _delete(f'/components/{HOST_COMPONENT}/faults') + self.assertEqual(response.status_code, 503, response.text) + + def test_02_a_refused_record_clear_is_500_naming_its_code(self): + with open(FAULT_MANAGER_GATE, 'w', encoding='utf-8') as gate: + gate.write('open') + _poll(lambda: len(_statuses(f'/components/{HOST_COMPONENT}/faults')) == 2, + what='the fault manager to list both records') + + response = _delete(f'/components/{HOST_COMPONENT}/faults') + self.assertEqual(response.status_code, 500, response.text) + body = response.json() + self.assertEqual(body['parameters']['failed_codes'], [REFUSED_CODE], body) + self.assertEqual( + _statuses(f'/components/{HOST_COMPONENT}/faults'), + {ACCEPTED_CODE: 'CLEARED', REFUSED_CODE: 'CONFIRMED'}, + 'the record the fault manager accepted must stay cleared') + + # One app's collection: the same answer for the one record it holds. + response = _delete(f'/apps/{OWNER_B}/faults') + self.assertEqual(response.status_code, 500, response.text) + self.assertEqual(response.json()['parameters']['failed_codes'], [REFUSED_CODE]) + + +@launch_testing.post_shutdown_test() +class TestShutdown(unittest.TestCase): + + def test_exit_codes(self, proc_info): + for info in proc_info: + self.assertIn(info.returncode, ALLOWED_EXIT_CODES, + f'{info.process_name} exited with code {info.returncode}') diff --git a/src/ros2_medkit_integration_tests/test/features/test_faults_owner_identity.test.py b/src/ros2_medkit_integration_tests/test/features/test_faults_owner_identity.test.py new file mode 100644 index 000000000..e0b430cd9 --- /dev/null +++ b/src/ros2_medkit_integration_tests/test/features/test_faults_owner_identity.test.py @@ -0,0 +1,625 @@ +#!/usr/bin/env python3 +# Copyright 2026 bburda +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Fault records of one code under SOVD entities, over the whole HTTP stack. + +A fault record is the pair (``fault_code``, owner). A per_source code has one +record per reporting source; any other code has one shared record that every +source reports into. An entity shows each code once: + +* An App shows the records of its own source and of its sub-sources + (``/``) as one entry. A Component or an Area shows every record + of the code under it as one entry. +* An entry of several records is their merge, and ``x-medkit.records`` names + each record: its App, its source and its status. A shared record is named by + its ``reporting_sources``, never by an owner. +* An entry of one record is that record, as the global list shows it. A shared + record looks as it always has: no owner field anywhere. +* DELETE of a code clears every record of it under the entity. +* The global list ``GET /faults`` is not an entity: one item per record. + +Every scenario runs for a per_source code and for a shared one. + +@verifies REQ_INTEROP_012 +@verifies REQ_INTEROP_013 +@verifies REQ_INTEROP_014 +@verifies REQ_INTEROP_015 +""" + +import atexit +import json +import os +import shutil +import tempfile +import threading +import time +import unittest + +from ament_index_python.packages import get_package_share_directory +import launch_testing +import rclpy +from rclpy.node import Node +import requests +from ros2_medkit_msgs.msg import Fault +from ros2_medkit_msgs.srv import ReportFault + +from ros2_medkit_test_utils.constants import ALLOWED_EXIT_CODES +from ros2_medkit_test_utils.gateway_test_case import GatewayTestCase +from ros2_medkit_test_utils.launch_helpers import create_test_launch + + +OWNER_A = 'owner-a' +OWNER_B = 'owner-b' +# Its id extends OWNER_A's, so a prefix match would give it OWNER_A's sub-sources. +NEAR_A = 'owner-a2' +SUB_X = f'{OWNER_A}/x' +SUB_Y = f'{OWNER_A}/y' +HOST_COMPONENT = 'shared-code-hub' +CELL_AREA = 'shared-code-cell' +# per_source codes. SHARED_CODE keeps its name for the recording cases. +SHARED_CODE = 'SHARED_OVERFLOW' +PER_SOURCE = { + 'app': 'APP_PER_SOURCE', + 'sub': 'SUB_PER_SOURCE', + 'merge': 'MERGE_PER_SOURCE', + 'bulk': 'BULK_PER_SOURCE', + 'global': 'GLOBAL_PER_SOURCE', + # Reported only in test_08, so every frame of it comes from that test's reports. + 'stream': 'STREAM_PER_SOURCE', +} +# Default codes: one shared record whatever the number of sources. +SHARED = { + 'pair': 'PAIR_SHARED', + 'solo': 'SOLO_SHARED', + 'sub': 'SUB_SHARED', + 'merge': 'MERGE_SHARED', + 'bulk': 'BULK_SHARED', +} +PRIME_CODE = 'OWNER_IDENTITY_PRIME' +STREAM_CODE = PER_SOURCE['stream'] +FAULT_TIMEOUT = 30.0 +REPORT_TIMEOUT = 2.0 +CONFIG_DIR = tempfile.mkdtemp(prefix='medkit_owner_identity_') +atexit.register(shutil.rmtree, CONFIG_DIR, True) +THRESHOLDS_PATH = os.path.join(CONFIG_DIR, 'fault_thresholds.yaml') +# Keys a shared record never carries. +OWNER_KEYS = ('source_id', 'owner', 'x-medkit') + + +def generate_test_description(): + manifest_path = os.path.join( + get_package_share_directory('ros2_medkit_gateway'), + 'config', 'examples', 'fault_owner_identity_manifest.yaml', + ) + with open(THRESHOLDS_PATH, 'w') as f: + for code in (SHARED_CODE, *PER_SOURCE.values()): + f.write(f'{code}:\n per_source: true\n') + return create_test_launch( + demo_nodes=[], + fault_manager=True, + # Negative threshold: a record confirms on its second FAILED event, and + # each owner debounces on its own reports. + fault_manager_params={ + 'confirmation_threshold': -2, + 'fault_thresholds.config_file': THRESHOLDS_PATH, + }, + gateway_params={ + 'discovery.mode': 'hybrid', + 'discovery.manifest_path': manifest_path, + 'discovery.manifest_strict_validation': False, + # Locking off, so the per-code DELETE publishes no lock contract and + # test_07 reads a 409 on it only if the route declares one itself. + 'locking.enabled': False, + # The fault stream writes at least once a second, so closing it in + # test_08 returns at once instead of after a 30 s keepalive. + 'sse.keepalive_interval_sec': 1, + }, + ) + + +class TestFaultsOwnerIdentity(GatewayTestCase): + """Each entity shows a code once; the global list shows each record.""" + + MIN_EXPECTED_APPS = 3 + REQUIRED_APPS = {OWNER_A, OWNER_B, NEAR_A} + + @classmethod + def setUpClass(cls): + rclpy.init() + cls._reporter = Node('fault_owner_identity_reporter') + cls._report_client = cls._reporter.create_client( + ReportFault, '/fault_manager/report_fault' + ) + super().setUpClass() + assert cls._report_client.wait_for_service(timeout_sec=15.0), \ + 'report_fault service not available' + + @classmethod + def tearDownClass(cls): + cls._reporter.destroy_node() + rclpy.shutdown() + + def _report(self, source_id, *, fault_code=SHARED_CODE, + event_type=ReportFault.Request.EVENT_FAILED, times=4): + """Report under one source, one request at a time. + + Each report waits for its answer, so a burst cannot overrun the + service's request queue, which drops the oldest requests. The reply + path is not discovery-matched by wait_for_service, so an early answer + can be lost even though the record was written: a report that gets no + answer in time is not an error. The behavioural assertion is the HTTP + poll, which retries. + """ + for _ in range(times): + req = ReportFault.Request() + req.fault_code = fault_code + req.event_type = event_type + req.severity = Fault.SEVERITY_ERROR + req.description = f'reported by {source_id}' + req.source_id = source_id + future = self._report_client.call_async(req) + rclpy.spin_until_future_complete(self._reporter, future, timeout_sec=REPORT_TIMEOUT) + + def _records(self, code, *, status='all'): + """Items of the global list for `code`.""" + body = self.get_json(f'/faults?status={status}') + return [item for item in body.get('items', []) if item.get('fault_code') == code] + + def _wait_records(self, code, statuses): + """Wait until the global list holds one record of `code` per expected status. + + `statuses` maps a record's first reporting source to the status it must + reach. Returns those records keyed the same way. + """ + deadline = time.monotonic() + FAULT_TIMEOUT + found = {} + while time.monotonic() < deadline: + found = {item['reporting_sources'][0]: item for item in self._records(code)} + if all(found.get(src, {}).get('status') == want for src, want in statuses.items()): + return found + time.sleep(0.2) + self.fail(f'{code}: records {statuses} not reached; global list has {found}') + + def _raise(self, code, sources, *, confirmed=True): + times = 4 if confirmed else 1 + for source in sources: + self._report(source, fault_code=code, times=times) + + def _entries(self, entity, code, *, status=None): + query = f'?status={status}' if status else '' + body = self.get_json(f'/{entity}/faults{query}') + return [item for item in body.get('items', []) if item.get('fault_code') == code] + + def _assert_record_view_of_shared(self, item, sources): + for key in OWNER_KEYS: + self.assertNotIn(key, item, f'a shared record carries no {key}: {item}') + self.assertEqual(sorted(item['reporting_sources']), sorted(sources), item) + + def _assert_detail_of_shared(self, detail, sources): + self.assertNotIn('owner', detail['x-medkit'], detail) + self.assertNotIn('records', detail['x-medkit'], detail) + self.assertEqual(sorted(detail['x-medkit']['reporting_sources']), sorted(sources)) + + def _wait_status(self, code, source, want): + deadline = time.monotonic() + FAULT_TIMEOUT + status = None + while time.monotonic() < deadline: + status = next((item['status'] for item in self._records(code) + if source in item['reporting_sources']), None) + if status == want: + return + time.sleep(0.2) + self.fail(f'{code} of {source}: status {status}, expected {want}') + + def _raise_both(self): + self._report(OWNER_A) + self._report(OWNER_B) + self.wait_for_fault(f'/apps/{OWNER_A}', SHARED_CODE, max_wait=FAULT_TIMEOUT) + self.wait_for_fault(f'/apps/{OWNER_B}', SHARED_CODE, max_wait=FAULT_TIMEOUT) + + # Method order is alphabetical. Each case raises codes of its own. + + def test_01_a_shared_code_shows_no_owner(self): + """A shared record looks the same on an App, a Component and the global list. + + @verifies REQ_INTEROP_012 + @verifies REQ_INTEROP_013 + """ + pair, solo = SHARED['pair'], SHARED['solo'] + self._raise(pair, [OWNER_A, OWNER_B]) + self._raise(solo, [OWNER_A]) + self.poll_endpoint_until( + '/faults', lambda d: d if sum( + 1 for i in d.get('items', []) + if i.get('fault_code') == pair and sorted(i.get('reporting_sources', [])) + == [OWNER_A, OWNER_B]) == 1 else None, timeout=FAULT_TIMEOUT) + self.wait_for_fault(f'/apps/{OWNER_A}', solo, max_wait=FAULT_TIMEOUT) + + records = self._records(pair) + self.assertEqual(len(records), 1, records) + self._assert_record_view_of_shared(records[0], [OWNER_A, OWNER_B]) + + entries = self._entries(f'components/{HOST_COMPONENT}', pair) + self.assertEqual(len(entries), 1, entries) + self._assert_record_view_of_shared(entries[0], [OWNER_A, OWNER_B]) + self._assert_detail_of_shared( + self.get_json(f'/components/{HOST_COMPONENT}/faults/{pair}'), [OWNER_A, OWNER_B]) + + # An App holds a shared record only when every source is its own. + self.assertEqual(self._entries(f'apps/{OWNER_A}', pair), []) + self.get_json(f'/apps/{OWNER_A}/faults/{pair}', expected_status=404) + entries = self._entries(f'apps/{OWNER_A}', solo) + self.assertEqual(len(entries), 1, entries) + self._assert_record_view_of_shared(entries[0], [OWNER_A]) + self._assert_detail_of_shared(self.get_json(f'/apps/{OWNER_A}/faults/{solo}'), [OWNER_A]) + + def test_02_an_app_addresses_the_record_of_its_source(self): + """Two sources of a per_source code: each App sees, reads and clears its own. + + @verifies REQ_INTEROP_012 + @verifies REQ_INTEROP_013 + @verifies REQ_INTEROP_015 + """ + code = PER_SOURCE['app'] + self._raise(code, [OWNER_A]) + self._raise(code, [OWNER_B], confirmed=False) + records = self._wait_records(code, {OWNER_A: 'CONFIRMED', OWNER_B: 'PREFAILED'}) + + entries = self._entries(f'apps/{OWNER_A}', code) + self.assertEqual(len(entries), 1, entries) + self.assertNotIn('source_id', entries[0], 'vendor fields live inside x-medkit only') + self.assertEqual(entries[0]['reporting_sources'], [OWNER_A]) + self.assertEqual(entries[0]['occurrence_count'], records[OWNER_A]['occurrence_count']) + self.assertEqual(entries[0]['x-medkit']['records'], + [{'app_id': OWNER_A, 'source_id': OWNER_A, 'status': 'CONFIRMED'}]) + + detail = self.get_json(f'/apps/{OWNER_A}/faults/{code}') + self.assertEqual(detail['x-medkit']['owner'], OWNER_A) + self.assertEqual(detail['x-medkit']['reporting_sources'], [OWNER_A]) + self.assertEqual(detail['x-medkit']['status_raw'], 'CONFIRMED') + self.assertEqual(detail['x-medkit']['occurrence_count'], + records[OWNER_A]['occurrence_count']) + # B's record is PREFAILED: a detail of B that merged both records or + # served A's says CONFIRMED. + detail_b = self.get_json(f'/apps/{OWNER_B}/faults/{code}') + self.assertEqual(detail_b['x-medkit']['owner'], OWNER_B) + self.assertEqual(detail_b['x-medkit']['status_raw'], 'PREFAILED') + + self.delete_request(f'/apps/{OWNER_A}/faults/{code}') + self._wait_status(code, OWNER_A, 'CLEARED') + self._wait_status(code, OWNER_B, 'PREFAILED') + + def test_03_sub_sources_of_one_app_are_one_entry(self): + """Records under / merge into the App's one entry, not a neighbour's. + + @verifies REQ_INTEROP_012 + @verifies REQ_INTEROP_013 + @verifies REQ_INTEROP_015 + """ + code, shared = PER_SOURCE['sub'], SHARED['sub'] + self._raise(code, [SUB_X]) + self._raise(code, [SUB_Y], confirmed=False) + self._raise(shared, [SUB_X, SUB_Y]) + records = self._wait_records(code, {SUB_X: 'CONFIRMED', SUB_Y: 'PREFAILED'}) + self.wait_for_fault(f'/apps/{OWNER_A}', shared, max_wait=FAULT_TIMEOUT) + total = sum(r['occurrence_count'] for r in records.values()) + + entries = self._entries(f'apps/{OWNER_A}', code) + self.assertEqual(len(entries), 1, entries) + entry = entries[0] + self.assertEqual(entry['status'], 'CONFIRMED') + self.assertEqual(entry['occurrence_count'], total) + self.assertEqual(entry['reporting_sources'], [SUB_X, SUB_Y]) + self.assertNotIn('source_id', entry) + self.assertEqual(entry['x-medkit']['records'], [ + {'app_id': OWNER_A, 'source_id': SUB_X, 'status': 'CONFIRMED'}, + {'app_id': OWNER_A, 'source_id': SUB_Y, 'status': 'PREFAILED'}, + ]) + + detail = self.get_json(f'/apps/{OWNER_A}/faults/{code}') + self.assertEqual(detail['item']['status']['aggregatedStatus'], 'active') + self.assertEqual(detail['x-medkit']['occurrence_count'], total) + self.assertEqual(detail['x-medkit']['status_raw'], 'CONFIRMED') + self.assertNotIn('owner', detail['x-medkit']) + self.assertEqual(len(detail['x-medkit']['records']), 2, detail) + + shared_entries = self._entries(f'apps/{OWNER_A}', shared) + self.assertEqual(len(shared_entries), 1, shared_entries) + self._assert_record_view_of_shared(shared_entries[0], [SUB_X, SUB_Y]) + + for listed in (code, shared): + self.assertEqual(self._entries(f'apps/{NEAR_A}', listed, status='all'), [], + f'{NEAR_A} must not list a record under {OWNER_A}') + + self.delete_request(f'/apps/{OWNER_A}/faults/{code}') + self._wait_status(code, SUB_X, 'CLEARED') + self._wait_status(code, SUB_Y, 'CLEARED') + + def test_04_a_component_and_an_area_merge_every_app(self): + """Three Apps and a sub-source: one entry per code on the Component and the Area. + + @verifies REQ_INTEROP_012 + @verifies REQ_INTEROP_013 + @verifies REQ_INTEROP_015 + """ + code, shared = PER_SOURCE['merge'], SHARED['merge'] + sources = [OWNER_A, OWNER_B, NEAR_A, SUB_X] + self._raise(code, sources) + self._raise(shared, sources) + records = self._wait_records(code, dict.fromkeys(sources, 'CONFIRMED')) + self.poll_endpoint_until( + f'/components/{HOST_COMPONENT}/faults', lambda d: d if any( + i.get('fault_code') == shared and len(i.get('reporting_sources', [])) == 4 + for i in d.get('items', [])) else None, timeout=FAULT_TIMEOUT) + total = sum(r['occurrence_count'] for r in records.values()) + expected_records = sorted([ + {'app_id': OWNER_A, 'source_id': OWNER_A, 'status': 'CONFIRMED'}, + {'app_id': OWNER_A, 'source_id': SUB_X, 'status': 'CONFIRMED'}, + {'app_id': OWNER_B, 'source_id': OWNER_B, 'status': 'CONFIRMED'}, + {'app_id': NEAR_A, 'source_id': NEAR_A, 'status': 'CONFIRMED'}, + ], key=lambda r: r['source_id']) + + for entity in (f'components/{HOST_COMPONENT}', f'areas/{CELL_AREA}'): + entries = self._entries(entity, code) + self.assertEqual(len(entries), 1, f'{entity}: {entries}') + self.assertEqual(entries[0]['occurrence_count'], total, entity) + self.assertEqual(entries[0]['reporting_sources'], sorted(sources), entity) + self.assertEqual(sorted(entries[0]['x-medkit']['records'], + key=lambda r: r['source_id']), expected_records, entity) + shared_entries = self._entries(entity, shared) + self.assertEqual(len(shared_entries), 1, f'{entity}: {shared_entries}') + self._assert_record_view_of_shared(shared_entries[0], sources) + + detail = self.get_json(f'/components/{HOST_COMPONENT}/faults/{code}') + self.assertEqual(detail['x-medkit']['occurrence_count'], total) + self.assertEqual(len(detail['x-medkit']['records']), 4, detail) + self._assert_detail_of_shared( + self.get_json(f'/areas/{CELL_AREA}/faults/{shared}'), sources) + + self.delete_request(f'/components/{HOST_COMPONENT}/faults/{code}') + for source in sources: + self._wait_status(code, source, 'CLEARED') + self.delete_request(f'/areas/{CELL_AREA}/faults/{shared}') + self._wait_status(shared, OWNER_B, 'CLEARED') + + def test_05_the_bulk_clear_takes_every_record_under_the_entity(self): + """DELETE /apps/{id}/faults clears its records and its sub-sources', nobody else's. + + @verifies REQ_INTEROP_014 + """ + code, shared = PER_SOURCE['bulk'], SHARED['bulk'] + self._raise(code, [OWNER_A, SUB_X, OWNER_B]) + self._raise(shared, [OWNER_A]) + self._wait_records(code, {OWNER_A: 'CONFIRMED', SUB_X: 'CONFIRMED', OWNER_B: 'CONFIRMED'}) + self._wait_records(shared, {OWNER_A: 'CONFIRMED'}) + + self.delete_request(f'/apps/{OWNER_A}/faults') + + self._wait_status(code, OWNER_A, 'CLEARED') + self._wait_status(code, SUB_X, 'CLEARED') + self._wait_status(shared, OWNER_A, 'CLEARED') + self._wait_status(code, OWNER_B, 'CONFIRMED') + + def test_06_the_global_list_shows_one_item_per_record(self): + """GET /faults: a record of a per_source code names its App and source. + + @verifies REQ_INTEROP_012 + """ + code = PER_SOURCE['global'] + self._raise(code, [OWNER_A, OWNER_B]) + records = self._wait_records(code, {OWNER_A: 'CONFIRMED', OWNER_B: 'CONFIRMED'}) + + self.assertEqual(len(self._records(code)), 2) + for source, item in records.items(): + self.assertNotIn('source_id', item, 'vendor fields live inside x-medkit only') + self.assertEqual(item['x-medkit']['records'], + [{'app_id': source, 'source_id': source, 'status': 'CONFIRMED'}]) + self.assertNotIn('owner', item) + + def test_07_the_openapi_document_declares_the_record_contract(self): + """The served spec says what the routes now do. + + @verifies REQ_INTEROP_013 + """ + spec = self.poll_endpoint_until('/docs', lambda d: d if 'openapi' in d else None) + + item = spec['components']['schemas']['FaultListItem'] + self.assertIn('x-medkit', item['properties'], + 'the published fault item must declare x-medkit') + self.assertNotIn('source_id', item['properties'], + 'a fault item carries vendor fields only inside x-medkit') + + route = spec['paths']['/apps/{app_id}/faults/{fault_code}'] + # With locking off the DELETE carries no lock contract, so a 409 on + # either method would be the route's own. + root = self.get_json('') + self.assertIs( + root['capabilities']['locking'], False, + 'this launch must run with locking off, or the DELETE 409 below may be the lock one', + ) + self.assertNotIn('x-medkit-lock-guarded', route['delete']) + for method in ('get', 'delete'): + self.assertNotIn( + '409', route[method]['responses'], + f'the per-entity fault {method} never answers 409 for a code', + ) + + def test_076_every_recording_download_declares_its_409(self): + """The download routes answer 409 on an ambiguous code and say so. + + A GET carries no lock marker, so a 409 on these operations is there + only because the route declares the ambiguous-fault refusal. + + @verifies REQ_INTEROP_072 + """ + spec = self.poll_endpoint_until('/docs', lambda d: d if 'openapi' in d else None) + + downloads = { + path: item['get'] for path, item in spec['paths'].items() + if path.endswith('/bulk-data/{category_id}/{file_id}') and 'get' in item + } + # One per entity type that serves bulk data, so the loop below cannot + # pass by matching nothing. + self.assertEqual(len(downloads), 6, sorted(downloads)) + for path, operation in sorted(downloads.items()): + self.assertIn( + '409', operation['responses'], + f'GET {path} answers 409 x-medkit-ambiguous-fault and has to declare it', + ) + + def test_077_a_bare_code_bag_url_refuses_to_pick_an_owner(self): + """A recording URL carrying a bare fault code names no single record. + + Two owners of one code in the component's scope means the compatibility + URL (the pre-recording-id form, which carries a fault code) addresses + neither record, and each owner keeps its own recordings. Serving the + lowest-sorting owner's bytes under that URL would never say whose they + were, so it answers the same 409 the fault routes do. Rosbag capture is + off in this launch, so the assertion is on the refusal the resolution + makes before any bag is looked up. + + @verifies REQ_INTEROP_072 + """ + self._raise_both() + + response = requests.get( + f'{self.BASE_URL}/components/{HOST_COMPONENT}/bulk-data/rosbags/{SHARED_CODE}', + timeout=10, + ) + + self.assertEqual(response.status_code, 409, response.text) + body = response.json() + self.assertEqual(body.get('vendor_code'), 'x-medkit-ambiguous-fault', body) + self.assertEqual( + sorted(body['parameters']['owners']), [OWNER_A, OWNER_B], body) + + # One owner in scope is one record, so that entity's own URL is not + # ambiguous - it 404s only because no bag was captured. + single = requests.get( + f'{self.BASE_URL}/apps/{OWNER_A}/bulk-data/rosbags/{SHARED_CODE}', timeout=10 + ) + self.assertNotEqual( + single.status_code, 409, + 'one owner in scope must not read as ambiguous', + ) + + def test_08_stream_frames_name_the_record_they_describe(self): + """Each frame carries the entity hint of its own record's owner.""" + frames = [] + stop_event = threading.Event() + response = requests.get( + f'{self.BASE_URL}/faults/stream', stream=True, timeout=(5, 60) + ) + self.assertEqual(response.status_code, 200) + pump = threading.Thread( + target=self._pump_stream, args=(response, frames, stop_event), daemon=True + ) + pump.start() + try: + self._prime_stream(frames) + self._report(OWNER_A, fault_code=STREAM_CODE) + self._report(OWNER_B, fault_code=STREAM_CODE) + + seen = {} + deadline = time.monotonic() + FAULT_TIMEOUT + while time.monotonic() < deadline and len(seen) < 2: + for frame in list(frames): + data = frame.get('data') + if data is None: + continue + payload = json.loads(data) + fault = payload.get('fault', {}) + if fault.get('fault_code') != STREAM_CODE: + continue + hint = payload.get('x-medkit') + self.assertIsNotNone( + hint, f'a frame for {STREAM_CODE} carried no entity hint: {payload}' + ) + self.assertEqual( + hint['entity_id'], fault.get('source_id'), + 'the hint must name the record the frame describes', + ) + seen[hint['entity_id']] = hint + time.sleep(0.2) + + self.assertEqual( + sorted(seen), [OWNER_A, OWNER_B], + f'both owners must appear on the stream, saw: {sorted(seen)}', + ) + for hint in seen.values(): + self.assertEqual(hint['entity_type'], 'apps') + finally: + stop_event.set() + response.close() + pump.join(timeout=5) + + @staticmethod + def _pump_stream(response, frames, stop_event): + """Collect SSE frames as dicts of field -> value.""" + current = {} + try: + for line in response.iter_lines(decode_unicode=True): + if stop_event.is_set(): + break + if line is None: + continue + if line == '': + if current: + frames.append(current) + current = {} + continue + if line.startswith(':'): + continue # keepalive comment + key, _, value = line.partition(':') + current[key.strip()] = value.strip() + except Exception: # noqa: BLE001 - closed socket on test teardown + pass + + def _prime_stream(self, frames): + """Block until the fault event pipeline demonstrably reaches the stream. + + /fault_manager/events is reliable but volatile: an event published + before the gateway's subscription has matched the fault manager's + publisher is lost outright. Repeating a sacrificial record until one of + its frames arrives proves service -> fault manager -> events -> SSE. + """ + deadline = time.monotonic() + FAULT_TIMEOUT + while time.monotonic() < deadline: + self._report(OWNER_A, fault_code=PRIME_CODE, times=2) + settle = min(time.monotonic() + 1.0, deadline) + while time.monotonic() < settle: + for frame in list(frames): + data = frame.get('data') + if data is None: + continue + if json.loads(data).get('fault', {}).get('fault_code') == PRIME_CODE: + return + time.sleep(0.1) + raise AssertionError( + f'no event for priming fault {PRIME_CODE} on /faults/stream within ' + f'{FAULT_TIMEOUT}s, events pipeline never went live' + ) + + +@launch_testing.post_shutdown_test() +class TestProcessOutput(unittest.TestCase): + """All processes exited cleanly (SIGTERM allowed for SSE teardown).""" + + def test_exit_codes(self, proc_info): + for process_name in proc_info.process_names(): + self.assertIn( + proc_info[process_name].returncode, ALLOWED_EXIT_CODES, + f'{process_name} exited with {proc_info[process_name].returncode}', + ) diff --git a/src/ros2_medkit_integration_tests/test/features/test_grouping_entity_aggregation.test.py b/src/ros2_medkit_integration_tests/test/features/test_grouping_entity_aggregation.test.py index d95ab3009..55dfeda6f 100644 --- a/src/ros2_medkit_integration_tests/test/features/test_grouping_entity_aggregation.test.py +++ b/src/ros2_medkit_integration_tests/test/features/test_grouping_entity_aggregation.test.py @@ -33,14 +33,16 @@ ADDRESSING SECOND, and it has exactly two cases, decided per collection rather than per handler: - a LEAF-OWNED item - a topic, an operation, a configuration key - belongs to - one leaf, and is addressed ":". - an AGGREGATE item - a fault, which has one code and a SET of reporting - sources - belongs to the aggregate itself, and is + a LEAF-OWNED item - a topic, an operation, a configuration key, a fault + record - belongs to one leaf, and is addressed + ":". A fault record is (fault_code, + reporting source), and the source is the leaf. + an AGGREGATE item - one a grouping computes over its members rather than + holds - belongs to the aggregate itself, and is addressed by its own id while naming its contributors. -Faults, logs and bulk-data are not exceptions to the model; they are the second -case. Which case a collection is, is declared once. +Logs and bulk-data are not exceptions to the model. Which case a collection is, +is declared once. WHAT THIS SUITE EXISTS TO PREVENT, all three measured on this topology: diff --git a/src/ros2_medkit_integration_tests/test/features/test_locking_faults_owners.test.py b/src/ros2_medkit_integration_tests/test/features/test_locking_faults_owners.test.py new file mode 100644 index 000000000..26713b840 --- /dev/null +++ b/src/ros2_medkit_integration_tests/test/features/test_locking_faults_owners.test.py @@ -0,0 +1,326 @@ +#!/usr/bin/env python3 +# Copyright 2026 bburda +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Integration test: global DELETE /faults honours the lock of every entity that owns a record. + +A record belongs to an entity when its owner equals one of the entity's owned +sources or is a path child of one: an app's node FQN, an external app's bare +id, an external component's bare id. The global clear skips the record when +any such entity is locked by another client. A name that only starts with an +owned source is not a path child, so its record is cleared. + +Hybrid discovery with a manifest: the only way to declare external apps and +components, and two apps bound to one node path (spelled two ways). +""" + +import os +import tempfile +import time +import unittest + +import launch_testing +import rclpy +from rclpy.node import Node +import requests +from ros2_medkit_msgs.msg import Fault +from ros2_medkit_msgs.srv import ReportFault + +from ros2_medkit_test_utils.constants import ALLOWED_EXIT_CODES +from ros2_medkit_test_utils.gateway_test_case import GatewayTestCase +from ros2_medkit_test_utils.launch_helpers import create_test_launch + +LIDAR_FQN = '/perception/lidar/lidar_sensor' +EXTERNAL_APP = 'plc-app' +EXTERNAL_COMPONENT = 'plc-rack' +TWIN_SOURCE = '/scope/t/twin' +UNOWNED_SOURCE = '/nobody/owns_this' + +MANIFEST_YAML = """\ +manifest_version: "1.0" +metadata: + name: locking-faults-owners + version: "1.0.0" +config: + unmanifested_nodes: ignore +areas: + - id: perception + name: Perception + namespace: /perception + - id: yard + name: Yard + namespace: /yard + - id: twins + name: Twins + namespace: /scope +components: + - id: lidar-unit + name: Lidar unit + area: perception + - id: plc-unit + name: PLC unit + area: yard + - id: plc-rack + name: PLC rack + area: yard + external: true + - id: twin-unit + name: Twin unit + area: twins +apps: + - id: lidar_sensor + name: Lidar sensor + is_located_on: lidar-unit + ros_binding: + node_name: lidar_sensor + namespace: /perception/lidar + - id: plc-app + name: PLC app + external: true + is_located_on: plc-unit + - id: twin-a + name: Twin A + is_located_on: twin-unit + ros_binding: + node_name: twin + namespace: /scope/t + # The same node path spelled with a trailing slash: an identical binding + # would be rejected as a duplicate (R010). + - id: twin-b + name: Twin B + is_located_on: twin-unit + ros_binding: + node_name: twin + namespace: /scope/t/ +""" + + +def _write_manifest(): + fd, path = tempfile.mkstemp(prefix='medkit_locking_faults_owners_', suffix='.yaml') + os.write(fd, MANIFEST_YAML.encode('utf-8')) + os.close(fd) + return path + + +def generate_test_description(): + """Launch a hybrid gateway with locking, the fault manager and the lidar node.""" + return create_test_launch( + demo_nodes=['lidar_sensor'], + gateway_params={ + 'locking.enabled': True, + 'locking.default_max_expiration': 3600, + 'discovery.mode': 'hybrid', + 'discovery.manifest_path': _write_manifest(), + 'discovery.manifest_strict_validation': False, + }, + fault_manager=True, + fault_manager_params={'confirmation_threshold': -2}, + lidar_faulty=False, + ) + + +class TestLockingFaultsOwners(GatewayTestCase): + """Global fault clear honours the lock of every entity whose scope holds the owner.""" + + MIN_EXPECTED_APPS = 4 + REQUIRED_APPS = {'lidar_sensor', EXTERNAL_APP, 'twin-a', 'twin-b'} + + @classmethod + def setUpClass(cls): + super().setUpClass() + rclpy.init() + cls._reporter = Node('locking_faults_owners_reporter') + cls._report_client = cls._reporter.create_client( + ReportFault, '/fault_manager/report_fault') + assert cls._report_client.wait_for_service(timeout_sec=15.0), \ + 'report_fault service not available' + + @classmethod + def tearDownClass(cls): + cls._reporter.destroy_node() + rclpy.shutdown() + + def _report(self, fault_code, source): + """Report FAILED until two reports are answered, so the fault confirms. + + A reply can be lost while the fault manager is still matching this + client, even though the request arrived; resending is harmless. + """ + req = ReportFault.Request() + req.fault_code = fault_code + req.event_type = ReportFault.Request.EVENT_FAILED + req.severity = Fault.SEVERITY_ERROR + req.description = 'locking test fault' + req.source_id = source + answered = 0 + for _ in range(8): + future = self._report_client.call_async(req) + deadline = time.monotonic() + 5.0 + while not future.done() and time.monotonic() < deadline: + rclpy.spin_once(self._reporter, timeout_sec=0.1) + if future.done(): + self.assertTrue(future.result().accepted) + answered += 1 + if answered == 2: + return + self.fail(f'report_fault {fault_code} answered {answered} of 2 times') + + def _lock(self, entity_path, client_id): + resp = requests.post( + f'{self.BASE_URL}{entity_path}/locks', + json={'lock_expiration': 300}, + headers={'X-Client-Id': client_id}, + timeout=10, + ) + self.assertEqual(resp.status_code, 201, resp.text) + lock_id = resp.json()['id'] + self.addCleanup( + requests.delete, + f'{self.BASE_URL}{entity_path}/locks/{lock_id}', + headers={'X-Client-Id': client_id}, + timeout=10, + ) + + def _global_codes(self): + resp = requests.get(f'{self.BASE_URL}/faults', timeout=10) + self.assertEqual(resp.status_code, 200, resp.text) + return {f['fault_code'] for f in resp.json().get('items', [])} + + def _assert_global_clear(self, entity_path, spared, cleared): + """Clear globally as another client: `spared` codes stay, `cleared` codes go. + + `cleared` holds a control fault from a source no entity owns, which + proves the clear ran. The per-entity DELETE of a spared fault must + refuse too. + """ + for fault_code in spared: + resp = requests.delete( + f'{self.BASE_URL}{entity_path}/faults/{fault_code}', + headers={'X-Client-Id': 'client_b'}, + timeout=10, + ) + self.assertEqual(resp.status_code, 409, resp.text) + + resp = requests.delete( + f'{self.BASE_URL}/faults', + headers={'X-Client-Id': 'client_b'}, + timeout=10, + ) + self.assertEqual(resp.status_code, 204, resp.text) + + codes = self._global_codes() + for fault_code in cleared: + self.assertNotIn( + fault_code, codes, + f'global clear kept {fault_code}, which no locked entity owns') + for fault_code in spared: + self.assertIn( + fault_code, codes, + f'global clear removed {fault_code} while {entity_path} is locked by another ' + f'client') + + # @verifies REQ_INTEROP_100 + def test_global_clear_skips_subpath_source_of_locked_app(self): + """Owners below the locked app's FQN are spared, a name sharing only its prefix is not.""" + self._report('LOCK_SUBPATH', f'{LIDAR_FQN}/diagnostics') + self._report('LOCK_SUBPATH2', f'{LIDAR_FQN}/diagnostics/deep') + # Starts with the app's FQN but is not a path child of it. + self._report('LOCK_NAME_PREFIX', f'{LIDAR_FQN}_extra/diag') + self._report('LOCK_SUBPATH_CONTROL', UNOWNED_SOURCE) + self.wait_for_fault('/apps/lidar_sensor', 'LOCK_SUBPATH') + self.wait_for_fault('/apps/lidar_sensor', 'LOCK_SUBPATH2') + + self._lock('/apps/lidar_sensor', 'client_a') + self._assert_global_clear('/apps/lidar_sensor', ['LOCK_SUBPATH', 'LOCK_SUBPATH2'], + ['LOCK_NAME_PREFIX', 'LOCK_SUBPATH_CONTROL']) + + # @verifies REQ_INTEROP_100 + def test_global_clear_by_the_lock_holder_clears_its_locked_records(self): + """The lock holder's own global clear removes the records its locks cover.""" + self._report('LOCK_HOLDER', f'{LIDAR_FQN}/holder') + self._report('LOCK_HOLDER_TWIN', TWIN_SOURCE) + self.wait_for_fault('/apps/lidar_sensor', 'LOCK_HOLDER') + self.wait_for_fault('/apps/twin-a', 'LOCK_HOLDER_TWIN') + + self._lock('/apps/lidar_sensor', 'client_a') + self._lock('/apps/twin-a', 'client_a') + resp = requests.delete( + f'{self.BASE_URL}/faults', + headers={'X-Client-Id': 'client_a'}, + timeout=10, + ) + self.assertEqual(resp.status_code, 204, resp.text) + + codes = self._global_codes() + for fault_code in ('LOCK_HOLDER', 'LOCK_HOLDER_TWIN'): + self.assertNotIn( + fault_code, codes, + f'the global clear of the lock holder kept {fault_code}') + + # @verifies REQ_INTEROP_100 + def test_global_clear_skips_bare_id_source_of_locked_external_app(self): + """A source equal to a locked external app's id is covered by that app's lock.""" + self._report('LOCK_EXTERNAL', EXTERNAL_APP) + self._report('LOCK_EXTERNAL_CONTROL', UNOWNED_SOURCE) + self.wait_for_fault(f'/apps/{EXTERNAL_APP}', 'LOCK_EXTERNAL') + + self._lock(f'/apps/{EXTERNAL_APP}', 'client_a') + self._assert_global_clear(f'/apps/{EXTERNAL_APP}', ['LOCK_EXTERNAL'], + ['LOCK_EXTERNAL_CONTROL']) + + # @verifies REQ_INTEROP_100 + def test_global_clear_skips_bare_id_source_of_locked_external_component(self): + """A source equal to a locked external component's id, or below it, is covered.""" + self._report('LOCK_RACK', EXTERNAL_COMPONENT) + self._report('LOCK_RACK_SLOT', f'{EXTERNAL_COMPONENT}/slot_1') + self._report('LOCK_RACK_CONTROL', UNOWNED_SOURCE) + self.wait_for_fault(f'/components/{EXTERNAL_COMPONENT}', 'LOCK_RACK') + self.wait_for_fault(f'/components/{EXTERNAL_COMPONENT}', 'LOCK_RACK_SLOT') + + self._lock(f'/components/{EXTERNAL_COMPONENT}', 'client_a') + self._assert_global_clear(f'/components/{EXTERNAL_COMPONENT}', + ['LOCK_RACK', 'LOCK_RACK_SLOT'], ['LOCK_RACK_CONTROL']) + + def _assert_twin_lock_spares(self, twin): + code = f'LOCK_{twin.upper().replace("-", "_")}' + control = f'{code}_CONTROL' + self._report(code, TWIN_SOURCE) + self._report(control, UNOWNED_SOURCE) + self.wait_for_fault(f'/apps/{twin}', code) + + self._lock(f'/apps/{twin}', 'client_a') + self._assert_global_clear(f'/apps/{twin}', [code], [control]) + + # @verifies REQ_INTEROP_100 + def test_global_clear_skips_source_of_locked_twin_a(self): + """Two apps own one source: the lock of the first is honoured.""" + self._assert_twin_lock_spares('twin-a') + + # @verifies REQ_INTEROP_100 + def test_global_clear_skips_source_of_locked_twin_b(self): + """Two apps own one source: the lock of the second is honoured.""" + self._assert_twin_lock_spares('twin-b') + + +@launch_testing.post_shutdown_test() +class TestShutdown(unittest.TestCase): + + def test_exit_codes(self, proc_info): + """Check all processes exited cleanly.""" + for info in proc_info: + self.assertIn( + info.returncode, + ALLOWED_EXIT_CODES, + f'{info.process_name} exited with code {info.returncode}', + ) diff --git a/src/ros2_medkit_integration_tests/test/features/test_rosbag_list_fault_states.test.py b/src/ros2_medkit_integration_tests/test/features/test_rosbag_list_fault_states.test.py new file mode 100644 index 000000000..889cbc004 --- /dev/null +++ b/src/ros2_medkit_integration_tests/test/features/test_rosbag_list_fault_states.test.py @@ -0,0 +1,247 @@ +#!/usr/bin/env python3 +# Copyright 2026 bburda +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""An entity's rosbag listing keeps a recording whatever state its fault is in. + +A recording is made when its fault confirms and outlives that state. The +download of a recording checks only that the fault is in the entity's scope, +so the listing must not drop a recording when its fault is pending again +(PREFAILED or PREPASSED), cleared, healed or muted as a correlated symptom. + +The debounce counter starts at -1, confirms at -2 and heals at +2; a +confirmed fault stays CONFIRMED until the counter reaches +2. A fault +reactivated after a clear starts again at -1, so two PASSED reports move it +to +1, PREPASSED. The fault manager mutes ``BAGSYM`` as a symptom of +``BAGROOT`` and keeps a recording when its fault is cleared +(``snapshots.rosbag.auto_cleanup`` off). Every +fault is reported from a path child of the ``/bagstates/bag_owner`` node, +which this process creates, so they all belong to its app. +""" + +import os +import tempfile +import threading +import time +import unittest + +from launch import LaunchDescription +from launch.actions import TimerAction +import launch_testing +import launch_testing.actions +import rclpy +from rclpy.executors import SingleThreadedExecutor +from rclpy.node import Node +from ros2_medkit_msgs.msg import Fault +from ros2_medkit_msgs.srv import ClearFault, ReportFault + +from ros2_medkit_test_utils.constants import ALLOWED_EXIT_CODES, FAULT_TIMEOUT, ROSBAG_TIMEOUT +from ros2_medkit_test_utils.gateway_test_case import GatewayTestCase +from ros2_medkit_test_utils.launch_helpers import ( + create_fault_manager_node, + create_gateway_node, +) +from ros2_medkit_test_utils.service_round_trip import answer_once_each + +from std_msgs.msg import Float32 + + +OWNER_APP = 'bag_owner' +SOURCE = '/bagstates/bag_owner/diag' +SIGNAL_TOPIC = '/bagstates/signal' + +PREFAILED = 'STATE_PREFAILED' +PREPASSED = 'STATE_PREPASSED' +CLEARED = 'STATE_CLEARED' +HEALED = 'STATE_HEALED' +ROOT = 'BAGROOT' +MUTED = 'BAGSYM' + +CORRELATION_YAML = """\ +correlation: + enabled: true + default_window_ms: 30000 + rules: + - id: bag_cascade + name: Bag cascade + mode: hierarchical + root_cause: + codes: ["BAGROOT"] + symptoms: + - codes: ["BAGSYM"] + window_ms: 30000 + mute_symptoms: true + auto_clear_with_root: false +""" + + +def _write_correlation_config(): + fd, path = tempfile.mkstemp(prefix='medkit_bag_states_', suffix='.yaml') + os.write(fd, CORRELATION_YAML.encode('utf-8')) + os.close(fd) + return path + + +def generate_test_description(): + gateway = create_gateway_node(extra_params={'discovery.mode': 'runtime_only'}) + fault_manager = create_fault_manager_node( + rosbag_topics=[SIGNAL_TOPIC], + snapshot_topics=[SIGNAL_TOPIC], + extra_params={ + 'confirmation_threshold': -2, + 'healing_enabled': True, + 'healing_threshold': 2, + # The default deletes a fault's recording when the fault is cleared. + 'snapshots.rosbag.auto_cleanup': False, + 'correlation.config_file': _write_correlation_config(), + }, + ) + return ( + LaunchDescription([ + gateway, + TimerAction(period=2.0, actions=[fault_manager]), + launch_testing.actions.ReadyToTest(), + ]), + {'gateway_node': gateway}, + ) + + +class TestRosbagListFaultStates(GatewayTestCase): + """The rosbag listing follows fault scope, not fault status.""" + + MIN_EXPECTED_APPS = 1 + REQUIRED_APPS = {OWNER_APP} + + @classmethod + def setUpClass(cls): + rclpy.init() + cls._owner = Node(OWNER_APP, namespace='/bagstates') + cls._report_client = cls._owner.create_client(ReportFault, '/fault_manager/report_fault') + cls._clear_client = cls._owner.create_client(ClearFault, '/fault_manager/clear_fault') + # The capture ring buffer needs data before a fault confirms, and the + # fault manager resolves an explicit topic's type only at startup. + signal = cls._owner.create_publisher(Float32, SIGNAL_TOPIC, 10) + cls._owner.create_timer(0.05, lambda: signal.publish(Float32(data=1.0))) + cls._executor = SingleThreadedExecutor() + cls._executor.add_node(cls._owner) + cls._spin_thread = threading.Thread(target=cls._executor.spin, daemon=True) + cls._spin_thread.start() + + super().setUpClass() + assert cls._report_client.wait_for_service(timeout_sec=30.0), \ + 'report_fault service not available' + answer_once_each( + cls._owner, (cls._report_client, cls._clear_client), FAULT_TIMEOUT, spin=False) + + @classmethod + def tearDownClass(cls): + cls._executor.shutdown() + cls._spin_thread.join(timeout=5) + cls._owner.destroy_node() + rclpy.shutdown() + + @staticmethod + def _report_request(code, source, passed): + request = ReportFault.Request() + request.fault_code = code + request.event_type = (ReportFault.Request.EVENT_PASSED if passed + else ReportFault.Request.EVENT_FAILED) + request.severity = Fault.SEVERITY_ERROR + request.description = 'rosbag listing state test' + request.source_id = source + return request + + @staticmethod + def _call(client, request, answer_within): + future = client.call_async(request) + deadline = time.monotonic() + answer_within + while not future.done() and time.monotonic() < deadline: + time.sleep(0.02) + return future.result() if future.done() else None + + def _report(self, code, times=1, passed=False): + for _ in range(times): + answer = self._call(self._report_client, + self._report_request(code, SOURCE, passed), 10.0) + self.assertTrue(answer is not None and answer.accepted, f'report {code} failed') + + def _clear(self, code): + request = ClearFault.Request() + request.fault_code = code + request.source_id = SOURCE + answer = self._call(self._clear_client, request, 10.0) + self.assertTrue(answer is not None and answer.success, f'clear {code} failed') + + def _status(self, code): + """Status the fault manager holds for ``code``, read without a list filter.""" + detail = self.get_json(f'/apps/{OWNER_APP}/faults/{code}') + return detail.get('x-medkit', {}).get('status_raw') + + def _bag_codes(self): + data = self.get_json(f'/apps/{OWNER_APP}/bulk-data/rosbags') + return {code for item in data.get('items', []) + for code in item.get('x-medkit', {}).get('fault_codes', [])} + + def _wait_for_bags(self, codes): + deadline = time.monotonic() + ROSBAG_TIMEOUT + while time.monotonic() < deadline: + listed = self._bag_codes() + if codes <= listed: + return + time.sleep(0.5) + self.fail(f'bags of {sorted(codes - listed)} never listed; listed {sorted(listed)}') + + # @verifies REQ_INTEROP_072 + def test_recordings_stay_listed_in_every_fault_state(self): + """Pending, cleared, healed and muted faults keep their recordings listed.""" + for code in (PREFAILED, PREPASSED, CLEARED, HEALED, ROOT, MUTED): + self._report(code, times=2) + every = {PREFAILED, PREPASSED, CLEARED, HEALED, MUTED} + self._wait_for_bags(every) + + # PREFAILED: a new outage after a clear, one FAILED short of confirming. + self._clear(PREFAILED) + self._report(PREFAILED) + # PREPASSED: the same new outage, then two PASSED reports, one short of healing. + self._clear(PREPASSED) + self._report(PREPASSED) + self._report(PREPASSED, times=2, passed=True) + self._clear(CLEARED) + self._report(HEALED, times=4, passed=True) + + # Preconditions: each fault is in the state this test is about. The + # detail route does not go through the list status filter. + self.assertEqual( + {code: self._status(code) for code in (PREFAILED, PREPASSED, CLEARED, HEALED)}, + {PREFAILED: 'PREFAILED', PREPASSED: 'PREPASSED', CLEARED: 'CLEARED', + HEALED: 'HEALED'}) + faults = self.get_json('/faults?status=all&include_muted=true') + muted = {m.get('fault_code') for m in faults.get('x-medkit', {}).get('muted_faults', [])} + self.assertIn(MUTED, muted) + + listed = self._bag_codes() + self.assertTrue(every <= listed, + f'bags of {sorted(every - listed)} dropped; listed {sorted(listed)}') + + +@launch_testing.post_shutdown_test() +class TestShutdown(unittest.TestCase): + + def test_exit_codes(self, proc_info): + """Check all processes exited cleanly (SIGTERM allowed).""" + for info in proc_info: + self.assertIn( + info.returncode, ALLOWED_EXIT_CODES, + f'{info.process_name} exited with code {info.returncode}' + ) diff --git a/src/ros2_medkit_integration_tests/test/features/test_rosbag_owner_download.test.py b/src/ros2_medkit_integration_tests/test/features/test_rosbag_owner_download.test.py new file mode 100644 index 000000000..c42628495 --- /dev/null +++ b/src/ros2_medkit_integration_tests/test/features/test_rosbag_owner_download.test.py @@ -0,0 +1,205 @@ +#!/usr/bin/env python3 +# Copyright 2026 bburda +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""A recording downloads for the owner of the record it is attached to. + +Two external apps under one component report the same fault code, each under +its own entity id, so the fault manager keeps two records of that code and +captures a recording for each. A recording belongs to the (fault code, owner) +records it is attached to, so over the recording-id URL: + +* each app downloads its own recording and gets ``404`` on the other's, in both + directions, although both apps own a record of the same code +* the component hosting both apps downloads both recordings +* after one app's record is cleared, that app still gets ``404`` on the other + app's recording, and still downloads its own + +The unit layer pins the same rule against a stub fault manager +(``PluginClearOwnerTest.EachOwnerDownloadsOnlyItsOwnRecordingById``). This runs +it against the real fault manager and its real rosbag rows. +""" + +import atexit +import os +import shutil +import tempfile +import time +import unittest + +from ament_index_python.packages import get_package_share_directory +import launch_testing +import rclpy +from rclpy.node import Node +import requests +from ros2_medkit_msgs.msg import Fault +from ros2_medkit_msgs.srv import ReportFault + +from ros2_medkit_test_utils.constants import ALLOWED_EXIT_CODES +from ros2_medkit_test_utils.gateway_test_case import GatewayTestCase +from ros2_medkit_test_utils.launch_helpers import create_test_launch + + +OWNER_A = 'owner-a' +OWNER_B = 'owner-b' +HOST_COMPONENT = 'shared-code-hub' +SHARED_CODE = 'SHARED_RECORDED' +RECORDING_TIMEOUT = 30.0 +# SHARED_CODE is per_source: without it the fault manager keeps one record that +# both owners share. +CONFIG_DIR = tempfile.mkdtemp(prefix='medkit_rosbag_owner_') +atexit.register(shutil.rmtree, CONFIG_DIR, True) +THRESHOLDS_PATH = os.path.join(CONFIG_DIR, 'fault_thresholds.yaml') + + +def generate_test_description(): + manifest_path = os.path.join( + get_package_share_directory('ros2_medkit_gateway'), + 'config', 'examples', 'fault_owner_identity_manifest.yaml', + ) + with open(THRESHOLDS_PATH, 'w') as f: + f.write(f'{SHARED_CODE}:\n per_source: true\n') + return create_test_launch( + # The lidar publishes the scan topic the fault manager records, so each + # confirmation leaves a real bag behind. + demo_nodes=['lidar_sensor'], + fault_manager=True, + fault_manager_params={ + 'confirmation_threshold': -1, # a single report confirms + # Keep a cleared record's recording, so the app that cleared can + # still be shown downloading its own bag after the clear. + 'snapshots.rosbag.auto_cleanup': False, + 'fault_thresholds.config_file': THRESHOLDS_PATH, + }, + gateway_params={ + 'discovery.mode': 'hybrid', + 'discovery.manifest_path': manifest_path, + 'discovery.manifest_strict_validation': False, + }, + ) + + +class TestRosbagOwnerDownload(GatewayTestCase): + """Each owner of one fault code downloads only its own recording by id.""" + + MIN_EXPECTED_APPS = 2 + REQUIRED_APPS = {OWNER_A, OWNER_B} + + @classmethod + def setUpClass(cls): + rclpy.init() + cls._reporter = Node('rosbag_owner_download_reporter') + cls._report_client = cls._reporter.create_client( + ReportFault, '/fault_manager/report_fault' + ) + super().setUpClass() + assert cls._report_client.wait_for_service(timeout_sec=15.0), \ + 'report_fault service not available' + + @classmethod + def tearDownClass(cls): + cls._reporter.destroy_node() + rclpy.shutdown() + + def _report(self, source_id): + """Report SHARED_RECORDED under one owner, fire and forget.""" + req = ReportFault.Request() + req.fault_code = SHARED_CODE + req.event_type = ReportFault.Request.EVENT_FAILED + req.severity = Fault.SEVERITY_ERROR + req.description = f'reported by {source_id}' + req.source_id = source_id + self._report_client.call_async(req) + rclpy.spin_once(self._reporter, timeout_sec=0.1) + + def _recordings_listed_for(self, app_id): + listing = self.get_json(f'/apps/{app_id}/bulk-data/rosbags') + return sorted( + item['id'] for item in listing.get('items', []) + if SHARED_CODE in item.get('x-medkit', {}).get('fault_codes', []) + ) + + def _record_for(self, app_id): + """Raise the app's own record and wait for its one finished recording.""" + deadline = time.monotonic() + RECORDING_TIMEOUT + ids = [] + while time.monotonic() < deadline: + # Re-sent until the recording is listed: the report is fire and + # forget, and a repeat inside one occurrence changes nothing. + self._report(app_id) + ids = self._recordings_listed_for(app_id) + if ids: + break + time.sleep(0.5) + self.assertEqual(len(ids), 1, f'{app_id} should hold exactly one recording, got {ids}') + return ids[0] + + def test_01_each_owner_downloads_only_its_own_recording(self): + """Two owners of one code, a recording each, over the recording-id URL. + + @verifies REQ_INTEROP_072 + """ + # One after the other, so the second confirmation falls outside the + # first recording's post-roll and gets a recording of its own instead + # of attaching to the first one as a burst would. + rec_a = self._record_for(OWNER_A) + rec_b = self._record_for(OWNER_B) + self.assertNotEqual(rec_a, rec_b, 'the two records share one recording') + + self._assert_downloads({ + (f'/apps/{OWNER_A}', rec_a): 200, + (f'/apps/{OWNER_A}', rec_b): 404, + (f'/apps/{OWNER_B}', rec_b): 200, + (f'/apps/{OWNER_B}', rec_a): 404, + (f'/components/{HOST_COMPONENT}', rec_a): 200, + (f'/components/{HOST_COMPONENT}', rec_b): 200, + }) + + # Clearing owner-a's record keeps owner-a in its own scope with a + # CLEARED record of the code. That must not make owner-b's recording + # owner-a's, and owner-a's own recording stays served. + self.delete_request(f'/apps/{OWNER_A}/faults/{SHARED_CODE}', expected_status=204) + self._assert_downloads({ + (f'/apps/{OWNER_A}', rec_b): 404, + (f'/apps/{OWNER_A}', rec_a): 200, + (f'/apps/{OWNER_B}', rec_a): 404, + (f'/apps/{OWNER_B}', rec_b): 200, + }) + + def _assert_downloads(self, expected): + """Download each (entity path, recording id) and report every wrong status at once.""" + wrong = [] + for (entity_path, recording_id), want in expected.items(): + response = requests.get( + f'{self.BASE_URL}{entity_path}/bulk-data/rosbags/{recording_id}', + timeout=15, + ) + if response.status_code != want: + wrong.append( + f'{entity_path} downloading {recording_id} answered ' + f'{response.status_code}, expected {want}' + ) + self.assertEqual(wrong, [], '\n'.join(wrong)) + + +@launch_testing.post_shutdown_test() +class TestProcessOutput(unittest.TestCase): + """All processes exited cleanly.""" + + def test_exit_codes(self, proc_info): + for process_name in proc_info.process_names(): + self.assertIn( + proc_info[process_name].returncode, ALLOWED_EXIT_CODES, + f'{process_name} exited with {proc_info[process_name].returncode}', + ) diff --git a/src/ros2_medkit_integration_tests/test/features/test_triggers_faults.test.py b/src/ros2_medkit_integration_tests/test/features/test_triggers_faults.test.py index 223c617cd..d27422d57 100644 --- a/src/ros2_medkit_integration_tests/test/features/test_triggers_faults.test.py +++ b/src/ros2_medkit_integration_tests/test/features/test_triggers_faults.test.py @@ -22,17 +22,25 @@ """ +import json import threading +import time import unittest import launch_testing import launch_testing.actions +import rclpy +from rclpy.node import Node import requests +from ros2_medkit_msgs.msg import Fault +from ros2_medkit_msgs.srv import ReportFault -from ros2_medkit_test_utils.constants import ALLOWED_EXIT_CODES, API_BASE_PATH +from ros2_medkit_test_utils.constants import ALLOWED_EXIT_CODES, API_BASE_PATH, FAULT_TIMEOUT from ros2_medkit_test_utils.gateway_test_case import GatewayTestCase from ros2_medkit_test_utils.launch_helpers import create_test_launch +LIDAR_FQN = '/perception/lidar/lidar_sensor' + def generate_test_description(): return create_test_launch( @@ -51,6 +59,66 @@ class TestTriggersFaults(GatewayTestCase): MIN_EXPECTED_APPS = 1 REQUIRED_APPS = {'lidar_sensor'} + @classmethod + def setUpClass(cls): + super().setUpClass() + rclpy.init() + cls._reporter = Node('triggers_faults_reporter') + cls._report_client = cls._reporter.create_client( + ReportFault, '/fault_manager/report_fault') + assert cls._report_client.wait_for_service(timeout_sec=15.0), \ + 'report_fault service not available' + + @classmethod + def tearDownClass(cls): + cls._reporter.destroy_node() + rclpy.shutdown() + + def _report(self, fault_code): + """Report a FAILED event from the lidar node; the reply is not needed.""" + req = ReportFault.Request() + req.fault_code = fault_code + req.event_type = ReportFault.Request.EVENT_FAILED + req.severity = Fault.SEVERITY_ERROR + req.description = 'trigger stream probe' + req.source_id = LIDAR_FQN + self._report_client.call_async(req) + rclpy.spin_once(self._reporter, timeout_sec=0.1) + + def _open_stream(self, events_url): + """Read an SSE stream on a thread; returns (state, stop, thread). + + ``state`` gets the response headers under 'headers', every data + payload under 'events', and any request error under 'error'. A + trigger stream sends its headers with the first frame, which is why + the read timeout exceeds the 15 s keepalive. + """ + state = {'headers': None, 'events': [], 'error': None} + stop = threading.Event() + + def read_stream(): + try: + with requests.get(events_url, stream=True, timeout=(5, 30)) as resp: + state['headers'] = { + 'status': resp.status_code, + 'content_type': resp.headers.get('Content-Type'), + 'cache_control': resp.headers.get('Cache-Control'), + } + for line in resp.iter_lines(decode_unicode=True): + if stop.is_set(): + return + if line and line.startswith('data:'): + state['events'].append(json.loads(line[len('data:'):])) + except requests.exceptions.RequestException as e: + state['error'] = repr(e) + + thread = threading.Thread(target=read_stream, daemon=True) + thread.start() + return state, stop, thread + + def _events_url(self, trig): + return f'{self.BASE_URL}{trig["event_source"].removeprefix(API_BASE_PATH)}' + def _create_fault_trigger(self, entity_type='apps', entity_id='lidar_sensor', *, multishot=True, lifetime=None): """Create a trigger on faults collection for the given entity.""" @@ -172,66 +240,52 @@ def test_05_delete_fault_trigger(self): # ------------------------------------------------------------------ # @verifies REQ_INTEROP_097 - def test_10_sse_endpoint_connects_and_streams(self): - """SSE events endpoint accepts connections and streams keepalives. + def test_10_sse_endpoint_delivers_fault_events(self): + """The trigger's event stream answers and delivers a fault event of its app. - Verifies the SSE transport works for fault triggers - the connection - is established and the server sends keepalive comments. Actual fault - events depend on the fault subscriber's entity_id matching the - trigger's entity_id, which requires correct reporting_source mapping. + The lidar node's own FQN is the reporting source, so the lidar app's + trigger must deliver the event. The stream is read on a thread and a + fresh fault is reported until one arrives: the trigger buffers events, + so reporting before the stream connects loses nothing. """ - trig = self._create_fault_trigger(multishot=True, lifetime=60) + trig = self._create_fault_trigger(multishot=True, lifetime=120) self.addCleanup(self._delete_trigger, trig['id']) - events_url = ( - f'{self.BASE_URL}{trig["event_source"].removeprefix(API_BASE_PATH)}' - ) - - # Verify SSE connection works - read at least one line (data or keepalive) - stream_connected = threading.Event() - received_lines = [] - - def read_stream(): - try: - with requests.get(events_url, stream=True, timeout=10) as resp: - stream_connected.set() - for line in resp.iter_lines(decode_unicode=True): - if line: - received_lines.append(line) - break - except requests.exceptions.RequestException: - pass - finally: - stream_connected.set() - - thread = threading.Thread(target=read_stream, daemon=True) - thread.start() - - self.assertTrue( - stream_connected.wait(timeout=5), - 'SSE stream failed to connect within 5s', - ) - thread.join(timeout=10) + state, stop, _ = self._open_stream(self._events_url(trig)) + self.addCleanup(stop.set) + code = f'TRIGGER_STREAM_PROBE_{trig["id"]}' + deadline = time.monotonic() + FAULT_TIMEOUT + delivered = False + while time.monotonic() < deadline and not delivered: + self._report(code) + delivered = any(e.get('payload', {}).get('fault_code') == code + for e in list(state['events'])) + time.sleep(0.5) + + self.assertIsNone(state['error'], f'event stream failed: {state["error"]}') + self.assertIsNotNone(state['headers'], 'event stream never answered') + self.assertEqual(state['headers']['status'], 200, state['headers']) + received = [e.get('payload', {}).get('fault_code') for e in list(state['events'])] + self.assertTrue(delivered, f'no event for {code}; received {received}') # @verifies REQ_INTEROP_097 def test_11_sse_fault_event_has_correct_headers(self): """SSE events endpoint for faults returns correct headers.""" - trig = self._create_fault_trigger(lifetime=60) + trig = self._create_fault_trigger(lifetime=120) self.addCleanup(self._delete_trigger, trig['id']) - events_url = ( - f'{self.BASE_URL}{trig["event_source"].removeprefix(API_BASE_PATH)}' - ) - - try: - r = requests.get(events_url, stream=True, timeout=3) - self.assertEqual( - r.headers.get('Content-Type'), 'text/event-stream' - ) - self.assertEqual(r.headers.get('Cache-Control'), 'no-cache') - r.close() - except requests.exceptions.ReadTimeout: - pass + state, stop, _ = self._open_stream(self._events_url(trig)) + self.addCleanup(stop.set) + self._report(f'TRIGGER_HEADERS_PROBE_{trig["id"]}') + deadline = time.monotonic() + FAULT_TIMEOUT + while state['headers'] is None and state['error'] is None and time.monotonic() < deadline: + time.sleep(0.2) + + self.assertIsNone(state['error'], f'event stream failed: {state["error"]}') + self.assertIsNotNone(state['headers'], 'event stream never answered') + self.assertEqual(state['headers']['status'], 200, state['headers']) + self.assertEqual(state['headers']['content_type'], 'text/event-stream') + self.assertEqual(state['headers']['cache_control'], 'no-cache') # ------------------------------------------------------------------ # Component-scoped fault trigger diff --git a/src/ros2_medkit_integration_tests/test/features/test_triggers_logs_markers.test.py b/src/ros2_medkit_integration_tests/test/features/test_triggers_logs_markers.test.py new file mode 100644 index 000000000..9711b3965 --- /dev/null +++ b/src/ros2_medkit_integration_tests/test/features/test_triggers_logs_markers.test.py @@ -0,0 +1,659 @@ +#!/usr/bin/env python3 +# Copyright 2026 bburda +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Entries that a trigger's log_settings writes never fire a logs trigger. + +A trigger with log_settings writes a marker entry each time it fires. The +marker is logged under the trigger's entity id, so GET /{entity}/logs lists it +where that logger name is in the entity's log. It fires no logs trigger: not +its own, not another one on the same entity, not one above or below it. + +Each case creates its triggers, publishes five real /rosout entries under a +logger its entities list, each listed before the next is logged, then one +closing entry. A trigger delivers in log order, so once the closing entry +arrives no earlier marker can still fire it. The case then compares per +trigger the events it delivered with what its entity's GET lists: one event per +real entry, none for a marker, and one marker per event wherever the entity +lists the trigger's markers. + +Two gateways share one ROS graph: runtime (runtime_only) and manifest +(manifest_only), with a fault manager for the faults trigger case. The graph +nodes log only at startup. +""" + +import json +import os +import tempfile +import threading +import time +import unittest +import uuid + +from launch import LaunchDescription +import launch_ros.actions +import launch_testing +import launch_testing.actions +from rcl_interfaces.msg import Log +import rclpy +from rclpy.qos import DurabilityPolicy, QoSProfile, ReliabilityPolicy +import requests +from ros2_medkit_msgs.msg import Fault +from ros2_medkit_msgs.srv import ReportFault + +from ros2_medkit_test_utils.constants import ( + ALLOWED_EXIT_CODES, + API_BASE_PATH, + get_test_port, + get_time_scale, +) +from ros2_medkit_test_utils.coverage import get_coverage_env +from ros2_medkit_test_utils.launch_helpers import create_fault_manager_node, create_gateway_node +from ros2_medkit_test_utils.service_round_trip import answer_once_each + +RUNTIME_PORT = get_test_port(0) +MANIFEST_PORT = get_test_port(1) +RUNTIME_URL = f'http://localhost:{RUNTIME_PORT}{API_BASE_PATH}' +MANIFEST_URL = f'http://localhost:{MANIFEST_PORT}{API_BASE_PATH}' + +TIME_SCALE = get_time_scale() +HTTP_TIMEOUT_SEC = 5.0 * TIME_SCALE +DISCOVERY_TIMEOUT_SEC = 60.0 * TIME_SCALE +SETTLE_TIMEOUT_SEC = 15.0 * TIME_SCALE +DELIVERY_TIMEOUT_SEC = 20.0 * TIME_SCALE +POLL_INTERVAL_SEC = 0.5 + + +def _level(constant): + """Return a Log severity constant as an int. Humble declares the constants as byte.""" + return constant if isinstance(constant, int) else ord(constant) + + +REAL_ENTRIES = 5 + +RUN = uuid.uuid4().hex[:10] +HOST = '{host}' + +# The look-alike's stamp tells it apart from markers in the test only. +LOOKALIKE_STAMP_SEC = 1000000000 +LOOKALIKE_TIMESTAMP = '2001-09-09T01:46:40.' + +# (executable, node name, namespace) +GRAPH_NODES = [ + ('demo_calibration_service', 'loop_root', '/'), + ('demo_calibration_service', 'loop_ns', '/loopns'), + ('demo_calibration_service', 'zapp', '/zone'), + ('demo_calibration_service', 'papp', '/plant'), +] + +MANIFEST_YAML = """\ +manifest_version: "1.0" +metadata: + name: "log trigger markers" + version: "1.0.0" +config: + unmanifested_nodes: ignore +areas: + - id: zone + name: "Zone" + - id: plant-area + name: "Plant area" + namespace: /plant +components: + - id: ecu + name: "ECU" + area: zone + - id: plant-ecu + name: "Plant ECU" + area: plant-area + namespace: /plant +apps: + - id: zapp + name: "Zone app" + is_located_on: ecu + ros_binding: + node_name: zapp + namespace: /zone + - id: papp + name: "Plant app" + is_located_on: plant-ecu + ros_binding: + node_name: papp + namespace: /plant +""" + +RUNTIME_PATHS = ['apps/loop_root', 'apps/loop_ns', 'functions/loopns'] +MANIFEST_PATHS = [ + 'areas/zone', 'areas/plant-area', 'components/ecu', 'components/plant-ecu', 'apps/zapp', +] + +_MANIFEST_FILES = [] + + +def _write_manifest(): + fd, path = tempfile.mkstemp(suffix='.yaml', prefix='test_triggers_logs_markers_') + with os.fdopen(fd, 'w') as handle: + handle.write(MANIFEST_YAML) + _MANIFEST_FILES.append(path) + return path + + +def generate_test_description(): + streams = {'sse.max_clients': 8, 'server.http_thread_pool_size': 16} + gateways = [ + create_gateway_node(port=RUNTIME_PORT, name='gw_markers_runtime', + extra_params=dict(streams, **{'discovery.mode': 'runtime_only'})), + create_gateway_node(port=MANIFEST_PORT, name='gw_markers_manifest', + extra_params=dict(streams, **{ + 'discovery.mode': 'manifest_only', + 'discovery.manifest_path': _write_manifest(), + 'discovery.manifest_strict_validation': False, + })), + ] + env = get_coverage_env('ros2_medkit_integration_tests') + nodes = [ + launch_ros.actions.Node( + package='ros2_medkit_integration_tests', + executable=executable, + name=name, + namespace=namespace, + output='screen', + additional_env=env, + sigterm_timeout='30', + sigkill_timeout='15', + ) + for executable, name, namespace in GRAPH_NODES + ] + fault_manager = create_fault_manager_node(rosbag_enabled=False) + return ( + LaunchDescription( + gateways + nodes + [fault_manager, launch_testing.actions.ReadyToTest()]), + {'gateway_node': gateways[0]}, + ) + + +# -------------------------------------------------------------------------- +# Helpers +# -------------------------------------------------------------------------- + +def _poll(predicate, *, what, timeout=DISCOVERY_TIMEOUT_SEC): + deadline = time.monotonic() + timeout + last = None + while time.monotonic() < deadline: + try: + last = predicate() + if last: + return last + except requests.exceptions.RequestException as exc: + last = exc + time.sleep(0.3) + raise AssertionError(f'timed out after {timeout}s waiting for {what}; last={last!r}') + + +def _status(url): + return requests.get(url, timeout=HTTP_TIMEOUT_SEC).status_code + + +def _listing(base_url, path): + r = requests.get(f'{base_url}/{path}/logs', timeout=HTTP_TIMEOUT_SEC) + r.raise_for_status() + return r.json().get('items', []) + + +def _count(base_url, path, message): + """Return how many entries GET /{path}/logs lists with exactly this message.""" + return sum(1 for i in _listing(base_url, path) if i.get('message') == message) + + +def _num(entry_id): + return int(str(entry_id).removeprefix('log_')) + + +def _configure_logs(base_url, path): + r = requests.put(f'{base_url}/{path}/logs/configuration', + json={'max_entries': 10000}, timeout=HTTP_TIMEOUT_SEC) + assert r.status_code == 204, f'PUT {base_url}/{path}/logs/configuration: {r.text}' + + +def _marker_text(label): + return f'{RUN}:mark-{label}' + + +def _is_marker_of(message, label): + return str(message).startswith(_marker_text(label) + ' ') + + +def _describe(entry): + return (entry.get('id'), entry.get('severity'), entry.get('context', {}).get('node'), + str(entry.get('message', ''))[:80]) + + +def _is_lookalike(entry, lookalike): + return (entry.get('message') == lookalike[0] + and str(entry.get('timestamp', '')).startswith(LOOKALIKE_TIMESTAMP)) + + +class TriggerStream: + """A trigger, by default a logs trigger with log_settings, and its event reader.""" + + def __init__(self, base_url, path, label, *, collection='logs', marker=True): + self.base_url = base_url + self.path = path + self.label = label + self.collection = collection + self.marker = marker + self.trigger_id = None + self.payloads = [] + self.error = None + self._lock = threading.Lock() + self._resp = None + self._closed = False + + def start(self): + body = { + 'resource': f'{API_BASE_PATH}/{self.path}/{self.collection}', + 'trigger_condition': {'condition_type': 'OnChange'}, + 'multishot': True, + 'lifetime': 600, + } + if self.marker: + body['log_settings'] = {'severity': 'info', 'marker': _marker_text(self.label)} + r = requests.post(f'{self.base_url}/{self.path}/triggers', json=body, + timeout=HTTP_TIMEOUT_SEC) + assert r.status_code == 201, f'POST {self.base_url}/{self.path}/triggers: {r.text}' + trigger = r.json() + self.trigger_id = trigger['id'] + events_url = self.base_url + trigger['event_source'].removeprefix(API_BASE_PATH) + self._resp = requests.get(events_url, stream=True, + timeout=(HTTP_TIMEOUT_SEC, 120.0 * TIME_SCALE)) + assert self._resp.status_code == 200, f'GET {events_url}: {self._resp.status_code}' + threading.Thread(target=self._read, daemon=True).start() + return self + + def _read(self): + try: + for line in self._resp.iter_lines(chunk_size=None, decode_unicode=True): + if line and line.startswith('data: '): + payload = json.loads(line[len('data: '):]).get('payload') + with self._lock: + self.payloads.append(payload if isinstance(payload, dict) else {}) + except Exception as exc: + if not self._closed: + self.error = exc + + def snapshot(self): + with self._lock: + return list(self.payloads) + + def marker_message(self): + """Return the message of this trigger's marker entries.""" + metadata = { + 'condition_type': 'OnChange', + 'resource': f'{API_BASE_PATH}/{self.path}/{self.collection}', + 'trigger_id': self.trigger_id, + } + return _marker_text(self.label) + ' ' + json.dumps( + metadata, sort_keys=True, separators=(',', ':')) + + def close(self): + self._closed = True + if self.trigger_id: + try: + requests.delete(f'{self.base_url}/{self.path}/triggers/{self.trigger_id}', + timeout=HTTP_TIMEOUT_SEC) + except requests.exceptions.RequestException: + pass + if self._resp is not None: + self._resp.close() + + +class RosoutProbe: + """Publish rcl_interfaces/Log entries on /rosout from one publisher.""" + + def __init__(self): + rclpy.init() + self.node = rclpy.create_node('marker_probe') + qos = QoSProfile(depth=1000, reliability=ReliabilityPolicy.RELIABLE, + durability=DurabilityPolicy.VOLATILE) + self.pub = self.node.create_publisher(Log, '/rosout', qos) + self.report_client = self.node.create_client(ReportFault, '/fault_manager/report_fault') + + def publish(self, logger_name, message, *, source=True, stamp_sec=None): + msg = Log() + msg.stamp = self.node.get_clock().now().to_msg() + if stamp_sec is not None: + msg.stamp.sec = stamp_sec + msg.stamp.nanosec = 0 + msg.level = _level(Log.INFO) + msg.name = logger_name + msg.msg = message + if source: + msg.file = 'test_triggers_logs_markers.test.py' + msg.function = 'publish' + msg.line = 1 + self.pub.publish(msg) + + def report_fault(self, code, source_id): + """Report a CRITICAL fault, which the fault manager confirms at once. Sent once.""" + assert self.report_client.wait_for_service(timeout_sec=DISCOVERY_TIMEOUT_SEC), \ + 'report_fault service not available' + answer_once_each(self.node, (self.report_client,), DISCOVERY_TIMEOUT_SEC) + req = ReportFault.Request() + req.fault_code = code + req.event_type = ReportFault.Request.EVENT_FAILED + req.severity = Fault.SEVERITY_CRITICAL + req.description = 'marker test fault' + req.source_id = source_id + future = self.report_client.call_async(req) + rclpy.spin_until_future_complete(self.node, future, timeout_sec=HTTP_TIMEOUT_SEC) + return future.result() is not None and future.result().accepted + + def close(self): + self.node.destroy_node() + rclpy.shutdown() + + +# -------------------------------------------------------------------------- +# Cases +# -------------------------------------------------------------------------- + +class TestLogSettingsMarkers(unittest.TestCase): + """A marker is listed where its logger name is in scope and fires nothing.""" + + @classmethod + def setUpClass(cls): + for path in RUNTIME_PATHS: + _poll(lambda p=path: _status(f'{RUNTIME_URL}/{p}') == 200, what=f'runtime {path}') + for path in MANIFEST_PATHS: + _poll(lambda p=path: _status(f'{MANIFEST_URL}/{p}') == 200, what=f'manifest {path}') + + def one_component(): + items = requests.get(f'{RUNTIME_URL}/components', + timeout=HTTP_TIMEOUT_SEC).json().get('items', []) + return items[0]['id'] if len(items) == 1 else None + cls.host = _poll(one_component, what='the runtime host component') + + cls.probe = RosoutProbe() + cls.addClassCleanup(cls.probe.close) + _poll(lambda: cls.probe.pub.get_subscription_count() >= 2, what='2 /rosout subscribers') + for url, path, logger in ((RUNTIME_URL, 'apps/loop_root', 'loop_root'), + (MANIFEST_URL, 'areas/zone', 'zone/zapp')): + token = f'{RUN}:warm-{uuid.uuid4().hex[:6]}' + count = [0] + + def listed(u=url, p=path, lg=logger, t=token): + count[0] += 1 + cls.probe.publish(lg, f'{t}-{count[0]}') + time.sleep(0.2) + return any(str(i.get('message', '')).startswith(t) for i in _listing(u, p)) + _poll(listed, what=f'{url}/{path} to list a warm-up entry') + + def _path(self, path): + return path.replace(HOST, self.host) + + def _measure(self, url, streams, lists, watermarks, case, kicks, lookalike): + """Return the problems one snapshot of the listings and the events shows.""" + labels = [s.label for s in streams] + events = {s.label: s.snapshot() for s in streams} + listings = {p: _listing(url, p) for p in watermarks} + + def marker_like(entry): + return any(_is_marker_of(entry.get('message', ''), lb) for lb in labels) + + problems = [] + for path, items in listings.items(): + mark = watermarks[path] + fresh = [i for i in items if _num(i['id']) > mark] + mimic = {i['id'] for i in fresh if lookalike and _is_lookalike(i, lookalike)} + real = {i['id'] for i in fresh if not marker_like(i)} + fresh_markers = len(fresh) - len(real) - len(mimic) + if lookalike and lookalike[1] == path and len(mimic) != 1: + problems.append(f'{path}: GET lists the look-alike {len(mimic)} times') + # One marker per firing of each trigger whose markers this entity + # lists; a firing before the watermark left its marker below it. + want_markers = 0 + for label in lists.get(path, set()): + older = sum(1 for i in items if _num(i['id']) <= mark + and _is_marker_of(i.get('message', ''), label)) + want_markers += len(events[label]) - older + published = sum(1 for i in items + if str(i.get('message', '')).startswith(f'{RUN}:real-{case}-')) + if published != REAL_ENTRIES * len(kicks): + problems.append(f'{path}: GET lists {published} of the ' + f'{REAL_ENTRIES * len(kicks)} real entries published') + if fresh_markers != want_markers: + problems.append(f'{path}: {len(fresh)} new entries listed = {len(real)} real + ' + f'{len(mimic)} look-alike + {fresh_markers} markers; expected ' + f'{want_markers} markers, one per firing') + + for stream in streams: + path = stream.path + payloads = events[stream.label] + tag = f'{stream.label} on {path}' + items = listings[path] + fresh = [i for i in items if _num(i['id']) > watermarks[path]] + real = {i['id'] for i in fresh if not marker_like(i)} + is_mimic = bool(lookalike) and lookalike[2] == stream.label + mimic = {i['id'] for i in fresh if is_mimic and _is_lookalike(i, lookalike)} + delivered = [p.get('id') for p in payloads + if p.get('id') and _num(p['id']) > watermarks[path]] + extra = set(delivered) - real + missing = real - set(delivered) + if len(delivered) != len(set(delivered)): + problems.append(f'{tag}: {len(delivered) - len(set(delivered))} ' + f'entries delivered twice') + if missing: + shown = [_describe(i) for i in fresh if i['id'] in missing][:3] + problems.append(f'{tag}: {len(missing)} real entries its GET lists ' + f'were not delivered, e.g. {shown}') + want_extra = 1 if is_mimic else 0 + if len(extra) != want_extra or not extra <= mimic: + problems.append(f'{tag}: {len(extra)} deliveries of entries that are not ' + f'real entries its GET lists (expected {want_extra}: the ' + f'look-alike)') + marker_events = sum( + 1 for p in payloads + if marker_like(p) and not (lookalike and _is_lookalike(p, lookalike))) + if marker_events: + problems.append(f'{tag}: {marker_events} events for markers') + if stream.label not in lists.get(path, set()): + own = sum(1 for i in items if _is_marker_of(i.get('message', ''), stream.label)) + if own: + problems.append(f'{tag}: GET lists {own} of its markers, expected none') + if stream.error is not None: + problems.append(f'{tag}: stream error {stream.error!r}') + return problems + + def _run(self, url, triggers, kicks, lists, case, lookalike_of=None): + """Create the triggers, log for at least 5 s, then compare events and GET.""" + triggers = [(label, self._path(path)) for label, path in triggers] + lists = {self._path(p): set(v) for p, v in lists.items()} + paths = sorted({p for _, p in triggers}) + for path in paths: + _configure_logs(url, path) + streams = [] + try: + for label, path in triggers: + streams.append(TriggerStream(url, path, label).start()) + watermarks = {p: max((_num(i['id']) for i in _listing(url, p)), default=0) + for p in paths} + lookalike = None + if lookalike_of: + stream = next(s for s in streams if s.label == lookalike_of[0]) + lookalike = (stream.marker_message(), stream.path, stream.label, lookalike_of[1]) + + for n in range(REAL_ENTRIES): + message = f'{RUN}:real-{case}-{n}' + for kick in kicks: + self.probe.publish(kick, message) + if lookalike and n == REAL_ENTRIES // 2: + # Same logger, level, message and empty source as a marker. + self.probe.publish(lookalike[3], lookalike[0], source=False, + stamp_sec=LOOKALIKE_STAMP_SEC) + _poll(lambda m=message: all(_count(url, p, m) == len(kicks) for p in paths), + what=f'every listing to hold {message}') + + closing = f'{RUN}:done-{case}' + for kick in kicks: + self.probe.publish(kick, closing) + deadline = time.monotonic() + SETTLE_TIMEOUT_SEC + clean_polls = 0 + problems = [] + while time.monotonic() < deadline and clean_polls < 2: + time.sleep(POLL_INTERVAL_SEC) + if all(_count(url, p, closing) == len(kicks) for p in paths): + problems = self._measure(url, streams, lists, watermarks, case, kicks, + lookalike) + else: + problems = [f'{closing} is not listed everywhere yet'] + clean_polls = clean_polls + 1 if not problems else 0 + same = [] + if lookalike: + same = [i for i in _listing(url, lookalike[1]) if i.get('message') == lookalike[0]] + finally: + for stream in streams: + stream.close() + self.assertFalse(problems, '; '.join(problems)) + if lookalike: + shapes = {json.dumps([i.get('severity'), i.get('context')], sort_keys=True) + for i in same} + self.assertTrue(len(same) >= 2 and len(shapes) == 1, + f'the look-alike must share message, severity and context with a ' + f'listed marker, or it proves nothing: {same[:3]}') + + # L1: one trigger per entity kind. + + # @verifies REQ_INTEROP_097 + def test_l1_root_namespace_app(self): + self._run(RUNTIME_URL, [('root-app', 'apps/loop_root')], ['loop_root'], + {'apps/loop_root': {'root-app'}}, 'root-app') + + # @verifies REQ_INTEROP_097 + def test_l1_namespaced_app(self): + self._run(RUNTIME_URL, [('ns-app', 'apps/loop_ns')], ['loopns.loop_ns'], + {}, 'ns-app') + + # @verifies REQ_INTEROP_097 + def test_l1_host_component(self): + self._run(RUNTIME_URL, [('host', f'components/{HOST}')], ['loopns/loop_ns'], + {}, 'host') + + # @verifies REQ_INTEROP_097 + def test_l1_runtime_function(self): + self._run(RUNTIME_URL, [('function', 'functions/loopns')], ['loopns.loop_ns'], + {}, 'function') + + # @verifies REQ_INTEROP_097 + def test_l1_area_with_default_namespace(self): + self._run(MANIFEST_URL, [('area', 'areas/zone')], ['zone/zapp'], + {'areas/zone': {'area'}}, 'area') + + # @verifies REQ_INTEROP_097 + def test_l1_area_with_namespace(self): + self._run(MANIFEST_URL, [('ns-area', 'areas/plant-area')], ['plant.papp'], + {}, 'ns-area') + + # @verifies REQ_INTEROP_097 + def test_l1_component_with_default_fqn(self): + self._run(MANIFEST_URL, [('component', 'components/ecu')], ['zone.zapp'], + {'components/ecu': {'component'}}, 'component') + + # @verifies REQ_INTEROP_097 + def test_l1_component_with_namespace(self): + self._run(MANIFEST_URL, [('ns-component', 'components/plant-ecu')], ['plant/papp'], + {}, 'ns-component') + + # L2: triggers that list each other's markers. + + # @verifies REQ_INTEROP_097 + def test_l2_two_triggers_on_one_app(self): + self._run(RUNTIME_URL, [('twin-a', 'apps/loop_root'), ('twin-b', 'apps/loop_root')], + ['loop_root'], {'apps/loop_root': {'twin-a', 'twin-b'}}, 'twins') + + # @verifies REQ_INTEROP_097 + def test_l2_app_and_the_host_component_above_it(self): + self._run(RUNTIME_URL, + [('pair-app', 'apps/loop_root'), ('pair-host', f'components/{HOST}')], + ['loop_root'], + {'apps/loop_root': {'pair-app'}, f'components/{HOST}': {'pair-app'}}, + 'app-host') + + # @verifies REQ_INTEROP_097 + def test_l2_component_and_the_app_on_it(self): + self._run(MANIFEST_URL, + [('pair-ecu', 'components/ecu'), ('pair-zapp', 'apps/zapp')], + ['zone/zapp'], {'components/ecu': {'pair-ecu'}}, 'ecu-app') + + # A marker written by a faults trigger. + + # @verifies REQ_INTEROP_097 + def test_marker_of_a_faults_trigger_fires_no_logs_trigger(self): + """The root app lists the faults trigger's marker; its logs trigger skips it.""" + path = 'apps/loop_root' + _configure_logs(RUNTIME_URL, path) + logs = TriggerStream(RUNTIME_URL, path, 'fault-logs', marker=False) + faults = TriggerStream(RUNTIME_URL, path, 'fault', collection='faults') + try: + logs.start() + faults.start() + self.assertTrue(self.probe.report_fault(f'MARKER_FAULT_{RUN.upper()}', '/loop_root'), + 'the fault manager refused the report') + + def markers(): + return [i for i in _listing(RUNTIME_URL, path) + if _is_marker_of(i.get('message', ''), 'fault')] + _poll(lambda: markers() and faults.snapshot(), + what='the faults trigger to fire and GET to list its marker', + timeout=DELIVERY_TIMEOUT_SEC) + # Queued after the markers, so delivered after the logs trigger judged them. + sentinel = f'{RUN}:fault-sentinel' + self.probe.publish('loop_root', sentinel) + _poll(lambda: any(p.get('message') == sentinel for p in logs.snapshot()), + what='the logs trigger to deliver the sentinel', timeout=DELIVERY_TIMEOUT_SEC) + + def settled(): + first = (len(markers()), len(faults.snapshot())) + time.sleep(POLL_INTERVAL_SEC) + return first == (len(markers()), len(faults.snapshot())) and first + listed, firings = _poll(settled, what='the fault events to settle', + timeout=SETTLE_TIMEOUT_SEC) + fired = [_describe(p) for p in logs.snapshot() + if _is_marker_of(p.get('message', ''), 'fault')] + finally: + logs.close() + faults.close() + self.assertEqual(fired, [], 'a faults trigger marker fired the logs trigger') + self.assertEqual(listed, firings, 'GET lists one marker per firing of the faults trigger') + + # L4: a real entry with a marker's content still fires. + + # @verifies REQ_INTEROP_097 + def test_l4_real_entry_that_looks_like_a_marker_fires(self): + self._run(RUNTIME_URL, [('mimic', 'apps/loop_root')], ['loop_root'], + {'apps/loop_root': {'mimic'}}, 'mimic', lookalike_of=('mimic', 'loop_root')) + + +@launch_testing.post_shutdown_test() +class TestShutdown(unittest.TestCase): + + def test_exit_codes(self, proc_info): + """Check all processes exited cleanly (SIGTERM allowed).""" + for info in proc_info: + self.assertIn( + info.returncode, ALLOWED_EXIT_CODES, + f'{info.process_name} exited with code {info.returncode}', + ) + for path in _MANIFEST_FILES: + if os.path.exists(path): + os.unlink(path) diff --git a/src/ros2_medkit_integration_tests/test/features/test_triggers_logs_restore_before_discovery.test.py b/src/ros2_medkit_integration_tests/test/features/test_triggers_logs_restore_before_discovery.test.py new file mode 100644 index 000000000..92f7ced66 --- /dev/null +++ b/src/ros2_medkit_integration_tests/test/features/test_triggers_logs_restore_before_discovery.test.py @@ -0,0 +1,538 @@ +#!/usr/bin/env python3 +# Copyright 2026 bburda +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Feature test: a restored logs trigger fires for entries logged before discovery. + +A persistent logs trigger is restored while the gateway is constructed, and its +entity may reach the discovery cache much later. An entry its entity's +GET /logs lists once the entity is discovered must reach the trigger, even when +it was logged before that, and must reach no other entity's trigger. Both +triggers have log_settings and write one marker per firing, the first one +during the replay. A marker fires no trigger. It is logged under the trigger's +entity id, so the root app lists its own markers and neither app lists the +namespaced app's. + +Two nodes named late_logger make the owner rule visible. Node +/late/late_logger gets the app id late_late_logger, because the root node +/late_logger holds the bare name, so the last segment of its logger is the id +of the other app. A rule that guesses the owner from that segment sends +the namespaced node's entries to late_logger and none to late_late_logger. + +The restarted gateway runs on a DDS domain of its own. Its nodes are started by +the test, root node first so the ids never change, after the test has published +an entry under each node's logger name, so each entry provably arrives before +its entity exists. +""" + +import json +import os +import re +import tempfile +import threading +import time +import unittest +import uuid + +from launch import LaunchDescription +from launch.actions import ExecuteProcess, RegisterEventHandler, TimerAction +from launch.event_handlers import OnProcessExit +import launch_ros.actions +import launch_testing +import launch_testing.actions +from rcl_interfaces.msg import Log +import rclpy +from rclpy.duration import Duration +from rclpy.qos import DurabilityPolicy, QoSProfile, ReliabilityPolicy +import requests + +from ros2_medkit_test_utils.constants import ( + ALLOWED_EXIT_CODES, + API_BASE_PATH, + get_test_domain_id, + get_test_port, + get_time_scale, +) +from ros2_medkit_test_utils.coverage import get_coverage_env +from ros2_medkit_test_utils.launch_helpers import create_gateway_node + +PORT_PRIMARY = get_test_port(0) +PORT_RESTARTED = get_test_port(1) +BASE_URL_PRIMARY = f'http://localhost:{PORT_PRIMARY}{API_BASE_PATH}' +BASE_URL_RESTARTED = f'http://localhost:{PORT_RESTARTED}{API_BASE_PATH}' + +PRIMARY_DOMAIN = get_test_domain_id(0) +RESTARTED_DOMAIN = get_test_domain_id(1) + +TIME_SCALE = get_time_scale() +HTTP_TIMEOUT_SEC = 5.0 * TIME_SCALE +DISCOVERY_TIMEOUT_SEC = 60.0 * TIME_SCALE +DELIVERY_TIMEOUT_SEC = 20.0 * TIME_SCALE + + +def _level(constant): + """Return a Log severity constant as an int. Humble declares the constants as byte.""" + return constant if isinstance(constant, int) else ord(constant) + + +# Logs one INFO line at startup and nothing after. +NODE_EXECUTABLE = 'demo_calibration_service' +# The startup line reaches the gateway only when /rosout matched first, and it fires the +# trigger like any entry. At warn the node publishes nothing, so the test's entries are +# the only entries of each app. +NODE_LOG_LEVEL = 'warn' +# Root node: its logger name is its app id. +NAMESAKE_APP = 'late_logger' +# /late/late_logger: collides with the root node, so its id carries the namespace. +OWNER_APP = 'late_late_logger' +OWNER_NAMESPACE = '/late' +# Slash form: the last segment is NAMESAKE_APP. +OWNER_LOGGER = 'late/late_logger' + +RUN = uuid.uuid4().hex[:10] +MARKER = re.compile(rf'{RUN}:([A-Za-z0-9_.-]+)') + +DB_PATH = os.path.join( + tempfile.gettempdir(), + f'test_triggers_logs_restore_before_discovery_{os.getpid()}.db', +) +GATE_RESTART = os.path.join( + tempfile.gettempdir(), + f'test_triggers_logs_restore_before_discovery_restart_{os.getpid()}', +) +GATE_NAMESAKE = os.path.join( + tempfile.gettempdir(), + f'test_triggers_logs_restore_before_discovery_namesake_{os.getpid()}', +) +GATE_OWNER = os.path.join( + tempfile.gettempdir(), + f'test_triggers_logs_restore_before_discovery_owner_{os.getpid()}', +) + + +def _gate_process(name, path): + return ExecuteProcess( + cmd=['sh', '-c', f'while [ ! -e "{path}" ]; do sleep 0.2; done'], + name=name, + output='screen', + ) + + +def _app_node(domain, namespace): + env = get_coverage_env('ros2_medkit_integration_tests') + env['ROS_DOMAIN_ID'] = str(domain) + return launch_ros.actions.Node( + package='ros2_medkit_integration_tests', + executable=NODE_EXECUTABLE, + name='late_logger', + namespace=namespace, + arguments=['--ros-args', '--log-level', NODE_LOG_LEVEL], + output='screen', + additional_env=env, + sigterm_timeout='30', + sigkill_timeout='15', + ) + + +def generate_test_description(): + primary = create_gateway_node( + name=f'ros2_medkit_gateway_{PORT_PRIMARY}', + port=PORT_PRIMARY, + extra_params={ + 'triggers.enabled': True, + 'triggers.storage.path': DB_PATH, + 'triggers.on_restart_behavior': 'reset', + }, + extra_env={'ROS_DOMAIN_ID': str(PRIMARY_DOMAIN)}, + ) + restarted = create_gateway_node( + name=f'ros2_medkit_gateway_{PORT_RESTARTED}', + port=PORT_RESTARTED, + extra_params={ + 'triggers.enabled': True, + 'triggers.storage.path': DB_PATH, + 'triggers.on_restart_behavior': 'restore', + 'refresh_interval_ms': 1000, + }, + extra_env={'ROS_DOMAIN_ID': str(RESTARTED_DOMAIN)}, + ) + + restart_gate = _gate_process('restart_gate', GATE_RESTART) + namesake_gate = _gate_process('namesake_gate', GATE_NAMESAKE) + owner_gate = _gate_process('owner_gate', GATE_OWNER) + + return ( + LaunchDescription([ + primary, + restart_gate, + RegisterEventHandler( + OnProcessExit(target_action=restart_gate, on_exit=[restarted]), + ), + namesake_gate, + RegisterEventHandler( + OnProcessExit(target_action=namesake_gate, + on_exit=[_app_node(RESTARTED_DOMAIN, '/')]), + ), + owner_gate, + RegisterEventHandler( + OnProcessExit(target_action=owner_gate, + on_exit=[_app_node(RESTARTED_DOMAIN, OWNER_NAMESPACE)]), + ), + TimerAction( + period=2.0, + actions=[ + _app_node(PRIMARY_DOMAIN, '/'), + _app_node(PRIMARY_DOMAIN, OWNER_NAMESPACE), + launch_testing.actions.ReadyToTest(), + ], + ), + ]), + {'primary': primary, 'restarted': restarted}, + ) + + +def _remove_gates(): + for path in (GATE_RESTART, GATE_NAMESAKE, GATE_OWNER): + if os.path.exists(path): + try: + os.unlink(path) + except OSError: + pass + + +def _open_gate(path): + with open(path, 'w', encoding='utf-8') as gate: + gate.write('open') + + +def _poll(predicate, *, what, timeout=DISCOVERY_TIMEOUT_SEC): + deadline = time.monotonic() + timeout + last = None + while time.monotonic() < deadline: + try: + last = predicate() + if last: + return last + except requests.exceptions.RequestException as exc: + last = exc + time.sleep(0.3) + raise AssertionError(f'timed out after {timeout}s waiting for {what}; last={last!r}') + + +def _at_least(items, count): + """Return `items` when it holds `count` or more entries, else None.""" + return items if len(items) >= count else None + + +def _id_number(entry_id): + """Return n of a log entry id "log_"; n grows with every entry the gateway logs.""" + return int(entry_id.removeprefix('log_')) + + +def _status(url): + return requests.get(url, timeout=HTTP_TIMEOUT_SEC).status_code + + +def _listed(base_url, app_id): + r = requests.get(f'{base_url}/apps/{app_id}/logs', timeout=HTTP_TIMEOUT_SEC) + r.raise_for_status() + found = {} + for item in r.json().get('items', []): + match = MARKER.search(item.get('message', '')) + if match: + found.setdefault(match.group(1), []).append(item.get('id')) + return found + + +def _listed_past(base_url, app_id, marker, last_id): + """Return the listing of `app_id` once it holds `marker` with an id past `last_id`.""" + found = _listed(base_url, app_id) + return found if any(_id_number(i) > last_id for i in found.get(marker, [])) else None + + +def _markers(payloads): + """Return [(marker, entry id), ...] in delivery order.""" + out = [] + for payload in payloads: + if isinstance(payload, dict): + match = MARKER.search(payload.get('message', '')) + if match: + out.append((match.group(1), payload.get('id'))) + return out + + +class EventReader: + """Read a trigger's SSE stream on a daemon thread.""" + + def __init__(self, events_url): + self.payloads = [] + self.error = None + self._closed = False + self._cond = threading.Condition() + self._resp = requests.get(events_url, stream=True, + timeout=(HTTP_TIMEOUT_SEC, 120.0 * TIME_SCALE)) + assert self._resp.status_code == 200, f'GET {events_url}: {self._resp.status_code}' + self._thread = threading.Thread(target=self._read, daemon=True) + self._thread.start() + + def _read(self): + try: + for line in self._resp.iter_lines(chunk_size=None, decode_unicode=True): + if line and line.startswith('data: '): + envelope = json.loads(line[len('data: '):]) + with self._cond: + self.payloads.append(envelope.get('payload')) + self._cond.notify_all() + except Exception as exc: + if not self._closed: + self.error = exc + + def markers(self): + with self._cond: + return _markers(self.payloads) + + def count(self): + """Return the number of events delivered so far, one per firing.""" + with self._cond: + return len(self.payloads) + + def others(self): + """Return the delivered payloads that carry no marker of this run.""" + with self._cond: + return [p for p in self.payloads + if not (isinstance(p, dict) and MARKER.search(p.get('message', '')))] + + def wait_for(self, marker, timeout): + deadline = time.monotonic() + timeout + with self._cond: + while not any(m == marker for m, _ in _markers(self.payloads)): + remaining = deadline - time.monotonic() + if remaining <= 0: + return False + self._cond.wait(timeout=min(remaining, 0.5)) + return True + + def ended(self, timeout): + """Return True once the gateway has ended the stream.""" + self._thread.join(timeout) + return not self._thread.is_alive() + + def close(self): + self._closed = True + self._resp.close() + + +class RosoutProbe: + """Publish /rosout entries on the restarted gateway's domain.""" + + def __init__(self, domain): + self.context = rclpy.Context() + rclpy.init(context=self.context, domain_id=domain) + self.node = rclpy.create_node('log_restore_probe', context=self.context) + qos = QoSProfile(depth=100, reliability=ReliabilityPolicy.RELIABLE, + durability=DurabilityPolicy.VOLATILE) + self.pub = self.node.create_publisher(Log, '/rosout', qos) + + def publish(self, case_id, logger_name): + msg = Log() + msg.stamp = self.node.get_clock().now().to_msg() + msg.level = _level(Log.INFO) + msg.name = logger_name + msg.msg = f'{RUN}:{case_id}' + msg.file = 'test_triggers_logs_restore_before_discovery.test.py' + msg.function = 'publish' + msg.line = 1 + self.pub.publish(msg) + + def wait_delivered(self): + """Return True once every subscriber has acknowledged every published entry.""" + return self.pub.wait_for_all_acked(Duration(seconds=DELIVERY_TIMEOUT_SEC)) + + def close(self): + self.node.destroy_node() + rclpy.shutdown(context=self.context) + + +class TestLogsTriggerRestoreBeforeDiscovery(unittest.TestCase): + """An entry logged before discovery reaches its owner's restored trigger only.""" + + trigger_ids = {} + + @classmethod + def setUpClass(cls): + cls.addClassCleanup(_remove_gates) + for app_id in (NAMESAKE_APP, OWNER_APP): + _poll(lambda a=app_id: _status(f'{BASE_URL_PRIMARY}/apps/{a}') == 200, + what=f'{app_id} on the primary gateway') + + # @verifies REQ_INTEROP_097 + def test_01_create_persistent_logs_triggers(self): + """POST a persistent logs trigger on each app, then restart.""" + for app_id in (NAMESAKE_APP, OWNER_APP): + r = requests.post( + f'{BASE_URL_PRIMARY}/apps/{app_id}/triggers', + json={ + 'resource': f'{API_BASE_PATH}/apps/{app_id}/logs', + 'trigger_condition': {'condition_type': 'OnChange'}, + 'multishot': True, + 'persistent': True, + 'lifetime': 3600, + 'log_settings': {'severity': 'info', 'marker': f'{RUN}:ls-{app_id}'}, + }, + timeout=HTTP_TIMEOUT_SEC, + ) + self.assertEqual(r.status_code, 201, r.text) + TestLogsTriggerRestoreBeforeDiscovery.trigger_ids[app_id] = r.json()['id'] + _open_gate(GATE_RESTART) + + def _open_stream(self, app_id): + r = requests.get( + f'{BASE_URL_RESTARTED}/apps/{app_id}/triggers/{self.trigger_ids[app_id]}', + timeout=HTTP_TIMEOUT_SEC) + self.assertEqual(r.status_code, 200, r.text) + return EventReader(BASE_URL_RESTARTED + + r.json()['event_source'].removeprefix(API_BASE_PATH)) + + def _discover(self, gate, app_id, marker): + _open_gate(gate) + _poll(lambda: _status(f'{BASE_URL_RESTARTED}/apps/{app_id}') == 200, + what=f'{app_id} on the restarted gateway') + return _poll(lambda: _listed(BASE_URL_RESTARTED, app_id).get(marker), + what=f'GET /apps/{app_id}/logs to list {marker}') + + # @verifies REQ_INTEROP_097 + def test_02_entries_before_discovery_reach_their_owner_only(self): + """Each early entry is delivered once, first, to the app that lists it.""" + self.assertEqual(len(self.trigger_ids), 2, 'test_01 must create both triggers') + _poll(lambda: _status(f'{BASE_URL_RESTARTED}/health') == 200, + what='the restarted gateway') + + probe = RosoutProbe(RESTARTED_DOMAIN) + readers = {} + problems_at_close = [] + try: + _poll(lambda: probe.pub.get_subscription_count() >= 1, + what='the restarted gateway to subscribe /rosout') + for app_id in (NAMESAKE_APP, OWNER_APP): + self.assertEqual(_status(f'{BASE_URL_RESTARTED}/apps/{app_id}'), 404, + f'{app_id} must not exist before its node starts') + probe.publish('owner-early', OWNER_LOGGER) + probe.publish('namesake-early', NAMESAKE_APP) + self.assertTrue(probe.wait_delivered(), + 'the gateway did not receive the early entries') + for app_id in (NAMESAKE_APP, OWNER_APP): + self.assertEqual(_status(f'{BASE_URL_RESTARTED}/apps/{app_id}'), 404, + f'{app_id} appeared before the early entries were logged') + + listed = { + NAMESAKE_APP: self._discover(GATE_NAMESAKE, NAMESAKE_APP, 'namesake-early'), + OWNER_APP: self._discover(GATE_OWNER, OWNER_APP, 'owner-early'), + } + self.assertEqual(_status(f'{BASE_URL_RESTARTED}/apps/{NAMESAKE_APP}'), 200) + self.assertNotIn('owner-early', _listed(BASE_URL_RESTARTED, NAMESAKE_APP)) + + for app_id in (NAMESAKE_APP, OWNER_APP): + readers[app_id] = self._open_stream(app_id) + probe.publish('owner-late', OWNER_LOGGER) + probe.publish('namesake-late', NAMESAKE_APP) + for app_id, marker in ((OWNER_APP, 'owner-late'), (NAMESAKE_APP, 'namesake-late')): + reader = readers[app_id] + self.assertTrue(reader.wait_for(marker, DELIVERY_TIMEOUT_SEC), + f'{app_id}: {marker} not delivered: {reader.markers()}, ' + f'stream error {reader.error!r}') + + # A firing writes its marker after it delivers the event. Once each firing + # so far has its marker listed, a marker with an id past the next entry's + # belongs to that entry or a later one. + fired = readers[NAMESAKE_APP].count() + _poll(lambda: _at_least( + _listed(BASE_URL_RESTARTED, NAMESAKE_APP).get(f'ls-{NAMESAKE_APP}', []), fired), + what=f'GET /apps/{NAMESAKE_APP}/logs to list a marker per delivered event') + # Each trigger delivers in log order, so once an entry logged after the + # others arrives, no earlier entry can still reach it. + probe.publish('owner-done', OWNER_LOGGER) + probe.publish('namesake-done', NAMESAKE_APP) + for app_id, marker in ((OWNER_APP, 'owner-done'), (NAMESAKE_APP, 'namesake-done')): + self.assertTrue(readers[app_id].wait_for(marker, DELIVERY_TIMEOUT_SEC), + f'{app_id}: {marker} not delivered: {readers[app_id].markers()}') + # One notifier worker dispatches the entries in log order and writes the + # markers of an entry before it takes the next. namesake-done is the last + # entry, so once its marker is listed no trigger writes another. + done_id = _id_number(dict(readers[NAMESAKE_APP].markers())['namesake-done']) + listings = { + NAMESAKE_APP: _poll( + lambda: _listed_past(BASE_URL_RESTARTED, NAMESAKE_APP, + f'ls-{NAMESAKE_APP}', done_id), + what=f'GET /apps/{NAMESAKE_APP}/logs to list the marker of namesake-done'), + OWNER_APP: _listed(BASE_URL_RESTARTED, OWNER_APP), + } + owner = readers[OWNER_APP].markers() + namesake = readers[NAMESAKE_APP].markers() + problems = [] + # The root app lists one marker per own entry. Neither app lists the namespaced + # app's markers: they are logged under its id, which neither app's log holds. + expected_markers = {NAMESAKE_APP: {f'ls-{NAMESAKE_APP}': 3}, OWNER_APP: {}} + for app_id, want in expected_markers.items(): + got = {name: len(ids) for name, ids in listings[app_id].items() + if name.startswith('ls-')} + if got != want: + problems.append(f'{app_id}: GET lists markers {got}, expected {want}: ' + f'{listings[app_id]}') + for app_id, reader in readers.items(): + if reader.others(): + problems.append(f'{app_id}: delivered entries the test never logged: ' + f'{reader.others()}') + if [i for m, i in owner if m == 'owner-early'] != listed[OWNER_APP]: + problems.append(f'{OWNER_APP}: early entry listed as {listed[OWNER_APP]}, ' + f'delivered {owner}') + if [m for m, _ in owner] != ['owner-early', 'owner-late', 'owner-done']: + problems.append(f'{OWNER_APP}: expected each own entry once, in log order, ' + f'got {owner}') + if [m for m, _ in namesake] != ['namesake-early', 'namesake-late', 'namesake-done']: + problems.append(f'{NAMESAKE_APP}: expected only its own entries, once, in log ' + f'order, got {namesake}') + self.assertFalse(problems, '; '.join(problems)) + finally: + # A trigger's stream wakes only for an event, the trigger's removal or its + # 15 s keepalive. Removing the triggers ends both streams now, so neither the + # close below nor the gateway's shutdown waits for a keepalive. + for app_id, reader in readers.items(): + requests.delete( + f'{BASE_URL_RESTARTED}/apps/{app_id}/triggers/{self.trigger_ids[app_id]}', + timeout=HTTP_TIMEOUT_SEC) + if not reader.ended(DELIVERY_TIMEOUT_SEC): + problems_at_close.append(f'{app_id}: the stream outlived its trigger') + for reader in readers.values(): + reader.close() + probe.close() + self.assertFalse(problems_at_close, '; '.join(problems_at_close)) + + +@launch_testing.post_shutdown_test() +class TestShutdown(unittest.TestCase): + + def test_exit_codes(self, proc_info): + """Check all processes exited cleanly (SIGTERM allowed).""" + for info in proc_info: + self.assertIn( + info.returncode, ALLOWED_EXIT_CODES, + f'{info.process_name} exited with code {info.returncode}', + ) + for path in (DB_PATH, f'{DB_PATH}-wal', f'{DB_PATH}-shm'): + if os.path.exists(path): + try: + os.unlink(path) + except OSError: + pass + _remove_gates() diff --git a/src/ros2_medkit_integration_tests/test/features/test_triggers_logs_scope.test.py b/src/ros2_medkit_integration_tests/test/features/test_triggers_logs_scope.test.py new file mode 100644 index 000000000..9f6ea32a4 --- /dev/null +++ b/src/ros2_medkit_integration_tests/test/features/test_triggers_logs_scope.test.py @@ -0,0 +1,880 @@ +#!/usr/bin/env python3 +# Copyright 2026 bburda +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""A trigger on /{entity}/logs fires for the entries GET /{entity}/logs lists. + +Checked with the gateway's own log buffer (no LogProvider plugin) and far +below the change-queue and pending-event bounds. + +Four gateways share one ROS graph: + +- runtime: runtime_only. Two ``temp_sensor`` nodes collide, ``rpm_sensor`` does + not, ``engine_temp`` is a root-namespace node. +- hybrid, manifest: one manifest, app ids differ from node names. The app id + ``engine_temp`` equals the root node's name. Areas with a subarea, two + components, a function hosted by an app and a component. +- capped: runtime_only with ``logs.buffer_size`` 3, so the distinct-logger cap + (buffer size x 10) drops entries from new logger names. + +The test publishes /rosout entries under the logger names nodes use (dot form), +the slash form, child loggers and a root-namespace name. Each stream is checked +three ways: the entries its trigger delivered, the entries GET lists, and the +owners written in the tables below. A sentinel entry published after every +probe proves each stream live, so an absent event is a real absence. +""" + +import json +import os +import re +import tempfile +import threading +import time +import unittest +import uuid + +from launch import LaunchDescription +import launch_ros.actions +import launch_testing +import launch_testing.actions +from rcl_interfaces.msg import Log +import rclpy +from rclpy.duration import Duration +from rclpy.qos import DurabilityPolicy, QoSProfile, ReliabilityPolicy +import requests + +from ros2_medkit_test_utils.constants import ( + ALLOWED_EXIT_CODES, + API_BASE_PATH, + get_test_port, + get_time_scale, +) +from ros2_medkit_test_utils.coverage import get_coverage_env +from ros2_medkit_test_utils.launch_helpers import create_gateway_node + +RUNTIME_PORT = get_test_port(0) +HYBRID_PORT = get_test_port(1) +MANIFEST_PORT = get_test_port(2) +CAPPED_PORT = get_test_port(3) + + +def _base_url(port): + return f'http://localhost:{port}{API_BASE_PATH}' + + +RUNTIME_URL = _base_url(RUNTIME_PORT) +HYBRID_URL = _base_url(HYBRID_PORT) +MANIFEST_URL = _base_url(MANIFEST_PORT) +CAPPED_URL = _base_url(CAPPED_PORT) + +TIME_SCALE = get_time_scale() +HTTP_TIMEOUT_SEC = 5.0 * TIME_SCALE +DISCOVERY_TIMEOUT_SEC = 60.0 * TIME_SCALE +DELIVERY_TIMEOUT_SEC = 20.0 * TIME_SCALE +POLL_INTERVAL_SEC = 0.3 + +# Every marker this run writes starts with RUN, so entries of other runs and +# of the demo nodes never match. +RUN = uuid.uuid4().hex[:10] +MARKER = re.compile(rf'{RUN}:([A-Za-z0-9_.-]+)') + + +def _level(constant): + """Return a Log severity constant as an int. Humble declares the constants as byte.""" + return constant if isinstance(constant, int) else ord(constant) + + +INFO = _level(Log.INFO) +WARN = _level(Log.WARN) + +PROBE_NODE = 'log_trigger_probe' + +# (executable, node name, namespace) +GRAPH_NODES = [ + ('demo_engine_temp_sensor', 'temp_sensor', '/powertrain/engine'), + ('demo_engine_temp_sensor', 'temp_sensor', '/backup'), + ('demo_rpm_sensor', 'rpm_sensor', '/powertrain/engine'), + ('demo_rpm_sensor', 'engine_temp', '/'), + # Logs only at startup, so its buffer on the capped gateway holds what the + # test writes. + ('demo_calibration_service', 'cap_probe', '/capped'), +] + +MANIFEST_YAML = """\ +manifest_version: "1.0" +metadata: + name: "log trigger scope" + version: "1.0.0" +config: + unmanifested_nodes: ignore +areas: + - id: powertrain + name: "Powertrain" + namespace: /powertrain + subareas: + - id: engine + name: "Engine" + namespace: /powertrain/engine + - id: backup-area + name: "Backup" + namespace: /backup +components: + - id: engine-ecu + name: "Engine ECU" + area: engine + namespace: /powertrain + - id: backup-ecu + name: "Backup ECU" + area: backup-area +apps: + - id: engine_temp + name: "Engine temperature" + is_located_on: engine-ecu + ros_binding: + node_name: temp_sensor + namespace: /powertrain/engine + - id: engine-rpm + name: "Engine RPM" + is_located_on: engine-ecu + ros_binding: + node_name: rpm_sensor + namespace: /powertrain/engine + - id: backup-temp + name: "Backup temperature" + is_located_on: backup-ecu + ros_binding: + node_name: temp_sensor + namespace: /backup +functions: + - id: engine-watch + name: "Engine watch" + hosted_by: + - engine-rpm + - backup-ecu +""" + +# (case id, logger name, level) +PROBES = [ + ('tw1-dot', 'powertrain.engine.temp_sensor', INFO), + ('tw1-slash', 'powertrain/engine/temp_sensor', INFO), + ('tw1-child-dot', 'powertrain.engine.temp_sensor.worker', INFO), + # Child logger whose last segment is the runtime app id of another node. + ('tw1-child-slash', 'powertrain/engine/temp_sensor/rpm_sensor', INFO), + ('tw2-warn', 'backup.temp_sensor', WARN), + ('tw2-info', 'backup.temp_sensor', INFO), + ('rpm-dot', 'powertrain.engine.rpm_sensor', INFO), + ('rpm-slash', 'powertrain/engine/rpm_sensor', INFO), + # Root-namespace node. Namesake of the manifest app id engine_temp. + ('root', 'engine_temp', INFO), + ('ns-token', 'powertrain', INFO), + # Under the engine-ecu fqn, not under the /powertrain/engine namespace. + ('ecu-child', 'powertrain.engine-ecu.fw', INFO), + ('stranger', 'unrelated/node', INFO), +] +PROBE_IDS = {case for case, _, _ in PROBES} + +# Published after every probe. Each has the logger name and level of the +# probe it maps to, so it has the same owners. +SENTINELS = { + 's-tw1': 'tw1-dot', + 's-tw2': 'tw2-warn', + 's-rpm': 'rpm-dot', + 's-root': 'root', +} +_PROBE_BY_ID = {case: (name, level) for case, name, level in PROBES} + +HOST = '{host}' + +# Owners per the log routes' rule: exact sources (slash or dot form) for every +# entity type, plus the namespace prefix for components and areas. +RUNTIME_EXPECTED = { + 'apps/powertrain_engine_temp_sensor': {'tw1-dot', 'tw1-slash'}, + # severity_filter is set to warning on this app. + 'apps/backup_temp_sensor': {'tw2-warn'}, + 'apps/rpm_sensor': {'rpm-dot', 'rpm-slash'}, + 'apps/engine_temp': {'root'}, + f'components/{HOST}': { + 'tw1-dot', 'tw1-slash', 'tw2-warn', 'tw2-info', 'rpm-dot', 'rpm-slash', 'root', + }, + 'functions/powertrain': {'tw1-dot', 'tw1-slash', 'rpm-dot', 'rpm-slash'}, + 'functions/backup': {'tw2-warn', 'tw2-info'}, +} +MANIFEST_EXPECTED = { + 'apps/engine_temp': {'tw1-dot', 'tw1-slash'}, + 'apps/engine-rpm': {'rpm-dot', 'rpm-slash'}, + 'apps/backup-temp': {'tw2-warn', 'tw2-info'}, + 'components/engine-ecu': {'tw1-dot', 'tw1-slash', 'rpm-dot', 'rpm-slash', 'ecu-child'}, + 'components/backup-ecu': {'tw2-warn', 'tw2-info'}, + 'areas/powertrain': { + 'tw1-dot', 'tw1-slash', 'tw1-child-dot', 'tw1-child-slash', 'rpm-dot', 'rpm-slash', + 'ns-token', 'ecu-child', + }, + 'areas/engine': { + 'tw1-dot', 'tw1-slash', 'tw1-child-dot', 'tw1-child-slash', 'rpm-dot', 'rpm-slash', + }, + 'areas/backup-area': {'tw2-warn', 'tw2-info'}, + 'functions/engine-watch': {'rpm-dot', 'rpm-slash', 'tw2-warn', 'tw2-info'}, +} +GATEWAY_EXPECTED = { + RUNTIME_URL: RUNTIME_EXPECTED, + HYBRID_URL: MANIFEST_EXPECTED, + MANIFEST_URL: MANIFEST_EXPECTED, +} + + +def _expected_sentinels(owned): + return {s for s, twin in SENTINELS.items() if twin in owned} + + +_MANIFEST_PATHS = [] + + +def _write_manifest(): + fd, path = tempfile.mkstemp(suffix='.yaml', prefix='test_triggers_logs_scope_') + with os.fdopen(fd, 'w') as handle: + handle.write(MANIFEST_YAML) + _MANIFEST_PATHS.append(path) + return path + + +def generate_test_description(): + manifest_path = _write_manifest() + # Up to nine streams per gateway at once; one stream pins one HTTP worker. + streams = {'sse.max_clients': 12, 'server.http_thread_pool_size': 20} + + def gateway(name, port, params): + merged = dict(streams) + merged.update(params) + return create_gateway_node(port=port, name=name, extra_params=merged) + + manifest_params = { + 'discovery.manifest_path': manifest_path, + 'discovery.manifest_strict_validation': False, + } + gateways = [ + gateway('gw_runtime', RUNTIME_PORT, {'discovery.mode': 'runtime_only'}), + gateway('gw_hybrid', HYBRID_PORT, dict( + manifest_params, **{ + 'discovery.mode': 'hybrid', + 'discovery.merge_pipeline.gap_fill.allow_heuristic_apps': False, + })), + gateway('gw_manifest', MANIFEST_PORT, dict( + manifest_params, **{'discovery.mode': 'manifest_only'})), + gateway('gw_capped', CAPPED_PORT, { + 'discovery.mode': 'runtime_only', + 'logs.buffer_size': 3, + }), + ] + + env = get_coverage_env('ros2_medkit_integration_tests') + nodes = [ + launch_ros.actions.Node( + package='ros2_medkit_integration_tests', + executable=executable, + name=name, + namespace=namespace, + output='screen', + additional_env=env, + sigterm_timeout='30', + sigkill_timeout='15', + ) + for executable, name, namespace in GRAPH_NODES + ] + + return ( + LaunchDescription(gateways + nodes + [launch_testing.actions.ReadyToTest()]), + {'gateway_node': gateways[0]}, + ) + + +# -------------------------------------------------------------------------- +# Helpers +# -------------------------------------------------------------------------- + +def _poll(predicate, *, what, timeout=DISCOVERY_TIMEOUT_SEC): + deadline = time.monotonic() + timeout + last = None + while time.monotonic() < deadline: + try: + last = predicate() + if last: + return last + except requests.exceptions.RequestException as exc: + last = exc + time.sleep(POLL_INTERVAL_SEC) + raise AssertionError(f'timed out after {timeout}s waiting for {what}; last={last!r}') + + +def _status(url): + return requests.get(url, timeout=HTTP_TIMEOUT_SEC).status_code + + +def _wait_for_entities(base_url, paths): + for path in paths: + _poll(lambda p=path: _status(f'{base_url}/{p}') == 200, + what=f'{base_url}/{path}') + + +def _host_component(base_url): + def one_component(): + items = requests.get(f'{base_url}/components', timeout=HTTP_TIMEOUT_SEC).json() + items = items.get('items', []) + return items[0]['id'] if len(items) == 1 else None + return _poll(one_component, what=f'{base_url} to serve its host component') + + +def _listed(base_url, path): + """Return the RUN markers GET /{path}/logs lists, with their entry ids.""" + r = requests.get(f'{base_url}/{path}/logs', timeout=HTTP_TIMEOUT_SEC) + r.raise_for_status() + found = {} + for item in r.json().get('items', []): + match = MARKER.search(item.get('message', '')) + if match: + found.setdefault(match.group(1), []).append(item.get('id')) + return found + + +def _configure_logs(base_url, path, **fields): + body = {'max_entries': 10000} + body.update(fields) + r = requests.put(f'{base_url}/{path}/logs/configuration', json=body, + timeout=HTTP_TIMEOUT_SEC) + assert r.status_code == 204, f'PUT {base_url}/{path}/logs/configuration: {r.text}' + + +class EventStream: + """One trigger plus a thread reading its SSE event stream.""" + + def __init__(self, base_url, path, body=None): + self.base_url = base_url + self.path = path + self.body = body or { + 'resource': f'{API_BASE_PATH}/{path}/logs', + 'trigger_condition': {'condition_type': 'OnChange'}, + 'multishot': True, + 'lifetime': 600, + } + self.trigger_id = None + self._events_url = None + self.payloads = [] + self.error = None + self._cond = threading.Condition() + self._resp = None + self._closed = False + self._thread = None + + def start(self): + return self.create().connect() + + def create(self): + """POST the trigger; its events queue until connect() reads them.""" + r = requests.post(f'{self.base_url}/{self.path}/triggers', json=self.body, + timeout=HTTP_TIMEOUT_SEC) + assert r.status_code == 201, f'POST {self.base_url}/{self.path}/triggers: {r.text}' + trigger = r.json() + self.trigger_id = trigger['id'] + self._events_url = self.base_url + trigger['event_source'].removeprefix(API_BASE_PATH) + return self + + def connect(self): + events_url = self._events_url + self._resp = requests.get(events_url, stream=True, + timeout=(HTTP_TIMEOUT_SEC, 120.0 * TIME_SCALE)) + assert self._resp.status_code == 200, f'GET {events_url}: {self._resp.status_code}' + self._thread = threading.Thread(target=self._read, daemon=True) + self._thread.start() + return self + + def _read(self): + try: + for line in self._resp.iter_lines(chunk_size=None, decode_unicode=True): + if not line or not line.startswith('data: '): + continue + envelope = json.loads(line[len('data: '):]) + with self._cond: + self.payloads.append(envelope.get('payload')) + self._cond.notify_all() + except Exception as exc: + if not self._closed: + self.error = exc + + def markers(self): + """Return {marker: [entry id, ...]} over the log entries delivered so far.""" + with self._cond: + payloads = list(self.payloads) + found = {} + for payload in payloads: + if not isinstance(payload, dict): + continue + match = MARKER.search(payload.get('message', '')) + if match: + found.setdefault(match.group(1), []).append(payload.get('id')) + return found + + def ordered_markers(self): + """Return the markers of the delivered entries in delivery order.""" + with self._cond: + payloads = list(self.payloads) + return [m.group(1) for m in (MARKER.search(p.get('message', '')) + for p in payloads if isinstance(p, dict)) if m] + + def event_count(self): + with self._cond: + return len(self.payloads) + + def wait_until(self, predicate, timeout): + deadline = time.monotonic() + timeout + with self._cond: + while not predicate(): + remaining = deadline - time.monotonic() + if remaining <= 0: + return False + self._cond.wait(timeout=min(remaining, 0.5)) + return True + + def close(self): + self._closed = True + if self.trigger_id: + try: + requests.delete(f'{self.base_url}/{self.path}/triggers/{self.trigger_id}', + timeout=HTTP_TIMEOUT_SEC) + except requests.exceptions.RequestException: + pass + if self._resp is not None: + self._resp.close() + + +class RosoutProbe: + """Publish rcl_interfaces/Log entries on /rosout from one publisher.""" + + def __init__(self): + rclpy.init() + self.node = rclpy.create_node(PROBE_NODE) + qos = QoSProfile(depth=1000, reliability=ReliabilityPolicy.RELIABLE, + durability=DurabilityPolicy.VOLATILE) + self.pub = self.node.create_publisher(Log, '/rosout', qos) + # Warm-up entries of an earlier publisher prove nothing about this one. + self.token = uuid.uuid4().hex[:6] + + def publish(self, case_id, logger_name, level=INFO): + msg = Log() + msg.stamp = self.node.get_clock().now().to_msg() + msg.level = level + msg.name = logger_name + msg.msg = f'{RUN}:{case_id}' + msg.file = 'test_triggers_logs_scope.test.py' + msg.function = 'publish' + msg.line = 1 + self.pub.publish(msg) + + def wait_delivered(self): + """Return True once every gateway has acknowledged every published entry.""" + return self.pub.wait_for_all_acked(Duration(seconds=DELIVERY_TIMEOUT_SEC)) + + def wait_for_gateways(self, count): + _poll(lambda: self.pub.get_subscription_count() >= count, + what=f'{count} /rosout subscribers') + + def warm_up(self, checks, logger_name): + """Publish on logger_name until every (base_url, path) GET lists it.""" + attempt = 0 + prefix = f'warm-{self.token}-' + + def all_listed(): + nonlocal attempt + attempt += 1 + self.publish(f'{prefix}{attempt}', logger_name) + time.sleep(0.2) + return all( + any(m.startswith(prefix) for m in _listed(url, path)) for url, path in checks + ) + _poll(all_listed, what=f'{checks} to list a warm-up entry') + + def close(self): + self.node.destroy_node() + rclpy.shutdown() + + +# -------------------------------------------------------------------------- +# Distinct-logger cap: a dropped entry is never listed, so no trigger fires +# -------------------------------------------------------------------------- + +class TestCappedRingBuffer(unittest.TestCase): + """An entry the ring buffer drops fires no trigger (capped gateway).""" + + @classmethod + def setUpClass(cls): + _wait_for_entities(CAPPED_URL, ['apps/cap_probe']) + cls.probe = RosoutProbe() + # Class cleanups also run when setUpClass fails part-way. + cls.addClassCleanup(cls.probe.close) + cls.probe.wait_for_gateways(4) + cls.probe.warm_up([(CAPPED_URL, 'apps/cap_probe')], 'capped.cap_probe') + cls.stream = EventStream(CAPPED_URL, 'apps/cap_probe').start() + cls.addClassCleanup(cls.stream.close) + + # @verifies REQ_INTEROP_097 + def test_dropped_entry_fires_no_trigger(self): + """Fill the distinct-logger cap, then log under a new name the app owns.""" + # Cap is 3 x 10 = 30 distinct logger names; 40 new ones reach it. + for i in range(40): + self.probe.publish(f'junk-{i}', f'junk/n{i}') + self.assertTrue(self.probe.wait_delivered(), + 'the gateways did not receive the junk entries') + self.probe.publish('cap-drop', 'capped/cap_probe') + self.probe.publish('cap-sentinel', 'capped.cap_probe') + + arrived = self.stream.wait_until( + lambda: 'cap-sentinel' in self.stream.markers(), DELIVERY_TIMEOUT_SEC) + listed = _listed(CAPPED_URL, 'apps/cap_probe') + received = self.stream.markers() + + self.assertIn('cap-sentinel', listed, 'the sentinel went to an existing buffer') + self.assertNotIn('cap-drop', listed, + 'the cap did not engage, so this test measures nothing') + self.assertNotIn('cap-drop', received, + 'a trigger fired for an entry GET never lists') + self.assertTrue(arrived, f'sentinel not delivered; received={sorted(received)}') + + +# -------------------------------------------------------------------------- +# Owner rule across discovery modes, entity types and logger-name forms +# -------------------------------------------------------------------------- + +class TestLogTriggerScope(unittest.TestCase): + """Each trigger delivers exactly the probes its entity's GET lists.""" + + @classmethod + def setUpClass(cls): + _wait_for_entities(RUNTIME_URL, [ + p for p in RUNTIME_EXPECTED if HOST not in p]) + cls.host = _host_component(RUNTIME_URL) + for url in (HYBRID_URL, MANIFEST_URL): + _wait_for_entities(url, list(MANIFEST_EXPECTED)) + + cls.probe = RosoutProbe() + # Class cleanups also run when setUpClass fails part-way. + cls.addClassCleanup(cls.probe.close) + cls.probe.wait_for_gateways(4) + cls.probe.warm_up([ + (RUNTIME_URL, 'apps/rpm_sensor'), + (HYBRID_URL, 'apps/engine-rpm'), + (MANIFEST_URL, 'apps/engine-rpm'), + ], 'powertrain.engine.rpm_sensor') + + cls.expected = {} + for url, table in GATEWAY_EXPECTED.items(): + for path, owned in table.items(): + cls.expected[(url, path.replace(HOST, cls.host))] = owned + + for url, path in cls.expected: + _configure_logs(url, path) + _configure_logs(RUNTIME_URL, 'apps/backup_temp_sensor', severity_filter='warning') + + cls.streams = {} + for key in cls.expected: + cls.streams[key] = EventStream(*key).start() + cls.addClassCleanup(cls.streams[key].close) + + for case_id, name, level in PROBES: + cls.probe.publish(case_id, name, level) + for sentinel, twin in SENTINELS.items(): + name, level = _PROBE_BY_ID[twin] + cls.probe.publish(sentinel, name, level) + + deadline = time.monotonic() + DELIVERY_TIMEOUT_SEC + for key, stream in cls.streams.items(): + want = _expected_sentinels(cls.expected[key]) + stream.wait_until(lambda s=stream, w=want: w <= set(s.markers()), + max(0.0, deadline - time.monotonic())) + + cls.received = {key: stream.markers() for key, stream in cls.streams.items()} + cls.listed = {key: _listed(*key) for key in cls.expected} + + def _problems(self, key): + owned = self.expected[key] + want = owned | _expected_sentinels(owned) + tracked = PROBE_IDS | set(SENTINELS) + received = {m: ids for m, ids in self.received[key].items() if m in tracked} + listed = {m for m in self.listed[key] if m in tracked} + + problems = [] + if self.streams[key].error is not None: + problems.append(f'stream failed: {self.streams[key].error!r}') + if listed != want: + problems.append(f'GET lists {sorted(listed)}, expected {sorted(want)}') + extra = set(received) - want + if extra: + problems.append(f'delivered, not owned: {sorted(extra)}') + missing = want - set(received) + if missing: + problems.append(f'owned, not delivered: {sorted(missing)}') + repeated = {m: len(ids) for m, ids in received.items() if len(ids) != 1} + if repeated: + problems.append(f'delivered more than once: {repeated}') + return problems + + # @verifies REQ_INTEROP_097 + def test_events_match_get_listing(self): + """Every stream delivers exactly the probes its GET lists, once each.""" + for key in sorted(self.expected): + with self.subTest(gateway=key[0], entity=key[1]): + problems = self._problems(key) + self.assertFalse(problems, f'{key[1]}: ' + '; '.join(problems)) + + # @verifies REQ_INTEROP_097 + def test_namesake_app_gets_no_event(self): + """An app whose id is the last segment of another logger gets no event.""" + cases = [ + ((RUNTIME_URL, 'apps/rpm_sensor'), 'tw1-child-slash', 's-rpm'), + ((HYBRID_URL, 'apps/engine_temp'), 'root', 's-tw1'), + ((MANIFEST_URL, 'apps/engine_temp'), 'root', 's-tw1'), + ] + for key, namesake, sentinel in cases: + with self.subTest(gateway=key[0], entity=key[1], probe=namesake): + received = self.received[key] + self.assertNotIn(namesake, self.listed[key]) + self.assertNotIn(namesake, received, + f'{key[1]} got an event for the logger of another node') + self.assertIn(sentinel, received, 'stream not proven live') + + +# -------------------------------------------------------------------------- +# Entries a trigger's log_settings writes are listed and fire nothing +# -------------------------------------------------------------------------- + +class TestLogSettingsEntries(unittest.TestCase): + """log_settings entries are listed where GET covers them and fire no logs trigger.""" + + RPM_TOPIC = '/powertrain/engine/rpm' + ROOT_TOPIC = '/rpm' + + @classmethod + def _data_ready(cls, app, topic): + def ready(): + r = requests.get(f'{RUNTIME_URL}/apps/{app}/data', timeout=HTTP_TIMEOUT_SEC) + return r.status_code == 200 and any( + item.get('id') == topic for item in r.json().get('items', [])) + _poll(ready, what=f'{app} to expose {topic}') + + @classmethod + def setUpClass(cls): + _wait_for_entities(RUNTIME_URL, ['apps/rpm_sensor', 'apps/engine_temp']) + cls.host = _host_component(RUNTIME_URL) + cls._data_ready('rpm_sensor', cls.RPM_TOPIC) + cls._data_ready('engine_temp', cls.ROOT_TOPIC) + cls.probe = RosoutProbe() + # Class cleanups also run when setUpClass fails part-way. + cls.addClassCleanup(cls.probe.close) + cls.probe.wait_for_gateways(4) + cls.probe.warm_up([(RUNTIME_URL, 'apps/rpm_sensor')], 'powertrain.engine.rpm_sensor') + + cls.log_paths = ['apps/rpm_sensor', 'apps/engine_temp', f'components/{cls.host}'] + for path in cls.log_paths: + _configure_logs(RUNTIME_URL, path) + cls.logs = {p: EventStream(RUNTIME_URL, p).start() for p in cls.log_paths} + for stream in cls.logs.values(): + cls.addClassCleanup(stream.close) + + def data_trigger(app, topic, marker): + return EventStream(RUNTIME_URL, f'apps/{app}', body={ + 'resource': f'{API_BASE_PATH}/apps/{app}/data{topic}', + 'trigger_condition': {'condition_type': 'OnChange'}, + 'multishot': True, + 'lifetime': 600, + 'log_settings': {'severity': 'warning', 'marker': f'{RUN}:{marker}'}, + }).start() + + cls.data = [ + data_trigger('rpm_sensor', cls.RPM_TOPIC, 'ls-rpm'), + data_trigger('engine_temp', cls.ROOT_TOPIC, 'ls-root'), + ] + for stream in cls.data: + cls.addClassCleanup(stream.close) + + # @verifies REQ_INTEROP_097 + def test_log_settings_entries_are_listed_and_fire_nothing(self): + """Entry named engine_temp: listed by the root app and the host. rpm_sensor: by none.""" + for stream in self.data: + self.assertTrue(stream.wait_until(lambda s=stream: s.event_count() >= 3, + DELIVERY_TIMEOUT_SEC), + f'data trigger on {stream.path} did not fire') + self.probe.publish('ls-s-rpm', 'powertrain.engine.rpm_sensor') + self.probe.publish('ls-s-root', 'engine_temp') + + sentinels = { + 'apps/rpm_sensor': 'ls-s-rpm', + 'apps/engine_temp': 'ls-s-root', + f'components/{self.host}': 'ls-s-root', + } + for path, sentinel in sentinels.items(): + stream = self.logs[path] + arrived = stream.wait_until( + lambda s=stream, m=sentinel: m in s.markers(), DELIVERY_TIMEOUT_SEC) + received = stream.markers() + listed = _listed(RUNTIME_URL, path) + with self.subTest(entity=path): + self.assertTrue(arrived, f'stream not proven live: {sorted(received)}') + self.assertEqual(received.get('ls-root', []) + received.get('ls-rpm', []), [], + f'{path}: a log_settings entry fired a logs trigger') + if path != 'apps/rpm_sensor': + self.assertTrue(listed.get('ls-root'), + f'{path}: GET does not list the ls-root entries') + + self.assertNotIn('ls-rpm', _listed(RUNTIME_URL, 'apps/rpm_sensor')) + self.assertIn('ls-root', _listed(RUNTIME_URL, 'apps/engine_temp'), + 'the root app lists its own log_settings entries') + + +# -------------------------------------------------------------------------- +# severity_filter changes under a live trigger +# -------------------------------------------------------------------------- + +class TestSeverityFilterChange(unittest.TestCase): + """The trigger follows the entity's severity_filter as it is changed.""" + + PATH = 'apps/rpm_sensor' + LOGGER = 'powertrain.engine.rpm_sensor' + + @classmethod + def setUpClass(cls): + _wait_for_entities(RUNTIME_URL, [cls.PATH]) + cls.probe = RosoutProbe() + # Class cleanups also run when setUpClass fails part-way. + cls.addClassCleanup(cls.probe.close) + cls.probe.wait_for_gateways(4) + cls.probe.warm_up([(RUNTIME_URL, cls.PATH)], cls.LOGGER) + cls.stream = EventStream(RUNTIME_URL, cls.PATH).start() + cls.addClassCleanup(cls.stream.close) + + @classmethod + def tearDownClass(cls): + _configure_logs(RUNTIME_URL, cls.PATH, severity_filter='debug') + + def _round(self, severity_filter, probes, sentinel): + _configure_logs(RUNTIME_URL, self.PATH, severity_filter=severity_filter) + for case_id, level in probes: + self.probe.publish(case_id, self.LOGGER, level) + self.probe.publish(sentinel, self.LOGGER, _level(Log.FATAL)) + arrived = self.stream.wait_until( + lambda: sentinel in self.stream.markers(), DELIVERY_TIMEOUT_SEC) + self.assertTrue(arrived, f'{sentinel} not delivered') + + # @verifies REQ_INTEROP_097 + def test_filter_endpoints_and_change(self): + """Fatal admits only fatal; back to debug admits debug again.""" + self._round('fatal', [('sf-error', _level(Log.ERROR)), ('sf-fatal', _level(Log.FATAL))], + 'sf-s1') + self._round('debug', [('sf-debug', _level(Log.DEBUG))], 'sf-s2') + + received = set(self.stream.markers()) + self.assertEqual(received & {'sf-error', 'sf-fatal', 'sf-debug', 'sf-s1', 'sf-s2'}, + {'sf-fatal', 'sf-s1', 'sf-debug', 'sf-s2'}) + # GET applies the filter at read time, so it is checked under each one. + _configure_logs(RUNTIME_URL, self.PATH, severity_filter='fatal') + self.assertNotIn('sf-error', _listed(RUNTIME_URL, self.PATH)) + self.assertIn('sf-fatal', _listed(RUNTIME_URL, self.PATH)) + _configure_logs(RUNTIME_URL, self.PATH, severity_filter='debug') + self.assertIn('sf-debug', _listed(RUNTIME_URL, self.PATH)) + + # @verifies REQ_INTEROP_097 + def test_levels_without_a_severity_name(self): + """A level between the named ones fires exactly when GET lists it.""" + rounds = [ + ('info', [('lv-info-25', 25), ('lv-info-19', 19)], 'lv-s1'), + ('debug', [('lv-debug-0', 0), ('lv-debug-9', 9)], 'lv-s2'), + ('fatal', [('lv-fatal-99', 99), ('lv-fatal-49', 49)], 'lv-s3'), + ] + expected_listed = {'lv-info-25', 'lv-fatal-99'} + listed = set() + for severity_filter, probes, sentinel in rounds: + self._round(severity_filter, probes, sentinel) + round_ids = {case_id for case_id, _ in probes} + listed |= set(_listed(RUNTIME_URL, self.PATH)) & round_ids + + probe_ids = {case_id for _, probes, _ in rounds for case_id, _ in probes} + received = set(self.stream.markers()) & probe_ids + self.assertEqual(listed, expected_listed, 'GET under each filter') + self.assertEqual(received, listed, 'delivered != listed') + + +# -------------------------------------------------------------------------- +# Events no client has read +# -------------------------------------------------------------------------- + +class TestUnreadEventCap(unittest.TestCase): + """A trigger keeps the newest 100 events no client has read (capped gateway).""" + + PATH = 'apps/cap_probe' + # Its node logs only at startup, so every queued event is one of ours. + LOGGER = 'capped.cap_probe' + COUNT = 150 + CAP = 100 + + @classmethod + def setUpClass(cls): + _wait_for_entities(CAPPED_URL, [cls.PATH]) + cls.probe = RosoutProbe() + # Class cleanups also run when setUpClass fails part-way. + cls.addClassCleanup(cls.probe.close) + cls.probe.wait_for_gateways(4) + cls.probe.warm_up([(CAPPED_URL, cls.PATH)], cls.LOGGER) + + # @verifies REQ_INTEROP_097 + def test_newest_events_survive_past_the_cap(self): + """150 unread events leave the newest 100, oldest dropped, in order.""" + unread = EventStream(CAPPED_URL, self.PATH).create() + # Every change reaches both triggers in the same dispatch, so once this + # stream has the last entry the unread trigger has judged all of them. + barrier = EventStream(CAPPED_URL, self.PATH).start() + try: + names = [f'uc-{i:03d}' for i in range(self.COUNT)] + for i, name in enumerate(names): + self.probe.publish(name, self.LOGGER) + if i % 50 == 49: + time.sleep(0.2) + self.assertTrue(barrier.wait_until(lambda: names[-1] in barrier.markers(), + DELIVERY_TIMEOUT_SEC), + f'barrier stream got {len(barrier.markers())} markers') + self.assertEqual([m for m in barrier.ordered_markers() if m.startswith('uc-')], names, + 'every entry reached the gateway once, in order') + + unread.connect() + self.probe.publish('uc-sentinel', self.LOGGER) + self.assertTrue(unread.wait_until(lambda: 'uc-sentinel' in unread.markers(), + DELIVERY_TIMEOUT_SEC)) + got = unread.ordered_markers() + self.assertEqual(len(got), self.CAP + 1, f'{len(got)} events') + self.assertEqual(got, names[-self.CAP:] + ['uc-sentinel']) + finally: + barrier.close() + unread.close() + + +@launch_testing.post_shutdown_test() +class TestShutdown(unittest.TestCase): + + def test_exit_codes(self, proc_info): + """Check all processes exited cleanly (SIGTERM allowed).""" + for path in _MANIFEST_PATHS: + try: + os.unlink(path) + except OSError: + pass + for info in proc_info: + self.assertIn( + info.returncode, ALLOWED_EXIT_CODES, + f'{info.process_name} exited with code {info.returncode}' + ) diff --git a/src/ros2_medkit_plugins/ros2_medkit_graph_watchdog/README.md b/src/ros2_medkit_plugins/ros2_medkit_graph_watchdog/README.md index c6dc29f34..8f502c8e8 100644 --- a/src/ros2_medkit_plugins/ros2_medkit_graph_watchdog/README.md +++ b/src/ros2_medkit_plugins/ros2_medkit_graph_watchdog/README.md @@ -230,13 +230,14 @@ directly - the obvious choice - makes them reachable from no endpoint at all: it. There is no server-level `/faults/{code}` route either, so the flat `/faults` list was the only place these faults existed. -Hanging each fault on the FQN of the node it is ABOUT does not work either: fault -identity in the store is `fault_code` alone (`fault_code TEXT PRIMARY KEY`) while -`reporting_sources` accumulates into a set on that one record, and `fault_in_source_scope` -requires EVERY source to be in scope - so two dead nodes would hide -`GRAPH_NODE_DISAPPEARED` from both of their `/apps//faults` pages. One owned entity -per code keeps exactly one reporting source per record; the affected nodes are named in -the description. +Hanging each fault on the FQN of the node it is ABOUT is a different design, not an +impossible one: with the code configured `per_source`, a record is (`fault_code`, +reporting source), so raising `GRAPH_NODE_DISAPPEARED` under each dead node's FQN would +give each of them its own record on its own `/apps//faults` page. This detector aggregates by choice. A +graph-level condition is one condition however many nodes it names, the affected set +changes every tick, and per-node records would make the operator reconstruct the graph +view from a list raising and clearing under them. One owned entity per code keeps the +condition addressable as one thing, and the affected nodes are named in the description. ### Detectors @@ -311,9 +312,8 @@ means the subscriber does not constrain that policy, so it is always compatible. and depth are not RxO-compatibility dimensions and are deliberately not checked. **Aggregated fault, not per-topic**, via the shared `AggregatedFault` helper -(`aggregated_fault.hpp`): the fault_manager -identifies a fault by `fault_code` alone, so one `GRAPH_QOS_MISMATCH` per mismatched -topic would collide into a single record under the shared code. One graph-level fault +(`aggregated_fault.hpp`): a mismatch is a property of the graph, not of one topic, and +the mismatched set changes every tick. One graph-level fault enumerates every currently-mismatched topic; it clears (`EVENT_PASSED`) on every tick where nothing is mismatched, subject to the same `healing_enabled` requirement described in "Closing the loop" above to actually reach HEALED. @@ -428,8 +428,8 @@ plugins: ``` **Aggregated fault, not per-pair**, via the same `AggregatedFault` helper `qos_mismatch` -uses - the fault_manager identifies a fault by `fault_code` alone, so one `GRAPH_ORPHAN` -per near-miss pair would collide into a single record under the shared code. One +uses, and for the same reason: an orphaned pair is a property of the graph and the +orphaned set changes every tick. One graph-level fault enumerates every currently-orphaned pair, keyed by the canonical ` <-> ` string; it clears (`EVENT_PASSED`) on every tick where nothing is orphaned, subject to the same `healing_enabled` requirement @@ -486,8 +486,8 @@ whoever reads the fault. **Aggregated fault, not per-node.** All drift in the graph is one graph-level `GRAPH_PARAM_DRIFT`, not a fault per owning node, for the same reason as the other -detectors: the fault_manager identifies a fault by `fault_code` alone, so per-node faults -would collide into a single record. The description enumerates every currently-drifted +detectors: drift is a graph-level condition and the drifted set changes every tick. +The description enumerates every currently-drifted `(node, parameter)` pair, up to a cap of 480 characters - past that the text is truncated. Each app's own contribution is trimmed to 150 characters before that, so one node drifting on many parameters cannot fill the 480 by itself, and the entries are ordered `expect` violations first, @@ -1059,9 +1059,9 @@ the tick thread, the only thread that ever pumps these callbacks, so nothing it queued is delivered afterwards. **Aggregated fault, not per-node**, via the shared `AggregatedFault` helper - the same -rationale as every other detector here: the fault_manager identifies a fault by -`fault_code` alone, so one `GRAPH_NODE_INACTIVE` per stuck node would collide into a -single record under the shared code. Three graph-level faults, each enumerating every +rationale as every other detector here: a stuck node is reported as part of a +graph-level condition whose affected set changes every tick. +Three graph-level faults, each enumerating every currently affected node for ITS OWN code, each description capped independently at 480 characters with a truncation marker. Like `orphan` and `param_drift`, all three are fixed-severity `AggregatedFault` members at class scope (`aggregated_inactive_`, @@ -1081,7 +1081,7 @@ independently.** The description used to list affected nodes in fqn order interesting but not once the cap is full: a fleet sharing `require_active: ["controller_server"]` across a dozen robots fills the 480-char cap from the alphabetically-earliest ones, and a THIRTEENTH robot going inactive afterward would -be silently invisible forever - one shared `fault_code`, one record, no way to tell the +be silently invisible forever - one aggregated fault, one description, no way to tell the operator which of thirteen actually broke. The tracker reports which fqns entered EACH fault's content on THIS tick - `newly_affected`, `newly_unreadable`, `newly_not_managed` - and each list orders only its OWN fault's `AggregatedFault::emit_ordered` call, since the @@ -1542,10 +1542,11 @@ SAME occurrence continuing, not a new one, so restarting the dead node fast enou before anyone acknowledges it will not move the count. The honest limits, read off this branch's fault-manager storage rather than assumed: the -per-fault rosbag store enforces `fault_code` as UNIQUE, so a fault can hold at most one -recording at a time - a later confirmation's capture replaces the earlier one on disk rather -than accumulating a history. The freeze frame is the same shape: one row per `fault_code`, -overwritten on every capture, so a fifth occurrence's captured values overwrite the first's. +rosbag store is unique on `(fault_code, owner, file_path)`, so one record keeps a history of +recordings and the cap governs how many. The freeze frame is one row per record +(`(fault_code, owner)`), overwritten on every capture, so a fifth occurrence's captured +values overwrite the first's. Both are per record, so another source reporting the same +`per_source` code keeps its own. What survives every occurrence by default is the count itself; a hash-chained record of every raise/clear/heal transition also exists, but only once the fault manager's own `audit_log.enabled` is turned on, which it is not by default. Recordings and the freeze @@ -1579,10 +1580,9 @@ operator, not by the passage of a restart. Re-seeding the tracker from the fault store at startup would change it, and today it cannot be done. `/fault_manager/list_faults` would tell this detector that a `GRAPH_NODE_DISAPPEARED` -record is outstanding and that it is among its reporting sources - but not WHICH nodes it -names, which is the only thing that would make a fresh instance's silence meaningful. The fault -manager keeps one record per `fault_code`, and this detector folds every dead key into that one -record's description. That description is this detector's own deterministic text, so for a +record is outstanding and that it owns it - but not WHICH nodes it names, which is the only +thing that would make a fresh instance's silence meaningful. This detector raises one +aggregated fault under one owner and folds every dead key into that record's description. That description is this detector's own deterministic text, so for a record that never hit `kMaxDescriptionChars` the key list could in principle be read back out of it - the detector does not, because past the cap the remainder is collapsed into a count and the names are gone for good, and a re-seeding rule that works only for small faults is worse than diff --git a/src/ros2_medkit_plugins/ros2_medkit_graph_watchdog/design/graph_watchdog.rst b/src/ros2_medkit_plugins/ros2_medkit_graph_watchdog/design/graph_watchdog.rst index d9a8a84c0..10b1da81c 100644 --- a/src/ros2_medkit_plugins/ros2_medkit_graph_watchdog/design/graph_watchdog.rst +++ b/src/ros2_medkit_plugins/ros2_medkit_graph_watchdog/design/graph_watchdog.rst @@ -13,7 +13,7 @@ below). Structure --------- - **Plugin shell** (``GraphWatchdogPlugin``): loads via the gateway plugin ABI - (v7). In ``set_context`` it casts the context with ``as_ros_plugin_context``, + (v8). In ``set_context`` it casts the context with ``as_ros_plugin_context``, creates one ``rclcpp::Client`` on the gateway node, and starts a dedicated tick thread. The tick is deliberately NOT a gateway wall timer: detectors do blocking parameter/service reads, and the gateway's small @@ -196,9 +196,9 @@ reported by a live endpoint, which always carries the resolved profile) never ra **Aggregation via the shared helper.** ``qos_mismatch`` was the first detector to use the ``AggregatedFault`` helper (``aggregated_fault.hpp``); ``orphan`` and ``param_drift`` go through the same one, so no detector reimplements the level-triggered raise/clear -pattern. The rationale: the fault_manager identifies a fault by -``fault_code`` alone, so one ``GRAPH_QOS_MISMATCH`` per mismatched topic would collide -into a single record under the shared code. The detector keeps one ``AggregatedFault`` +pattern. The rationale: a mismatch is a property of the graph, not of one topic, and +the mismatched set changes every tick, so a fault per topic would raise and clear +under the operator rather than describe a condition. The detector keeps one ``AggregatedFault`` instance for the whole graph and, each tick, hands it every currently-mismatched topic's description (keyed by topic name so a repeat mismatch on the same topic overwrites rather than duplicates); an empty map on a clean tick clears @@ -687,7 +687,7 @@ log wants to know WHICH of the two is happening. **Three independent faults, not one shared record.** ``GRAPH_NODE_INACTIVE``, ``GRAPH_NODE_UNREADABLE`` and ``GRAPH_NODE_NOT_MANAGED`` are each raised through the shared ``AggregatedFault`` helper every ``GRAPH_*`` detector uses (one graph-level -record per code, since the fault_manager identifies a fault by ``fault_code`` alone), +record per code, because the condition is graph-level and its affected set moves), but as three SEPARATE, fixed-severity class members - ``GRAPH_NODE_INACTIVE`` always ``SEVERITY_ERROR``, the other two always ``SEVERITY_WARN`` - the same shape ``orphan_detector`` and ``param_drift_detector`` use, rather than one record whose @@ -735,7 +735,7 @@ independently.** The description used to list affected nodes in fqn order interesting but not once the cap is full: a fleet sharing ``require_active: ["controller_server"]`` across a dozen robots fills the 480-char cap from the alphabetically-earliest ones, and a THIRTEENTH robot going inactive afterward -would be silently invisible forever - one shared ``fault_code``, one record, no way to +would be silently invisible forever - one aggregated fault, one description, no way to tell the operator which of thirteen actually broke. The tracker reports which fqns entered EACH fault's content on THIS tick - ``newly_affected``, ``newly_unreadable``, ``newly_not_managed`` - and each list orders only its OWN fault's diff --git a/src/ros2_medkit_plugins/ros2_medkit_graph_watchdog/include/ros2_medkit_graph_watchdog/aggregated_fault.hpp b/src/ros2_medkit_plugins/ros2_medkit_graph_watchdog/include/ros2_medkit_graph_watchdog/aggregated_fault.hpp index a1fc860fc..b3692d2ed 100644 --- a/src/ros2_medkit_plugins/ros2_medkit_graph_watchdog/include/ros2_medkit_graph_watchdog/aggregated_fault.hpp +++ b/src/ros2_medkit_plugins/ros2_medkit_graph_watchdog/include/ros2_medkit_graph_watchdog/aggregated_fault.hpp @@ -37,12 +37,15 @@ namespace ros2_medkit_graph_watchdog { /// `/faults/{code}` route either, so the flat `/faults` list was the only place these /// faults existed. /// -/// Hanging each fault on the FQN of the node it is ABOUT does not work either: -/// fault identity in the store is `fault_code` alone (`fault_code TEXT PRIMARY KEY`), -/// and `reporting_sources` accumulates into a set on that one record while -/// `fault_in_source_scope` requires EVERY source to be in scope. Two dead nodes would -/// therefore hide GRAPH_NODE_DISAPPEARED from both of their `/apps//faults` pages. -/// One owned entity per code keeps exactly one source per record. +/// Hanging each fault on the FQN of the node it is ABOUT is a different design, +/// not an impossible one: with the code configured per_source, a record is +/// (fault_code, reporting source), so raising GRAPH_NODE_DISAPPEARED under each +/// dead node's FQN would give each of them its own record on its own +/// `/apps//faults` page. This detector aggregates by +/// choice. A graph-level condition is one condition however many nodes it names, +/// the affected set changes every tick, and per-node records would make the +/// operator reconstruct the graph view from a list that raises and clears under +/// them. One owned entity per code keeps the condition addressable as one thing. /// /// Same shape as the ADS plugin's device entity, for the same reason. inline constexpr const char * kGraphWatchdogEntityId = "graph_watchdog"; diff --git a/src/ros2_medkit_plugins/ros2_medkit_graph_watchdog/test/e2e/test_node_death_e2e.test.py b/src/ros2_medkit_plugins/ros2_medkit_graph_watchdog/test/e2e/test_node_death_e2e.test.py index b09efd6ad..7489ae198 100644 --- a/src/ros2_medkit_plugins/ros2_medkit_graph_watchdog/test/e2e/test_node_death_e2e.test.py +++ b/src/ros2_medkit_plugins/ros2_medkit_graph_watchdog/test/e2e/test_node_death_e2e.test.py @@ -143,6 +143,10 @@ observed_enclaves, wait_observed, ) +from ros2_medkit_test_utils.service_round_trip import ( # noqa: E402 + answer_once_each, + call_until_answered, +) # No default on purpose - a default makes this file FAIL OPEN (see harness-consuming # siblings' identical rationale): if the CTest ENVIRONMENT property never reaches the @@ -593,9 +597,10 @@ def _get_lifecycle_label(client_node, service_name, timeout=10.0): try: if not client.wait_for_service(timeout_sec=timeout): return None - future = client.call_async(GetState.Request()) - rclpy.spin_until_future_complete(client_node, future, timeout_sec=timeout) - result = future.result() + try: + result = call_until_answered(client_node, client, GetState.Request(), timeout) + except AssertionError: + return None return None if result is None else result.current_state.label finally: client_node.destroy_client(client) @@ -631,10 +636,16 @@ def _poll_lifecycle_label(client_node, service_name, expected_label, timeout=30. def _call_change_state_once(client_node, service_name, transition_id, timeout=30.0): - """One ``ChangeState`` call against `service_name`. ``True`` on ``result.success``.""" + """ + One ``ChangeState`` call against `service_name`. ``True`` on ``result.success``. + + The new client first gets one answer to the default request, transition 0 (create), + which a running node refuses without changing state. + """ client = client_node.create_client(ChangeState, service_name) if not client.wait_for_service(timeout_sec=timeout): return False + answer_once_each(client_node, (client,), timeout) request = ChangeState.Request() request.transition.id = transition_id future = client.call_async(request) @@ -1334,15 +1345,12 @@ class TestNodeDeathRestartLoopOccurrences(unittest.TestCase): occurrence"), which is why this scenario acknowledges explicitly between cycles instead of waiting for an organic heal. - Deliberately does NOT assert the "more than one recording" half of this row. The - per-fault rosbag store enforces `fault_code` as UNIQUE (`sqlite_fault_storage.cpp`'s - `store_rosbag_file_locked`: "INSERT OR REPLACE INTO rosbag_files ... (fault_code is - UNIQUE)" - the SAME fault_code can hold at most one recording ROW, structurally, and a - re-confirm deletes the previous bag file from disk). No config key in the fault manager - lifts this cap today: recording more than one rosbag per fault_code needs the storage - schema itself to change. Asserting a recording count here would either assert something - trivially true for the wrong reason (no captures happen at all without a detector) or - something structurally impossible to ever pass - neither is written. + Deliberately does NOT assert the "more than one recording" half of this row. How many + recordings one record keeps is governed by the fault manager's own retention + configuration, not by anything this scenario drives, so asserting a count here would + assert the retention default rather than the detector. What this scenario is about is + the occurrence count, which is what the acknowledge-between-cycles sequence above + establishes. """ def test_repeated_kills_reach_matching_occurrence_count(self, target_node): diff --git a/src/ros2_medkit_plugins/ros2_medkit_graph_watchdog/test/test_node_death_integration.cpp b/src/ros2_medkit_plugins/ros2_medkit_graph_watchdog/test/test_node_death_integration.cpp index 81a875e5b..6208ec1ae 100644 --- a/src/ros2_medkit_plugins/ros2_medkit_graph_watchdog/test/test_node_death_integration.cpp +++ b/src/ros2_medkit_plugins/ros2_medkit_graph_watchdog/test/test_node_death_integration.cpp @@ -1193,9 +1193,9 @@ TEST_F(NodeDeathIntegrationTest, UngatedClearStaysSuppressedWhileAnUnrelatedNode // Why it is not simply fixed here: the guard exists because a fresh instance cannot tell "the // node came back" from "I never saw that node". Answering that needs the KEYS the outstanding // record names, and they are not reachable. /fault_manager/list_faults would tell this detector -// that a GRAPH_NODE_DISAPPEARED record is outstanding and that it is among its reporting -// sources, but not which nodes it names: fault_manager aggregates one record per fault_code, -// and this detector merges every dead key into that one record's description, capped at +// that a GRAPH_NODE_DISAPPEARED record is outstanding and that this plugin owns it, but not +// which nodes it names: this detector raises one aggregated fault under one owner and merges +// every dead key into that record's description, capped at // AggregatedFault::kMaxDescriptionChars with the remainder collapsed into a count. Acting on // the record's mere existence would clear it for a node that is still dead and simply never // re-observed, which is the ungated heal the sibling tests above forbid. diff --git a/src/ros2_medkit_plugins/ros2_medkit_opcua/README.md b/src/ros2_medkit_plugins/ros2_medkit_opcua/README.md index 09c033313..1d511b42b 100644 --- a/src/ros2_medkit_plugins/ros2_medkit_opcua/README.md +++ b/src/ros2_medkit_plugins/ros2_medkit_opcua/README.md @@ -192,7 +192,7 @@ A write-capable build restores everything above; nothing else differs. | GET | `/apps/{id}/x-plc-data` | All OPC-UA values for entity (with units, types, timestamps) | | GET | `/apps/{id}/x-plc-data/{name}` | Single data point value | | POST | `/apps/{id}/x-plc-operations/set_{name}` | Write value to PLC (`{"value": 75.0}`) - write-capable build only; not registered otherwise | -| GET | `/components/{id}/x-plc-status` | Connection state, poll stats, active alarms, and `write_capable` - the write surface of the plugin object itself | +| GET | `/components/{id}/x-plc-status` | Connection state, poll stats, active alarms, and `write_capable` - the write surface of the plugin object itself. Served for the plugin's own component only. Any other component answers 404 `resource-not-found` | ### Standard SOVD (provided by gateway) @@ -255,6 +255,14 @@ build, `true` only in one built with `-DMEDKIT_OPCUA_READ_ONLY=OFF`. An absent `x-plc-operations` capability alone does not say this, because a write-capable build whose node map marks nothing writable shows the same absence. +The status describes the plugin's own OPC UA session, so it is served only under +the component the plugin introspects (`openplc_runtime` above). When the plugin +re-derives its component id from the device, the id `/components` still lists +keeps answering until the next discovery refresh lists the new one. Any other +component in the gateway, for example one another plugin introspects, answers +404 `resource-not-found`, the same answer the data route gives an entity with no +mapped points. + ## Finding Node IDs on your PLC The plugin identifies PLC tags by OPC-UA node IDs in the canonical string @@ -411,13 +419,14 @@ nodes: # is not listed in `active_alarms` on `x-plc-status`. Out-of-range values are # clamped with a warning. # -# Fault codes must be globally unique across ALL fault sources - every `alarm` / -# `status_bits` / `fault_enum` entry AND every `event_alarms` entry (including -# each of its `mappings[].fault_code`) - regardless of which entity owns them. The fault manager keys and clears faults by -# fault_code alone, so a code reused on two sources (even on different entities, -# even one polled and one event-driven) would flap raise/clear or clear the other -# source's fault. The loader rejects the whole file at load with an actionable -# error naming both sources. +# Fault codes must be unique across ALL fault sources in this file - every +# `alarm` / `status_bits` / `fault_enum` entry AND every `event_alarms` entry +# (including each of its `mappings[].fault_code`) - regardless of which entity +# owns them. The reason is this plugin's own shared `FaultTransitionTracker`, +# which is keyed by fault_code alone: two sources emitting one code would +# alternately raise and clear it every cycle, before either report is sent. The +# loader rejects the whole file at load with an actionable error naming both +# sources. # Native OPC-UA AlarmConditionType events (issue #386). Subscribes to alarms # defined inside the PLC (Siemens Program_Alarm / ProDiag, Beckhoff TF6100, @@ -621,9 +630,9 @@ no node-map file required. SourceNode + EventType + Message. The SourceNode is folded into every tier so two distinct conditions sharing a ConditionName/SourceName but sitting on different sources (e.g. two identical FB instances each raising - "Overpressure") never collapse onto one code - the fault manager keys/clears - by code alone, so a collision would let one condition's clear wipe the - other's still-active fault. Message is NOT folded into the slug tiers, so a + "Overpressure") never collapse onto one code - the shared transition tracker + is keyed by code alone, so a collision would flap raise/clear between the two + conditions every cycle. Message is NOT folded into the slug tiers, so a condition whose Message differs between its active and inactive notifications still maps to one code. The hash tier additionally folds in the Message deliberately - a real Siemens S7-1500 multiplexes every `Program_Alarm` of @@ -1257,10 +1266,12 @@ GET /api/v1/apps/tank_process/faults ``` When the value returns below threshold, the plugin calls the fault manager's -`~/clear_fault` service (`/fault_manager/clear_fault`) for that fault code. That -is the same service an operator's -`DELETE /api/v1/apps/{app_id}/faults/{fault_code}` ends up calling, so a device -de-assert is a clear like any other. It drops the fault's value snapshots unless +`~/clear_fault` service (`/fault_manager/clear_fault`) for that fault code and +the entity that reported it, which together address one record. That is the same +service an operator's `DELETE /api/v1/apps/{app_id}/faults/{fault_code}` ends up +calling, so a device de-assert is a clear like any other. That REST route reaches the plugin +through `FaultProvider::clear_fault_record` with the owner the gateway resolved, so on a +component route the clear names the hosted app that reported the record. It drops the fault's value snapshots unless `snapshots.retain_on_clear` is set, and its rosbag recording unless `snapshots.rosbag.auto_cleanup` is off or `snapshots.rosbag.max_bags_per_fault` keeps a history. diff --git a/src/ros2_medkit_plugins/ros2_medkit_opcua/design/index.rst b/src/ros2_medkit_plugins/ros2_medkit_opcua/design/index.rst index d752f0f6e..a246d10b0 100644 --- a/src/ros2_medkit_plugins/ros2_medkit_opcua/design/index.rst +++ b/src/ros2_medkit_plugins/ros2_medkit_opcua/design/index.rst @@ -102,7 +102,13 @@ edge-triggered callbacks: - **Active -> fault reported** via ``/fault_manager/report_fault`` with the configured severity and message -- **Cleared -> fault cleared** via ``/fault_manager/clear_fault`` by fault code +- **Cleared -> fault cleared** via ``/fault_manager/clear_fault`` by fault code and the + entity the plugin reported it under, which together name one record + +A REST ``DELETE /{entity}/faults/{code}`` on an entity this plugin owns reaches +``FaultProvider::clear_fault_record`` with the owner the gateway resolved, and the clear +names that owner. For a component route it is one of the component's hosted apps, not the +component itself. The two-argument ``clear_fault`` falls back to the addressed entity. The plugin keeps per-fault state only long enough to detect edges; the fault manager owns persistence and fault lifecycle. diff --git a/src/ros2_medkit_plugins/ros2_medkit_opcua/include/ros2_medkit_opcua/opcua_plugin.hpp b/src/ros2_medkit_plugins/ros2_medkit_opcua/include/ros2_medkit_opcua/opcua_plugin.hpp index bd02bf9e0..990f11ea2 100644 --- a/src/ros2_medkit_plugins/ros2_medkit_opcua/include/ros2_medkit_opcua/opcua_plugin.hpp +++ b/src/ros2_medkit_plugins/ros2_medkit_opcua/include/ros2_medkit_opcua/opcua_plugin.hpp @@ -34,6 +34,7 @@ #include #include +#include #include #include #include @@ -209,6 +210,8 @@ class OpcuaPlugin : public ros2_medkit_gateway::GatewayPlugin, const std::string & fault_code) override; tl::expected clear_fault(const std::string & entity_id, const std::string & fault_code) override; + tl::expected + clear_fault_record(const std::string & entity_id, const std::string & fault_code, const std::string & owner) override; // Resolve the SOVD severity bucket for an event alarm. An explicit configured // override wins; with none configured the raw OPC-UA event Severity (1-1000) @@ -647,7 +650,10 @@ class OpcuaPlugin : public ros2_medkit_gateway::GatewayPlugin, // Report/clear fault via ROS 2 service (private helpers, not the FaultProvider overrides) void send_report_fault(const std::string & entity_id, const std::string & fault_code, const std::string & severity_str, const std::string & message); - // ``source_id`` is the entity id the fault is reported under. + // ``source_id`` is the reporting source that owns the record. On the polled + // and event paths it is the entity this plugin reported under. On the REST + // path it is the owner the gateway resolved, which for a component route is + // one of its hosted apps. A refusal in the reply is logged. // ``origin`` says why the clear is being sent, which decides both the wire // flag and how the pending buffer ranks it. See ClearOrigin. void send_clear_fault(const std::string & fault_code, const std::string & source_id, @@ -874,6 +880,12 @@ class OpcuaPlugin : public ros2_medkit_gateway::GatewayPlugin, // they hold pointers/refs into node_map_. mutable std::shared_mutex node_map_mutex_; + // Component ids the last two introspect() calls returned. The gateway lists a + // new id only after it merges that result, so the older id stays served until + // then. Guarded by exposed_components_mutex_. + mutable std::mutex exposed_components_mutex_; + std::array exposed_component_ids_; + // OpcuaClient::connection_generation the last auto_browse walk ran against (0 // = never walked). The poll thread re-walks when the live generation differs, // mirroring device_identity_generation_. Written on the set_context thread diff --git a/src/ros2_medkit_plugins/ros2_medkit_opcua/src/node_map.cpp b/src/ros2_medkit_plugins/ros2_medkit_opcua/src/node_map.cpp index bbdcb358a..42456cc28 100644 --- a/src/ros2_medkit_plugins/ros2_medkit_opcua/src/node_map.cpp +++ b/src/ros2_medkit_plugins/ros2_medkit_opcua/src/node_map.cpp @@ -907,8 +907,8 @@ bool NodeMap::load(const std::string & yaml_path) { auto_alarms_.severity_bands.warning_min = read_band("warning", auto_alarms_.severity_bands.warning_min); warn_unknown_keys(bands_node, "auto_alarms.severity_bands", {"critical", "error", "warning"}); } - if (!(auto_alarms_.severity_bands.critical_min >= auto_alarms_.severity_bands.error_min && - auto_alarms_.severity_bands.error_min >= auto_alarms_.severity_bands.warning_min)) { + if (auto_alarms_.severity_bands.critical_min < auto_alarms_.severity_bands.error_min || + auto_alarms_.severity_bands.error_min < auto_alarms_.severity_bands.warning_min) { RCLCPP_WARN(logger, "auto_alarms.severity_bands must satisfy critical >= error >= warning - " "resetting to the default bands (801/501/201)"); @@ -976,24 +976,24 @@ bool NodeMap::load(const std::string & yaml_path) { } } - // Global fault-code uniqueness validation. fault_manager keys and clears - // faults by ``fault_code`` ALONE (clear_fault(fault_code) / - // get_fault(fault_code)); the poller shares one ``FaultTransitionTracker`` - // that is likewise keyed by code alone. So a fault_code is a global - // identifier and must be unique across EVERY source that can emit it, - // regardless of entity_id: + // Global fault-code uniqueness validation. The reason is THIS PLUGIN's own + // ``FaultTransitionTracker``, which the poller shares across every node-map + // entry and which is keyed by ``fault_code`` alone: two entries emitting + // one code would alternately raise and clear it every cycle, because the + // tracker cannot tell their edges apart. So a fault_code must be unique + // across EVERY source this file declares, regardless of entity_id: // // * every polled detection fault (threshold + each status_bits bit + // each fault_enum code + the enum catch-all), and // * every native ``event_alarms`` subscription (issue #386). // - // Two sources sharing a code - even a polled code on entity_a and an - // event_alarms code on entity_b - would collide at fault_manager: one - // source's clear wipes the other's fault, or the two flap raise/clear - // every cycle. The (entity_id, fault_code) pair the earlier check keyed on - // is NOT sufficient, because the fault manager never sees entity_id in its - // key. Reject the whole file at load with an actionable error so the intent - // - one code, one source - is enforced before anything runs. + // The fault manager is not the only reason: this plugin reports and clears + // under the owning entity_id, so for a per_source code two entries sharing + // it are two records there and neither clear touches the other. The shared + // tracker breaks on a collision whatever the fault manager keeps, and it + // breaks before either report is sent. Reject the whole file + // at load with an actionable error so the intent - one code, one source - + // is enforced before anything runs. { namespace fd = ros2_medkit::fault_detection; struct EmittedFault { @@ -1006,9 +1006,9 @@ bool NodeMap::load(const std::string & yaml_path) { if (!inserted) { RCLCPP_ERROR(rclcpp::get_logger("opcua.node_map"), "fault_code '%s' is emitted by more than one source (%s on entity '%s' and %s on " - "entity '%s'); fault codes must be globally unique across all detection entries and " - "event_alarms because the shared fault manager is keyed by code alone (a collision " - "clears the other source's fault or flaps raise/clear) - rename one of them", + "entity '%s'). Fault codes must be unique across all detection entries and " + "event_alarms in this file because the shared fault-transition tracker is keyed by " + "code alone (a collision flaps raise/clear every cycle). Rename one of them.", code.c_str(), it->second.pipeline, it->second.entity_id.c_str(), pipeline, entity_id.c_str()); return false; } diff --git a/src/ros2_medkit_plugins/ros2_medkit_opcua/src/opcua_plugin.cpp b/src/ros2_medkit_plugins/ros2_medkit_opcua/src/opcua_plugin.cpp index 81fe57d2e..e187dd923 100644 --- a/src/ros2_medkit_plugins/ros2_medkit_opcua/src/opcua_plugin.cpp +++ b/src/ros2_medkit_plugins/ros2_medkit_opcua/src/opcua_plugin.cpp @@ -1060,6 +1060,11 @@ IntrospectionResult OpcuaPlugin::introspect(const IntrospectionInput & /*input*/ comp.identity = device_identity_; } + { + std::lock_guard lock(exposed_components_mutex_); + exposed_component_ids_[1] = std::move(exposed_component_ids_[0]); + exposed_component_ids_[0] = comp.id; + } result.new_entities.components.push_back(std::move(comp)); // Register x-plc-status on the PLC runtime component only, so non-PLC @@ -1288,6 +1293,22 @@ void OpcuaPlugin::handle_plc_status(const PluginRequest & req, PluginResponse & } std::shared_lock node_map_lock(node_map_mutex_); + // Everything below describes this plugin's own OPC UA session, so it is served + // for the components the plugin exposes: its node map's id and the ids its last + // two introspect() calls returned. A re-derived id keeps answering under the id + // /components still lists. Any other component gets the 404 the data route + // gives an entity with no mapped points. + bool own_component = component_id == node_map_.component_id(); + if (!own_component) { + std::lock_guard lock(exposed_components_mutex_); + own_component = std::find(exposed_component_ids_.begin(), exposed_component_ids_.end(), component_id) != + exposed_component_ids_.end(); + } + if (!own_component) { + res.send_error(404, ERR_RESOURCE_NOT_FOUND, "No PLC status for component: " + component_id); + return; + } + auto snap = poller_->snapshot(); nlohmann::json j; @@ -1529,7 +1550,7 @@ void OpcuaPlugin::on_event_alarm(const AlarmEventDelivery & delivery) { log_info("AlarmCondition HEALED (latched, awaiting ack/confirm): " + delivery.fault_code); break; case AlarmAction::ClearFault: - log_info("AlarmCondition CLEARED: " + delivery.fault_code); + log_info("AlarmCondition CLEARED: " + delivery.fault_code + " on " + delivery.entity_id); // The device itself reported the condition cleared (Part 9 lifecycle), so // this IS a resolution at the source and the correlation engine may act on // it. DeviceAlarm is also the default, spelled out here because this is @@ -1593,8 +1614,36 @@ void OpcuaPlugin::send_clear_fault(const std::string & fault_code, const std::st auto request = std::make_shared( make_clear_fault_request(fault_code, source_id, clear_skips_correlation(origin))); - send_or_buffer({PendingFaultDispatch::Kind::Clear, fault_code, clear_is_link_state(origin), [this, request]() { - fault_clients_->clear->async_send_request(request); + // The dispatch stays fire-and-forget for the caller, but a refusal (no such + // record for that owner) reaches the log. The reply callback can run while + // the plugin is destroyed, so it holds nothing of the plugin: it logs through + // a copy of the log sink and carries the source and the code by value. + auto warn = [sink = log_sink()](const std::string & msg) { + if (sink) { + sink(PluginLogLevel::kWarn, msg); + } + }; + using ClearFuture = rclcpp::Client::SharedFuture; + // async_send_request only accepts a callback whose parameter is exactly + // SharedFuture, by value, so the future cannot be taken by reference. + // NOLINTNEXTLINE(performance-unnecessary-value-param) + auto on_reply = [warn = std::move(warn), source_id, fault_code](ClearFuture future) { + try { + const auto & response = future.get(); + if (!response->success) { + warn("ClearFault refused for '" + fault_code + "' of source '" + source_id + "': " + response->message); + } + } catch (const std::exception & e) { + warn("ClearFault reply for '" + fault_code + "' of source '" + source_id + "' failed: " + e.what()); + } catch (...) { + warn("ClearFault reply for '" + fault_code + "' of source '" + source_id + "' failed"); + } + }; + + send_or_buffer({PendingFaultDispatch::Kind::Clear, fault_code, clear_is_link_state(origin), + [this, request, on_reply = std::move(on_reply)]() { + auto callback = on_reply; + fault_clients_->clear->async_send_request(request, std::move(callback)); }}); } @@ -2902,7 +2951,16 @@ tl::expected OpcuaPlugin::list_data( items.push_back(std::move(item)); } - return dto::DataListResult{nlohmann::json{{"items", std::move(items)}}}; + // Link state travels with the values, the same envelope the plugin's own data + // route serves. Without it a reader cannot tell a live value from the frozen + // last-known one the poller keeps across an outage: the fault-trigger engine + // holds a rule's state on a down link, and it can only do that if the content + // says the link is down. + nlohmann::json envelope{{"items", std::move(items)}}; + envelope["connected"] = snap.connected; + envelope["timestamp"] = std::chrono::system_clock::to_time_t(snap.timestamp); + + return dto::DataListResult{std::move(envelope)}; } tl::expected OpcuaPlugin::read_data(const std::string & entity_id, @@ -3291,11 +3349,10 @@ tl::expected OpcuaPlugin::list_fau item["severity"] = f.value("severity", 0); item["description"] = f.value("description", ""); item["status"] = f.value("status", ""); - // Fault records carry reporting_sources, not source_id. On this - // entity-scoped list prefer the requested entity when it is among the - // sources - reporting_sources[0] is ordering-dependent and can name a - // different co-reporting entity; fall back to the first source only - // when the entity itself never reported. + // A record carries its first reporting source as source_id, for a + // per_source record its owner. Only a record read without source_id falls + // back to reporting_sources: the requested entity when it is listed + // there, else the first entry. std::string source_id = f.value("source_id", ""); if (source_id.empty() && f.contains("reporting_sources") && f["reporting_sources"].is_array()) { for (const auto & src : f["reporting_sources"]) { @@ -3332,8 +3389,16 @@ tl::expected OpcuaPlugin::get_fa return tl::make_unexpected(FaultProviderErrorInfo{FaultProviderError::Internal, "plugin not initialized", 503}); } - // list_entity_faults returns a bare JSON array of fault objects scoped to - // this entity (see PluginContext contract). + // The entity match is the LIST, not a comparison here. list_entity_faults + // returns a bare JSON array already scoped to this entity (see PluginContext + // contract): the fault manager's records filtered by the entity's resolved + // source set, plus the peers' answers for this same entity. So an item of + // this code in that array is a record this entity owns, and another entity's + // record of the same code never reaches this loop. + // + // Comparing the item's source_id to entity_id instead would be wrong, not + // merely redundant: a COMPONENT owns the records its hosted apps reported, + // whose owner is the app id, and the comparison would 404 them. auto faults = ctx_->list_entity_faults(entity_id); if (faults.is_array()) { for (const auto & f : faults) { @@ -3343,12 +3408,20 @@ tl::expected OpcuaPlugin::get_fa } } - return tl::make_unexpected( - FaultProviderErrorInfo{FaultProviderError::FaultNotFound, "Fault not found: " + fault_code, 404}); + return tl::make_unexpected(FaultProviderErrorInfo{FaultProviderError::FaultNotFound, + "Fault not found: " + fault_code + " on entity " + entity_id, 404}); } tl::expected OpcuaPlugin::clear_fault(const std::string & entity_id, const std::string & fault_code) { + // The code-only form of the contract names no owner, so the addressed entity + // stands in, which is the source this plugin reports its own faults under. + return clear_fault_record(entity_id, fault_code, ""); +} + +tl::expected +OpcuaPlugin::clear_fault_record(const std::string & entity_id, const std::string & fault_code, + const std::string & owner) { if (!ctx_ || !fault_clients_) { return tl::make_unexpected(FaultProviderErrorInfo{FaultProviderError::Internal, "plugin not initialized", 503}); } @@ -3359,7 +3432,13 @@ tl::expected OpcuaPlugin::clear_f // apps in other entities, and a plugin-owned entity must not be the hole in // that rule: the request goes through this provider instead, so the same flag // has to be set here. - send_clear_fault(fault_code, entity_id, ClearOrigin::ScopedOperator); + // + // The gateway resolved which record this route addresses, so its owner is + // what the clear names. The entity in the URL is not it: a component owns the + // records its hosted apps reported. Only when the gateway resolved nothing (a + // record it does not hold) does the addressed entity stand in, which is what + // this plugin's own polled clears use. + send_clear_fault(fault_code, owner.empty() ? entity_id : owner, ClearOrigin::ScopedOperator); return dto::FaultClearResult{ nlohmann::json{{"status", "cleared"}, {"fault_code", fault_code}, {"entity_id", entity_id}}}; } diff --git a/src/ros2_medkit_plugins/ros2_medkit_opcua/src/opcua_poller.cpp b/src/ros2_medkit_plugins/ros2_medkit_opcua/src/opcua_poller.cpp index b8d74a275..f165f10b5 100644 --- a/src/ros2_medkit_plugins/ros2_medkit_opcua/src/opcua_poller.cpp +++ b/src/ros2_medkit_plugins/ros2_medkit_opcua/src/opcua_poller.cpp @@ -743,13 +743,14 @@ bool OpcuaPoller::should_clear_after_refresh(SovdAlarmStatus last_status, const } // True when ANOTHER tracked condition instance carries the same fault_code and -// is currently Confirmed. Siemens A&C mints a fresh ConditionId (GUID) for -// every activation of the same Program_Alarm, so a stale instance's heal / -// reconcile-clear races the new instance's raise; because fault_manager keys -// by fault_code alone, letting the stale clear through would wipe the fault -// the live instance just asserted (seen on a real 1505SP: CONFIRMED then -// HEALED 170 us apart, alarm ends invisible). Caller must hold -// conditions_mutex_. +// is currently Confirmed. A server may mint a fresh ConditionId (GUID) for +// every activation of the same alarm, so a stale instance's heal or +// reconcile-clear races the new instance's raise. Both instances report under +// the entity that owns the code (the node map keeps every code on one entity), +// so they address one fault record, and letting the stale clear through would +// clear the record the live instance just raised (seen on a real controller: +// CONFIRMED then HEALED 170 us apart, the alarm ending invisible). Caller must +// hold conditions_mutex_. bool OpcuaPoller::same_code_active_elsewhere_locked(const std::string & fault_code, const std::string & condition_id_str) const { for (const auto & [cid, runtime] : conditions_) { diff --git a/src/ros2_medkit_plugins/ros2_medkit_opcua/test/test_node_map.cpp b/src/ros2_medkit_plugins/ros2_medkit_opcua/test/test_node_map.cpp index 29f1a2313..de04913fa 100644 --- a/src/ros2_medkit_plugins/ros2_medkit_opcua/test/test_node_map.cpp +++ b/src/ros2_medkit_plugins/ros2_medkit_opcua/test/test_node_map.cpp @@ -999,9 +999,11 @@ component_id: test TEST_F(NodeMapTest, RejectsCodeAcrossPipelinesDifferentEntities) { // Global-by-code uniqueness (issue #481): a polled detection code on // entity_a and an event_alarms code on entity_b share the SAME fault_code. - // fault_manager keys and clears by code alone, so the two collide even - // though the entities differ; the (entity_id, fault_code) pair the earlier - // guard keyed on let this through. The loader must reject the whole file. + // The poller's shared fault-transition tracker keys by code alone, so the + // two collide there even though the entities differ (the fault manager + // keeps them as two records only for a per_source code). The + // (entity_id, fault_code) pair the earlier guard keyed on let this through. + // The loader must reject the whole file. std::string path = "/tmp/test_node_map_cross_entity_pipeline.yaml"; std::ofstream f(path); f << R"( diff --git a/src/ros2_medkit_plugins/ros2_medkit_opcua/test/test_opcua_plugin.cpp b/src/ros2_medkit_plugins/ros2_medkit_opcua/test/test_opcua_plugin.cpp index 90949fa5d..6a857cf78 100644 --- a/src/ros2_medkit_plugins/ros2_medkit_opcua/test/test_opcua_plugin.cpp +++ b/src/ros2_medkit_plugins/ros2_medkit_opcua/test/test_opcua_plugin.cpp @@ -33,6 +33,7 @@ #include #include #include +#include #include #include #include @@ -44,6 +45,7 @@ #include #include #include +#include #include #include @@ -57,9 +59,24 @@ namespace ros2_medkit_gateway { +// What the route tests hand the stubs below. The provider tests never build a +// request or a response, so a null impl answers empty and records nothing. +struct StubRequest { + std::vector path_params; // index 0 is the full match +}; +struct StubResponse { + int status = 0; + std::string error_code; + nlohmann::json body; +}; + PluginRequest::PluginRequest(const void * impl) : impl_(impl) { } -std::string PluginRequest::path_param(size_t /*index*/) const { +std::string PluginRequest::path_param(size_t index) const { + const auto * req = static_cast(impl_); + if (req != nullptr && index < req->path_params.size()) { + return req->path_params[index]; + } return {}; } std::string PluginRequest::header(const std::string & /*name*/) const { @@ -79,10 +96,21 @@ std::string PluginRequest::query_param(const std::string & /*name*/) const { PluginResponse::PluginResponse(void * impl) : impl_(impl) { } -void PluginResponse::send_json(const nlohmann::json & /*data*/) { +void PluginResponse::send_json(const nlohmann::json & data) { + auto * res = static_cast(impl_); + if (res != nullptr) { + res->status = 200; + res->body = data; + } } -void PluginResponse::send_error(int /*status*/, const std::string & /*error_code*/, const std::string & /*message*/, +void PluginResponse::send_error(int status, const std::string & error_code, const std::string & message, const nlohmann::json & /*parameters*/) { + auto * res = static_cast(impl_); + if (res != nullptr) { + res->status = status; + res->error_code = error_code; + res->body = {{"message", message}}; + } } // -- FakePluginContext -- @@ -230,6 +258,22 @@ component_id: test_runtime // -- DataProvider tests -- +// The provider's list envelope carries link state, the same shape the plugin's +// own data route serves. Without it a reader cannot tell a live value from the +// frozen last-known one the poller keeps across an outage, and the fault-trigger +// engine's link-down guard (content_reports_disconnected) never fires for a +// provider-served entity: a threshold rule then evaluates on a stale number for +// the whole outage. The fixture endpoint never connects, so the poller reports +// disconnected here. +TEST_F(OpcuaPluginTest, ListDataEnvelopeCarriesLinkState) { + auto result = plugin_.list_data("tank"); + + ASSERT_TRUE(result.has_value()); + ASSERT_TRUE(result->content.contains("connected")) << "the list envelope must report link state"; + EXPECT_FALSE(result->content["connected"].get()) << "the fixture endpoint never connects"; + ASSERT_TRUE(result->content.contains("timestamp")) << "the list envelope must date its values"; +} + TEST_F(OpcuaPluginTest, ListDataReturnsItems) { auto result = plugin_.list_data("tank"); ASSERT_TRUE(result.has_value()); @@ -634,6 +678,55 @@ TEST_F(OpcuaPluginAlarmsFitnessTest, IntrospectSkipsXPlcDataForAlarmsFallbackEnt EXPECT_TRUE(tank_got_x_plc_data) << "data-bearing entities must keep x-plc-data"; } +// -- Route tests -- + +namespace { + +// Calls the plugin's GET route under components/ for ``component_id`` and +// returns what the handler sent. +StubResponse get_component_status(OpcuaPlugin & plugin, const std::string & component_id) { + StubResponse recorded; + const auto routes = plugin.get_routes(); + const auto route = std::find_if(routes.begin(), routes.end(), [](const GatewayPlugin::PluginRoute & r) { + return r.method == "GET" && r.pattern.rfind("components/", 0) == 0; + }); + if (route == routes.end()) { + ADD_FAILURE() << "the plugin registers no GET route under components/"; + return recorded; + } + // The handler reads capture group 1 only. + StubRequest request{{"", component_id}}; + PluginRequest req(&request); + PluginResponse res(&recorded); + route->handler(req, res); + return recorded; +} + +} // namespace + +// The status route describes this plugin's own OPC UA session: its endpoint, +// its mode and whether it is connected. In a gateway that loads several +// plugins, other components share the entity tree, and that session says +// nothing about them. So the route serves the plugin's own component and +// answers any other one with the 404 the data route gives an entity with no +// mapped points. The own component is asked on the same plugin first, which +// shows that this harness does record a served status. +TEST_F(OpcuaPluginTest, StatusRouteServesOnlyThePluginsOwnComponent) { + ctx_.entities["other_device"] = {SovdEntityType::COMPONENT, "other_device", "/other_area", + "/other_area/other_device"}; + + const StubResponse own = get_component_status(plugin_, "test_runtime"); + ASSERT_EQ(own.status, 200) << own.body.dump(); + EXPECT_EQ(own.body.value("component_id", ""), "test_runtime"); + ASSERT_TRUE(own.body.contains("connected")) << "the plugin's own component must report link state"; + EXPECT_FALSE(own.body["connected"].get()) << "the fixture endpoint never connects"; + + const StubResponse other = get_component_status(plugin_, "other_device"); + EXPECT_EQ(other.status, 404) << "a component the plugin does not own was answered with " << other.body.dump(); + EXPECT_EQ(other.error_code, ERR_RESOURCE_NOT_FOUND); + EXPECT_FALSE(other.body.contains("endpoint_url")) << "the plugin's session must not be reported under another id"; +} + // -- FaultProvider tests -- TEST_F(OpcuaPluginTest, ListFaultsEmpty) { @@ -705,6 +798,27 @@ TEST_F(OpcuaPluginTest, GetFaultFound) { EXPECT_EQ(result->content.value("fault_code", ""), "PLC_LOW_LEVEL"); } +// A fault code is half of a record's identity, so a detail read on entity A must +// not serve entity B's record of the same code. What keeps them apart is the +// entity-scoped list this scan runs over, not a comparison inside the scan, and +// that is what this pins: two owners of one code, and only the addressed +// entity's record comes back. +TEST_F(OpcuaPluginTest, GetFaultServesOnlyTheAddressedEntitysRecordOfASharedCode) { + ctx_.all_faults = {{"faults", + {{{"fault_code", "SHARED_CODE"}, {"source_id", "other_tank"}, {"severity", 2}}, + {{"fault_code", "SHARED_CODE"}, {"source_id", "tank"}, {"severity", 3}}}}}; + + auto result = plugin_.get_fault("tank", "SHARED_CODE"); + + ASSERT_TRUE(result.has_value()); + EXPECT_EQ(result->content.value("source_id", ""), "tank") << "served another owner's record of the same code"; + EXPECT_EQ(result->content.value("severity", 0), 3); + + auto other = plugin_.get_fault("other_tank", "SHARED_CODE"); + ASSERT_TRUE(other.has_value()); + EXPECT_EQ(other->content.value("source_id", ""), "other_tank"); +} + // -- configure() validation (issue #481) -- TEST(OpcuaPluginConfigureTest, ThrowsOnInvalidNodeMap) { @@ -2500,8 +2614,369 @@ struct ScopedExecutorSpin { ScopedExecutorSpin & operator=(ScopedExecutorSpin &&) = delete; }; -// The SOVD DELETE /faults/{code} route lands on FaultProvider::clear_fault(), -// which buffers a dispatch into pending_reports_ - the SAME vector the poll +// The plugin raises under the entity it polled, so its clear has to name the +// same owner: a record is (fault_code, source_id), and a clear carrying no +// source reaches whichever record the store resolves - on a box running a second +// plugin instance against another device, that is the other device's still-active +// fault. +TEST(OpcuaPluginFaultIdentity, ClearFaultSendsTheOwningEntityAsSourceId) { + ScopedRclcpp rclcpp_scope; + auto node = std::make_shared("opcua_clear_owner_plugin"); + auto fault_manager = std::make_shared("opcua_clear_owner_faultmgr"); + + std::mutex seen_mutex; + std::vector> cleared; // (fault_code, source_id) + auto report_srv = fault_manager->create_service( + "/fault_manager/report_fault", [](const std::shared_ptr &, + const std::shared_ptr & res) { + res->accepted = true; + }); + auto clear_srv = fault_manager->create_service( + "/fault_manager/clear_fault", + [&cleared, &seen_mutex](const std::shared_ptr & req, + const std::shared_ptr & res) { + { + std::lock_guard lock(seen_mutex); + cleared.emplace_back(req->fault_code, req->source_id); + } + res->success = true; + }); + + const std::string yaml_path = "/tmp/test_opcua_clear_owner_nodemap.yaml"; + { + std::ofstream f(yaml_path); + f << R"( +area_id: owner_plc +component_id: owner_runtime +nodes: + - node_id: "ns=2;i=1" + entity_id: tank + data_name: level + data_type: float +)"; + } + + OpcuaPlugin plugin; + nlohmann::json config; + config["node_map_path"] = yaml_path; + config["endpoint_url"] = "opc.tcp://127.0.0.1:1"; // nothing listening, the fault sink drives the drain + config["poll_interval_ms"] = 100; + plugin.configure(config); + + RealNodePluginContext ctx(node.get()); + ctx.entities["tank"] = {SovdEntityType::APP, "tank", "/owner_plc", "/owner_plc/owner_runtime/tank"}; + plugin.set_context(ctx); + + ScopedExecutorSpin spinner({node, fault_manager}); + + auto probe = node->create_client("/fault_manager/report_fault"); + const auto ready_deadline = std::chrono::steady_clock::now() + std::chrono::seconds(10); + while (!probe->service_is_ready() && std::chrono::steady_clock::now() < ready_deadline) { + std::this_thread::sleep_for(std::chrono::milliseconds(20)); + } + ASSERT_TRUE(probe->service_is_ready()) << "stub ReportFault server never became discoverable"; + + // The addressed entity is the COMPONENT, the resolved owner is the hosted app. + // A record is (fault_code, owner), so the owner is what has to travel: sending + // the component id addresses a record no source owns and the fault manager + // declines it. + // Through the FaultProvider interface, which is how the gateway reaches it: a + // plugin that did not override clear_fault_record would land on the default, + // which clears by code with the addressed entity and drops the owner. + ros2_medkit_gateway::FaultProvider & provider = plugin; + static_cast(provider.clear_fault_record("owner_runtime", "SHARED_CODE", "tank")); + + const auto deadline = std::chrono::steady_clock::now() + std::chrono::seconds(10); + while (std::chrono::steady_clock::now() < deadline) { + { + std::lock_guard lock(seen_mutex); + if (!cleared.empty()) { + break; + } + } + static_cast(provider.clear_fault_record("owner_runtime", "SHARED_CODE", "tank")); + std::this_thread::sleep_for(std::chrono::milliseconds(50)); + } + + spinner.stop(); + plugin.shutdown(); + std::remove(yaml_path.c_str()); + + std::lock_guard lock(seen_mutex); + ASSERT_FALSE(cleared.empty()) << "no ClearFault reached the stub fault manager"; + EXPECT_EQ(cleared.front().first, "SHARED_CODE"); + EXPECT_EQ(cleared.front().second, "tank") + << "the clear must name the record's owner the gateway resolved, not the addressed entity"; +} + +// The gateway wires every plugin's log sink through PluginManager, which this +// suite does not link. This subclass wires one the test owns instead. +class SinkedOpcuaPlugin : public OpcuaPlugin { + public: + using GatewayPlugin::set_logger; +}; + +// Records the warnings a plugin log sink receives. The state is shared with the +// sink it hands out, so a sink copied into a callback stays valid on its own. +class WarningLog { + public: + std::function sink() const { + return [state = state_](PluginLogLevel level, const std::string & msg) { + if (level == PluginLogLevel::kWarn) { + std::lock_guard lock(state->mutex); + state->warnings.push_back(msg); + } + }; + } + + /// The first warning containing `needle`, or "" when none arrives in time. + std::string wait_for(const std::string & needle, std::chrono::milliseconds timeout) const { + const auto deadline = std::chrono::steady_clock::now() + timeout; + do { + if (auto found = find(needle); !found.empty()) { + return found; + } + std::this_thread::sleep_for(std::chrono::milliseconds(20)); + } while (std::chrono::steady_clock::now() < deadline); + return find(needle); + } + + std::string find(const std::string & needle) const { + std::lock_guard lock(state_->mutex); + for (const auto & warning : state_->warnings) { + if (warning.find(needle) != std::string::npos) { + return warning; + } + } + return ""; + } + + private: + struct State { + std::mutex mutex; + std::vector warnings; + }; + std::shared_ptr state_ = std::make_shared(); +}; + +// A clear the fault manager declined used to leave no trace: the dispatch never +// read its reply, so the REST route answered as though it had worked while the +// record stayed CONFIRMED. The reply is read now, and a refusal is logged as a +// warning that names the code and the owner the clear was addressed to. +TEST(OpcuaPluginFaultIdentity, ARefusedClearIsReadAndNotSilent) { + ScopedRclcpp rclcpp_scope; + auto node = std::make_shared("opcua_refused_clear_plugin"); + auto fault_manager = std::make_shared("opcua_refused_clear_faultmgr"); + + auto report_srv = fault_manager->create_service( + "/fault_manager/report_fault", [](const std::shared_ptr &, + const std::shared_ptr & res) { + res->accepted = true; + }); + auto clear_srv = fault_manager->create_service( + "/fault_manager/clear_fault", [](const std::shared_ptr & req, + const std::shared_ptr & res) { + res->success = false; + res->message = "Fault not found: " + req->fault_code + " (source " + req->source_id + ")"; + }); + + const std::string yaml_path = "/tmp/test_opcua_refused_clear_nodemap.yaml"; + { + std::ofstream f(yaml_path); + f << R"( +area_id: refused_plc +component_id: refused_runtime +nodes: + - node_id: "ns=2;i=1" + entity_id: tank + data_name: level + data_type: float +)"; + } + + WarningLog log; + SinkedOpcuaPlugin plugin; + plugin.set_logger(log.sink()); // before any plugin thread starts + nlohmann::json config; + config["node_map_path"] = yaml_path; + config["endpoint_url"] = "opc.tcp://127.0.0.1:1"; + config["poll_interval_ms"] = 100; + plugin.configure(config); + + RealNodePluginContext ctx(node.get()); + ctx.entities["tank"] = {SovdEntityType::APP, "tank", "/refused_plc", "/refused_plc/refused_runtime/tank"}; + plugin.set_context(ctx); + + ScopedExecutorSpin spinner({node, fault_manager}); + + auto probe = node->create_client("/fault_manager/report_fault"); + const auto ready_deadline = std::chrono::steady_clock::now() + std::chrono::seconds(10); + while (!probe->service_is_ready() && std::chrono::steady_clock::now() < ready_deadline) { + std::this_thread::sleep_for(std::chrono::milliseconds(20)); + } + ASSERT_TRUE(probe->service_is_ready()) << "stub ReportFault server never became discoverable"; + + // Addressed to the component, owned by the app: the warning has to name the + // owner the clear carried, which is not the entity in the request. + std::string warning; + const auto deadline = std::chrono::steady_clock::now() + std::chrono::seconds(10); + while (warning.empty() && std::chrono::steady_clock::now() < deadline) { + static_cast(plugin.clear_fault_record("refused_runtime", "SHARED_CODE", "tank")); + warning = log.wait_for("ClearFault refused", std::chrono::milliseconds(200)); + } + + spinner.stop(); + plugin.shutdown(); + std::remove(yaml_path.c_str()); + + ASSERT_FALSE(warning.empty()) << "a refused clear left no warning in the plugin's log"; + EXPECT_NE(warning.find("'SHARED_CODE'"), std::string::npos) << warning; + EXPECT_NE(warning.find("source 'tank'"), std::string::npos) << "the warning must name the owner: " << warning; +} + +// A ClearFault reply that comes back after the plugin is gone must not reach +// the plugin. shutdown() joins the plugin's fault-client executor and drops the +// clients with their pending requests, so a reply in flight at shutdown is +// dropped. The test holds one clear in flight, shuts the plugin down, destroys +// it, then has the stub answer: the sink the plugin had receives nothing. Two +// controls: a refusal answered while the plugin lives reaches that sink, and a +// test-owned request answered after the dropped one gets its reply, so the stub +// did send it. +TEST(OpcuaPluginFaultIdentity, AClearReplyNeverReadsThePluginAgain) { + ScopedRclcpp rclcpp_scope; + auto node = std::make_shared("opcua_reply_lifetime_plugin"); + auto fault_manager = std::make_shared("opcua_reply_lifetime_faultmgr"); + + using ClearFault = ros2_medkit_msgs::srv::ClearFault; + struct Pending { + std::shared_ptr header; + std::string fault_code; + }; + std::mutex pending_mutex; + std::vector pending; + + auto report_srv = fault_manager->create_service( + "/fault_manager/report_fault", [](const std::shared_ptr &, + const std::shared_ptr & res) { + res->accepted = true; + }); + // Deferred: the request is held and answered by the test. + auto clear_srv = fault_manager->create_service( + "/fault_manager/clear_fault", + [&pending, &pending_mutex](const std::shared_ptr> & /*service*/, + const std::shared_ptr & header, + const std::shared_ptr & req) { + std::lock_guard lock(pending_mutex); + pending.push_back(Pending{header, req->fault_code}); + }); + // A refusal for the held request of `fault_code`. The requester may be gone: + // the response is sent all the same. + auto answer = [&](const std::string & fault_code) { + std::lock_guard lock(pending_mutex); + for (auto it = pending.begin(); it != pending.end(); ++it) { + if (it->fault_code == fault_code) { + ClearFault::Response response; + response.success = false; + response.message = "Fault not found: " + fault_code; + try { + clear_srv->send_response(*it->header, response); + } catch (const std::exception &) { + // A requester that is gone can make the send fail. That is a dropped reply too. + } + pending.erase(it); + return true; + } + } + return false; + }; + auto wait_received = [&](const std::string & fault_code) { + const auto deadline = std::chrono::steady_clock::now() + std::chrono::seconds(10); + while (std::chrono::steady_clock::now() < deadline) { + { + std::lock_guard lock(pending_mutex); + for (const auto & p : pending) { + if (p.fault_code == fault_code) { + return true; + } + } + } + std::this_thread::sleep_for(std::chrono::milliseconds(20)); + } + return false; + }; + // The control requester, on the test's executor. Created before it spins. + auto control = node->create_client("/fault_manager/clear_fault"); + + const std::string yaml_path = "/tmp/test_opcua_reply_lifetime_nodemap.yaml"; + { + std::ofstream f(yaml_path); + f << R"( +area_id: lifetime_plc +component_id: lifetime_runtime +nodes: + - node_id: "ns=2;i=1" + entity_id: tank + data_name: level + data_type: float +)"; + } + + WarningLog log; + auto plugin = std::make_unique(); + plugin->set_logger(log.sink()); // before any plugin thread starts + nlohmann::json config; + config["node_map_path"] = yaml_path; + config["endpoint_url"] = "opc.tcp://127.0.0.1:1"; + config["poll_interval_ms"] = 100; + plugin->configure(config); + + RealNodePluginContext ctx(node.get()); + ctx.entities["tank"] = {SovdEntityType::APP, "tank", "/lifetime_plc", "/lifetime_plc/lifetime_runtime/tank"}; + plugin->set_context(ctx); + + ScopedExecutorSpin spinner({node, fault_manager}); + + auto probe = node->create_client("/fault_manager/report_fault"); + const auto ready_deadline = std::chrono::steady_clock::now() + std::chrono::seconds(10); + while (!probe->service_is_ready() && std::chrono::steady_clock::now() < ready_deadline) { + std::this_thread::sleep_for(std::chrono::milliseconds(20)); + } + ASSERT_TRUE(probe->service_is_ready()) << "stub ReportFault server never became discoverable"; + + // Control: while the plugin lives, a refusal reaches its sink. + static_cast(plugin->clear_fault_record("tank", "LIVE_CODE", "tank")); + ASSERT_TRUE(wait_received("LIVE_CODE")) << "the live clear never reached the stub"; + ASSERT_TRUE(answer("LIVE_CODE")); + ASSERT_FALSE(log.wait_for("'LIVE_CODE'", std::chrono::seconds(10)).empty()) + << "positive control: a refusal answered while the plugin lives never reached its sink"; + + // The clear under test is in flight when the plugin goes. + static_cast(plugin->clear_fault_record("tank", "LATE_CODE", "tank")); + ASSERT_TRUE(wait_received("LATE_CODE")) << "the late clear never reached the stub"; + plugin->shutdown(); + plugin.reset(); + + // Control: a request answered after LATE_CODE's reply gets its own, so the stub + // sent LATE_CODE's reply before this one. + auto control_request = std::make_shared(); + control_request->fault_code = "CONTROL_CODE"; + control_request->source_id = "tank"; + auto control_reply = control->async_send_request(control_request); + ASSERT_TRUE(wait_received("CONTROL_CODE")) << "the control request never reached the stub"; + ASSERT_TRUE(answer("LATE_CODE")); + ASSERT_TRUE(answer("CONTROL_CODE")); + ASSERT_EQ(control_reply.wait_for(std::chrono::seconds(10)), std::future_status::ready) + << "the stub's reply to the control request never arrived"; + + spinner.stop(); + std::remove(yaml_path.c_str()); + + EXPECT_TRUE(log.find("'LATE_CODE'").empty()) + << "a reply that arrived after the plugin was destroyed reached the plugin's sink"; +} + +// The SOVD DELETE /faults/{code} route lands on FaultProvider::clear_fault_record(), +// which buffers a dispatch into pending_reports_, the SAME vector the poll // thread drains in publish_values()/flush_pending_reports(). Before the fix the // buffer had no lock, so a push_back that reallocated the vector while another // thread iterated/swapped it corrupted the heap (the crash observed in @@ -2511,10 +2986,10 @@ struct ScopedExecutorSpin { // in send_or_buffer() and the batch.swap(pending_reports_) drain in // flush_pending_reports(). The drain only runs once report->service_is_ready() // is true, so the test stands up a real ReportFault/ClearFault server (a stub -// fault_manager) to open that gate; only then does every clear_fault() both push +// fault_manager) to open that gate. Only then does every clear both push // into the buffer AND swap it out under concurrent pushes from the other worker // threads. It must complete without heap corruption and is clean under -// ThreadSanitizer - the DDS/rclcpp machinery it drives is covered by +// ThreadSanitizer. The DDS/rclcpp machinery it drives is covered by // tsan_suppressions.txt, the same paths the gateway service tests exercise. TEST(OpcuaPluginConcurrency, ClearFaultBufferIsThreadSafe) { ScopedRclcpp rclcpp_scope; @@ -2596,7 +3071,8 @@ component_id: race_runtime // push_back against swap on the shared buffer. // The result is deliberately dropped: this test races the shared // pending-report buffer, it does not assert on any single clear. - static_cast(plugin.clear_fault("tank", "RACE_" + std::to_string(t) + "_" + std::to_string(i++ & 0x3f))); + static_cast( + plugin.clear_fault_record("tank", "RACE_" + std::to_string(t) + "_" + std::to_string(i++ & 0x3f), "tank")); } }); } diff --git a/src/ros2_medkit_plugins/ros2_medkit_sovd_service_interface/README.md b/src/ros2_medkit_plugins/ros2_medkit_sovd_service_interface/README.md index c08ade622..d6b5099a8 100644 --- a/src/ros2_medkit_plugins/ros2_medkit_sovd_service_interface/README.md +++ b/src/ros2_medkit_plugins/ros2_medkit_sovd_service_interface/README.md @@ -13,6 +13,8 @@ Gateway plugin exposing medkit entity data via ROS 2 services. Enables ROS 2 nod Service prefix is configurable via `plugins.sovd_service_interface.service_prefix` parameter (default: `/medkit`). +`list_entity_faults` answers one `Fault` per fault record in the entity's scope. Each carries the record's `owner` as the fault manager sets it: the reporting source of a `per_source` record, `*` for a shared one. Pass it as `source_id` to the fault manager's single-record services. + ## Usage Load as a gateway plugin: diff --git a/src/ros2_medkit_plugins/ros2_medkit_sovd_service_interface/src/sovd_service_interface.cpp b/src/ros2_medkit_plugins/ros2_medkit_sovd_service_interface/src/sovd_service_interface.cpp index e42704158..bb38bb649 100644 --- a/src/ros2_medkit_plugins/ros2_medkit_sovd_service_interface/src/sovd_service_interface.cpp +++ b/src/ros2_medkit_plugins/ros2_medkit_sovd_service_interface/src/sovd_service_interface.cpp @@ -65,29 +65,29 @@ void SovdServiceInterface::set_context(PluginContext & context) { list_entities_srv_ = node->create_service( service_prefix_ + "/list_entities", - [this](const std::shared_ptr req, - std::shared_ptr res) { + [this](const std::shared_ptr & req, + const std::shared_ptr & res) { handle_list_entities(req, res); }); list_faults_srv_ = node->create_service( service_prefix_ + "/list_entity_faults", - [this](const std::shared_ptr req, - std::shared_ptr res) { + [this](const std::shared_ptr & req, + const std::shared_ptr & res) { handle_list_entity_faults(req, res); }); get_data_srv_ = node->create_service( service_prefix_ + "/get_entity_data", - [this](const std::shared_ptr req, - std::shared_ptr res) { + [this](const std::shared_ptr & req, + const std::shared_ptr & res) { handle_get_entity_data(req, res); }); get_capabilities_srv_ = node->create_service( service_prefix_ + "/get_capabilities", - [this](const std::shared_ptr req, - std::shared_ptr res) { + [this](const std::shared_ptr & req, + const std::shared_ptr & res) { handle_get_capabilities(req, res); }); @@ -110,8 +110,8 @@ void SovdServiceInterface::shutdown() { } void SovdServiceInterface::handle_list_entities( - const std::shared_ptr request, - std::shared_ptr response) { + const std::shared_ptr & request, + const std::shared_ptr & response) { if (shutdown_requested_.load()) { response->success = false; response->error_message = "Plugin shutting down"; @@ -167,8 +167,8 @@ void SovdServiceInterface::handle_list_entities( } void SovdServiceInterface::handle_list_entity_faults( - const std::shared_ptr request, - std::shared_ptr response) { + const std::shared_ptr & request, + const std::shared_ptr & response) { if (shutdown_requested_.load()) { response->success = false; response->error_message = "Plugin shutting down"; @@ -218,6 +218,17 @@ void SovdServiceInterface::handle_list_entity_faults( } } } + // The fault manager's owner of the record. A peer gateway's HTTP item of + // one per_source record names it in x-medkit.records instead. + if (fault_json.contains("owner") && fault_json["owner"].is_string()) { + fault.owner = fault_json["owner"].get(); + } else if (const auto xm = fault_json.find("x-medkit"); xm != fault_json.end() && xm->is_object()) { + const auto records = xm->value("records", nlohmann::json::array()); + if (records.is_array() && records.size() == 1 && records[0].is_object() && records[0].contains("source_id") && + records[0]["source_id"].is_string()) { + fault.owner = records[0]["source_id"].get(); + } + } response->faults.push_back(std::move(fault)); } } @@ -231,8 +242,8 @@ void SovdServiceInterface::handle_list_entity_faults( } void SovdServiceInterface::handle_get_entity_data( - const std::shared_ptr request, - std::shared_ptr response) { + const std::shared_ptr & request, + const std::shared_ptr & response) { if (shutdown_requested_.load()) { response->success = false; response->error_message = "Plugin shutting down"; @@ -259,8 +270,8 @@ void SovdServiceInterface::handle_get_entity_data( } void SovdServiceInterface::handle_get_capabilities( - const std::shared_ptr request, - std::shared_ptr response) { + const std::shared_ptr & request, + const std::shared_ptr & response) { if (shutdown_requested_.load()) { response->success = false; response->error_message = "Plugin shutting down"; diff --git a/src/ros2_medkit_plugins/ros2_medkit_sovd_service_interface/src/sovd_service_interface.hpp b/src/ros2_medkit_plugins/ros2_medkit_sovd_service_interface/src/sovd_service_interface.hpp index 274cd3116..9c8ce8413 100644 --- a/src/ros2_medkit_plugins/ros2_medkit_sovd_service_interface/src/sovd_service_interface.hpp +++ b/src/ros2_medkit_plugins/ros2_medkit_sovd_service_interface/src/sovd_service_interface.hpp @@ -45,17 +45,18 @@ class SovdServiceInterface : public GatewayPlugin { ~SovdServiceInterface() override; private: - void handle_list_entities(const std::shared_ptr request, - std::shared_ptr response); + void handle_list_entities(const std::shared_ptr & request, + const std::shared_ptr & response); - void handle_list_entity_faults(const std::shared_ptr request, - std::shared_ptr response); + void + handle_list_entity_faults(const std::shared_ptr & request, + const std::shared_ptr & response); - void handle_get_entity_data(const std::shared_ptr request, - std::shared_ptr response); + void handle_get_entity_data(const std::shared_ptr & request, + const std::shared_ptr & response); - void handle_get_capabilities(const std::shared_ptr request, - std::shared_ptr response); + void handle_get_capabilities(const std::shared_ptr & request, + const std::shared_ptr & response); RosPluginContext * context_{nullptr}; std::string service_prefix_{"/medkit"}; diff --git a/src/ros2_medkit_plugins/ros2_medkit_sovd_service_interface/test/test_sovd_service_interface.cpp b/src/ros2_medkit_plugins/ros2_medkit_sovd_service_interface/test/test_sovd_service_interface.cpp index 8d69b44a3..724339043 100644 --- a/src/ros2_medkit_plugins/ros2_medkit_sovd_service_interface/test/test_sovd_service_interface.cpp +++ b/src/ros2_medkit_plugins/ros2_medkit_sovd_service_interface/test/test_sovd_service_interface.cpp @@ -433,7 +433,17 @@ TEST_F(SovdServiceInterfaceTest, ListEntityFaults) { {"description", "Motor temperature exceeds threshold"}, {"status", "CONFIRMED"}, {"occurrence_count", 3}, - {"reporting_sources", {"temp_sensor"}}}); + {"reporting_sources", {"temp_sensor"}}, + {"owner", "temp_sensor"}}); + // A shared record: the fault manager's owner is "*". + faults.push_back( + {{"fault_code", "BUS_OFF"}, {"status", "CONFIRMED"}, {"reporting_sources", {"temp_sensor"}}, {"owner", "*"}}); + // A peer gateway's item of one per_source record names it in x-medkit.records. + faults.push_back( + {{"fault_code", "PEER_FAULT"}, + {"status", "CONFIRMED"}, + {"reporting_sources", {"peer_sensor"}}, + {"x-medkit", {{"records", nlohmann::json::array({{{"source_id", "peer_sensor"}, {"status", "CONFIRMED"}}})}}}}); context_->set_entity_faults("temp_sensor", faults); auto client = node_->create_client("/test_medkit/list_entity_faults"); @@ -447,11 +457,15 @@ TEST_F(SovdServiceInterfaceTest, ListEntityFaults) { auto response = future.get(); ASSERT_TRUE(response->success); - ASSERT_EQ(response->faults.size(), 1u); + ASSERT_EQ(response->faults.size(), 3u); EXPECT_EQ(response->faults[0].fault_code, "MOTOR_OVERHEAT"); EXPECT_EQ(response->faults[0].severity, 2u); EXPECT_EQ(response->faults[0].status, "CONFIRMED"); EXPECT_EQ(response->faults[0].occurrence_count, 3u); + // Each Fault names its record, as the fault manager's Fault does. + EXPECT_EQ(response->faults[0].owner, "temp_sensor"); + EXPECT_EQ(response->faults[1].owner, "*"); + EXPECT_EQ(response->faults[2].owner, "peer_sensor"); } TEST_F(SovdServiceInterfaceTest, ListEntityFaultsNotFound) {