Add support to track the duration of fuzz task steps by brun0-matheus · Pull Request #5555 · google/clusterfuzz

brun0-matheus · 2026-10-27T19:42:57Z

Bug: b/577973994

Overview

Instruments the code to be able to easily emit metrics related to the duration of each fuzz_task step. The metrics are emitted both to Cloud Monitoring (using only a combination of fuzzer name and job, to keep the cardinality low) and Big Query.

Changes

Create TimeStatTracker, which is a reusable context manager that measures the time spent in its code context, saving it to a local dictionary (used to send the measurements to FuzzTaskOutput protobuf later on) and emitting it to a supplied Cloud Monitoring metric.
Add fuzz_task_steps_duration to FuzzTaskOutput protobuf, which is a map of string to interval duration, which allows to easily extend the code to support additional steps.
Add the has_field method in Metric, in order to make sure the metric supplied to TimeStatTracker has the correct fields.
Create the FUZZ_TASK_STEPS_DURATION metric in Cloud Monitoring. It contains the same labels as FUZZING_SESSION_DURATION, with the addition of the step and success labels.
Add the fuzz_task_steps_duration field in the JobRun schema (BigQuery), which contains the duration of all steps. Currently it only contains the bad_build_check duration.
Simplify the load_bigquery_stats test. Now it uses the schema defined in fuzzer_stats.py, avoiding duplicated code.

Tests

Additional unit tests created. I ran tests in the dev environment with the bad_build check implemented.
Evidence that the CloudMonitoring metric is being tracked: https://screenshot.corp.google.com/2u8b1d3psrv68.
Evidence that the BigQuery metric is being tracked: https://screenshot-v2.corp.google.com/73759kq0eje1o.

ViniciustCosta

Let's do the bad_build_check metric in a separate PR and only add the time tracker structure and plumbing needed for the metrics in this one, so it's easier to review.

ViniciustCosta · 2026-10-30T13:39:36Z

+  pass
+
+
+class TimeStatTracker:


This largely overlaps with TimeMetricTracker just above. Could we extend that class instead? Or maybe remove the old class as it is only used, if I remember correctly, by the corpus sync metric you added?

I'd rather not keep two trackers that differ only slightly.

Yes, I suggest removing TimeMetricTracker in a future PR.

ViniciustCosta · 2026-10-30T13:47:13Z

    testcase_execution_duration: datetime.timedelta = None,
-    fuzzing_duration: datetime.timedelta = None):
+    fuzzing_duration: datetime.timedelta = None,
+    fuzz_task_steps_duration=None):


nit: please add a type hint like the other params: fuzz_task_steps_duration: dict[str, datetime.timedelta] | None = None.

ViniciustCosta · 2026-10-30T13:47:50Z


    self.gcs_corpus = None
    self.fuzz_task_output = uworker_msg_pb2.FuzzTaskOutput()  # pylint: disable=no-member
+    self.tracker = monitor.TimeStatTracker(


nit: self.tracker is too generic on FuzzingSession. Maybe self.steps_time_tracker?

ViniciustCosta · 2026-10-30T13:48:49Z

+    start = time.time()
+    success = True
+    try:
+      yield None


nit: yield None -> yield

ViniciustCosta · 2026-10-30T13:50:30Z

    if fuzzing_duration is not None:
      self._stats_data['fuzzing_duration'] = _timedelta_to_duration_string(
          fuzzing_duration)
+    if fuzz_task_steps_duration:


If we keep the fixed RECORD, please drop and log keys that aren't in SCHEMA here for visibility, since these keys come from the uworker output.

ViniciustCosta · 2026-10-30T13:52:41Z

-                                known_crash_count, groups, testcases_generated,
-                                testcase_generation_duration,
-                                testcase_execution_duration, fuzzing_duration)
+  job_run = fuzzer_stats.JobRun(


very nit: maybe in a follow-up PR we can try to use keyword args here to avoid ordering errors as this function now receives too many args.

ViniciustCosta · 2026-10-30T14:03:02Z

    self.fuzz_task_output.fuzzing_duration.FromTimedelta(
        datetime.timedelta(seconds=fuzzing_session_duration_seconds))

+    for name, duration in self.tracker.stats.items():


iiuc stats are only getting copied into fuzz_task_output at the very end of run, so any early return drops every step tracked so far. Not sure if there is any successful early returns, in which case we could write each step into as soon as it finishes.

FYI JobRun only reaches BQ for successfull sessions, while the Cloud Monitoring metric is emitted for every session that reaches a tracked step. So the two will have distinct values, e.g. failed or errored sessions only show up in Monitoring. Is that fine for how these step durations will be analysed? Maybe worth noting in a comment near the BQ field

Yes, this is the expected behavior and there are no successful early returns. Actually, I'd prefer to have both of them (CloudMonitoring and BigQuery) track unsuccessful runs as well, but this is very hard to do with BQ because of how CF works, so I kept this implementation only for CM.
I agree to adding a comment in the BQ and CM metrics description to clarify this.

ViniciustCosta · 2026-10-30T15:01:22Z

+    except:
+      success = False
+      raise
+    finally:


Are there cases when the same step runs more than once (loops, retries)? Right now it warns and overwrites. Should durations be summed instead?

Yes, there are. In this case I opted to make a tracker for each loop (as in PR #5569). Maybe summing it up is a simpler solution.

As a matter of fact, I opted for the first solution (make a tracker for each loop) in order to get the stats for TestcaseRun, but later on (PR #5571) I had to do the same for blackbox fuzzing, so now I don't think it's the best solution anymore.

ViniciustCosta · 2026-10-30T15:09:02Z

+    self.stats: dict[str, datetime.timedelta] = {}
+    self.metric = metric
+    self.common_labels = dict(common_labels) if common_labels else {}
+    if metric:


This checks a hard-coded metric at runtime on every FuzzingSession. Isn't a unit test enough?

I don't get it. Are you referring to the if metric part? I wanted to make this generic and work for any metric, so it's easy to extend to other code parts.

ViniciustCosta · 2026-10-30T15:13:17Z

  // Total fuzzing session time. Includes time spent updating the fuzzer, build
  // and test corpus set up, and crash processing.
  optional google.protobuf.Duration fuzzing_duration = 23;
+  map<string, google.protobuf.Duration> fuzz_task_steps_duration = 24;


Please add a comment like the above field have.

Also nit: I think steps_duration is enough inside FuzzTaskOutput

decoNR · 2026-10-30T18:45:24Z

        'type': 'INTERVAL',
        'mode': 'NULLABLE'
+    }, {
+        'name':


Out of curiosity, did you look into possibly adding another table instead of adding a struct column?

No. I believe this place is good enough, since it's basically extra stats to the fuzz task run.

decoNR · 2026-10-30T19:00:01Z

  # TODO(metzman): Replace fuzz_task_output.fully_qualified_fuzzer_name` with
  # `fuzz_task_input.fuzz_target` instead.


It seems that this comment should be after the code you added.

brun0-matheus · 2026-10-30T19:27:01Z

Let's do the bad_build_check metric in a separate PR and only add the time tracker structure and plumbing needed for the metrics in this one, so it's easier to review.

Done, now the bad build check PR is #5574.

brun0-matheus added 3 commits October 27, 2026 18:32

Add steps duration field to FuzzTaskOutput proto

6582dcd

Create TimeStatTracker and track bad_build time

675521c

Add and modify unittests

cfee67e

brun0-matheus requested a review from ViniciustCosta October 27, 2026 19:42

brun0-matheus self-assigned this Oct 27, 2026

brun0-matheus requested a review from a team as a code owner October 27, 2026 19:43

brun0-matheus added this pull request to stack #5565 October 29, 2026 18:47

ViniciustCosta requested changes Oct 30, 2026

View reviewed changes

ViniciustCosta requested a review from decoNR October 30, 2026 15:14

brun0-matheus added 5 commits October 30, 2026 17:59

Remove bad_build check duration metric

cfa5f2e

do nits

f1592a4

rename proto fuzz_task_steps_duration attribute

301fca1

rename FuzzingSession.tracker

c794463

Filter non-existing step names in JobRun

d6a0d77

decoNR reviewed Oct 30, 2026

View reviewed changes

fix unittests

57af416

brun0-matheus mentioned this pull request Oct 30, 2026

[Fuzz task steps duration] Add bad_build_check duration metric #5574

Open

		# TODO(metzman): Replace fuzz_task_output.fully_qualified_fuzzer_name` with
		# `fuzz_task_input.fuzz_target` instead.

Conversation

brun0-matheus commented Oct 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Changes

Tests

Uh oh!

ViniciustCosta left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

brun0-matheus commented Oct 30, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

brun0-matheus commented Oct 27, 2026 •

edited

Loading