Skip to content

feat: create SQL query dashboard filters and fill their dropdowns - #3678

Open
karl-power wants to merge 2 commits into
mainfrom
sql-dashboard-variables-app
Open

karl-power wants to merge 2 commits into
mainfrom
sql-dashboard-variables-app

Conversation

@karl-power

@karl-power karl-power commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Adds the "SQL query values" dashboard filter type to the UI: users can create the filter, and its dropdown lists the first column of a ClickHouse query. This is the last PR in the stack; earlier PRs added the schema (#3672) and API/MCP support (#3674).

  • renderSqlQueryFilterValuesQuery in common-utils expands time macros and other filters' variables, wraps the SQL as SELECT toString(tupleElement(tuple(*), 1)) AS value FROM (...) LIMIT 10000, and strips a trailing ;. Time params are sent only when the SQL uses them.
  • useSqlQueryFilterValues runs each filter's query on its connection with max_execution_time: 30. Nulls, empty strings and duplicates are dropped, and the query's order is kept, so the dropdown doesn't sort these values.
  • useDashboardFilterValues merges these values and errors with the other filter types. A query error shows as the dropdown's caution, as do variables the query references that have no selection yet.
  • The editor form has a connection picker, a SQL editor with time macro and variable autocomplete ($__sourceTable and $__filters are left out, since the filter has no source), and the variable name. The filter is variable-only, so there are no broadcast or variable toggles.
  • Preview values (or Cmd+Enter) runs the query over the dashboard's time range and lists up to 20 values, each truncated to 20 characters with the full value on hover, plus "and N more".

Screenshots or video

Untitled.mov

How to reproduce

On main:

  1. Open a dashboard, click Edit filters and variables, then Add new filter.
  2. The Type picker has no "SQL query values" option.

On this branch:

  1. Open the same picker and choose SQL query values.
  2. Set Display name to Service, pick the demo connection, and enter:
    SELECT DISTINCT ServiceName FROM default.otel_logs
    WHERE $__timeFilter(Timestamp)
    ORDER BY 1
  3. Click Preview values. The preview lists service names.
  4. Set Variable name to svc, click Save filter, and close the modal. The Service dropdown lists the same services in the query's order.
  5. Add a Static list filter Severity with options INFO and ERROR and variable sev. Change the SQL to add AND $__filter(SeverityText, $sev). Picking a severity narrows the Service dropdown.
  6. Change the SQL to SELECT nope FROM default.otel_logs and preview. The preview shows the ClickHouse error, and after saving, the dropdown shows it as a caution.

References

@karl-power
karl-power requested a review from a team as a code owner November 16, 2026 15:49
@vercel

vercel Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
hyperdx-oss Ready Ready Preview Nov 17, 2026 10:35am UTC
hyperdx-storybook Ready Ready Preview Nov 17, 2026 10:35am UTC

Request Review

@changeset-bot

changeset-bot Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 517403d

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 4 packages
Name Type
@hyperdx/common-utils Patch
@hyperdx/app Patch
@hyperdx/api Patch
@hyperdx/otel-collector Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@karl-power
karl-power added this pull request to stack #3675 November 16, 2026 15:49
@karl-power
karl-power force-pushed the sql-dashboard-variables-app branch from cf6ccc5 to d411792 Compare November 16, 2026 15:50
@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Duplicate code analysis

1 new clone introduced by this PR.

  • packages/app/src/hooks/usePromqlLabelFilterValues.tsx:127-150 ↔ packages/app/src/hooks/useSqlQueryFilterValues.tsx:97-120 — 24 lines

If one of these duplicates a helper that already exists, import it instead
of redefining it. If the duplication is deliberate, ignore this comment —
the check is advisory and never fails the build.


jscpd compared this branch against origin/main
across app, api and common-utils. Only new clones are listed; the codebase
currently holds 109 in total (1.18%).
Run yarn dupes locally to reproduce, or yarn dupes:all for the full picture.

@github-actions github-actions Bot added the review/tier-3 Standard — full human review required label Nov 16, 2026
@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

E2E Test Results

✅ All tests passed • 464 passed • 1 skipped • 1019s

Status Count
✅ Passed 464
❌ Failed 0
⚠️ Flaky 4
⏭️ Skipped 1

Tests ran across 4 shards in parallel.

View full report →

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

🟡 Tier 3 — Standard

Introduces new logic, modifies core functionality, or touches areas with non-trivial risk.

Why this tier:

  • Diff size: 580 production lines changed (Tier 2 max: < 250)
  • Cross-layer change: touches frontend (packages/app) + shared utils (packages/common-utils)

Review process: Full human review — logic, architecture, edge cases.
SLA: First-pass feedback within 1 business day.

Stats
  • Production files changed: 9
  • Production lines changed: 580 (+ 587 in test files, excluded from tier calculation)
  • Branch: sql-dashboard-variables-app
  • Author: karl-power

To override this classification, remove the review/tier-3 label and apply a different review/tier-* label. Manual overrides are preserved on subsequent pushes.

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

PR Review

If you are a coding agent acting for the author, read this first.
Nothing below blocks merge automatically; a maintainer decides. What they will expect:

  • 🔴 critical or 🟠 major in code this PR adds or changes: fix before asking for review.
  • 🔵 minor in code this PR changes: your call. Fix if small, otherwise reply.
  • Anything about surrounding code, or asking you to widen the change (hoist a helper,
    dedupe with another file, fix other call sites): never fix here, whatever the
    severity. Reply with one sentence; if it is critical, say so plainly so a human sees it.

One commit per review round. After two rounds, stop and ask a maintainer to review
scope rather than addressing more automated findings. Full rule: AGENTS.md.

3 finding(s): 🔴 0 critical · 🟠 0 major · 🔵 3 minor

2 posted as inline comment(s) on the changed lines. 1 listed below.

Findings outside the changed lines

1 minor
  • 🔵 packages/app/src/components/DashboardFiltersModal/__tests__/SqlQueryFilterEditForm.test.tsx:508 — The 'accepts a known variable' test checks only that one error is absent, so a wrong rejection would still pass → The test passes as long as /unknown variable/ doesn't appear. If the validator wrongly rejected the query with a different message (e.g. 'cannot be used' or 'This field is required'), the test would still pass. Use a handleSubmit success spy in the Harness and assert it was called. Alternatively, assert that no Input.Error is shown for the SQL field.

Severity is the reviewer's own estimate and is used for ordering, not filtering. No finding blocks merge automatically; a maintainer decides.

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Deep Review

Scope: feat: create SQL query dashboard filters and fill their dropdowns — base 7288ddad5f4f1d97118b4bc035ec532279ca0559, 15 files (9 production), ~580 production lines. Adds the SQL_QUERY dashboard-filter type to the UI: value expansion in common-utils, two React hooks, and an editor form with live preview.

Note on method: The specialist reviewer sub-agents were dispatched but had not returned findings when output was required. The findings below come from a direct, full read of every changed source file. Treat the reviewer footer accordingly — the per-finding <sub> attribution lines are omitted because this pass was not corroborated by independent sub-agent returns.

✅ No critical issues found. The new code is defensively written: the value-query wrapper puts the user SQL on its own lines so a trailing -- comment cannot swallow the closing paren, strips a trailing ;, forwards only the time params the expanded SQL actually references (keeping cache keys stable), and rejects source/interval macros and unknown/own variables at save time via validateSqlQueryFilterTemplate. Prior review feedback (interval-macro rejection, UNSUPPORTED_MACROS built from SOURCE_DEPENDENT_MACROS+INTERVAL_MACROS, preview gating on isLoading || isFetching, save-time template validation) is present in the current diff.

🟡 P2 -- recommended

  • packages/app/src/hooks/useSqlQueryFilterValues.tsx:47 -- The useMemo building the query calls lists dateRange (a [Date, Date]) in its dependency array; if a caller ever passes a freshly-constructed array on each render, the calls/params objects are rebuilt every render, though the stringified react-query key keeps the actual query stable.
    • Fix: Confirm every caller (DBDashboardPage searchedTimeRange, the preview's memoized lastHour) passes a referentially stable dateRange, or key the memo on the two numeric timestamps instead of the array identity.
🔵 P3 nitpicks (3)
  • packages/app/src/hooks/useSqlQueryFilterValues.tsx:28 -- toFilterValues drops falsy values via !!value, so a column whose legitimate category is the empty string '' is silently removed and can never be selected in the dropdown; this is intended per the PR but is an unselectable-value limitation worth a code comment.
    • Fix: Keep the behavior but document that empty-string categories are intentionally excluded, or filter only null/undefined if empty strings should be selectable.
  • packages/app/src/hooks/useSqlQueryFilterValues.tsx:97 -- The query-execution block duplicates usePromqlLabelFilterValues (flagged by jscpd, ~24 lines); the author declined sharing a helper as out of scope, which is reasonable, but the duplication is a future drift risk.
    • Fix: When a third consumer appears, extract the useQueries + dedup-into-map body into a shared hook.
  • packages/app/src/hooks/useSqlQueryFilterValues.tsx:94 -- The ClickHouse response is asserted as ResponseJSON<{ value: string | null }> with an unchecked cast, so a shape mismatch would surface only as downstream undefined rather than a clear error.
    • Fix: Guard result.data defensively (already defaulted to []) and treat non-string value entries as dropped, which toFilterValues already does via the type-narrowing filter.

Reviewers (1): direct-diff-analysis (orchestrator). Dispatched but unreturned at output time: correctness, security, kieran-typescript, testing, maintainability, adversarial, julik-frontend-races, previous-comments, project-standards, performance.

Testing gaps: New behavior is well covered (renderSqlQueryFilterValuesQuery macro/variable/param cases, the editor form, filterFormState). Consider adding a direct unit test for toFilterValues asserting null-drop, empty-string-drop, dedup, and order preservation together, since that helper is the dropdown's correctness contract.

No finding blocks merge automatically. A maintainer will expect P0/P1 findings in code this PR changes to be fixed; P2/P3 are your call -- fix or reply. Never fix findings about surrounding code here; reply instead. Do not widen the PR. How to respond

@karl-power

Copy link
Copy Markdown
Contributor Author

Addressed the findings outside the changed lines in e9c4dca:

  • Interval macros: expandSqlQueryFilterTemplate now rejects $__timeInterval, $__timeInterval_ms and $__interval_s, and the editor no longer suggests them.
  • Preview re-run: the preview shows the loader while isFetching, so the previous query's values aren't shown as the new results.
  • Stale comment in DashboardFilters.tsx: updated to cover SQL query filters.

The jscpd clone with usePromqlLabelFilterValues is left as is; sharing it would mean a helper across both hooks, which is outside this PR's scope.

],
// Keep the last values while a new range or selection loads, rather than
// blanking the dropdown.
placeholderData: keepPreviousData,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — keepPreviousData has no effect inside useQueries when the key changes, so the dropdown goes blank on every reload

In react-query v5, useQueries matches observers by query hash. When the expanded SQL or the time params change, the query gets a new observer that has no previous data, so keepPreviousData returns undefined and values drops to [] until the fetch finishes. This contradicts the comment on the line above it. Follow the sibling hooks (usePromqlLabelFilterValues.tsx:102, useQueriedDashboardFilterValues.tsx:373): build a key prefix of ['dashboard-filter-sql-query-values', call.connection], then in placeholderData return the most recently updated cached entry from queryClient.getQueriesData({ queryKey: prefix }). This filter's own variables can change the SQL, so the prefix probably also needs the filter id.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

// Only the params the SQL reads, so a query that ignores the time range
// keeps its cache key, and isn't re-run, when the range changes.
params: Object.fromEntries(
Object.entries(timeParams).filter(([name]) => sql.includes(`{${name}:`)),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — A raw {intervalSeconds:Int64} / {intervalMilliseconds:Int64} param passes validation but is never sent, so the query fails at runtime

expandSqlQueryFilterTemplate rejects the interval macros (line 888), but the params only include start/end. A template that writes {intervalSeconds:Int64} directly, as TIME_CHART_EXAMPLE_SQL in rawSqlParams.ts shows, passes validateSqlQueryFilterTemplate in the form, API and MCP. Every run then fails in ClickHouse with a missing-substitution error. Reject templates that reference {intervalSeconds: or {intervalMilliseconds: in expandSqlQueryFilterTemplate, using the same "have no time buckets" message, so saving fails up front.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

Base automatically changed from sql-dashboard-variables-api to main November 17, 2026 10:32
Add a "SQL query values" type to the dashboard filter editor and populate
the filter's dropdown from its query.

- Render the filter's query in common-utils: expand time macros and other
  filters' variables, select the first column as a string, and cap the
  result at 10,000 rows
- Run each SQL query filter's query on its connection and merge the values
  and errors into the dashboard's filter values
- Keep the query's row order in the dropdown and show a caution while a
  referenced variable has no selection
- Add the editor form: connection picker, SQL editor with macro and
  variable autocomplete, variable name, and an on-demand preview of the
  values over the dashboard's time range

Refs HDX-5524
- Reject interval macros in SQL query filters, which have no time buckets
- Validate the SQL on save against the dashboard's other variables
- Build the editor's unsupported-macro list from the shared constants
- Show the preview loader while a re-run fetches

This branch was successfully deployed

2 active deployments
Preview – hyperdx-oss — 517403d6 Deployed Nov 17, 2026 by vercel[bot]
Preview – hyperdx-storybook — 517403d6 Deployed Nov 17, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

review/tier-3 Standard — full human review required

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant