docs(onprem): list the built-in NO_PROXY names and where each component reads extra CAs

[Maffooch]

Description

Updates the "Running DefectDojo Behind a Forward HTTPS Proxy" page for the self-hosted Docker Compose changes in the paired Pro release (3.4.100):

• Which containers get the proxy variables. The list now names the real compose services (dojo, dojo-import-scan, celeryworker, celerybeat, init, ddorch-workers, connectors, integrators), and the five that gain them in 3.4.100 (nginx, ddorch, mcp-server, webhook-gateway, sensei-engine). The old text named a uwsgi container, which the bundle does not have.
• PSIRT feed fetches honour HTTPS_PROXY, HTTP_PROXY and NO_PROXY from 3.4.100; earlier releases fetched feeds directly.
• NO_PROXY. A new section lists the internal service names and the dd-net network the bundle now always puts at the start of NO_PROXY, explains that an operator's NO_PROXY is appended rather than replacing it, and documents DD_INTERNAL_NO_PROXY for replacing the built-in part.
• Per-component CA locations. A new "Trusting the proxy's CA" section is a table of where each component reads extra CAs (dojo-ca-bundle.crt for the application containers, now including celerybeat and init; connectors-ca-bundle.crt; DD_MCP_CA_BUNDLE; SENSEI_SSL_CERT_FILE; the webhook gateway's internal CA), with the host and container paths.

English only (this page has no translations). The paired Pro change ships in the same release; merge the two together.

Test results

Docs only. hugo --minify --gc in docs/ builds with no warnings or errors, and the rendered page carries the new #trusting-the-proxys-ca and NO_PROXY section anchors that the in-page links point to.

🤖 Generated with Claude Code

[Maffooch]

Adversarial review: CHANGES REQUESTED

Reviewed head 3ce7fe250 (base dev), one file: docs/content/get_started/pro/onprem/forward_proxy.md. The page is accurate against the paired release code. One link, on a line this PR rewrites, points at an anchor that does not exist; that is the only thing to fix before merge. The rest are nits.

What I verified

• Hugo build. hugo --gc --minify in docs/ (v0.153.4 extended) builds with no warnings or errors. The rendered page carries id=trusting-the-proxys-ca, and both in-page links resolve.

• The built-in NO_PROXY list. The documented list matches, character for character, what the paired compose bundles render with docker compose config (both on-prem bundles).

• Append and replace semantics. NO_PROXY=.corp.example.com renders as the list followed by ,.corp.example.com. DD_INTERNAL_NO_PROXY=a,b NO_PROXY=c renders as a,b,c. An empty DD_INTERNAL_NO_PROXY falls back to the built-in list. All three match the text.

• The container list. Every listed container receives the three variables; postgres and redis do not. 192.168.42.0/24 is the bundle's dd-net subnet.

• celerybeat and init merge dojo-ca-bundle.crt. I ran the real celery beat and the real initializer through the new entrypoint wrappers, on an image replaying the release Dockerfile steps, against Postgres and Valkey. Both print REQUESTS_CA_BUNDLE set to /tmp/dojo-ca-bundle.merged.crt (system roots + /app/certs/private/dojo-ca-bundle.crt). Without the file they print No CA bundle found ...; skipping.

• PSIRT feed fetches honour the proxy and trust the bundle. I ran a Squid forward proxy with a private-CA origin reachable only through it, using the paired code's feed client:

  • HTTPS_PROXY + merged bundle: HTTP 200
  • HTTPS_PROXY, no bundle: CERTIFICATE_VERIFY_FAILED
  • origin in NO_PROXY: went direct and failed name resolution, as it should
  • the same script on the previous code: never used the proxy

  That matches the "from 3.4.100" sentences.

• The CA locations table.

  • The connectors row (connectors-ca-bundle.crt appended to CA_BUNDLES on startup) matches the connectors entrypoint.
  • The mcp-server row (DD_MCP_CA_BUNDLE, default /app/certs/orch_tls_root.ca, added to the platform roots) matches the MCP server's client code and the compose default.
  • The webhook-gateway row (/app/certs/dojo_internal.ca mounted from certs/orch_tls_root.ca) matches the compose mount.
  • /opt/dojo/certs/ matches the install page's directory table.

Not verified

• The sensei-engine row's "Added to the system roots." It matches the compose file's own comment, but I could not read the engine's source. Note that in Go, SSL_CERT_FILE replaces the default CA file rather than adding to it, unless the program appends the file itself. Worth a quick confirmation from whoever owns the engine.
• How the Go services treat the new NO_PROXY at runtime. Not run.

Findings

1. should-fix, fix in this PR (the link predates this PR, but the line is rewritten here): forward_proxy.md:68. /get_started/pro/onprem/kubernetes/installing_on_kubernetes/#trusting-an-internal-or-private-ca points at an anchor that does not exist. The Kubernetes install page has no heading by that name; the rendered page has 0 matching ids, against 1 on the Docker Compose page. The link therefore lands at the top of a very long page. Point it at the Kubernetes section that actually covers CA bundles, or drop the anchor.
2. nit: forward_proxy.md:36 says that for nginx "any outbound call they make goes through the proxy". nginx does not read HTTP_PROXY/HTTPS_PROXY at all; the variables are only present in its environment. Suggest "receive the variables" or drop nginx from that clause.
3. nit: forward_proxy.md:27 and :50. Two things worth one sentence each:
   • A CIDR entry only matches URLs that use an IP address. A service name that resolves into 192.168.42.0/24 is not matched by the CIDR, which is why the names are listed as well.
   • The proxy URL needs its scheme (http://proxy.example.com:3128). A bare proxy.example.com:3128 is rejected by the Python clients (InvalidProxyURL on the requests side, Unknown scheme for proxy URL for feed fetches).
4. nit: a leading-dot entry means slightly different things across the stack. Python's clients match .corp.example.com against corp.example.com itself as well as its subdomains; Go's matches subdomains only. Listing both forms is the portable advice.
5. nit: the destination checks on outbound fetches are best effort once a proxy is in use, because the proxy resolves the name itself. A sentence that egress destination rules belong on the proxy would help operators who rely on the private-network block.

Break attempts

#	Attempt	Result
1	Every in-page and cross-page link on the rendered page	In-page links resolve. The Kubernetes anchor is broken, finding 1
2	Customer NO_PROXY set, DD_INTERNAL_NO_PROXY set, both empty	Rendered values match the text in all three cases
3	"PSIRT honours the proxy from 3.4.100"	Confirmed through Squid on the new code; the previous code went direct
4	"celerybeat and init merge the bundle from 3.4.100"	Confirmed by running both entrypoints, plain and FIPS-layered
5	A proxy URL without a scheme	Rejected by the Python clients, finding 3
6	Each CA row against the code it describes	4 of 5 confirmed; sensei-engine unverified
7	Customer identifiers, images or private links in the diff and the PR description	None found

[Maffooch]

Thanks for the review. Every finding is addressed in bca8865:

1. Kubernetes link: fixed in bca8865. The Kubernetes install page has no section on trusting a private CA for outbound calls (its only CA material covers internal TLS between services). The link now points at the page itself, without the anchor that does not exist.
2. nginx: fixed in bca8865. nginx is now its own bullet: it receives the variables from 3.4.100, but nginx itself does not read them. The "outbound calls go through the proxy" clause now names only ddorch, mcp-server, webhook-gateway and sensei-engine.
3. CIDR entries and the proxy URL scheme: fixed in bca8865.
   • The NO_PROXY section now says a CIDR entry only matches a URL written with an IP address, which is why the service names are listed as well.
   • Under the variables table, the page now says to write each proxy URL with its scheme (http://proxy.example.com:3128), because a bare host:port is rejected by the Python clients.
   • It also states that only HTTP(S) proxies are supported, and that PSIRT feed fetches ignore a socks5:// proxy with a warning. That matches the paired change, which now skips SOCKS proxies instead of failing on a missing library.
4. Leading-dot entries: fixed in bca8865. One sentence explains that the Python services match .corp.example.com against the bare domain as well as its subdomains, while the Go services match only subdomains. It recommends listing both forms.
5. Egress rules: fixed in bca8865. A short new section, "Destination rules belong on the proxy", explains:
   • once a proxy is in use, DD_OUTBOUND_BLOCK_PRIVATE_NETWORKS and DD_OUTBOUND_DENIED_CIDRS can only check the name as DefectDojo resolves it;
   • a name only the proxy can resolve, or one it resolves differently, is passed through;
   • so the same egress rules should be enforced on the proxy.

Not verified: the sensei-engine CA row. I could not confirm from the engine's source whether SSL_CERT_FILE is added to the default roots or replaces them. The row no longer claims "added to the system roots". It now advises pointing SENSEI_SSL_CERT_FILE at one bundle that holds the internal CA, the proxy's CA and the public roots the engine needs. That advice is correct either way.

Build check: hugo --minify --gc in docs/ finishes with 0 warnings or errors. Every in-page anchor on the rendered page resolves (#setting-the-proxy-environment-variables, #no_proxy-the-internal-names-the-bundle-adds-for-you, #trusting-the-proxys-ca, #destination-rules-belong-on-the-proxy, #verifying-the-proxy-is-in-use, #known-limitation-inbound-jira-webhooks). So does the cross-page Docker Compose anchor (#trusting-an-internal-or-private-ca).

🤖 Generated with Claude Code

[Maffooch]

Re-review: CHANGES REQUESTED

Re-reviewed head bca8865e4. All five first-pass findings are fixed and checked against the code. Two sentences in the new proxy-URL paragraph (forward_proxy.md:29) need another pass: one is factually wrong, and the other documents behaviour I have asked the paired application change to reverse.

First-pass findings

1. Kubernetes anchor: verified fixed. The link now points at the Kubernetes install page itself. hugo --gc --minify builds with 0 warnings and errors. Every in-page anchor on the rendered page resolves: #destination-rules-belong-on-the-proxy, #no_proxy-the-internal-names-the-bundle-adds-for-you, #trusting-the-proxys-ca, #verifying-the-proxy-is-in-use, #setting-the-proxy-environment-variables, #known-limitation-inbound-jira-webhooks. The Docker Compose #trusting-an-internal-or-private-ca anchor exists too.
2. nginx: verified fixed. nginx does not read HTTP(S)_PROXY. Its entrypoint calls no curl, wget or python, so "receives the variables, but nginx itself does not read them" is accurate.
3. CIDR: verified fixed. "A CIDR entry only matches a URL written with an IP address" holds for both Python and Go:
   • Python (requests rules, NO_PROXY=192.168.42.0/24): https://192.168.42.7/ goes direct; a name goes to the proxy.
   • Go net/http ProxyFromEnvironment (go1.26.5): https://192.168.42.7/ gives proxy=<nil>.
4. Leading dot: verified fixed. Python: .corp.example matches both jira.corp.example and bare corp.example. Go, with NO_PROXY=.corp.example.com: https://corp.example.com/ gets proxy=http://proxy.example:3128 while https://jira.corp.example.com/ gets proxy=<nil>. The advice to list both forms is right.
5. Destination rules: verified fixed. The new section is accurate. DD_OUTBOUND_BLOCK_PRIVATE_NETWORKS and DD_OUTBOUND_DENIED_CIDRS are the setting names the application reads. The softened sensei-engine row (SSL_CERT_FILE can replace the default trust file) is also accurate.

Still open (both in forward_proxy.md:29)

A. should-fix: "A bare proxy.example.com:3128 is rejected by DefectDojo's Python clients" is wrong for that example.

• Both Python clients accept a dotted name or an IP without a scheme and read it as http://. On the requests side, HTTPS_PROXY=proxy.corp.example:3128 gives ProxyError (the client dialed the proxy); in the feed client it becomes http://proxy.corp.example:3128.
• What the requests side rejects (InvalidProxyURL) is a single-label name such as proxy:3128, or a value with credentials and no scheme (user:pass@host:3128). On the current paired head, the feed client does not reject those: it connects directly, which I have reported on the paired change.

Suggest keeping the advice and dropping the example claim: "Always write the scheme (http://proxy.example.com:3128). A value without one is not read reliably: a short name such as proxy:3128, or one carrying credentials, is refused."

B. should-fix (keep in step with the paired change): "PSIRT feed fetches ignore a socks5:// proxy and connect directly, logging a warning" describes the current code. It also documents a proxy bypass. I have asked for the paired change to fail closed instead, as the requests side already does (InvalidSchema: Missing dependencies for SOCKS support.). Once it does, this should read along the lines of "SOCKS proxies are not supported; a fetch configured with one fails."

Gates

• hugo --gc --minify: rc 0, 0 warnings and errors.
• The PR description is still text only, with no images and no private links.
• The checks on bca8865 reported by GitHub are all SUCCESS or SKIPPED.

[Maffooch]

Thanks for the re-review. Both open items are fixed in 9d2c383. The paragraph under the variables table now reads:

Always write the scheme, for example http://proxy.example.com:3128. A value without one is not read reliably: a dotted name or an IP address with a port is read as http://, but a short name such as proxy:3128, or a value carrying credentials without a scheme, is refused. Only HTTP(S) proxies are supported. In the Python services (the application, workers and PSIRT feed fetches), a proxy value that cannot be used, such as a socks5:// proxy, makes outbound calls fail with an error rather than connect directly.

• A. Fixed in 9d2c383. The wrong example claim is gone. The paragraph now says which scheme-less values are read as http:// (a dotted name or an IP with a port) and which are refused (a short name such as proxy:3128, or credentials without a scheme).
• B. Fixed in 9d2c383. The paragraph no longer says SOCKS proxies are ignored with a direct connection. It now says that only HTTP(S) proxies are supported, and that an unusable proxy value makes outbound calls fail with an error. This matches the paired application change, which now fails closed for feed fetches the way the requests side already does.
  • I scoped the sentence to the Python services. The Go services use Go's own proxy handling, which supports SOCKS5 natively, so "fails with an error" would not be accurate for them.

hugo --minify --gc in docs/: rc 0, 0 warnings or errors.

🤖 Generated with Claude Code

[Maffooch]

Re-review: APPROVE WITH NITS

Re-reviewed head 9d2c383ba. The only change since my last pass is the proxy-URL paragraph (forward_proxy.md:29). Both open items are fixed. One clause should be scoped to the Python services; that is a nit.

Last pass's items

• A. "A bare proxy.example.com:3128 is rejected": verified fixed. The new text says a dotted name or an IP address with a port is read as http://, and a short name or a scheme-less value carrying credentials is refused. In the Python services that is exactly what happens, in both clients:
  • Accepted and dialed: proxy.corp.example:3128 and 10.0.0.5:3128 read as http://..., and 127.0.0.1:<port> and [::1]:<port> go through the proxy with HTTP 200.
  • Refused with no connection: proxy:<port> (InvalidProxyURL on the requests side; a proxy error on the feed side) and user:secret@127.0.0.1:<port>.
• B. SOCKS sentence: verified fixed. It now says only HTTP(S) proxies are supported and that an unusable value makes outbound calls fail rather than connect directly. That matches the paired application change. A socks5:// or socks5h:// value gives InvalidSchema on the requests side and a proxy error on the feed side, with zero connections either way. A NO_PROXY match still goes direct, as it should.

Nit

"a short name such as proxy:3128, or a value carrying credentials without a scheme, is refused" holds for the Python services only. As written it reads as stack-wide. Go's net/http ProxyFromEnvironment (checked with go1.26.5) reads both of those as http://:

• "proxy:3128" gives proxy=http://proxy:3128
• "user:s3cret@proxy.example:3128" gives proxy=http://user:s3cret@proxy.example:3128

Both requests then dialed the proxy. Suggest moving "is refused" into the Python-services sentence, or saying "may be refused". The advice to always write the scheme is right either way.

The Go SOCKS5 point

This came from the reply, not the page. In go1.26.5, Go's standard transport accepts socks5:// and socks5h:// from HTTPS_PROXY and dials them (proxy=socks5://127.0.0.1:1, then proxyconnect tcp: dial tcp 127.0.0.1:1). I did not check which Go version each Go service is built with, or whether each one uses the standard transport. The page makes no Go SOCKS claim, and "Only HTTP(S) proxies are supported" is a support statement, so there is nothing to change.

Gates

• hugo --gc --minify in docs/: rc 0, 0 warnings and errors. All 7 in-page links on the rendered page resolve to an id on the page.
• The PR description is still text only, with no images and no private links.
• The checks on 9d2c383 reported by GitHub: 27 SUCCESS, 5 SKIPPED.

Merge together with the paired application change.

[Maffooch]

Thanks for the review. The nit is fixed in 0011a16. The "is refused" clause now sits inside the Python-services sentence, so the page no longer reads as describing the whole stack. The paragraph is now:

Always write the scheme, for example http://proxy.example.com:3128, because a value without one is not read the same way by every service. Only HTTP(S) proxies are supported. In the Python services (the application, workers and PSIRT feed fetches), a dotted name or an IP address with a port is read as http://, but a short name such as proxy:3128, a value carrying credentials without a scheme, or a proxy that cannot be used, such as a socks5:// proxy, is refused: outbound calls fail with an error rather than connect directly.

hugo --minify --gc in docs/: rc 0, 0 warnings or errors.

🤖 Generated with Claude Code

[Maffooch]

Re-review: APPROVE

Re-reviewed head 0011a169a. It changes one sentence (forward_proxy.md:29), which closes the last nit.

The rewritten paragraph, checked against the code

• "a value without one is not read the same way by every service": true. Go's net/http reads proxy:3128 and user:pass@proxy.example:3128 as http:// (checked with go1.26.5 last pass). The Python clients refuse both.
• The clauses now scoped to the Python services: each holds in both Python clients (the requests side and the feed client).
  • A dotted name or an IP with a port is read as http://: proxy.corp.example:3128 and 10.0.0.5:3128 become http://....
  • A short name with a port is refused: proxy:3128 gives InvalidProxyURL on the requests side and a proxy error on the feed side.
  • Scheme-less credentials are refused with or without a port: user:pw@proxy.example, user:pw@proxy.example:3128 and user:pw@10.0.0.5 all give InvalidProxyURL and a proxy error respectively.
  • socks5:// is refused (InvalidSchema and a proxy error).
  • Nothing is sent in any refused case; a listener counted 0 connections.
  • The refusal message names only the variable, never its value.
• Not covered by the sentence, but consistent with it: a bare name with no port (proxy, proxy.example) is read as http:// by both Python clients. Since the paragraph opens with "Always write the scheme", nothing needs adding.

Gates

• hugo --gc --minify in docs/: rc 0, 0 warnings and errors. All 7 in-page links on the rendered page resolve to an id.
• The checks on 0011a16 reported by GitHub: 27 pass, 5 skipping.
• The PR description is still text only, with no images and no private links.

Merge together with the paired application change. That change still needs a merge with its base branch and a full CI run before it can land.