Skip to content

feat(postgres): set up dual-writes - #6316

Open
michaelkedar wants to merge 2 commits into
google:masterfrom
michaelkedar:💽✍️✍️

Hidden character warning

The head ref may contain hidden characters: "\ud83d\udcbd\u270d\ufe0f\u270d\ufe0f"
Open

michaelkedar wants to merge 2 commits into
google:masterfrom
michaelkedar:💽✍️✍️

Conversation

@michaelkedar

Copy link
Copy Markdown
Member

(Sorry this PR is so big, a lot of things ended up having to be done at the same time)
Set up dual-writing infrastructure for Postgres in the places where vulns get written:

  • Implemented the remaining VulnerabilityStore for postgres (splitting the queries.sql into files corresponding to the go files they're used in)
  • Propagated data needed from importer (and recoverer) to worker to enable raw_vulnerabilities and historic_vulnerabilities to be written with all needed fields
    • Doing this meant passing a compressed JSON object instead of a compressed proto over the Pub/Sub for the raw vulnerability. We need to make sure there are no jobs in Pub/Sub while this is being merged (i.e. pause the importers).
  • Implemented Dual-write wrapper, to write to Datastore/GCS, then Postgres. Using this for workers and vanir signatures.
  • Updated relations cron-job to also mutate in Postgres (this wasn't using the database interfaces, and I have plans to change how this is done when postgres becomes the source of truth)
  • Configured all the GKE images with the POSTGRES_* env vars (I've already set up the GKE secret for the db password)
  • tests and such

I may have to make a follow-up PR fairly quickly to bump the CPUs of the postgres if affected commits are too plentiful.

In a future PR, I plan to re-jig the tests so they spin up postgres in a docker container instead of trying to mock all the endpoints, so we can make sure our queries actually work properly.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant