#10376 Investigation: Dry-running DB Migrations
Closed: Fixed 2 years ago by kevin. Opened 2 years ago by nphilipp.

@abompard and I had an interesting conversation in today’s Infra IRC meeting (from about 16:22 onwards), namely about dry-running migrations in order to spot issues with migration scripts when let loose on actual data.

One idea that came up was to have standby copies of production databases which can be used to test migrations before actually performing one in production, but we weren't sure how feasible this is. This should be investigated more thoroughly.

I noticed that the current "DB infrastructure" SOP doesn't describe migrations yet, so the points made in the conversation would be worthwhile to add there and if dry-running in standby copies (or otherwise) is considered feasible, this could be added there, too.


Metadata Update from @mohanboddu:
- Issue priority set to: Waiting on Assignee (was: Needs Review)
- Issue tagged with: low-gain, low-trouble, ops

2 years ago

So a few things:

  • we do have staging versions of most of our apps. We should test upgrades in staging before production and thus test the migration.

  • Many of our staging apps allow us to dump prod db's and load into staging before testing, which makes us more sure of data for the migration.

  • We do have backups made daily of our databases. These could be loaded into some vm and migrations tested.

I find tickets a poor place for discussion (They are much better for 'implement X' after a discussion).

Perhaps we should do a mailing list thread? Or even a short video meeting to discuss things?

@abompard suggested to put it in as a ticket, I guess to not lose track of it.

ok. I'm not sure what there is actionable here... personally, I would hope deploying on staging beforehand with a reasonably recent db sync from prod would be good enough... but if there's other actions here, let me know.

Lets go ahead and close this now, but if we want to have more discussion, open a thread on the list and we can action things out of that?

Thanks for bringing it up!

Metadata Update from @kevin:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

2 years ago

Login to comment on this ticket.

Metadata
Boards 1
ops Status: Backlog