Welcome toVigges Developer Community-Open, Learning,Share
Welcome To Ask or Share your Answers For Others

Categories

0 votes
463 views
in Technique[技术] by (71.8m points)

mysql - How to obfuscate existing mysqldump that is 20gb+

I'm working on an old Laravel project with a reasonably large data set (22Gb+) that cannot be replicated on a testing server with with migrations and seeds with faker factories.

There are daily mysqldumps performed early in the morning and stored on an s3 bucket.

I don't want to setup an extra mysqldump that obfuscates personal data and then import that to my test server as this would cause too much load on the production server. What I want is to use the test server to copy and manipulate that mysqldump and obfuscate any personal data then import it.

Options:

  • I could load the entire mysqldump into a DB and process the data. I don't like this as there would still be personal data being loaded into the DB as well as this being an expensive operation but might be my only option at this point.
  • Read stream / load file in a class then find and replace the data sets, I don't like this as loading in a 20gb+ file (even line by line) might make PHP run out of RAM and kill itself
  • ProxySQL has been suggested elsewhere but is well known not to play well with laravel

与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
Welcome To Ask or Share your Answers For Others

1 Answer

0 votes
by (71.8m points)
等待大神答复

与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
Welcome to Vigges Developer Community for programmer and developer-Open, Learning and Share
...