Linux PIPE, BZIP2 in parallel, multiple threads

I am getting old, my brain aint all that bright no more, for example i am so used to using

mysqldump –opt -u root –password=”LaooIa12@Hsu” mytodolist | gzip > mytodolist.bin.gz

So to compress the database while dumping it without having it hit the disk first in what is called piping.

the problem is that i didn’t think of checking whether pbzip2 supports this so that i can use all 6 CPU cores (12 vCPUs)

So, turns out that pbzip2 (parallel BZIP) does work

mysqldump –opt -u root –password=”LaooIa12@Hsu” mytodolist | pbzip2 -vc > mytodolist.bin.bz2

Now, i can get it done much much faster, 6-12 times faster

Leave a Reply

Your email address will not be published. Required fields are marked *