Portworx Backup cross site recovery

jboothomas
3 min readNov 23, 2020

In this blog I’ll cover how to use FlashBlade S3 replication with Portworx Backup to recover from a site failure.

Portworx Backup can use Pure Storage FlashBlade as a repository for backup images, providing the fastest time to recovery possible. Leveraging FlashBlade replication functionalities we can configure the recovery of an entire Portworx Backup dataset to an alternate site. So without further ado let’s get cross site recovery setup.

I have two sites (1 & 2) and within each site I have a running Kubernetes cluster each with an installation of portworx-store and portworx-backup. I also have a FlashBlade available at each site.

High level architecture

I first configure the buckets to use with px-backup, as per the diagram on site1 I configure a px-bckup-site1 and a px-bckup-site2rep bucket, and the opposite on site2.

I then configure replication from site1 to the site1rep bucket on site2. This is all well documented but the steps are: add the remote S3 account, followed by adding a replication ‘link’ (source — target — s3account).

Once done I have on the site1 FlashBlade:

and on the site2 FlashBlade:

Next we add for each site the local bucket to our portworx-backup, to be used as the main backup image repository for our daily backup jobs. Within our cloud settings we have for each site the local FlashBlade S3 account and the local bucket.

cloud settings (for site2)

I then create a backup job for an Elasticsearch cluster running on site1, I configured pre rules to freeze, flush and perform an Elasticsearch snapshot to a shared PVC prior to the backup job running, and a post rule to unfreeze the indice once the backup job is complete. As per the Portworx documentation: https://portworx.com/how-to-backup-and-restore-elasticsearch-on-kubernetes/

Using an S3 browser I can check my replicated data between the two sites:

local and remote replica buckets

I let the job run hourly, we can see all the backup images created:

Disaster strikes and site1 is down ! and I must recover my elastic search cluster from site1.

Within portworx backup on site2, all I have to do is add the replicated bucket for site1, my “pxbckup-site1rep” bucket.

Within px-central interface I can then go to my ‘All Backups’ and browse the backup images that are indexed from the added bucket (see the site1rep as the location).

imported images from replicated bucket

So simple !

All that is left to do is perform a restore operation from the desired image and portworx backup will recover my ElasticSearch from the replicated image data into site2.

restore job settings

The combination of Portworx backup and FlashBlade enables inter site recovery mechanisms, so you can safely and simply get your Kubernetes workloads back up and running.

--

--

jboothomas

Infrastructure engineering for modern data applications