This week I will continue with Amazon Web Services Storage Gateway maintenance tasks. In my previous post, I had explained how to configure a gateway-cached volume using Amazon Web Services Storage Gateway service. Today I will explain how to manage our disk capacities and how to resize our buffer,cache and storage volumes. Let’s start by storage volume.
Resize storage volume: Unfortunately resizing a storage volume is not supported by AWS. However there is a workaround to resize it.
- First we need to create a snapshot of the volume
- Then we create a new volume and preserve data using the snapshot id
I create the snapshot of my 500 GiB volume using AWS console.
I use the snapshot id ( snap-6ce005e8 ) and create a new 600 GiB volume and named it as awsbackup01.
Now I attach it using iscsi initiator using its target name iqn.1997-05.com.amazon:awsbackup01
As expected it is inactive now, I connect to it.
I rescan disks and successfully access the new volume. Since we used a 500GiB snapshot , the 100GiB part is unallocated.
I bring the disk online then extend it using 100GiB unallocated space.
Ok now I finished operations with my new volume. Next is deleting the old one.
I disconnect the old volume using iscsi initiator.
Then I delete the old volume using AWS console.
And this was the last step. We successfully resized our storage volume and now it’s up to you configuring your applications to use this disk.
Resize buffer volume: Before explaining how to resize buffer volume, let’s talk about AWS buffer volume calculation formula.
I will give a real calculation example using my veeam backup report.
I had attached my 500GiB volume to veeam server and backed up one of my servers. Since this was the first ( full ) backup it copied all of the vm image data. Now I use these values and calculate my buffer volume.
Application throughput -> 38Mb/s ( Veeam processing rate )
Network throughput -> 20Mb/s ( First it was 100Mb/s , then I decreased it to see how I fill the upload buffer )
Compression Factor -> 1x
Duration of writes -> 1:12 hours , let’s say 1 hour
((38 MB/sec) – (20 MB/sec * 1)) * (1 hours * 3600 seconds/hour) = 64800 megabytes . So I need minimum 65GiB buffer space. ( I use 150 GiB as AWS suggested ). If we check also Cloudwatch metrics I can see that there was still free space of my upload buffer when the backup job was running. I had also configured SNS notifications for my volumes’ usages.
You can use sizing formula and cloudwatch metrics ( also SNS notifications for usage alarms ) to size your upload buffer volume. Also I suggest you to also check the volume status of your storage volumes. If your upload buffer fills up , the volume status will be “Pass through” which means there is no enough space and data is not uploaded to AWS. If the volume status is “Bootstrapping” , it means the data is being uploaded and if the status is “Active” , the normal operations is resuming.
I don’t show how to add/remove upload buffer volumes but need to say some important tips.
To add capacity ( add new upload buffer disk ) , the storage gateway virtual machine needs to be “Powered ON”.
To remove capacity ( remove buffer disk ) , the storage gateway virtual machine needs to be “Powered OFF”.
So the steps should be as below:
- Find the device node using AWS console ( like SCSI (0:0) )
- Shutdown Storage gateway virtual machine
- Remove disk using vmware console ( I use vmware )
- Power on Storage gateway virtual machine
- Add new disk using console ( as in my previous post )
If you want to read more you can visit this link.
Resize cache volume: For the cache volume, AWS recommends at lease 1:1 of upload buffer volume. Here there is a problem I should tell you. If your data is filled up in your cache storage ( it is called dirty data, data not uploaded to AWS ). the write operations are BLOCKED. This means you should monitor and resize your cache storage as your data grows.
Again I don’t show how to add cache volumes but need to say some important tips.
To add capacity ( add new cache disk ) , the storage gateway virtual machine can be “Powered ON”.
Remove cache capacity is not allowed.
I hope this was helpful. In my next post, I will try to explain Storage gateway monitoring . If you have any question or comment, please feel free to write and don’t forget to share please.