Month: March 2020

In this blog post I’ll share how to setup Cloudian HyperStore 7.2.1, create an immutable bucket and how to make use of it in Veeam Backup & Replication v10 for testing purposes and getting your hands dirty with object lock which in essence will provide ransomware protection but first let me talk a little bit about why this is a big deal.

One of the most interesting and useful features in Veeam Backup & Replication version 10 is the “replacement for tapes”-feature called Immutability. Why does it replace tape I hear you ask. A few different reasons but the biggest to me is that you can protect your data from being tampered with just as with tape but unlike tape your data is still online and accesible. It’s built on an AWS S3 feature called Object lock, eventhough it’s originally from AWS that doesn’t mean that it’s only available for AWS users. There’s actually a growing list of object storage solutions and vendors implementing the latest AWS S3 API, both as cloud based solutions but also for on-premises solutions using either a hardware- or a software approach. Today the object lock functionality is supported by AWS S3 of course but also Zadara VPSA Object Storage (v 20.01 or later), Ceph (v14.2.6 or later) and Cloudian (v 7.2 or later). You can find the ever growing list of compatible object storage solutions that works with Veeam on this unofficial list which includes both object storage solutions with and without object lock functionality.

So let’s get back to basics, what exactly does the object lock feature do? Well, the short answer is that it write protects the data you save to on object storage solution for a period of time which you can define, making the data accessible to read (meaning online) but it cannot be changed or deleted untill that time has passed (so basically the equivalent of an offline tape but instantly accessible to recover from). “WORM” escentially, Write Once Read Many. Your data is still online and ready to be used if you need it but if your hit by some ransomware or malicious admin/hacker the data cannot be changed or deleted.

Want to kick the tires? Give it a spin? What you need: Veeam Backup & Replication v10, An Object Based storage solution, in this post I’ll be using Cloudian HyperStore, and an lab environment to deploy it all on. You will also need the AWS CLI when creating the S3 bucket in the Cloudian environment.

Download the AWS CLI, Cloudian HyperStore OVA and Veeam Backup & Replication from the links above. Install Veeam Backup & Replication on a VM.

Installing Cloudian HyperStore

I’m going to use a few new hostnames in my lab evenrionment, so first thing is to add those to my DNS server (I’m using a zone called vcsp.local where I’ll add the records):

• cloudian01 / 192.168.50.231
• cloudian02 / 192.168.50.232
• cloudian03 / 192.168.50.233
• cmc / 192.168.50.231, 192.168.50.232, 192.168.50.233
• iam / 192.168.50.231, 192.168.50.232, 192.168.50.233
• s3-nordics / 192.168.50.231, 192.168.50.232, 192.168.50.233
• s3-admin / 192.168.50.231, 192.168.50.232, 192.168.50.233
• s3-website-nordics / 192.168.50.231, 192.168.50.232, 192.168.50.233
• sqs / 192.168.50.231, 192.168.50.232, 192.168.50.233

By adding a hostname with multiple IP address you will get a basic “load-balancer” distributing connections between the different nodes. It’s not a real load-balancer but the DNS server will resolve a hostname to a new IP address for every request it gets, this is called DNS Round-Robin. If you don’t have a DNS server you can install a lighweight DNS server as part of the Cloudian deployment called dnsmasq, when you get to the part where you install HyperStore (the step below that says “. /cloudianInstall.sh force“), replace it with the command “. /cloudInstall.sh dnsmasq force” instead.

Configure the three nodes:

• Use Deploy OVF template in vSphere to install the Cloudian HyperStore OVA. Import the Cloudian HyperStore OVA 3 times using the names Cloudian01, Cloudian02 and Cloud03.

• Change resource of the VMs to 8 vCPU and 16 GB RAM (which is the minimum requirement but in this limited test environment it works with fewer resources, I’ve tested with 2 vCPU and 8 GB RAM and it seems to work ok)

• Power on VMs
• Logon as root / password using the console
  cd CloudianTools
• Run the following command and follow the steps
  ./system_setup.sh

• 1) Configure Networking

• 1) Ens160
  • Change IP address to static IP address
  • Set IP address, subnet mask, default gateway and DNS
  • Do you wish to save these settings? Yes
  • Will <IP address> be the address you use for hyperstore-ova in you survey file? Yes
  • Would you like to restart this interface to activate this new configuration? Yes

• P) Return to the Previous Menu
• D) Change Domain name
  • Do you want to change your domain name? Yes
  • New Domain Name: vcsp.local
• H) Change Hostname
  • Do you want to change your hostname? Yes
  • New Hostname: cloudian01
    (cloudian02 and cloudian03 for subsequent nodes)
• N) Restart Networking
  • Are you sure? Yes
• P) Return to the Previous Menu
• 2) Change Timezone
  • 8) Europe
  • 45) Sweden
  • Would you like to save this timezone setting? Yes
• 5) Change root Password
  • 1) Change root Password
    • New Password: <new root password>
    • Retype New Password: <new root password>
  • P) Return to the Previous Menu
• D) Download HyperStore Files (this step is only needed on the first node you install, i.e. cloudian01)
  • Select an option to download
    (if you only want to test object storage choose 1 but if you want to test object lock/immutability feature you need to choose 2)
    • 1) Download HyperStore GA files (HyperStore 7.1.7
    • 2) Download HyperStore EA files (HyperStore 7.2.1, required for Object Lock testing which is a licensed feature and not part of the trial license)

• P) Return to the Previous Menu
• X) Exit
• Continue to cloudian02 and do the same steps above and then cloudian03

Install Cloudian HyperStore On Master (cloudian01 only):

• On you PC/Mac using for instance WinSCP or CyberDuck, transfer your license file cloudian_<numbers>.lic to cloudian01 (there’s a trial license included in the download from the previous step that could be used but it doesn’t include the object lock functionality, you need to contact Cloudian to get a license to unlock that feature.
  • If needed, transfer the license file to /root/CloudianPackages/
    (or use the one already present in that folder, but again: it doesn’t provide Object Lock functionality)
• Logon to VM using root / <new root password>
• cd /root/CloudianPackages
• ./CloudianHyperStore-7.2.1.bin cloudian_<numbers>.lic

• cd /opt/cloudian-staging/7.2.1
• ./system_setup.sh
  • 4) Setup Survey.csv File
    • Would you like to create a survey file now? Yes
    • Would you like to add entries now? Yes
    • Region Name: nordics
    • Hostname: cloudian01
    • IP Address: 192.168.50.231
    • Data Center Name: DC1
    • Rack name (all nodes in a DC must use same rack name): rac1
    • Internal Interface (optional): <skip this option by pressing enter>
    • Would you like to add another entry? Yes/No (you can have a 1 node test bed if you like or add 2 additional nodes if you’d like to test object lock)

• P) Return to Previous Menu
  • S) Script Settings
    • 10) Generate SSH Key File
      • Install public key on cluster nodes? Yes
      • Enter password for each node

• P) Return to the Previous Menu
  • 6) Install & Configure Prerequisites
    • 1) Install & Configure Prerequisites
      • Would you like to perform this on all nodes listed in you survey file? Yes
      • P) Return to the Previous Menu
    • R) Run Pre-installation Checks
      • 1) Quite mode: (show only warning or failed tests)
      • R) Run Pre-Install Checks
        • You will most likely get warnings that you running a virtualized environment, perhaps not enough nodes in the cluster and not enough resources allocated to the VM. If other issues show up as failed they need to be addressed before proceeding.

• P) Return to the Previous Menu
  • W) Write sysctl Configuration
  • X) Exit
    • A reboot is required to apply some of the changes. Reboot cloudian01 now? Yes
• Wait for cloudian01 to reboot
• Logon as root / <new root password> using console or ssh
• cd /opt/cloudian-staging/7.2.1/
• . /cloudianInstall.sh force
  The “force” switch is required since we’re not using the recommended minimum resuorces, use “./cloudianInstall.sh dnsmasq force” if you need a DNS.

• 1) Install Cloudian HyperStore
  • Please enter survey file name: /opt/cloudian-staging/7.2.1/survey.csv
  • Would you like to use key ./cloudian-installation-key? Yes
  • Please enter your top level domain name: vcsp.local
  • Please enter the service metadata replication strategy for nordics: Accept default
  • Please enter your NTP time server(s): Accept default or enter to you preferred NTP Server
  • Region [Nordics] S3 service domain URLs: Accept default
  • Region [Nordics] S3 Web site end point: Accept default
  • Admin endpoint: Accept default
  • IAM endpoint: Accept default
  • Domain name for you Cloudian Management Console service: Accept default
  • Installation will kickoff and take a couple of minutes to complete

• 4) Advanced Configuration Options
  • e) Configure SSL for S3
    • a) Generate keystore for S3
      • Are these settings ok? Yes
    • b) Enable/Disable HTTPS for S3
      • Do you wish to enable HTTPS access on S3 server? Yes
    • x) Return to previous menu
  • x) Return to Main Menu
• 2) Cluster Management
  • b) Push Configuration Settings to Cluster
    • Enter a comma-separated list of hosts in nordics to execute agents on? [Empty for all]: Press enter
  • c) Manage Services
    • 5) S3 service
      • Type restart and press enter
    • X) Quit
  • D) Run validation tests
  • x) Return to Main Menu
• x) Exit
• Close the session
• Open a browser: https://cmc.vcsp.local:8443/
  • Login using admin / public
  • Click the top link displayed: No Storage Policies have been defined. Please create a Storage Policy to create a storage policy

• Click + Create Storage Policy in the top right corner
  • Give it a Policy Name and accept all other default settings
  • Click Save
• Click Users & Groups in the top bar
  • Click Manage Groups in in the top
  • Click + New Group
  • Give it a Group Name: Backupusers
  • Click Save

• Click Manage Users in the top bar
• Click + New user in the top right corner
  • Give it a User ID: veeam_backup_user
  • Add a password (min 9 chars)
  • Assign the Group Name created in earlier step: Backupusers

• In the field “Search For A User By ID:” type veeam_backup_user and click search
  • Click Security Credentials for the user veeam_backup_user
  • Copy Access Key ID and then click View Secret Key to access and copy it for use later

• Click Close
• Sign out of the CMC GUI
  

Enable object lock on Cloudian

• For more detail on Object lock you should read the document Cloudian-QuickStartGuide-Object-Lock.pdf, below is a summary of the steps outlined in that document that needs to be taken
  • Log into the Puppet Master node (should be cloudian01) as the root user.
  • Check to confirm that the HSH is currently disabled.
    • [root@cloudian01]# hsctl config get hsh.enabled
      False
  • Set hsh.enabled to true.
    • [root@cloudian01]# hsctl config set hsh.enabled=true
  • Push the configuration change out to the cluster.
    • [root@cloudian01]# hsctl config apply hsh
  • Confirm that HSH is now enabled.
    • [root@cloudian01]# hsctl config get hsh.enabled
      True

• HSH is now enabled in your system, but no users are yet able to log into it. To provision the default admin user for HSH do the following steps:
  • log into the CMC as the admin user with password public
  • Change the “admin” user’s password. in the top right corner, Admin->Security Credentials. This password change causes the system to create a corresponding HSH user.

• Once an HSH user has been created, that user can use SSH to log into any HyperStore node. Prefix sa_ should be applied to the admin account when logging on, so user should be sa_admin and password should be <new root password>. The prompt will appear as follows:
  sa_admin@cloudian01$
  • You can confirm that you are in the HyperStore shell by typing help:
    sa_admin@cloudian01$ help
  • Type exit and press enter to end session

• Log on to cloudian01 via console or ssh using root with <new root password>
• To disable root password access to all HyperStore nodes:
  • cd /opt/cloudian-staging/7.2.1
  • ./cloudianInstall.sh
  • 4) Advanced Configuration Options
    • m) Disable the root password
      • Do you wish to disable the root password on all Cluster nodes? Yes
    • X) Return to Main Menu
  • X) Exit
  • Type exit and press enter
  • Try to logon again using console or ssh and verify the root is no longer able to logon.

• To create a bucket with object lock this must be done using an API or using AWS command line interface, it can’t be done from the Cloudian CMC.
  On a management PC, download AWS S3 CLI to create a bucket with object lock.
  • Download the AWS CLI from: https://aws.amazon.com/cli/ and install where suitable.
  • Start a command line interface (cmd) and type:
    aws configure
    (access key and secret key from veeam_backup_user will be requested)
  • Next type:
    aws s3api --endpoint-url https://s3-nordics.vcsp.local --no-verify-ssl create-bucket --object-lock-enabled-for-bucket --bucket veeam_backup_bucket0001
  • Verify bucket creation and settings:
    aws s3api --endpoint-url https://s3-nordics.vcsp.local --no-verify-ssl get-object-lock-configuration --bucket veeam_backup_bucket0001

Create Veeam components

• Logon to Veeam Backup & Replication server
  • Start the Veeam Backup & replication GUI
    • Go to the Backup Infrastructure tab (bottom left)
      • Click Backup Repositories (top left)
      • Click Add Repository
        • Click Object Storage
        • Click S3 Compatible
        • Give it a name: Cloudian-nordics-with-object-lock and click Next
        • Enter Service Point: s3-nordics.vcsp.local
        • In the Credentials field, click Add
        • Add Access key and Secret Key copied from earlier steps for user veeam_backup_user
        • Click OK
        • Click Next
        • Click Continue for the Certificate Security Alert
        • In the Folder field, Click Browse… and create a folder for the backups
        • Click New folder and specify a name
        • Click OK
        • Set any restrictions to use:
          Limit object storage consumption to:
          Make recent backups immutable for:

• Click Next
• Click Finish

• Create a new repository to be used as the performance tier of a new Scale-Out Backup repository
  • The steps involved can be found on the Veeam HelpCenter pages
• Create a new Scale-Out Backup Repository
  • The steps involved can be found on the Veeam HelpCenter pages
    • Select performance tier just created
    • Select capacity tier: Cloudian-nordics-with-object-lock
      • Make sure to click “Copy backups to object storage as soon as the are created!

• Create a backup job and use Scale-Out backup repository above as target, and start it.
  
  Verify Object lock
  • Once backup job is finished
    • Go to Home
      • Under Backups find Object storage
      • Right click the backup job and select delete from disk
      • If everything is configured correctly you should get a failed attempt!

By using Veeam Backup & Replication version 10 in combination with a Scale-Out backup repository including an object based storage solution, we can make sure that our valuable data is protected, we get 2 backup copies automatically when using copy-mode, we get a second media type and we get a write protected copy with the immutability option. So in a single job we can actually adhere to the design princple we’ve talked about for a long time called the 3-2-1-rule. How cool is that!

So what we’ve now established is a solid solution that will protect your data no matter if it’s from malicious insiders or ransomware!

In the next blog post will be a follw up on this post where I’ll show you how easy it it’s to recover from a disaster including the Veeam Backup & Replication server, a total site failure. I’ll show you that as long as you have a copy of you backup available in object storag solution (not part of the site that filed of course), you can recover!