How To: Install Cloudian with object lock for Veeam Backup & Replication 10 = Ransomwareprotection

In this blog post I’ll share how to setup Cloudian HyperStore 7.2.1, create an immutable bucket and how to make use of it in Veeam Backup & Replication v10 for testing purposes and getting your hands dirty with object lock which in essence will provide ransomware protection but first let me talk a little bit about why this is a big deal.

One of the most interesting and useful features in Veeam Backup & Replication version 10 is the “replacement for tapes”-feature called Immutability. Why does it replace tape I hear you ask. A few different reasons but the biggest to me is that you can protect your data from being tampered with just as with tape but unlike tape your data is still online and accesible. It’s built on an AWS S3 feature called Object lock, eventhough it’s originally from AWS that doesn’t mean that it’s only available for AWS users. There’s actually a growing list of object storage solutions and vendors implementing the latest AWS S3 API, both as cloud based solutions but also for on-premises solutions using either a hardware- or a software approach. Today the object lock functionality is supported by AWS S3 of course but also Zadara VPSA Object Storage (v 20.01 or later), Ceph (v14.2.6 or later) and Cloudian (v 7.2 or later). You can find the ever growing list of compatible object storage solutions that works with Veeam on this unofficial list which includes both object storage solutions with and without object lock functionality.

So let’s get back to basics, what exactly does the object lock feature do? Well, the short answer is that it write protects the data you save to on object storage solution for a period of time which you can define, making the data accessible to read (meaning online) but it cannot be changed or deleted untill that time has passed (so basically the equivalent of an offline tape but instantly accessible to recover from). “WORM” escentially, Write Once Read Many. Your data is still online and ready to be used if you need it but if your hit by some ransomware or malicious admin/hacker the data cannot be changed or deleted.

Want to kick the tires? Give it a spin? What you need: Veeam Backup & Replication v10, An Object Based storage solution, in this post I’ll be using Cloudian HyperStore, and an lab environment to deploy it all on. You will also need the AWS CLI when creating the S3 bucket in the Cloudian environment.

Download the AWS CLI, Cloudian HyperStore OVA and Veeam Backup & Replication from the links above. Install Veeam Backup & Replication on a VM.

Installing Cloudian HyperStore

I’m going to use a few new hostnames in my lab evenrionment, so first thing is to add those to my DNS server (I’m using a zone called vcsp.local where I’ll add the records):

  • cloudian01 / 192.168.50.231
  • cloudian02 / 192.168.50.232
  • cloudian03 / 192.168.50.233
  • cmc / 192.168.50.231, 192.168.50.232, 192.168.50.233
  • iam / 192.168.50.231, 192.168.50.232, 192.168.50.233
  • s3-nordics / 192.168.50.231, 192.168.50.232, 192.168.50.233
  • s3-admin / 192.168.50.231, 192.168.50.232, 192.168.50.233
  • s3-website-nordics / 192.168.50.231, 192.168.50.232, 192.168.50.233
  • sqs / 192.168.50.231, 192.168.50.232, 192.168.50.233

By adding a hostname with multiple IP address you will get a basic “load-balancer” distributing connections between the different nodes. It’s not a real load-balancer but the DNS server will resolve a hostname to a new IP address for every request it gets, this is called DNS Round-Robin. If you don’t have a DNS server you can install a lighweight DNS server as part of the Cloudian deployment called dnsmasq, when you get to the part where you install HyperStore (the step below that says “. /cloudianInstall.sh force“), replace it with the command “. /cloudInstall.sh dnsmasq force” instead.

Configure the three nodes:

  • Use Deploy OVF template in vSphere to install the Cloudian HyperStore OVA. Import the Cloudian HyperStore OVA 3 times using the names Cloudian01, Cloudian02 and Cloud03.
  • Change resource of the VMs to 8 vCPU and 16 GB RAM (which is the minimum requirement but in this limited test environment it works with fewer resources, I’ve tested with 2 vCPU and 8 GB RAM and it seems to work ok)
  • Power on VMs
  • Logon as root / password using the console
    cd CloudianTools
  • Run the following command and follow the steps
    ./system_setup.sh
  • 1) Configure Networking
  • 1) Ens160
    • Change IP address to static IP address
    • Set IP address, subnet mask, default gateway and DNS
    • Do you wish to save these settings? Yes
    • Will <IP address> be the address you use for hyperstore-ova in you survey file? Yes
    • Would you like to restart this interface to activate this new configuration? Yes
  • P) Return to the Previous Menu
  • D) Change Domain name
    • Do you want to change your domain name? Yes
    • New Domain Name: vcsp.local
  • H) Change Hostname
    • Do you want to change your hostname? Yes
    • New Hostname: cloudian01
      (cloudian02 and cloudian03 for subsequent nodes)
  • N) Restart Networking
    • Are you sure? Yes
  • P) Return to the Previous Menu
  • 2) Change Timezone
    • 8) Europe
    • 45) Sweden
    • Would you like to save this timezone setting? Yes
  • 5) Change root Password
    • 1) Change root Password
      • New Password: <new root password>
      • Retype New Password: <new root password>
    • P) Return to the Previous Menu
  • D) Download HyperStore Files (this step is only needed on the first node you install, i.e. cloudian01)
    • Select an option to download
      (if you only want to test object storage choose 1 but if you want to test object lock/immutability feature you need to choose 2)
      • 1) Download HyperStore GA files (HyperStore 7.1.7
      • 2) Download HyperStore EA files (HyperStore 7.2.1, required for Object Lock testing which is a licensed feature and not part of the trial license)
  • P) Return to the Previous Menu
  • X) Exit
  • Continue to cloudian02 and do the same steps above and then cloudian03

Install Cloudian HyperStore On Master (cloudian01 only):

  • On you PC/Mac using for instance WinSCP or CyberDuck, transfer your license file cloudian_<numbers>.lic to cloudian01 (there’s a trial license included in the download from the previous step that could be used but it doesn’t include the object lock functionality, you need to contact Cloudian to get a license to unlock that feature.
    • If needed, transfer the license file to /root/CloudianPackages/
      (or use the one already present in that folder, but again: it doesn’t provide Object Lock functionality)
  • Logon to VM using root / <new root password>
  • cd /root/CloudianPackages
  • ./CloudianHyperStore-7.2.1.bin cloudian_<numbers>.lic
  • cd /opt/cloudian-staging/7.2.1
  • ./system_setup.sh
    • 4) Setup Survey.csv File
      • Would you like to create a survey file now? Yes
      • Would you like to add entries now? Yes
      • Region Name: nordics
      • Hostname: cloudian01
      • IP Address: 192.168.50.231
      • Data Center Name: DC1
      • Rack name (all nodes in a DC must use same rack name): rac1
      • Internal Interface (optional): <skip this option by pressing enter>
      • Would you like to add another entry? Yes/No (you can have a 1 node test bed if you like or add 2 additional nodes if you’d like to test object lock)
  • P) Return to Previous Menu
    • S) Script Settings
      • 10) Generate SSH Key File
        • Install public key on cluster nodes? Yes
        • Enter password for each node
  • P) Return to the Previous Menu
    • 6) Install & Configure Prerequisites
      • 1) Install & Configure Prerequisites
        • Would you like to perform this on all nodes listed in you survey file? Yes
        • P) Return to the Previous Menu
      • R) Run Pre-installation Checks
        • 1) Quite mode: (show only warning or failed tests)
        • R) Run Pre-Install Checks
          • You will most likely get warnings that you running a virtualized environment, perhaps not enough nodes in the cluster and not enough resources allocated to the VM. If other issues show up as failed they need to be addressed before proceeding.
  • P) Return to the Previous Menu
    • W) Write sysctl Configuration
    • X) Exit
      • A reboot is required to apply some of the changes. Reboot cloudian01 now? Yes
  • Wait for cloudian01 to reboot
  • Logon as root / <new root password> using console or ssh
  • cd /opt/cloudian-staging/7.2.1/
  • . /cloudianInstall.sh force
    The “force” switch is required since we’re not using the recommended minimum resuorces, use “./cloudianInstall.sh dnsmasq force” if you need a DNS.
  • 1) Install Cloudian HyperStore
    • Please enter survey file name: /opt/cloudian-staging/7.2.1/survey.csv
    • Would you like to use key ./cloudian-installation-key? Yes
    • Please enter your top level domain name: vcsp.local
    • Please enter the service metadata replication strategy for nordics: Accept default
    • Please enter your NTP time server(s): Accept default or enter to you preferred NTP Server
    • Region [Nordics] S3 service domain URLs: Accept default
    • Region [Nordics] S3 Web site end point: Accept default
    • Admin endpoint: Accept default
    • IAM endpoint: Accept default
    • Domain name for you Cloudian Management Console service: Accept default
    • Installation will kickoff and take a couple of minutes to complete
  • 4) Advanced Configuration Options
    • e) Configure SSL for S3
      • a) Generate keystore for S3
        • Are these settings ok? Yes
      • b) Enable/Disable HTTPS for S3
        • Do you wish to enable HTTPS access on S3 server? Yes
      • x) Return to previous menu
    • x) Return to Main Menu
  • 2) Cluster Management
    • b) Push Configuration Settings to Cluster
      • Enter a comma-separated list of hosts in nordics to execute agents on? [Empty for all]: Press enter
    • c) Manage Services
      • 5) S3 service
        • Type restart and press enter
      • X) Quit
    • D) Run validation tests
    • x) Return to Main Menu
  • x) Exit
  • Close the session
  • Open a browser: https://cmc.vcsp.local:8443/
    • Login using admin / public
    • Click the top link displayed: No Storage Policies have been defined. Please create a Storage Policy to create a storage policy
  • Click + Create Storage Policy in the top right corner
    • Give it a Policy Name and accept all other default settings
    • Click Save
  • Click Users & Groups in the top bar
    • Click Manage Groups in in the top
    • Click + New Group
    • Give it a Group Name: Backupusers
    • Click Save
  • Click Manage Users in the top bar
  • Click + New user in the top right corner
    • Give it a User ID: veeam_backup_user
    • Add a password (min 9 chars)
    • Assign the Group Name created in earlier step: Backupusers
  • In the field “Search For A User By ID:” type veeam_backup_user and click search
    • Click Security Credentials for the user veeam_backup_user
    • Copy Access Key ID and then click View Secret Key to access and copy it for use later
  • Click Close
  • Sign out of the CMC GUI

Enable object lock on Cloudian

  • For more detail on Object lock you should read the document Cloudian-QuickStartGuide-Object-Lock.pdf, below is a summary of the steps outlined in that document that needs to be taken
    • Log into the Puppet Master node (should be cloudian01) as the root user.
    • Check to confirm that the HSH is currently disabled.
      • [root@cloudian01]# hsctl config get hsh.enabled
        False
    • Set hsh.enabled to true.
      • [root@cloudian01]# hsctl config set hsh.enabled=true
    • Push the configuration change out to the cluster.
      • [root@cloudian01]# hsctl config apply hsh
    • Confirm that HSH is now enabled.
      • [root@cloudian01]# hsctl config get hsh.enabled
        True
  • HSH is now enabled in your system, but no users are yet able to log into it. To provision the default admin user for HSH do the following steps:
    • log into the CMC as the admin user with password public
    • Change the “admin” user’s password. in the top right corner, Admin->Security Credentials. This password change causes the system to create a corresponding HSH user.
  • Once an HSH user has been created, that user can use SSH to log into any HyperStore node. Prefix sa_ should be applied to the admin account when logging on, so user should be sa_admin and password should be <new root password>. The prompt will appear as follows:
    sa_admin@cloudian01$
    • You can confirm that you are in the HyperStore shell by typing help:
      sa_admin@cloudian01$ help
    • Type exit and press enter to end session
  • Log on to cloudian01 via console or ssh using root with <new root password>
  • To disable root password access to all HyperStore nodes:
    • cd /opt/cloudian-staging/7.2.1
    • ./cloudianInstall.sh
    • 4) Advanced Configuration Options
      • m) Disable the root password
        • Do you wish to disable the root password on all Cluster nodes? Yes
      • X) Return to Main Menu
    • X) Exit
    • Type exit and press enter
    • Try to logon again using console or ssh and verify the root is no longer able to logon.
  • To create a bucket with object lock this must be done using an API or using AWS command line interface, it can’t be done from the Cloudian CMC.
    On a management PC, download AWS S3 CLI to create a bucket with object lock.
    • Start a command line interface (cmd) and type:
      aws configure
      (access key and secret key from veeam_backup_user will be requested)
    • Next type:
      aws s3api --endpoint-url https://s3-nordics.vcsp.local --no-verify-ssl create-bucket --object-lock-enabled-for-bucket --bucket veeam_backup_bucket0001
    • Verify bucket creation and settings:
      aws s3api --endpoint-url https://s3-nordics.vcsp.local --no-verify-ssl get-object-lock-configuration --bucket veeam_backup_bucket0001

Create Veeam components

  • Logon to Veeam Backup & Replication server
    • Start the Veeam Backup & replication GUI
      • Go to the Backup Infrastructure tab (bottom left)
        • Click Backup Repositories (top left)
        • Click Add Repository
          • Click Object Storage
          • Click S3 Compatible
          • Give it a name: Cloudian-nordics-with-object-lock and click Next
          • Enter Service Point: s3-nordics.vcsp.local
          • In the Credentials field, click Add
          • Add Access key and Secret Key copied from earlier steps for user veeam_backup_user
          • Click OK
          • Click Next
          • Click Continue for the Certificate Security Alert
          • In the Folder field, Click Browse… and create a folder for the backups
          • Click New folder and specify a name
          • Click OK
          • Set any restrictions to use:
            Limit object storage consumption to:
            Make recent backups immutable for:
  • Click Next
  • Click Finish
  • Create a new repository to be used as the performance tier of a new Scale-Out Backup repository
  • Create a new Scale-Out Backup Repository
    • The steps involved can be found on the Veeam HelpCenter pages
      • Select performance tier just created
      • Select capacity tier: Cloudian-nordics-with-object-lock
        • Make sure to click “Copy backups to object storage as soon as the are created!
  • Create a backup job and use Scale-Out backup repository above as target, and start it.

    Verify Object lock
    • Once backup job is finished
      • Go to Home
        • Under Backups find Object storage
        • Right click the backup job and select delete from disk
        • If everything is configured correctly you should get a failed attempt!

By using Veeam Backup & Replication version 10 in combination with a Scale-Out backup repository including an object based storage solution, we can make sure that our valuable data is protected, we get 2 backup copies automatically when using copy-mode, we get a second media type and we get a write protected copy with the immutability option. So in a single job we can actually adhere to the design princple we’ve talked about for a long time called the 3-2-1-rule. How cool is that!

So what we’ve now established is a solid solution that will protect your data no matter if it’s from malicious insiders or ransomware!

In the next blog post will be a follw up on this post where I’ll show you how easy it it’s to recover from a disaster including the Veeam Backup & Replication server, a total site failure. I’ll show you that as long as you have a copy of you backup available in object storag solution (not part of the site that filed of course), you can recover!

Replica seeding to vCloud Director

One of the many use cases for Veeam Backup & Replication is disaster recovery, as the name of the product suggests it can certainly replicate virtual machines from a production environment to a secondary- or disaster recovery environment. While it is a very straight forward process running through a wizard selecting source and target environments and the start replicating the VM cross the network, you can even have your virtual machines replicated to a Veeam Cloud & Service Provider, VCSP, if you don’t have a disaster recovery site of your own. The VCSP can have a hypervisor environment built for either Microsoft Hyper-V, VMware vSphere or VMware vCloud Director. VMware vCloud Director is VMware’s multi-tenant solution to host Infrastructure as a Service and purpose built specifically for Service Providers.

In this post I’m describing the process of replicating VMs to a VCSP using a feature of Veeam Backup & Replication called Cloud Connect, I’m not going through how to setup Cloud Connect. If you need more information about the ins and outs of Cloud Connect please visit Luca Dell’Oca’s webpage about Cloud Connect.

In the hosted environment at the VCSP you can power on virtual machines if needed to keep your business going if there’s a catastrophic event at your own site for instance a lengthy power outage, load shedding, you can even create a fail-over plan dictating which virtual machines should be powered on and in which order they should start, making sure everything starts in the correct order.

Replicating over the network may not be optimal in all scenarios, at least not the first initial full replication cycle. Let’s say you have a few very large virtual machines that you want to protect by sending them to a disaster recovery site hosted by your Veeam Cloud Service Provider but it’s too big to actually be transferred over the network within the available backup window, what do you do?

In Veeam Backup & Replication you can seed an initial copy of the virtual machine to your service provider using some sort of transportable solution. USB drives, Tapes or solutions of that nature – using “sneaker net”. The basic concept is to get a copy of the virtual machine to the service provider so they can import the VM to their environment and when you start replicating over the network you just send the changes made to the VM that has occurred since you made the copy of the VM. No need for a full transfer of the VM cross the network!

So the 3 basic steps that needs to be taken:

  • Backup VM to a transportable storage device and send it to VCSP
  • The VCSP imports the VM to the correct Org vDC in vCloud Director
  • Set up a replication job at the customer site using the imported VM at the VSCP site as mapping VM

If the service provider has a multi-tenant virtualization layer, meaning built on VMware vCloud Director, the process is simple but has to been broken down into a few distinct steps. If you as a service provider are using VMware vCloud Director 9.7, these are the steps you take if the customer has Veem Backup & Replication installed that can be used:

Step 1 – Customer environment
Backup source VM (normal backup job or VeeamZip) to a portable storage solution. Either backup to C:\Backup and move the backupfile manually to the USB device or select “VeeamZIP…” and specify the target USB devices directly.
VeeamZIP

Step 2 – Customer environment

Step 3 – Customer environment

Step 4 – Customer environment
When the backup is completed it should be visible in the “Backups”-section in “Disk (VeeamZIP)”

Step 5 – Customer environment
Transfer the backupfile using a transportable storage solution (a USB drive can be used)

Step 6 – VCSP environment
Connect USB drive and import backup file to Veeam Backup & Replication running at the VCSP data center. Click “Import Backup” in the top section.

Step 7 – VCSP environment
Select the backupfile on the USB device and click “Open” (you may need to change the file type selector to “Backup files (*.vbk)” to see the backupfile.

Step 8 – VCSP environment
Now Veeam Backup & Replication will import the backup

Step 9 – VCSP environment
Right click the VM from the imported backup and select “Restore entire VM…”

Step 10 – VCSP environment

Step 11 – VCSP environment
Select “Restore to a new location, or with different settings”

Step 12 – VCSP environment
Click “Host…”

Step 13 – VCSP environment
Select a host or a cluster that is under vCloud Director management where the customer has a virtual datacenter (shows up as a resource source pool in the next few steps)

Step 14 – VCSP environment
Select the VM and click “Pool…”

Step 15 – VCSP environment
Select resource pool (Org vDC of the customer)

Step 16 – VCSP environment

Step 17 – VCSP environment

Step 18 – VCSP environment
Map network adapter to desired network in the Org vDC

Step 19 – VCSP environment

Step 20 – VCSP environment

Step 21 – VCSP environment

Step 22 – VCSP environment
Log on to vCloud Director using the flex UI (the HTML5 UI lacks the “import from vSphere” option.

If you as a service provider are using VMware vCloud Director 10 with the new HTML5 UI for providers, please note that “import from vSphere” is not available in the H5 UI. What’s even more annoying is that the flex UI has also been disable by default in vCD 10 so to be able to import the VM into the Org vDC of the customer you first need to enable the flex UI of vCD:

Enable the vCloud Director Web Console

Step 23 – VCSP environment
Import VM in vCD from vSphere

Step 24 – VCSP environment
Select “Move VM” and not “Copy VM” in Import wizard

Step 25 – VCSP environment

Step 26 – VCSP environment

Step 27 – Customer environment
Set up a new replication job at customer side

Step 28 – Customer environment
Select “Replica seeding (for low bandwidth DR sites)”

Step 29 – Customer environment
Select the source VM from the customer production hypervisor (the same used in step 1)

Step 30 – Customer environment
In the “Destination”-tab, for the “Host or cluster:”-selection. Choose “Cloud host…”

Step 31 – Customer environment
Select the Org vDC to use (same as in step 15)

Step 32 – Customer environment
Select vApp and Storage policy to be used

Step 33 – Customer environment
Select desired restore points to keep

Step 34 – Customer environment
Select desired replication mode

Step 35 – Customer environment
In the “Seeding”-tab. In the “Replica mapping” section. Select “Map replicas to exsiting VMs”, click on the VM and select edit.

Step 36 – Customer environment
Select the seeded VM from step 17

Step 37 – Customer environment

Step 38 – Customer environment
Set a desired replication schedule

Step 39 – Customer environment
If desired: Click “Run the job when I click Finish”
Click “Finish”

Step 40 – Customer environment
Verify that replication successfully finish

Step 41 – Customer environment
The replication job only transfers changed blocks since the backup/import was made