Migrating from a Windows ZVM to a Linux ZVMA

Zerto 9.7 is the final version of Zerto that supports running a ZVM from a Windows virtual machine. Beginning with Zerto 10.X, Zerto will run in a lightweight Debian Linux-based ZVM Appliance (or ZVMA). Because Zerto 10.X only runs on the Linux ZVMA, upgrading from 9.7 to 10.X requires a full “lift-and-shift” migration of the ZVM data from the Windows ZVM onto the Linux ZVMA. This whole process can be broken down into three phases, as detailed below.

While the ZVM appliance itself, and the management UI it provides, will still function the same way as previous versions – there is one significant change to the authentication. While the Windows ZVM depended on the vCenter it was bound to for LDAP authentication, the Linux ZVMA will require manually configuring LDAP for AD authentication into the ZVMA management web UI, or the use of a generic local admin account. If you would like to authenticate into the web UI of your private ZVMA using your private domain credentials, you will need to reference Zerto’s documentation for configuring Keycloak, linked at the bottom of this document. This step occurs after the full migration has been completed.

Note: This document is a guide. Zerto’s public documentation goes into more detail for certain steps. Please refer to the vendor’s documentation for further assistance alongside this document.

Overview

The migration can be broken down into these phases:

Prerequisites and Documentation

Prerequisites

There are few strict prerequisites required to be in place before the migration from the ZVM to the ZVMA can occur, and there are some additional steps we can perform to prepare. Please validate the following:

• Windows ZVM must be on 9.7u3 or above, and all VRAs must be upgraded to the latest version available.
  If the ZVM is not on 9.7u3 or above, it can NOT be migrated. Your Windows ZVM must be upgraded to 9.7u3 or above first before you can proceed with this document.

• TWO additional IP addresses from the subnet the Windows ZVM uses.
  Two IP addresses are needed to complete the whole process: These IPs must be from the same subnet the existing ZVM is currently using. After the migration is completed, the temporary IP addresses are no longer needed.

  • ZVMA IP Address: 1 IP will be used for the Linux ZVMA on initial deployment, which is eventually replaced by the Windows ZVM’s permanent IP address after the cutover.

  • Temp. Windows ZVM IP Address: 1 IP will be provided to the Migration Tool during the actual migration. The Windows ZVM is assigned this temp. IP during the cutover, so that it can be connected to while the migration is taking place. It will remain on the Windows ZVM after the cutover.

• Migration Tool should be downloaded to the Windows ZVM

• New accounts are documented internally. These new accounts are listed in the table below.

• Plan any necessary adjustments to monitoring, as all Zerto workloads will be transferred to the new Linux ZVM Appliance and the Windows ZVM will be deprecated.

• Plan any necessary adjustments to backup solutions, since this will be a brand new virtual appliance.

• The following subnet must not be used in your environment: 10.217.229.0/24. This is used for internal ZVM Appliance container communication.

  If it is in use, it will result in a migration error later. Please refer to the bottom of this document, or the Zerto Migration Tool document, for resolution steps.

Documentation

The ZVMA introduces new accounts which must be documented internally within your organization. These accounts will be configured during Phase One, but you can prepare these accounts by documenting the usernames and unique passwords for each listed below. Be sure to create good passwords and document these accounts internally within your organization.

These accounts must be documented internally within your organization. Expedient does not manage these accounts and does not have the means to recover them.

Phase One: ZVMA Deployment

The first phase of the whole process is to standup a brand new Linux-based ZVM Appliance (now referred to as a ZVMA). The ZVMA is a Linux-based virtual machine with the ZVM software installed, and is deployed using an OVA which is available on Zerto’s website or from the Expedient public repository. See the table below for locations where the OVA can be found. The ZVMA will be deployed on the same vCenter, cluster, and network as the existing Windows ZVM.

After initial deployment, there is some configuration required to prepare it for migration. After the VM is deployed, the prerequisite configuration is put in place on the ZVMA, and all new passwords are documented, then the ZVMA is considered “staged” and is ready to accept migrated data from the Windows based ZVM currently in production. Staging the ZVMA in this way is non-impacting, as it will use a different IP address when deployed than what the Windows ZVM uses in production.

Deploying the ZVMA

Using the table at the bottom of this document, download the ZVM 10 U2 P1 OVA for VMware Deployment to either a jumpbox or your desktop:

1. Login to the same vCenter as the existing Windows ZVM you have in production.

2. Begin an OVA-based deployment and use the following options:

   1. Select the same cluster as the Windows ZVM currently in production.

   2. The new ZVM’s name should be appended “-NEW” for now, so that the VM is named differently than the production Windows ZVM.

   3. Choose the same network as the existing ZVM. Remember, you will need 2 free IP addresses from this subnet for use throughout this whole process.

3. Once the wizard is complete, wait for the ZVMA to finish deploying.

Configuring the ZVMA for Migration

After the deployment has finished, we still need to do some basic configuring of the appliance as required for Zerto’s Migration Tool to work correctly later.

Note: If the remote console gives you input issues, such as typing the wrong key, use the VMRC instead of the web console.

Configuring the New ZVMA Shell

1. Power on the ZVMA and wait for the boot to complete

2. Console into the VM. Login as ‘zadmin’ with the default password of Zertodata123!

   1. You will be prompted to enter a new password. Enter a new password now. If you did not already document this new password, do so now.

3. Once logged in, you will be brought to the appliance shell. We want to enable SSH, change the hostname, and then set the network configuration.

   1. Press ‘7' at the main menu, then enter 'y’ to enable SSH. Once completed you should be brought back to the main menu.

   2. Press ‘9' at the main menu. Enter the hostname in all lowercase, with no underscores, and no domain. This should be the exact same name as the Windows ZVM. This operation will take a little while to complete.

4. To change the networking, enter '2' at the main menu to enter the Network Configuration submenu.

   1. Once in the Network Configuration submenu, enter '2' to change the static IP address of the appliance.

   2. Enter a new IP address for this ZVMA. The IP address should live on the same network as the existing Windows ZVM, but should NOT be the same IP address or in use. It must be a new IP address. The production IP will be cutover at a later time.

      1. Subnet, Gateway, and DNS can match what is already on the existing Windows ZVM. You can pull that information from the existing Windows ZVM if necessary.

5. After the networking is configured, the appliance will require a confirmation and then request permission to reboot. Let the ZVMA reboot.

6. Once the ZVMA comes back online, go to the web UI of the ZVMA using https://IP_OF_THE_APPLIANCE

   1. Enter admin as the username, and admin as the password. You will be prompted to change the password. Enter a new password now. If you did not already document this new password, do so now.

   2. Do not make any changes within the ZVMA web UI yet, except for changing this password. You will return to this UI after the migration has occurred.

Initial Keycloak Configuration

The keycloak admin password must be changed as well. After the reboot is completed, the ZVMA is now accessible via the web UI. Unlike the Windows ZVM, the ZVMA does not require a port at the end of the FQDN.

1. Open a browser and enter the keycloak admin interface of the appliance. Use the IP address you entered on the appliance, like so:

https://THE_NEW_IP_YOU_USED/auth

1. Once at the Keycloak Admin page, click on “Administration” and login with ‘admin’ as the username and ‘admin’ as the password.

2. You will be prompted to enter a new password. Enter a new password now. If you did not already document this new password, do so now.

At this point all the initial deployment steps should be completed. To begin the next phase, you may want to schedule downtime internally as Phase Two will involve a full Zerto site outage.

Phase Two: ZVM-to-ZVMA Cutover

At this point in the migration process, you have a fresh ZVMA deployed and configured alongside the existing Windows ZVM. During the next couple steps, you will run the Migration Tool from the Windows ZVM which will begin the migration. There will be a full Zerto service outage on the ZVM you’re migrating, when performing the migration. Any disaster recovery replication inbound or outbound of the ZVM you are migrating will be interrupted during the full duration of the cutover. There will be no impact on your actual virtual machines, but the DR service will be interrupted until the migration completes.

Reminder for this part of the process, you will need 1 IP address to enter into the tool as a temp. IP address on the Windows ZVM, and the passwords you’ve documented. In addition, if you have not staged the migration tool on the Windows ZVM yet, you must do so now so that you can run it from the ZVM. The download link is provided near the end of this document; you can either download it from the internet link or from Expedient’s public repository.

Migration

Reminders:

• Zerto recommends disabling any antivirus software that may be running on the Windows ZVM before proceeding with the upgrade process.

• Although Zerto provides a Recovery Workflow if the migration fails, it is still recommended to have a snapshot in place on the Windows ZVM.

1. RDP into the Windows ZVM that must be upgraded.

2. Run the Migration Tool as Administrator
   Note: If you receive a Windows Defender alert, choose “More” and then “Run Anyway” to continue.

3. When the launcher finishes loading, you must click on “Read Me” before wizard will allow you to continue. Click this and then close the window it opens, then click on “Next”
   This only opens a browser window to the Migration Tool documentation which is indexed at the bottom of this document if you need to refer to it.

4. On the “Target ZVM Appliance” screen, enter the IP address of the ZVMA you deployed in the previous step, as well as the zadmin account details you entered. Click on “Validate SSH Connectivity” which should finish pretty quickly. Click “Next”

5. On the “Alternate Host” screen, enter the temporary IP address you’ve selected, and all the relevant details. This IP address should be on the same subnet as the ZVM and ZVMA, and so the gateway, subnet, and DNS settings should all match. Remember, this IP will be the one you connect to the Windows ZVM on DURING the migration process. It will remain on the Windows ZVM after replaced, while the original IP of the Windows ZVM will be placed on the ZVMA.

6. Review the Summary screen and confirm the details are correct, and click “Migrate” when ready to begin. DOUBLE CHECK YOU HAVE A SNAPSHOT ON THE VM BEFORE PROCEEDING.
   This will cause an outage for the ZVM site when pressed and until the process is over which could take up to 20 minutes.

   1. Uncheck the “Upgrade VRAs” option if the Windows ZVM is large, but for very small sites this should be okay to leave on.

   2. Leave “Replace Peer Hostnames with IP Addresses” checked

7. After the migration begins, monitor the progress:

   1. Your RDP session will drop after the IP addresses are changed. You will need to reconnect to the ZVM by using the temporary IP you provided in the wizard.

   2. Services will be stopped on the Windows ZVM, beginning a Zerto outage for that site. This outage will last until the ZVMA is brought fully online at the end of the wizard.

   3. If the process fails, Zerto recommends using the “Recovery Workflow” to restore the Windows ZVM.

Regression Testing

Once the migration completes, you should be prompted to access the ZVMA’s web UI by using the IP address. Do so now. It is recommended to open an incognito window to prevent the web UI from hanging due to old sessions. You should be able to access the web UI without issue, though it may take a few minutes for the services to fully become online.

• Confirm the ZVMA web UI is accessible. Remember, you don’t need to use the :9669 port anymore.

• Under the VRAs, confirm they have been upgraded if you opted for automatic upgrades. If not, confirm they are at the very least online, connected, and VPGs are resyncing.

• Confirm all sites are connected as expected and that there are no compatibility issues.

Recovery and Troubleshooting

If the process fails, Zerto recommends using the “Recovery Workflow” to restore the Windows ZVM. You will need to delete the ZVMA and recreate it from scratch after recovery. See Zerto documentation for more details: https://help.zerto.com/bundle/Zerto.Migration.Utility.HTML.1.0/page/Post-migration.htm

Phase Three: Post-Migration Steps

After migrating from the Windows ZVM to the Linux ZVMA, and confirming your site connectivity and access is all healthy, you can proceed with some minor cleanup steps to finish the process off. In addition, if you did not upgrade the VRAs automatically, be sure to do that now. All VRAs must be upgraded after the ZVM has been cutover to the ZVMA.

VRA Upgrades

All VRAs on your newly migrated site must now be upgraded from 9.7 to 10U2P1. You may have selected automatic VRA upgrades when you were going through the Migration Tool wizard. If you did not, this can be done manually from within the web UI of the ZVMA. See the following documentation for that process: https://help.zerto.com/bundle/Admin.VC.HTML.10.0_U2/page/Upgrading_VRAs.htm

Cleanup

These steps apply to all Windows ZVMs and Linux ZVMAs:

1. The old Windows ZVM can be shut down and removed from inventory.

2. The Linux ZVMA can be renamed to remove the ‘-NEW’ suffix, if necessary.

3. Snapshots can be removed.

4. Confirm any monitoring solutions you might have in place for your ZVM has been adjusted and functions correctly for the new Linux ZVM Appliance.

Keycloak Configuration and AD Authentication (optional)

If you would like to configure your ZVMA to use AD authentication when logging into the web management portal, please reference the following Zerto documentation: https://help.zerto.com/bundle/Linux.ZVM.HTML.10.0_U1/page/User_Federation.htm

Downloads

The following table can be referenced for downloading the files needed to facilitate a ZVM-to-ZVMA upgrade.

For our public repo, the following username and password can be used:

Username: expedient

Password: publ1c

Known Issues

A list of known issues and their remediation steps or recommended actions can be listed here and will be updated as necessary.

ZORG of the VPG was not found in the storage error (VPG0063)

After migrating from a 9.7 to a 10U2P1 appliance you may see an error in your ZVMA saying, “ZORG of the VPG was not found in the storage”

Per Zerto this is a known issue in the migration utility, which causes a non-ZCM connected site to get a ZCM entry in its database. This is resolved by clearing this ZCM entry. The following steps can be taken on your ZVMA to remediate this error:

1. Snapshot the Appliance in question (including its memory) 

2. Open an SSH to the Appliance and exit to its shell by clicking '0' 

3. In the shell, execute the following command:

   kubectl exec -it $(kubectl get pods | awk '{print $1}' | grep zvm-db) -- /opt/mssql-tools/bin/sqlcmd -S localhost -U sa -P "zvmapp-5T^d#ak*Y6" -Q "use zvm_db; delete from CloudConfigurationIdentifierStorageObject;" 

4. Open the Appliance's appliance-manager by running 'appliance-manager' 

5. Restart the Appliance pods ('3') 

6. Refresh the ZVMA UI in question, in which after some time the warnings will disappear

If these steps do not clear the error you are seeing on your ZVMA, please reach out to Expedient so that we may work with the vendor for further troubleshooting.

Subnet In Use Error

If you encounter a validation error indicating that the Kubernetes (K8S) subnet range overlaps with the vCenter IP range, you are advised to run the steps below to resolve the issue. Once done, you will be able to run the migration utility again. If you do have such a subnet configuration, in order to avoid an IP subnet conflict between the Kubernetes (K8S) container and the vCenter, open the ZVMA VM console and run the following:

1. Modify flannel network configuration

   sudo nano /var/snap/microk8s/current/args/flannel-network-mgr-config

2. Change default IP range (ex: 10.100.0.0/16, or other subnet):

   {"Network": "10.100.0.0/16", "Backend": {"Type": "vxlan"}}

3. Restart k8s:

   microk8s stop
   microk8s start

4. Delete the cni0 network interface so flannel will recreate it:

   sudo ip link delete cni0 type bridge

5. Wait for microk8s to be ready:

   microk8s status --wait-ready

6. Wait for all pods to be ready:

   microk8s.kubectl wait --for=condition=Ready pods --all --timeout=300s

Unable to assign ZVM appliance with the ZVM Windows IP Address

If you encounter an error stating that the ZVMA could not be assigned with the Windows IP address, Zerto Support has given us the following steps as a workaround:

1. For each failed ZVMA migration attempt, the ZVMA must be completely redeployed from OVA. It is recommended to create a snapshot on your newly redeployed ZVMA BEFORE you first power it on, so that it can be reverted in the event of a failed migration attempt. This is is so you do not need to redeploy from OVA but instead can just revert the appliance to snapshot in the event of a failed migration attempt.

2. Setup the ZVMA shell as per our Phase One instructions listed above

3. Take a snapshot of the Windows ZVM VM before migration.

4. Download the latest "Zerto migration tool" from Zerto official portal: https://www.zerto.com/myzerto/support/downloads/

   1. In the migration tool folder create this .txt file and name it like this "tweaks.txt"

   2. In this file, please, copy/paste this tweak:

      SkipNetworkMigration = true

   3. Save the file

5. Repeat the migration again. When you run the Migration Utility again the "Alternative Host Network Details" window will not be shown and the migration of the network settings to the Linux ZVM will be skipped.
   
   Because the migration of the network settings is skipped, there are some additional steps to set the IP manually
   

6. To prevent a duplicate IP issue in the next step, power down the Windows ZVM once the migration completes successfully.

7. Use SSH or console into the Linux ZVMA and change the network settings by setting a Static IP with Option 2

   1. Configure the IP to match the original IP that the Windows ZVM was using, setting the subnet mask, gateway, and DNS servers appropriately per your environment.

   2. Confirm the changes and allow the ZVMA to reboot

The ZVMA should now power on with the IP of the original Windows ZVM and automatically reconnect any paired sites.

External Links

https://help.zerto.com/bundle/Zerto.Migration.Utility.HTML.1.0/page/Migration_Utility_Prerequisites.htm - Zerto Migration Tool Documentation

https://help.zerto.com/bundle/Linux.ZVM.HTML/page/ZVM_Linux_Deployment_Guide.htm - Zerto Linux Appliance Deployment Guide

https://help.zerto.com/bundle/Linux.ZVM.HTML.10.0_U1/page/User_Federation.htm - Zerto’s ‘Configure AD Authentication on ZVMA’ Documentation

https://help.zerto.com/bundle/Admin.VC.HTML.10.0_U2/page/Upgrading_VRAs.htm - Upgrading VRAs from within the Zerto management web UI