7 Replies Latest reply on Jan 22, 2019 10:48 AM by hockeyguyin714

    Storage groups cannot copy file

    Matrix__1970 Novice

      Hi all,

       

      I have 2 sites with AppVolumes 2.14. We have on these two vSAN datastores. We have created another NFS volumes that on the first site and a storage group with vSAN_site_1 and NFS_Original. This second NFS volumes is not attachable and it is mirrored on the second site on a volumes in Read Only mode (named NFS_Mirrored_Copy).

       

      In the second site, we have an analog storage group with vSAN_Site_2 and NFS_Mirrored_Copy.

       

      During replication on second site, we have errors like this in  AppVolumes:

       

       

      [2019-01-05 12:33:47 UTC] P1008DJ89466 ERROR   RvSphere: vSphere operation failed: Cannot complete file creation operation.

      [2019-01-05 12:33:47 UTC] P1008DJ89466  INFO   RvSphere: Failed to copy "[vSAN_Site_1] cloudvolumes/apps/Microsoft!20!Project!20!2013!20!32bit!20!ital.vmdk" to "[NFS_Mirrored_Copy] cloudvolumes/apps/Microsoft!20!Project!20!2013!20!32bit!20!ital.vmdk"

       

      Is this correct? Only the vSAN DS will attach appstacks. The NFS_Mirrored_Copy must be in R/W mode? And... the new AppStacks are always on NFS_Mirrored_Copy: this should acts as source for copy or I'm wrong?

       

      Thanks to all!

       

      Matrix

        • 1. Re: Storage groups cannot copy file
          sjesse Expert

          How are your storage groups set up in app volumes in both sites? In your second site do you have the nfs datastore selected in the correct location where its not trying to copy the appstacks to instead of being a source of where to copy the appstack from?

          • 2. Re: Storage groups cannot copy file
            hockeyguyin714 Enthusiast
            VMware Employees

            Both NFS stores should be set to non attachable in App Volumes - Storage.   The storage groups should be set to VSANSiteA+NFSSiteA in site A.  Site B should VSANSiteB+NFSSiteB in site B.  The replication sounds like you are doing this in some other function from NSFA to NSFB outside of App Volumes. If it is read only any updates will fail to NSF that is in read only mode.  Which is expected behavior as you have blocked writes so we won't even be able to modify the metadata file which we use for tracking and updates.  I would consider changing this setting of read only due to some issues you may have on SiteB on updating files.   The storage group can be setup to auto replicate from VSANSiteA to NFSSiteA.  The same could be done on SiteB as well. 

            • 3. Re: Storage groups cannot copy file
              Matrix__1970 Novice

              Thank you for your answer and to the others who have helped me.

               

              The behaviour of this scenario is that in Site B we wouldn't create AppStacks, but use that in SiteA. The idea is that we copy from SiteA (NFS on Site A) these AppStacks on NFSSIteB and, with the replication of Storage groups, use these AppStacks that will be copied in vSANSiteB. We don't know why vSANB try to copy AppStacks on NFSSiteB. This is our goal. for this we think that NFSSiteB can be in read only, because it acts only as source of these copies.

               

              On SiteA the replication is automatic. The copy from NFSSiteA to NFSSiteB is via Netappa Mirror. Storage group replication can be forced in only one direction, for example? Why vSAN try to write to NFS that could be the source?

               

              Thank you very much to all. Please, let me know.

               

              Matrix

              • 4. Re: Storage groups cannot copy file
                Matrix__1970 Novice

                Hello,

                 

                what do you mean when you say "In your second site do you have the nfs datastore selected in the correct location where its not trying to copy the appstacks to instead of being a source of where to copy the appstack from?"?

                 

                I have just added the two datastore on the sotrage group and marked the NFS as not attachable.

                 

                Thank you

                 

                Matrix

                • 5. Re: Storage groups cannot copy file
                  hockeyguyin714 Enthusiast
                  VMware Employees

                  So in your design you are assuming that the source never changes. AppStacks metadata file has to be updated as it tracks items like number of attachments, OS versions etc. The Data in the VMDK file does not change as it is attached as independent non persistent.  Your NFS will not be able to replicate that change because it is read only. It will update on the VSAN side but fail to update or replicate on the NETAPP side.  App Volumes does not have a source so it believes all storage luns are equal depending on which one has the latest data that is what it determines to overwrite.  The typical design recommendations for storage group is a single NFS shared between two sites so App Volumes would replicate via a Storage Group or multiple Storage Groups.  You are introducing something the product was not technically designed to account for.  I would recommend reviewing -  VMware App Volumes Deployment Considerations: VMware App Volumes 2.12 and Later

                   

                  Another option you could consider is manual replication after creating the AppStack on site B storage group to avoid the errors being ongoing.

                  • 6. Re: Storage groups cannot copy file
                    Matrix__1970 Novice

                    Hello,

                     

                    thank you for your answer. I think you mean that, even if we add (via copy/mirror) the NFS volume on Site1 to Site 2 (mirrored copy in read only), we must set in R/W the NFS on Site B because the replication process ivolves always write operations on all volumes in storage group. Correct? it's strange because logs (and vCenter) says me that vmdk files are unable to be  written on NFS (not only .metadata). But these messages can be generic.

                     

                    However, I have the same error if I do manual replication via Storage Group.

                     

                    Have a good day... if you wnat to explain me these doubts, I'll appreciate it.

                     

                    Matrix

                    • 7. Re: Storage groups cannot copy file
                      hockeyguyin714 Enthusiast
                      VMware Employees

                      Correct. So App Volumes Manager will consider the entire AppStack including the metadata as part of the AppStack.  The Metadata changing is typically what causes the replication to trigger in both directions for the Flat, Descriptor and Metatdata files.  The copy will include all 3 pieces not just 1.