VMware Cloud Community
Super6VCA
Expert
Expert

7.0.2 Failed Backup

Upgraded to 7.0.2 and since then my backups are failing with an error...BackupManager encountered an exception. See logs for details.  i have to admit i am not well versed at reading the log bundles.  Where in that bundle might i find the backup info to troubleshoot?  Any help is appreciated.  thanks

 

Perry 

Thank you, Perry
Reply
0 Kudos
21 Replies
Ajay1988
Expert
Expert

What backup r u using and does it support 7.0 U2 ? 

If you think your queries have been answered
Mark this response as "Correct" or "Helpful".

Regards,
AJ
Reply
0 Kudos
Anil0210
Enthusiast
Enthusiast

Hello Perry,

Please follow certain steps for me to understand this issue better.

1. ssh to the vcsa  - in the folder /var/log/vmware/applmgmt you see backup.log, backupscheduler.log, backupschedulercron.log.

2. "grep" for error, fail etc.

 

NOTE - Which protocol you are using ?

Few debugging to start on your own

1. Make sure you have proper permission to file servers

2. Check space in your file servers

3. Try to mount fileserver(nfs/smb) locally in vCSA, if you are using those protocols to check if mounting is fine.

-Anil

 

 

Reply
0 Kudos
Super6VCA
Expert
Expert

Using FTP.  Ran teh command you ask and got no return with grep.  

Thank you, Perry
Reply
0 Kudos
Ajay1988
Expert
Expert

Retry the backup for FTP and attach the backup.log file from VC here. 

If you think your queries have been answered
Mark this response as "Correct" or "Helpful".

Regards,
AJ
Reply
0 Kudos
Super6VCA
Expert
Expert

Here goes the backup log.  At least the important part.  thanks for the help!

Thank you, Perry
Reply
0 Kudos
muniba
Contributor
Contributor

Hi

Can you please share the error code.

Reply
0 Kudos
Super6VCA
Expert
Expert

error.JPG

Thank you, Perry
Reply
0 Kudos
Super6VCA
Expert
Expert

The odd part about this is that it's ben working fine up until a couple weeks ago.  Nothing has changed at that point but all of the sudden it stopped working.  Does it matter if i am running version 7

 

Error2.JPG

Thank you, Perry
Reply
0 Kudos
Ajay1988
Expert
Expert

backup.log
2021-04-23T18:49:16.572 [20210423-184810-17694817] [LotusBackup:PID-21365] [Lotus::BackupLotus:Lotus.py:61] INFO: stdout: VdcBackupDB: Server in state (8) in which it cannot perform backup. error(9125)
2021-04-23T18:49:16.573 [20210423-184810-17694817] [LotusBackup:PID-21365] [Lotus::BackupLotus:Lotus.py:68] ERROR: BackupLotus: command "/usr/lib/vmware-vmdir/bin/vdcbackup /storage/db/vmware-vmdir/ /tmp/backup_lotus/" failed. rc: 165.

Please follow my blog on the same issue here.
https://virtual-power.in/f/noticed-few-users-on-6770-hitting-issues-with-vami-backups

If you think your queries have been answered
Mark this response as "Correct" or "Helpful".

Regards,
AJ
Reply
0 Kudos
Super6VCA
Expert
Expert

Ajay,

No luck with your suggestion but thanks for the reply.  Got to a point I had to finally open a ticket with support.  Tech was on my vcenter for an hour adn was still stumped.  I had to send them a log bundle.  Still haven't heard back.  

Thank you, Perry
Reply
0 Kudos
Ajay1988
Expert
Expert

Hmm. Interesting. Does it fail with same error ?

SR number ? 

If you think your queries have been answered
Mark this response as "Correct" or "Helpful".

Regards,
AJ
Super6VCA
Expert
Expert

It did.  SR  21215999704

Thank you, Perry
Tags (1)
Reply
0 Kudos
Ajay1988
Expert
Expert

Same error. Doesn't look like you followed the complete blog.  Your vmdird is still in standalone mode . 

 

Point 4 from blog . https://virtual-power.in/f/noticed-few-users-on-6770-hitting-issues-with-vami-backups

backup.log
2021-04-26T18:15:57.319 [20210426-181512-17694817] [LotusBackup:PID-62583] [Lotus::BackupLotus:Lotus.py:61] INFO: stdout: VdcBackupDB: Server in state (8) in which it cannot perform backup. error(9125)
2021-04-26T18:15:57.320 [20210426-181512-17694817] [LotusBackup:PID-62583] [Lotus::BackupLotus:Lotus.py:68] ERROR: BackupLotus: command "/usr/lib/vmware-vmdir/bin/vdcbackup /storage/db/vmware-vmdir/ /tmp/backup_lotus/" failed. rc: 165.

vmdird-syslog.log
2021-04-26T15:28:33.724256+00:00 info vmdird t@139678593369920: VmDir State (8)
2021-04-26T15:28:33.724305+00:00 info vmdird t@139678593369920: Lotus Vmdird: running... state (8)

[\\services\vmdir]
+ "Arguments" REG_SZ "/usr/lib/vmware-vmdir/sbin/vmdird -s -l 0 -f /usr/lib/vmware-vmdir/share/config/vmdirschema.ldif -m standalone"
+ "dcAccount" REG_SZ "mwnvcs01.neuro.com"

If you think your queries have been answered
Mark this response as "Correct" or "Helpful".

Regards,
AJ
Reply
0 Kudos
Super6VCA
Expert
Expert

Ran the command   /usr/lib/vmware-vmdir/bin/vdcadmintool.  Option 6 shows   VmDir State is - (null).  choose option 5 then type NORMAL.  comes back with    State of Vmdir set to NORMAL.  Ran command /opt/likewise/bin/lwsm restart vmdir and the service restarts.  type   /usr/lib/vmware-vmdir/bin/vdcadmintool choose option 6 and it comes back with    VmDir State is - (null)

 

 

Thank you, Perry
Reply
0 Kudos
Ajay1988
Expert
Expert

vmdird null is bad.  Let support handle it then .

 

Last shot:-

Make vmdird state normal first. Don't restart service.
Run /opt/likewise/bin/lwregshell delete_value "[HKEY_THIS_MACHINE\Services\vmdir]" Arguments
Then /opt/likewise/bin/lwsm refresh and /opt/likewise/bin/lwsm restart vmdir


Run /opt/likewise/bin/lwregshell and then cd HKEY_THIS_MACHINE\services\vmdir\.
Then type ls and check "Arguments" information . Grab the output.
Re-check the vmdird state again. If normal ; re-run backup.

If you think your queries have been answered
Mark this response as "Correct" or "Helpful".

Regards,
AJ
Reply
0 Kudos
Super6VCA
Expert
Expert

Ajay

Just wanted to update you.  vmware support has still yet to figure out what is going on.  At one point we thought we had it working by doing a manual backup but after that it was right back to bad again.  

Thank you, Perry
Reply
0 Kudos
Ajay1988
Expert
Expert

Ok. I see . SO the moment vmdir  was set to "NORMAL" it worked. But after reboot   it again changed to  "standalone".

 

I would still like to check  point 4 and 5 from  https://virtual-power.in/f/noticed-few-users-on-6770-hitting-issues-with-vami-backups.

May be ask the Support Guy to help with this and see if the vmdird goes back to standalone after reboot

 

 

If you think your queries have been answered
Mark this response as "Correct" or "Helpful".

Regards,
AJ
Super6VCA
Expert
Expert

Ajay,

 

Here is teh scoop.  Seems like there was some corruption in the Directory Service in vCenter.  The tech ended up trying to recreate the issue in his lab and wound up copying some files to my vcenter.  Long story short, When we locked at the status of the DS it was set to STANDALONE (/usr/lib/vmware-vmafd/bin/dir-cli state get).  Once we et it to NORMAL (/usr/lib/vmware-vmafd/bin/dir-cli state set --state NORMAL --server-name localhost) we were able to take a manual backup.  I have to monitor the behavior for a bit but we think it is going to be fine.  All caused by teh upgrade 

Thank you, Perry
Tags (1)