VDP upgrade to 6.1.3 failed

[ascheltens22]

Hi,

I tried to upgrade from 6.1.2.19 to 6.1.3, in order to upgrade vmware to 6.5

It failed, because after upgrade I'm still in 6.1.2 and MCS service is down.

Itried to retart or force retore it, without success.

Here is what I found in log :

2017/01/19-19:47:50 dpnctl: INFO: subsystems to act upon: mcs

2017/01/19-19:47:50 dpnctl: INFO: subsystems not to act upon: maint, cron, firewall, axionfs, sched, avi, emt, gsan

2017/01/19-19:47:50 /bin/touch /var/lock/subsys/zzdpn

2017/01/19-19:47:50 INFO: environment variable USERNAME is undefined or empty

2017/01/19-19:47:50 /usr/bin/ssh-agent /bin/bash -c '/usr/bin/ssh-add /home/dpn/.ssh/dpnid && /tmp/dpnctl-run-self.5310'

2017/01/19-19:47:51 dpnctl: INFO: ======= dpnctl 7.2.80-118 (1.139) RESTARTING at 2017/01/19-20:47:51 CET (UTC +0100), running as admin =======

2017/01/19-19:47:51 dpnctl: INFO: log file time stamps are in UTC; local time is 2017/01/19-20:47:51 CET (UTC +0100)

2017/01/19-19:47:51 dpnctl: INFO: PATH=/opt/emc-tools/bin:/usr/local/avamar/bin:/sbin:/usr/sbin:/usr/local/avamar/bin:/bin:/usr/bin:/sbin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/usr/local/apache/bin:/usr/local/ssl/bin:/opt/vmware/bin

2017/01/19-19:47:51 dpnctl: INFO: Argument list: "--mcs_user=root" "start" "mcs"

2017/01/19-19:47:51 dpnctl: INFO: subsystems to act upon: mcs

2017/01/19-19:47:51 dpnctl: INFO: subsystems not to act upon: maint, cron, firewall, axionfs, sched, avi, emt, gsan

2017/01/19-19:47:51 INFO: environment variable USERNAME is undefined or empty

2017/01/19-19:47:51 [ pid = 5371 ]

2017/01/19-19:47:51 /usr/bin/ssh-add -l

2017/01/19-19:47:51 The agent has no identities.

2017/01/19-19:47:51 INFO: environment variable USERNAME is undefined or empty

2017/01/19-19:47:51 [ pid = 5371 ]

2017/01/19-19:47:51 /usr/bin/ssh-add -l

2017/01/19-19:47:51 The agent has no identities.

2017/01/19-19:47:51 /usr/bin/ssh-add /home/admin/.ssh/dpnid

2017/01/19-19:47:51 Identity added: /home/admin/.ssh/dpnid (/home/admin/.ssh/dpnid)

2017/01/19-19:47:51 [ pid = 5371 ]

2017/01/19-19:47:51 /usr/bin/ssh-add -l

2017/01/19-19:47:51 3072 dc:c8:3e:84:f5:99:88:b7:49:07:3f:b8:8b:ee:be:34 [MD5] /home/admin/.ssh/dpnid (RSA)

2017/01/19-19:47:51 dpnctl: INFO: no update context, so not checking for MCS and EMS restore indication

2017/01/19-19:47:51 gsan status check:

2017/01/19-19:47:51 - - - - - - - - - - - - - - - BEGIN

2017/01/19-19:47:51 server up: idle

2017/01/19-19:47:51 - - - - - - - - - - - - - - - END

2017/01/19-19:47:51 ([ -r /etc/profile ] && . /etc/profile ; /usr/local/avamar/bin/mcserver.sh --test ; echo $? >/tmp/dpnctl-get-mcs-status-status-5371) >/tmp/dpnctl-get-mcs-status-output-5371 2>&1

2017/01/19-19:47:53 dpnctl: INFO: "([ -r /etc/profile ] && . /etc/profile ; /usr/local/avamar/bin/mcserver.sh --test ; echo $? >/tmp/dpnctl-get-mcs-status-status-5371) >/tmp/dpnctl-get-mcs-status-output-5371 2>&1" - exit status 0

2017/01/19-19:47:53 /bin/cat /tmp/dpnctl-get-mcs-status-output-5371

2017/01/19-19:47:53 [ "/bin/cat /tmp/dpnctl-get-mcs-status-output-5371" exit status = 0 ]

2017/01/19-19:47:53 dpnctl: INFO: - - - - - - - - - - - - - - - BEGIN

2017/01/19-19:47:53 INFO: Administrator Server status: down

2017/01/19-19:47:53 dpnctl: INFO: - - - - - - - - - - - - - - - END

2017/01/19-19:47:53 /bin/cat /tmp/dpnctl-get-mcs-status-status-5371 2>&1

2017/01/19-19:47:53 [ "/bin/cat /tmp/dpnctl-get-mcs-status-status-5371 2>&1" exit status = 0 ]

2017/01/19-19:47:53 dpnctl: INFO: "[ -r /etc/profile ] && . /etc/profile ; /usr/local/avamar/bin/mcserver.sh --test" exit status = 2

2017/01/19-19:47:53 rm -f /tmp/dpnctl-get-mcs-status-status-5371 /tmp/dpnctl-get-mcs-status-output-5371

2017/01/19-19:47:53 dpnctl: INFO: "rm -f /tmp/dpnctl-get-mcs-status-status-5371 /tmp/dpnctl-get-mcs-status-output-5371" - exit status 0

2017/01/19-19:47:53 rpm -q --quiet connectemc

2017/01/19-19:47:53 dpnctl: INFO: "rpm -q --quiet connectemc" - exit status 1

2017/01/19-19:47:53 gsan status check:

2017/01/19-19:47:53 - - - - - - - - - - - - - - - BEGIN

2017/01/19-19:47:54 server up: idle

2017/01/19-19:47:54 - - - - - - - - - - - - - - - END

2017/01/19-19:47:54 dpnctl: INFO: Starting MCS...

2017/01/19-19:47:54 dpnctl: INFO: [no passthrough options found for program "mcserver.sh" in context "start"]

2017/01/19-19:47:54 dpnctl: INFO: To monitor progress, run in another window: tail -f /tmp/dpnctl-mcs-start-output-5371

2017/01/19-19:47:54 ([ -r /etc/profile ] && . /etc/profile ; /usr/local/avamar/bin/mcserver.sh --start ; echo $? >/tmp/dpnctl-mcs-start-status-5371) >/tmp/dpnctl-mcs-start-output-5371 2>&1

2017/01/19-19:48:02 dpnctl: INFO: "([ -r /etc/profile ] && . /etc/profile ; /usr/local/avamar/bin/mcserver.sh --start ; echo $? >/tmp/dpnctl-mcs-start-status-5371) >/tmp/dpnctl-mcs-start-output-5371 2>&1" - exit status 0

2017/01/19-19:48:02 /bin/cat /tmp/dpnctl-mcs-start-output-5371

2017/01/19-19:48:02 [ "/bin/cat /tmp/dpnctl-mcs-start-output-5371" exit status = 0 ]

2017/01/19-19:48:02 /bin/cat /tmp/dpnctl-mcs-start-status-5371 2>&1

2017/01/19-19:48:02 [ "/bin/cat /tmp/dpnctl-mcs-start-status-5371 2>&1" exit status = 0 ]

2017/01/19-19:48:02 output of "[ -r /etc/profile ] && . /etc/profile ; /usr/local/avamar/bin/mcserver.sh --start":

2017/01/19-19:48:02 - - - - - - - - - - - - - - - BEGIN

2017/01/19-19:48:02 Database server is running...

2017/01/19-19:48:02 INFO: Starting messaging service.

2017/01/19-19:48:02 INFO: Started messaging service.

2017/01/19-19:48:02 === BEGIN === check.mcs (prestart)

2017/01/19-19:48:02 check.mcs                        passed

2017/01/19-19:48:02 === PASS === check.mcs PASSED OVERALL (prestart)

2017/01/19-19:48:02 Starting Administrator Server at: Thu Jan 19 20:47:59 CET 2017

2017/01/19-19:48:02 Starting Administrator Server...

2017/01/19-19:48:02 java.net.SocketException: java.security.NoSuchAlgorithmException: Error constructing implementation (algorithm: Default, provider: SunJSSE, class: sun.security.ssl.SSLContextImpl$DefaultSSLContext)

2017/01/19-19:48:02 - - - - - - - - - - - - - - - END

2017/01/19-19:48:03 dpnctl: ERROR: error return from "[ -r /etc/profile ] && . /etc/profile ; /usr/local/avamar/bin/mcserver.sh --start" - exit status 1

2017/01/19-19:48:03 rm -f /tmp/dpnctl-mcs-start-status-5371 /tmp/dpnctl-mcs-start-output-5371

2017/01/19-19:48:03 ([ -r /etc/profile ] && . /etc/profile ; /usr/local/avamar/bin/mcserver.sh --test ; echo $? >/tmp/dpnctl-get-mcs-status-status-5371) >/tmp/dpnctl-get-mcs-status-output-5371 2>&1

2017/01/19-19:48:04 dpnctl: INFO: "([ -r /etc/profile ] && . /etc/profile ; /usr/local/avamar/bin/mcserver.sh --test ; echo $? >/tmp/dpnctl-get-mcs-status-status-5371) >/tmp/dpnctl-get-mcs-status-output-5371 2>&1" - exit status 0

2017/01/19-19:48:04 /bin/cat /tmp/dpnctl-get-mcs-status-output-5371

2017/01/19-19:48:04 [ "/bin/cat /tmp/dpnctl-get-mcs-status-output-5371" exit status = 0 ]

2017/01/19-19:48:04 dpnctl: INFO: - - - - - - - - - - - - - - - BEGIN

2017/01/19-19:48:04 INFO: Administrator Server status: down

2017/01/19-19:48:04 dpnctl: INFO: - - - - - - - - - - - - - - - END

2017/01/19-19:48:04 /bin/cat /tmp/dpnctl-get-mcs-status-status-5371 2>&1

2017/01/19-19:48:04 [ "/bin/cat /tmp/dpnctl-get-mcs-status-status-5371 2>&1" exit status = 0 ]

2017/01/19-19:48:04 dpnctl: INFO: "[ -r /etc/profile ] && . /etc/profile ; /usr/local/avamar/bin/mcserver.sh --test" exit status = 2

2017/01/19-19:48:04 rm -f /tmp/dpnctl-get-mcs-status-status-5371 /tmp/dpnctl-get-mcs-status-output-5371

2017/01/19-19:48:04 dpnctl: INFO: "rm -f /tmp/dpnctl-get-mcs-status-status-5371 /tmp/dpnctl-get-mcs-status-output-5371" - exit status 0

2017/01/19-19:48:04 dpnctl: INFO: [see log file "/usr/local/avamar/var/log/dpnctl.log"]

2017/01/19-19:48:04 rm -f /usr/local/avamar/var/dpnctl.lck

2017/01/19-19:48:04 ======= dpnctl 7.2.80-118 (1.139), running as admin, RETURNING at 2017-01-19 20:48:04 CET =======

2017/01/19-19:48:04 [user "admin"] program (pid 5371) exit status = 1 (error)

2017/01/19-19:48:04 ======= dpnctl 7.2.80-118 (1.139), running as root, ENDING at 2017-01-19 20:48:04 CET =======

2017/01/19-19:48:04 [user "root"] program (pid 5310) exit status = 1 (error)

I always see this : INFO: environment variable USERNAME is undefined or empty

I'm able to connect via the web client, but can't even call a rollback.

I also destroyed my snapshot, because at the end of the update process everything seemed to be good.

If someone can help me ! Otherwise I think I will have to delete and reinstall it, loosing my backups...

[SavkoorSuhas]

I would suggest you to raise a case with VM support for this.

I would need to look into this in detail. dpnctl log will not have much information on the MCS failure to start.

If you are on 6.1.2 now and if MCS is down, login to VDP SSH as admin mode and run

# mcserver.sh --start --verbose

If it fails, it will output the error as to why.

Suhas

[ascheltens22]

Hi,

Here is what the verbose "said" ^^ :

mcserver.sh --start --verbose

args = '--start' '--verbose'

---------------------------------

home_dir=/usr/local/avamar

db_dir=/usr/local/avamar/var/mc/server_data/postgres/data

data_dir=/usr/local/avamar/var/mc/server_data

tmp_dir=/usr/local/avamar/var/mc/server_tmp

postgres_dir=/usr/local/avamar/var/mc/server_data/postgres

prefs_dir=/usr/local/avamar/var/mc/server_data/prefs

mc_dir=/usr/local/avamar/var/mc

log_dir=/usr/local/avamar/var/mc/server_log

lib_dir=/usr/local/avamar/lib

bin_dir=/usr/local/avamar/bin

---------------------------------

Database server is running...

INFO: Starting messaging service.

INFO: Started messaging service.

/usr/java/latest/bin/java -Xmx1G -XX:+HeapDumpOnOutOfMemoryError -server -ea -cp /usr/local/avamar/lib/avamar-mcsecurity.jar:/usr/local/avamar/lib/commons-cli-1.2.jar:/usr/local/avamar/lib/commons-lang-2.6.jar:/usr/local/avamar/lib/commons-logging-adapters-1.1.jar:/usr/local/avamar/lib/rsa-jsafe-cryptojce-6.2.jar:/usr/local/avamar/lib/rsa-jsafe-cryptojcommon-6.2.jar:/usr/local/avamar/lib/rsa-jsafe-jcmFIPS-6.2.jar:/usr/local/avamar/lib/rsa-jsafe-util-6.2.jar:/usr/local/avamar/lib/spring-jdbc-3.1.1.RELEASE.jar:/usr/local/avamar/lib/spring-tx-3.1.1.RELEASE.jar:/usr/local/avamar/lib/mcserver.jar:/usr/local/avamar/var/mc/server_data/prefs:lib:/usr/local/avamar/lib/asn_server.jar:/usr/local/avamar/lib/mail.jar:/usr/local/avamar/lib/activation.jar:/usr/local/avamar/lib/xercesImpl.jar:/usr/local/avamar/lib/xmlsec.jar:/usr/local/avamar/lib/xmldsig.jar:lib/postgresql-8.4-702.jdbc4.jar:/usr/local/avamar/lib/jaxrpc.jar:/usr/local/avamar/lib/commons-logging-1.0.4.jar:/usr/local/avamar/lib/commons-validator.jar:/usr/local/avamar/lib/commons-discovery-0.2.jar:/usr/local/avamar/lib/commons-collections-3.2.1.jar:/usr/local/avamar/lib/saaj.jar:/usr/local/avamar/lib/wbem.jar:/usr/local/avamar/lib/wsdl4j-1.5.1.jar:/usr/local/avamar/lib/log4j-1.2.14.jar:/usr/local/avamar/lib/axis.jar:/usr/local/avamar/lib/vim25.jar:/usr/local/avamar/lib/commons-codec-1.3.jar:/usr/local/avamar/lib/jema_api.jar:/usr/local/avamar/lib/jetty-6.1.23.jar:/usr/local/avamar/lib/jetty-util-6.1.23.jar:/usr/local/avamar/lib/servlet-api-3.0.jar:/usr/local/avamar/lib/jsp-api-2.1.jar:/usr/local/avamar/lib/jsp-2.1.jar:/usr/local/avamar/lib/core-3.1.1.jar:/usr/local/avamar/lib/ant-1.6.5.jar:/usr/local/avamar/lib/apputils25.jar:/usr/local/avamar/lib/j2ssh-core-0.2.9.jar:/usr/local/avamar/lib/snmp4j-1.11.jar:/usr/local/avamar/lib/mccommons.jar:/usr/local/avamar/lib/user-authentication.jar:/usr/local/avamar/lib/nis-1.2.1.jar:/usr/local/avamar/lib/providerutil-1.2.1.jar:/usr/local/avamar/lib/commons-lang-2.6.jar:/usr/local/avamar/lib/commons-io-1.4.jar:/usr/local/avamar/lib/jutf7-1.0.0.jar:/usr/local/avamar/lib/i18n:/usr/local/avamar/lib/certj.jar:/usr/local/avamar/lib/rsa-jsafe-cryptojce-6.2.jar:/usr/local/avamar/lib/rsa-jsafe-cryptojcommon-6.2.jar:/usr/local/avamar/lib/rsa-jsafe-jcmFIPS-6.2.jar:/usr/local/avamar/lib/sslj.jar:/usr/local/avamar/lib/amqp-client-3.1.3.jar:/usr/local/avamar/lib/events-dispatcher-api-1.0.0.jar:/usr/local/avamar/lib/events-dispatcher-amqp-1.0.0.jar:/usr/local/avamar/lib/jackson-core-asl-1.9.13.jar:/usr/local/avamar/lib/jackson-mapper-asl-1.9.13.jar:/usr/local/avamar/lib/mcebms-client.jar:/usr/local/avamar/lib/httpclient-4.3.2.jar:/usr/local/avamar/lib/httpcore-4.3.1.jar:/usr/local/avamar/lib/javax.ws.rs-api-2.0-m10.jar:/usr/local/avamar/lib/rabbit-config.jar:/usr/local/avamar/lib/aopalliance-1.0.jar:/usr/local/avamar/lib/aspectjweaver-1.6.12.jar:/usr/local/avamar/lib/spring-aop-3.1.1.RELEASE.jar:/usr/local/avamar/lib/spring-asm-3.1.1.RELEASE.jar:/usr/local/avamar/lib/spring-beans-3.1.1.RELEASE.jar:/usr/local/avamar/lib/spring-context-3.1.1.RELEASE.jar:/usr/local/avamar/lib/spring-core-3.1.1.RELEASE.jar:/usr/local/avamar/lib/spring-expression-3.1.1.RELEASE.jar:/usr/local/avamar/lib/spring-amqp-1.2.0.RELEASE.jar:/usr/local/avamar/lib/spring-rabbit-1.2.0.RELEASE.jar:/usr/local/avamar/lib/spring-web-3.1.1.RELEASE.jar:/usr/local/avamar/lib/spring-jdbc-3.1.1.RELEASE.jar:/usr/local/avamar/lib/spring-tx-3.1.1.RELEASE.jar::/usr/local/avamar/lib/spring-security-config-3.1.0.RELEASE.jar:/usr/local/avamar/lib/spring-security-core-3.1.0.RELEASE.jar:/usr/local/avamar/lib/spring-security-crypto-3.1.0.RELEASE.jar:/usr/local/avamar/lib/spring-security-web-3.1.0.RELEASE.jar -Djava.rmi.server.hostname=vdp.mairlan -Djava.util.logging.config.file=/usr/local/avamar/lib/mcserver_logging.properties -Djava.security.egd=file:/dev/./urandom -Djava.io.tmpdir=/usr/local/avamar/var/mc/server_tmp -Djava.util.prefs.PreferencesFactory=com.avamar.mc.util.MCServerPreferencesFactory -Djavax.xml.parsers.DocumentBuilderFactory=org.apache.xerces.jaxp.DocumentBuilderFactoryImpl -Djavax.net.ssl.keyStore=lib/rmi_ssl_keystore -Djavax.net.ssl.trustStore=lib/rmi_ssl_keystore -Dfile.encoding=UTF-8 -Dlog4j.configuration=file:///usr/local/avamar/lib/log4j.properties  com.avamar.mc.CLI --start >> /usr/local/avamar/var/mc/server_log/mcserver.out 2>&1 &

=== BEGIN === check.mcs (prestart)

   checking if localhost is correctly defined in /etc/hosts

   checking if entry for VDP is correctly defined in /etc/hosts

   checking if javaDir preference is set: ("/usr/java/latest" ne "REPLACE_JAVADIR_HERE")

   checking if hfsaddr preference is set: ("VDP.mairlan" ne "REPLACE_HFSADDR_HERE")

   checking if rootAP preference is set: ("{AES}bWyZpzunGBlNABfMUFioN+2IOG/+XAs2d8NihAmlrAM=" ne "REPLACE_ROOT_AP_HERE")

   checking if MCUSERAP preference is set: ("{AES}bWyZpzunGBlNABfMUFioN+2IOG/+XAs2d8NihAmlrAM=" ne "REPLACE_MCUSER_AP_HERE")

   checking if hfsaddr preference is pingable: (mcsutils::ping( "VDP.mairlan" ))

   checking if local_hfsaddr preference is pingable: (mcsutils::ping( "VDP.mairlan" ))

   checking if the bin directory is in the path: ("/opt/emc-tools/bin:/usr/local/avamar/bin:/sbin:/usr/sbin:/usr/local/avamar/bin:/bin:/usr/bin:/sbin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/usr/local/apache/bin:/usr/local/ssl/bin:/opt/vmware/bin" =~ m/\/usr\/local\/avamar\/bin/)

   checking if probe.xml file exists: ('/usr/local/avamar/var/probe.xml' ne '')

   checking if probe.xml file is readable: (-r '/usr/local/avamar/var/probe.xml')

   checking if psql version is at least 8.2: ("8.3.23" ge "8.2")

   checking if postmaster version is at least 8.2: ("unknown" ge "8.2")

   checking if postgres version is at least 8.2: ("8.3.23" ge "8.2")

   checking if pg_ctl version is at least 8.2: ("8.3.23" ge "8.2")

   checking if postmaster is running make sure admin owns it

   checking if Perl version is at least version 5: ("5" le "5.10.0")

   checking if dpnserver version equal to dpnmcs version: ("7.2.80-118" eq "7.2.80-118"  && "7.2.80-118" ne "unknown")

   Executing: /usr/local/avamar/bin/avmaint --version

   checking if avmaint version equal to dpnmcs version: ("7.2.80-118" eq "7.2.80-118"  && "7.2.80-118" ne "unknown")

   Executing: /usr/local/avamar/bin/avmgr --version

   checking if avmgr version equal to dpnmcs version: ("7.2.80-118" eq "7.2.80-118"  && "7.2.80-118" ne "unknown")

   checking if ~/.ssh/dpnid is readable: (-r '/home/admin/.ssh/dpnid')

   checking if preferences.dtd file exists and is readable: (-r '/usr/local/avamar/var/mc/server_data/prefs/preferences.dtd')

   checking if mcserver.xml file exists and is readable: (-r '/usr/local/avamar/var/mc/server_data/prefs/mcserver.xml')

   checking if mcserver.xml file is parseable: ("/usr/local/avamar/" ne "")

   checking if var dir /usr/local/avamar/var directory exists: (-d '/usr/local/avamar/var')

   checking if mc dir /usr/local/avamar/var/mc directory exists: (-d '/usr/local/avamar/var/mc')

   checking if data dir /usr/local/avamar/var/mc/server_data directory exists: (-d '/usr/local/avamar/var/mc/server_data')

   checking if tmp dir /usr/local/avamar/var/mc/server_tmp directory exists: (-d '/usr/local/avamar/var/mc/server_tmp')

   checking if prefs dir /usr/local/avamar/var/mc/server_data/prefs directory exists: (-d '/usr/local/avamar/var/mc/server_data/prefs')

   checking if postgres dir /usr/local/avamar/var/mc/server_data/postgres directory exists: (-d '/usr/local/avamar/var/mc/server_data/postgres')

   checking if db dir /usr/local/avamar/var/mc/server_data/postgres/data directory exists: (-d '/usr/local/avamar/var/mc/server_data/postgres/data')

   checking if log dir /usr/local/avamar/var/mc/server_log directory exists: (-d '/usr/local/avamar/var/mc/server_log')

   checking if /usr/local/avamar/bin/avtar exists and is executable: (-e '/usr/local/avamar/bin/avtar')

   checking if /usr/local/avamar/bin/avmgr exists and is executable: (-e '/usr/local/avamar/bin/avmgr')

   checking if /usr/local/avamar/bin/avmaint exists and is executable: (-e '/usr/local/avamar/bin/avmaint')

   Executing: /usr/local/avamar/bin/avmgr logn

   Executing: /usr/local/avamar/bin/avmgr getu --path='' --format=xml

   Executing: /usr/local/avamar/bin/avmgr getl --path='' --format=xml

   Executing: /usr/local/avamar/bin/avmaint nodelist

   checking if accessmode allows full access: ('mhpu+0hpu+0hpu' matches /[m0]hpu\+0hpu\+0hpu/)

   Executing: /usr/local/avamar/bin/avmaint nodelist

   checking that the dispatcher is not suspended: ('false' matches /false/)

   Executing: /usr/local/avamar/bin/avtar --archives --path=/MC_BACKUPS --count=26

   checking if any clients have a null path

   Executing:  tar -cz --atime-preserve=system --dereference --ignore-failed-read --one-file-system --absolute-names -f /tmp/checkTarCmd.tgz /usr/local/avamar/var/mc/server_data/prefs/mcserver.xml

   Executing:  rm /tmp/checkTarCmd.tgz

check.mcs                        passed

=== PASS === check.mcs PASSED OVERALL (prestart)

Starting Administrator Server at: Fri Jan 20 07:39:38 CET 2017

Starting Administrator Server...

java.net.SocketException: java.security.NoSuchAlgorithmException: Error constructing implementation (algorithm: Default, provider: SunJSSE, class: sun.security.ssl.SSLContextImpl$DefaultSSLContext)

Seems to be a java error. Something must has done wrong during the upgrade. I'm going to try it again.

[SavkoorSuhas]

Could you run the below command to see what permissions are there for rmi_ssl_keystore and post the output?

ls -l /usr/local/avamar/lib/rmi_ssl_keystore

Suhas

[ascheltens22]

Thank you for your help.

Here's the result :

root@VDP:~/#: ls -l /usr/local/avamar/lib/rmi_ssl_keystore

-r--r--r-- 1 root admin 1251 Oct 14 14:57 /usr/local/avamar/lib/rmi_ssl_keystore

[SavkoorSuhas]

Can you chmod this file to 644

sudo chmod 644 /usr/local/avamar/lib/rmi_ssl_keystore

Once this is done, start the MCS using mcserver.sh --start --verbose

Suhas

[ascheltens22]

Hi, here's the result after the chmod.

admin@VDP:/root/#: mcserver.sh --start --verbose

args = '--start' '--verbose'

---------------------------------

home_dir=/usr/local/avamar

db_dir=/usr/local/avamar/var/mc/server_data/postgres/data

data_dir=/usr/local/avamar/var/mc/server_data

tmp_dir=/usr/local/avamar/var/mc/server_tmp

postgres_dir=/usr/local/avamar/var/mc/server_data/postgres

prefs_dir=/usr/local/avamar/var/mc/server_data/prefs

mc_dir=/usr/local/avamar/var/mc

log_dir=/usr/local/avamar/var/mc/server_log

lib_dir=/usr/local/avamar/lib

bin_dir=/usr/local/avamar/bin

---------------------------------

ERROR: /usr/local/avamar/var/mc/server_data/prefs/mcserver.xml is missing.

admin@VDP:/root/#:

Seems I lost one file since the last try ...

[SavkoorSuhas]

This is extremely weird.

Okay, there must be a default mcserver.xml file under /usr/local/avamar/lib/mcserver.xml
Copy this to server_data/prefs:

cp /usr/local/avamar/lib/mcserver.xml /usr/local/avamar/var/mc/server_data/prefs/

You can then try restarting the MCS.

If this fails, you might probably want to perform a rollback or open a support request with us.

Suhas

[ascheltens22]

Here is the result...

admin@VDP:/root/#: mcserver.sh --start --verbose

args = '--start' '--verbose'

---------------------------------

home_dir=/usr/local/avamar

db_dir=/usr/local/avamar/var/mc/server_data/postgres/data

data_dir=/usr/local/avamar/var/mc/server_data

tmp_dir=/usr/local/avamar/var/mc/server_tmp

postgres_dir=/usr/local/avamar/var/mc/server_data/postgres

prefs_dir=/usr/local/avamar/var/mc/server_data/prefs

mc_dir=/usr/local/avamar/var/mc

log_dir=/usr/local/avamar/var/mc/server_log

lib_dir=/usr/local/avamar/lib

bin_dir=/usr/local/avamar/bin

---------------------------------

ping: unknown host REPLACE_HFSADDR_HERE

ping: unknown host REPLACE_HFSADDR_HERE

ping: unknown host REPLACE_HFSADDR_HERE

ping: unknown host REPLACE_HFSADDR_HERE

ping: unknown host REPLACE_HFSADDR_HERE

Database server is running...

INFO: Starting messaging service.

INFO: Started messaging service.

/usr/java/latest/bin/java -Xmx1G -XX:+HeapDumpOnOutOfMemoryError -server -ea -cp /usr/local/avamar/lib/avamar-mcsecurity.jar:/usr/local/avamar/lib/commons-cli-1.2.jar:/usr/local/avamar/lib/commons-lang-2.6.jar:/usr/local/avamar/lib/commons-logging-adapters-1.1.jar:/usr/local/avamar/lib/rsa-jsafe-cryptojce-6.2.jar:/usr/local/avamar/lib/rsa-jsafe-cryptojcommon-6.2.jar:/usr/local/avamar/lib/rsa-jsafe-jcmFIPS-6.2.jar:/usr/local/avamar/lib/rsa-jsafe-util-6.2.jar:/usr/local/avamar/lib/spring-jdbc-3.1.1.RELEASE.jar:/usr/local/avamar/lib/spring-tx-3.1.1.RELEASE.jar:/usr/local/avamar/lib/mcserver.jar:/usr/local/avamar/var/mc/server_data/prefs:lib:/usr/local/avamar/lib/asn_server.jar:/usr/local/avamar/lib/mail.jar:/usr/local/avamar/lib/activation.jar:/usr/local/avamar/lib/xercesImpl.jar:/usr/local/avamar/lib/xmlsec.jar:/usr/local/avamar/lib/xmldsig.jar:lib/postgresql-8.4-702.jdbc4.jar:/usr/local/avamar/lib/jaxrpc.jar:/usr/local/avamar/lib/commons-logging-1.0.4.jar:/usr/local/avamar/lib/commons-validator.jar:/usr/local/avamar/lib/commons-discovery-0.2.jar:/usr/local/avamar/lib/commons-collections-3.2.1.jar:/usr/local/avamar/lib/saaj.jar:/usr/local/avamar/lib/wbem.jar:/usr/local/avamar/lib/wsdl4j-1.5.1.jar:/usr/local/avamar/lib/log4j-1.2.14.jar:/usr/local/avamar/lib/axis.jar:/usr/local/avamar/lib/vim25.jar:/usr/local/avamar/lib/commons-codec-1.3.jar:/usr/local/avamar/lib/jema_api.jar:/usr/local/avamar/lib/jetty-6.1.23.jar:/usr/local/avamar/lib/jetty-util-6.1.23.jar:/usr/local/avamar/lib/servlet-api-3.0.jar:/usr/local/avamar/lib/jsp-api-2.1.jar:/usr/local/avamar/lib/jsp-2.1.jar:/usr/local/avamar/lib/core-3.1.1.jar:/usr/local/avamar/lib/ant-1.6.5.jar:/usr/local/avamar/lib/apputils25.jar:/usr/local/avamar/lib/j2ssh-core-0.2.9.jar:/usr/local/avamar/lib/snmp4j-1.11.jar:/usr/local/avamar/lib/mccommons.jar:/usr/local/avamar/lib/user-authentication.jar:/usr/local/avamar/lib/nis-1.2.1.jar:/usr/local/avamar/lib/providerutil-1.2.1.jar:/usr/local/avamar/lib/commons-lang-2.6.jar:/usr/local/avamar/lib/commons-io-1.4.jar:/usr/local/avamar/lib/jutf7-1.0.0.jar:/usr/local/avamar/lib/i18n:/usr/local/avamar/lib/certj.jar:/usr/local/avamar/lib/rsa-jsafe-cryptojce-6.2.jar:/usr/local/avamar/lib/rsa-jsafe-cryptojcommon-6.2.jar:/usr/local/avamar/lib/rsa-jsafe-jcmFIPS-6.2.jar:/usr/local/avamar/lib/sslj.jar:/usr/local/avamar/lib/amqp-client-3.1.3.jar:/usr/local/avamar/lib/events-dispatcher-api-1.0.0.jar:/usr/local/avamar/lib/events-dispatcher-amqp-1.0.0.jar:/usr/local/avamar/lib/jackson-core-asl-1.9.13.jar:/usr/local/avamar/lib/jackson-mapper-asl-1.9.13.jar:/usr/local/avamar/lib/mcebms-client.jar:/usr/local/avamar/lib/httpclient-4.3.2.jar:/usr/local/avamar/lib/httpcore-4.3.1.jar:/usr/local/avamar/lib/javax.ws.rs-api-2.0-m10.jar:/usr/local/avamar/lib/rabbit-config.jar:/usr/local/avamar/lib/aopalliance-1.0.jar:/usr/local/avamar/lib/aspectjweaver-1.6.12.jar:/usr/local/avamar/lib/spring-aop-3.1.1.RELEASE.jar:/usr/local/avamar/lib/spring-asm-3.1.1.RELEASE.jar:/usr/local/avamar/lib/spring-beans-3.1.1.RELEASE.jar:/usr/local/avamar/lib/spring-context-3.1.1.RELEASE.jar:/usr/local/avamar/lib/spring-core-3.1.1.RELEASE.jar:/usr/local/avamar/lib/spring-expression-3.1.1.RELEASE.jar:/usr/local/avamar/lib/spring-amqp-1.2.0.RELEASE.jar:/usr/local/avamar/lib/spring-rabbit-1.2.0.RELEASE.jar:/usr/local/avamar/lib/spring-web-3.1.1.RELEASE.jar:/usr/local/avamar/lib/spring-jdbc-3.1.1.RELEASE.jar:/usr/local/avamar/lib/spring-tx-3.1.1.RELEASE.jar::/usr/local/avamar/lib/spring-security-config-3.1.0.RELEASE.jar:/usr/local/avamar/lib/spring-security-core-3.1.0.RELEASE.jar:/usr/local/avamar/lib/spring-security-crypto-3.1.0.RELEASE.jar:/usr/local/avamar/lib/spring-security-web-3.1.0.RELEASE.jar -Djava.rmi.server.hostname=replace_hfsaddr_here -Djava.util.logging.config.file=/usr/local/avamar/lib/mcserver_logging.properties -Djava.security.egd=file:/dev/./urandom -Djava.io.tmpdir=/usr/local/avamar/var/mc/server_tmp -Djava.util.prefs.PreferencesFactory=com.avamar.mc.util.MCServerPreferencesFactory -Djavax.xml.parsers.DocumentBuilderFactory=org.apache.xerces.jaxp.DocumentBuilderFactoryImpl -Djavax.net.ssl.keyStore=lib/rmi_ssl_keystore -Djavax.net.ssl.trustStore=lib/rmi_ssl_keystore -Dfile.encoding=UTF-8 -Dlog4j.configuration=file:///usr/local/avamar/lib/log4j.properties  com.avamar.mc.CLI --start >> /usr/local/avamar/var/mc/server_log/mcserver.out 2>&1 &

=== BEGIN === check.mcs (prestart)

   checking if localhost is correctly defined in /etc/hosts

   checking if entry for VDP is correctly defined in /etc/hosts

   checking if javaDir preference is set: ("/usr/java/latest" ne "REPLACE_JAVADIR_HERE")

   checking if hfsaddr preference is set: ("REPLACE_HFSADDR_HERE" ne "REPLACE_HFSADDR_HERE")

ERROR: check.mcs: [VDP] checking if hfsaddr preference is set CHECK FAILED: ("REPLACE_HFSADDR_HERE" ne "REPLACE_HFSADDR_HERE")

Must be a default mcserver.xml and perhaps i should try to modify it with good values ?

Otherwise may I try to installe 6.1.3 in place of the other (same name, same IP, etc) ? Will it retrieve all the datastores and job plannings back ?

[SavkoorSuhas]

The mcserver.xml file under /prefs folder should have these two entries for hfsaddr:
            <entry key="local_hfsaddr" value="vdp-dest.happycow.local" />

            <entry key="hfsaddr" value="vdp-dest.happycow.local" />

the value within quotes should be the FQDN of your vdp appliance. If this is reflected incorrectly, please change it and then try restarting MCS.

Suhas

[ascheltens22]

Nope,

Still have same error... Bad luck !

[SavkoorSuhas]

Well, for redeploying VDP, we usually redeploy the same version with existing drives.
But since we are moving from 6.1.2 to 6.1.3, this will be a minor upgrade, so it shouldn't really matter. So yes, you should be good, if you redeploy 6.1.3 and use data disks from 6.1.2

This will cause loss of backup jobs, but all your backup data will be safe.

Re-deploy VDP with existing storage ~ virtuallyPeculiar

If you would like to get the existing server up, then would recommend you to open a ticket with us.

I have tried most of the things I can, I would try a rollback to a last good known state or restore mcs from a flush. But this would be good to be done with VMware Support online.

Suhas

[ascheltens22]

Hi,

Didn't work either.

Well, finally I deleted everything and reinstalled the latest version and recreated th backup jobs.

No more backups !

Wish I had not deleted the first snapshot !

Thank you for your help !