App Volumes 4 Managers 2 and 3 hung install

[vJoeG]

We've installed the first AVM and got it configured but with some headaches. Frequently in the GUI a popup will appear and the web page will need to be refreshed. With patience, you can get through all the screens and be operational. Still not good

Then we worked on installed the 2nd and 3rd AVM but both hung late in the process and overnight made no progress.

Snippet from the svmanager_setup.log at the end

Caused by:

ODBC::Error: 42000 (262) [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]CREATE DATABASE permission denied in database 'master'.

F:/Program Files (x86)/CloudVolumes/Manager/vendor/bundle/ruby/2.5.0/gems/activerecord-sqlserver-adapter-5.1.6.odbc/lib/active_record/connection_adapters/sqlserver/database_statements.rb:304:in `do'

F:/Program Files (x86)/CloudVolumes/Manager/vendor/bundle/ruby/2.5.0/gems/activerecord-sqlserver-adapter-5.1.6.odbc/lib/active_record/connection_adapters/sqlserver/database_statements.rb:304:in `raw_connection_do'

F:/Program Files (x86)/CloudVolumes/Manager/vendor/bundle/ruby/2.5.0/gems/activerecord-sqlserver-adapter-5.1.6.odbc/lib/active_record/connection_adapters/sqlserver/database_statements.rb:243:in `block in do_execute'

F:/Program Files (x86)/CloudVolumes/Manager/vendor/bundle/ruby/2.5.0/gems/activerecord-5.1.7/lib/active_record/connection_adapters/abstract_adapter.rb:613:in `block (2 levels) in log'

F:/Program Files (x86)/CloudVolumes/Manager/vendor/bundle/ruby/2.5.0/gems/activerecord-5.1.7/lib/active_record/connection_adapters/abstract_adapter.rb:612:in `block in log'

F:/Program Files (x86)/CloudVolumes/Manager/vendor/bundle/ruby/2.5.0/gems/activesupport-5.1.7/lib/active_support/notifications/instrumenter.rb:21:in `instrument'

F:/Program Files (x86)/CloudVolumes/Manager/vendor/bundle/ruby/2.5.0/gems/activerecord-5.1.7/lib/active_record/connection_adapters/abstract_adapter.rb:604:in `log'

F:/Program Files (x86)/CloudVolumes/Manager/vendor/bundle/ruby/2.5.0/gems/activerecord-sqlserver-adapter-5.1.6.odbc/lib/active_record/connection_adapters/sqlserver/database_statements.rb:243:in `do_execute'

F:/Program Files (x86)/CloudVolumes/Manager/vendor/bundle/ruby/2.5.0/gems/activerecord-sqlserver-adapter-5.1.6.odbc/lib/active_record/connection_adapters/sqlserver/database_tasks.rb:10:in `create_database'

F:/Program Files (x86)/CloudVolumes/Manager/vendor/bundle/ruby/2.5.0/gems/activerecord-sqlserver-adapter-5.1.6.odbc/lib/active_record/tasks/sqlserver_database_tasks.rb:22:in `create'

F:/Program Files (x86)/CloudVolumes/Manager/vendor/bundle/ruby/2.5.0/gems/activerecord-5.1.7/lib/active_record/tasks/database_tasks.rb:117:in `create'

F:/Program Files (x86)/CloudVolumes/Manager/vendor/bundle/ruby/2.5.0/gems/activerecord-5.1.7/lib/active_record/tasks/database_tasks.rb:137:in `block in create_current'

F:/Program Files (x86)/CloudVolumes/Manager/vendor/bundle/ruby/2.5.0/gems/activerecord-5.1.7/lib/active_record/tasks/database_tasks.rb:305:in `block in each_current_configuration'

F:/Program Files (x86)/CloudVolumes/Manager/vendor/bundle/ruby/2.5.0/gems/activerecord-5.1.7/lib/active_record/tasks/database_tasks.rb:302:in `each'

F:/Program Files (x86)/CloudVolumes/Manager/vendor/bundle/ruby/2.5.0/gems/activerecord-5.1.7/lib/active_record/tasks/database_tasks.rb:302:in `each_current_configuration'

F:/Program Files (x86)/CloudVolumes/Manager/vendor/bundle/ruby/2.5.0/gems/activerecord-5.1.7/lib/active_record/tasks/database_tasks.rb:136:in `create_current'

F:/Program Files (x86)/CloudVolumes/Manager/vendor/bundle/ruby/2.5.0/gems/activerecord-5.1.7/lib/active_record/railties/databases.rake:27:in `block (2 levels) in <top (required)>'

F:/Program Files (x86)/CloudVolumes/Manager/vendor/bundle/ruby/2.5.0/gems/rake-12.3.3/exe/rake:27:in `<top (required)>'

Tasks: TOP => db:create

(See full trace by running task with --trace)

Ruby-ODBC: Force Encoding ruby result strings to UTF-8

create 0

Checking for db version:

Current version: 20191003173810

Migrating existing database

[sjesse]

ODBC::Error: 42000 (262) [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]CREATE DATABASE permission denied in database 'master'.

Did you use a service account or the SYSTEM account for database permissions? If the account doesn't have a role that allows database creation, you should pre create it and give that use db owner permissions on the database.

[vJoeG]

Yes, the account used was an AD service account for AV specifically.

Looking at the logs from server #1 there are the same errors. But we created an ODBC on that box prior to installing. But on box 2 and 3 we deleted that ODBC since it looked like with version 4 we don't need a pre-created ODBC any longer. At least in the installation guide that's the case.

I'm checking now to add the manually created ODBC and try installing again.

[Shreyskar]

Hi vJoeG,

Looking at the error and since you mentioned the error pops up sporadically, this looks to me an issue on SQL server itself. Perhaps SQL deadlock problem. You can enable debug loggin in appvolumes manager (VMware Knowledge Base ) and then you might see error something like below in production.log file on appvolumes manager:

ERROR Failed to query database during application server initialization: ODBC::Error: 40001 (1205) [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]Transaction (Process ID 107) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.: ALTER DATABASE "App_Volumes" SET READ_COMMITTED_SNAPSHOT ON

P27364 DEBUG    SQL:  (1.1ms) SELECT COUNT(*) FROM [delayed_jobs] WHERE (locked_at IS NOT NULL)

If so this can be resolved by configuring the database with  “READ_COMMITTED_SNAPSHOT” parameter however consult with your SQL team before making such changes.

Shreyskar

[doninside]

is it safe to set this “READ_COMMITTED_SNAPSHOT parameter? We saw that in our old appvolume 2.18 database is set to ON but it is not in our current Avv volumes 4, VERSION 2111 (4.5.0.77). Our DBA team is not so confident to change the behavior of the production database