Why Isn't VMWare Fusion Using All the CPU Allocated to the VMs It Runs

[danallenhtn]

My hope is you might be able to help me determine whether VMs are  using all the resources allocated to them, particularly the CPUs.

The performance of the Debian (Linux) VMs I run under VMWare Fusion is good, but it might be better under heavy loads.  Right now, one of the VMs is working at capacity on a long job.

According to MacOS Activity Monitor, the VMs and VMWare are using less than the CPU allocated.

This system specs and resource allocations to VMs and VMWare are show below:

Here are system monitors showing what the level of utilization is below the allocated levels, even though the system is working at capacity on a long complicated job.

What am I doing wrong?

Thank you!

[Technogeezer]

I would start at the guest.

Is your guest application a multi-threaded application that can take advantage of multiple CPUs?

Do you have it configured to do so?

And are you throwing enough work (through multiple connections/queries) to allow it to multi-thread. You say it's a long job - could that job be running on a single thread which would mean the aggregate performance of your system would be limited to that provided by a single core.

Also, is the work totally CPU bound or I/O bound? You aren't indicating what the I/O activity is - which could be limiting CPU usage.

[danallenhtn]

Thank you for taking a look at my question.  I don't know how to answer some of the questions you asked.  There is more than one kind of long-running task, so I am going to focus on one to see if I can make progress.

LONG RUNNING TASK 1 - LOADING  ~1,000 csv files into a table. Average # of records per file ~2,500.  ~2.5M records total.  Runtime ~6 minutes

The routine runs like this:

1. Before routine starts, ~1,000 csv files are stored in a directory on the guest server.

2. PHP script forms an array containing the names of the files.

SO FAR SO GOOD, NOTHING IS SLOW

THE FOLLOWING STEP TAKES ~6 minutes

START STEP

foreach (csvfile):
   Read/parse a line of the file into  a sql statement.
   sql statement is formed as: insert into target_table (fieldnamea, fieldnameb, fieldnamec...) values (valuea, valueb, valuec...)
   runSQL
endforeach

END STEP

I don't know anything about doing this in a multithreaded manner.  I did not know the programming has to be done in a particular way to take advantage of the multiple cpus. 

Obviously, I have some things to learn here.  I am going to chip at this problem over time, posting results here as I figure things out.

If there any other comments or questions I should be focusing on, please do not hesitate to post them.

Thank you again for providing perspective. 

[Technogeezer]

Yes, applications have to be designed in order to get max usage of multiple CPUs in any computer. In order to take advantage of multiple CPUs in a computer (physical or virtual), one or more of the following must be done:

• divide the work of 1 application into more than 1 simultaneously executing processes
• divide the work of 1 application into multiple independent threads within the same process
• run multiple different applications simultaneously.

Otherwise, an application is going to be single threaded and limited to the performance of a single CPU (yes there may be things being done by the operating system on your behalf, but your single threaded code would be limited to 1 CPU’s worth of performance) . 

Here’s what I think based on your description. You appear to be processing one file at a time, reading each line of each file, then inserting the data into the database one row at a time. This process is serial and hence single threaded. You may get a little use of multiple CPUs by the database but you are processing data serially. 

If so, then it’s explainable why you’re not seeing more usage from your multiple CPUs. You’ll likely have to modify what you’re doing for this application to do more work in parallel in order to get more CPU usage. That’s a more general application and database design question than a Fusion question because it’s applicable to any computer (physical or virtual) that has multiple CPUs.  

[danallenhtn]

I have to work in baby steps here.

Working at the mysql command line, enter a sql command.  The command is below.

Is there a way to make this take advantage of multiple cpus?

ENTERED AT THE MYSQL COMMAND LINE

insert into people_master (muid, cert_number, firstname, lastname)
select muid, cert_number, firstname, lastname from staging_table";

This sql statement inserts ~2.5 million rows into the table called people_master.

[danallenhtn]

I am guessing in the scenario I just posted, any programming for multiple cpus is going to have to have been done by the people who programmed the mysql binary.

[RDPetruska]

I would believe that is correct.  Any SQL scripting you are doing would still need to be processed through the SQL engine... and it's the engine which would need to be programmed for multi-threading.

[Technogeezer · Accepted solution]

The insert of one row of a table at a time may not be able to be multi-threaded in and of itself.

One strategy that might work is to try dividing the files you are loading into two sets. Run 2 copies of your script simultaneously, one for each set of files. Monitor CPU usage and elapsed time to see if this makes any difference. 

[ColoradoMarmot]

I would bet that inserting from a single file is single threaded.  Doing multiple at once is definitely more likely to run in parallel.

One other thing - definitely make sure that all the files are in the guest, and not via shared files.

[Technogeezer]

@RDPetruska from what the OP is posting, the database engine is MariaDB. According to the documentation it is capable of multi-threading. But you would have to open multiple connections/sessions from your application/scripting to the database to really take advantage of it. 

[danallenhtn]

This strikes me as a perfect approach for getting some multi-threaded action happening and benefits should be obtainable.

Thank you for your persistence  in formulating an answer to my questions.  I appreciate your efforts immensely.

It was not just this solution you formulated.  You took the time to walk me to this one step at a time, as a low level of sophistication, which in this case fits me.