DCasota
Expert
Expert

Homage to VMUG future homelab-aware vCenter Appliance setup error message(s)

Jump to solution

Lately with no Skyline integration, I wanted to stage/onboard a VMUG-based homelab. Well, in this pre-2030 scenario, Skyline has neither a button nor a conversational techtalk AI to do a smoothie device onboarding. Here the story what happened.

 

Yeah, iso. Remember?

VMware vCenter Server aka VCSA has been here for a while. Particularly VCSA8, I started installing that version on an ESXi8 laptop host, connected to home "internet". At that time, this was atrocious.

Like I said about Skyline, nothing augmented came back. A dinosaur VMUG homelab for testing purposes.

First, I've extracted the ova from the VMware-VCSA-all-8.0.0-20519528.iso and installed it directly on ESXi8. It always ended with an non-comprehensive error message on a webbrowser page https://ip:5480/configurev2.

Some research later, I've found out from communities thread entries that it would be better to enforce an environment already with configured dns server, reverse lookup, ntp, etc. What a mess. It didn't help.

Some research later, I've found out that instead of using the .ova, simply start on windows the ui installer and select install vCenter server. With a new issue about a .rpm in the 96%-97% installation phase (tested multiple times) and after some research, I found out that it may be better to switch to vcsa 8.0a as the newer version fixes some setup issues. Latest is greatest, right?

Well then, VMware-VCSA-all-8.0.0-20920323.iso downloaded, and using the Ui installer - static ip and gateway and with no fqdn - it failed again. This time with 'Failed to install VMware-spherelet_1_23-2.0.0-10658411.x86_64.rpm'.

 

I don't want to leave the media to do internet research every time an error occurs.

Guess what. No smoothie suggestions, no VMware techtalk AI in ESXi, just silence of the keyboard.

'Failed to install VMware-spherelet_1_23-2.0.0-10658411.x86_64.rpm' - what a message. vSpherelet runs on each ESXi node to control and communicate with the vSphere pod. That's good but how to give feedback about the issue? I mean it could mean the rpm is defect. 

It took a while until I admitted that my hardware was too small for understanding. But if Skyline next generation is truth, is 2023 the year of techtalk AI with onboarding/offboarding/mutations? I am a user with a spare 16gb ram laptop having tried to build a VMUG-based future homelab.

---

Hopefully you enjoyed the tiny story. Don't forget to ram.

0 Kudos
1 Solution

Accepted Solutions
Kinnison
Expert
Expert

Hi,


I raise my hands and give up, but one last thing I can say, just for fun I deployed (and it's not the first time) a VCSA appliance version 8.0 on a nested ESXi 8.0 host. I didn't measure the time with the stopwatch but it didn't take me more than thirty/forty minutes.


Regards,
Ferdinando

View solution in original post

0 Kudos
12 Replies
Kinnison
Expert
Expert

Hi,


The story is well written but the moral, from my point of view, is that if you want to use the latest VMware technologies you need a "pot of resources".
Otherwise I'm afraid it's just going to end up with wasted time.


Regards,
Ferdinando

0 Kudos
DCasota
Expert
Expert

Chronicling a journey, yes, it started a while ago. 

VMUG homelab vCenter Appliance setups: There are too many manual "if this, then do that".

Observe how young new engineers manage to install products. Where in ESXi DCUI can I get vCenter Server installed? Help>Tools and links>Get vCenter Server. There are too many spare links. Time wasted? No, not because they won't do it. That's my observation.

See the logfile attached in the first thread entry: I can remember a time where you had VMware scientist engineers and they meticulously documented knowledge base article after knowledge base article about logfile errors and solvings. To go back to that era is the wrong direction. KB suggestions should be included in the product. Skyline has started with that. And, with sort of conversational techtalk AI, that will become much more    atarimae .

Suggestions are welcome. Check the logfile. How to succeed this VMUG homelab vCenter Appliance setup ?

0 Kudos
Kinnison
Expert
Expert

@DCasota, hi,


From your logs I see this repeated warning:
"ServerFaultCode: Failed to authenticate with the guest operating system using the supplied credentials".


What I can say, based on my personal experience (i.e. circumscribed in my IT context) with the VCSA version 8.x object, is that from the completion of phase "one" before continuing with phase "two" it is better to wait a few minutes because the start-up of some services (with tiny deployment) takes a fair amount of time (more than before, compared with version 7.x).


Other than that I think the point is that when an install or upgrade fails there is no concrete (i mean really useful) description of the problem and/or what course of action "should be adopted other than starting over", hopefully the next time the same (or a different) problem does not occur again. Having the vCenter object unavailable for an unspecified time can lead to a consistent impact of everything that depends on it and thus, again from my very personal point of view, can be a sort of "denial of a service" no different from any other.


Regards,
Ferdinando

0 Kudos
DCasota
Expert
Expert

In time of Cross Cloud digitization recession pliers be aware when changing a default password.

Tiny setup does not work. Convince me.

Distinction for the vcsa8 setup: Whenever you configure a tiny setup as deployment size, VMware observability service fails to start. As said, pre-2030, you won't get help from Skyline.

Read 'Deploy the vCenter Server Appliance by Using the GUI', pages 6,7.
You shall set a password for the root user. Does that setup work? No, not. But before getting to the opinion edge of 'tiny setup works never ever', no, it may work. Pages 6,7 - keep in mind to synchronize clocks first. You set the password while deploying the virtual appliance.

If you started the old 6.7 way of extracting and deploying the ova, bad idea, first, you will see on the vcsa console during setup a message 'root password not set'. Having changed it, doesn't help - the setup fails. Luckily, there are two stage phases. But, you can't begin stage2 without having accomplished stage1 and for entering stage2 you must having changed the default root password. There are slight differences when you are on ipv6, dhcp ipv4 and static ipv4. For Microsoft clients, please, deploy the vCenter Server Appliance by Using the GUI. But, do not believe that on your Windows 10 client that bridging a wired LAN for the vSphere homelab and a WiFi LAN connected to the home router works. It does not. Well, it does, but the dhcp IP the vcsa gets is the same as on your Windows 10 client. This has to do with the underlying old Photon OS dhcp client implementation. The good news, this will change in the future.

Now, having started the setup of the vCenter Server Appliance by Using the GUI, tiny setup configuration, dhcp ipv4 and ipv6, with a vCenter Appliance root password VMware1!VMware1! and waiting 30 minutes, this setup will work, right? Guess what low blow, you shall not pass stage1.

Meanwhile, yeah, the logfile shows 'CertificatePeriod expired="true"', 'Failed to authenticate with the guest operating system using the supplied credentials' and, again, 'Failed to install VMware-spherelet_1_23-2.0.0-10658411.x86_64.rpm'.

---

The text above is not made by ChatGpt.

0 Kudos
Kinnison
Expert
Expert

@DCasota, hi,


I didn't say that the "tiny" deployment model doesn't work, but I tell you that at boot time the consumption of "CPU" and "RAM" resources is such as to often and willingly generate error messages for "exhaustion " of one, the other or both, and the time for starting the services can easily reach 15/20 minutes.


You see, when you update a vCenter object the service must first be stopped and then restarted and when you get to the so-called "post install" activity then, often, the virtual machine is restarted, you want to put that if the restart of the services / virtual machine takes five minutes instead of fifteen (or whatever it takes) the unavailability time of the services provided by the vCenter object is consequently reduced.


Now, forgive me, I didn't tell you to take my opinions at "face value", I can only tell you about my personal experiences.
Never pretended that you (or anyone else) have to believe it.


And sorry again, but now I'm really struggling to understand what you want to tell me, other than that I understand your frustration.


Regards,
Ferdinando

0 Kudos
DCasota
Expert
Expert

Mayday, mayday.

warn: User clicked on the ok button to Close Dialog.

Troubleshooting time.

VCSA comes with appliancesh. It's a tool which lists staged and installed packages like tdnf in Photon OS.

Enable ssh, then connect to the appliance. Type

appliancesh software-packages list >/tmp/packages

Strange. Most packages have an install date of 2022-05-05 and only a few packages have 2023-01-23. Why 2022-05-05 ?

In addition, the line VMware-spherelet_1_24 2.0.1 10658411 2023-01-23 is suspicious. Wasn't the error message 'Failed to install VMware-spherelet_1_23-2.0.0-10658411.x86_64.rpm' ?

 

---

to be continued. Maybe.

 

0 Kudos
Kinnison
Expert
Expert

Hi,


I raise my hands and give up, but one last thing I can say, just for fun I deployed (and it's not the first time) a VCSA appliance version 8.0 on a nested ESXi 8.0 host. I didn't measure the time with the stopwatch but it didn't take me more than thirty/forty minutes.


Regards,
Ferdinando

0 Kudos
DCasota
Expert
Expert

It was always the same. cat /var/log/firstboot/cloudvm.log ended up with


Installing : VMware-wcpovf-3.0.480-20884059.x86_64.rpm
Installing : VMware-spherelet-2.0.1-10658411.x86_64.rpm
Installing : VMware-spherelet_1_24-2.0.1-10658411.x86_64.rpm
Installing : VMware-spherelet_1_23-2.0.0-10658411.x86_64.rpm
USER PID ACCESS COMMAND
/var/lib/rpm/Packages:
2023-01-23T15:22:36.954Z: Failed to install VMware-spherelet_1_23-2.0.0-10658411.x86_64.rpm
2023-01-23T15:22:36.956Z: Starting ssh...
2023-01-23T15:25:32.886Z: Pruning sensitive information.

 

and cat /var/log/vmware/deploy/rpm_install_output.log ended with

Updating / installing...
VMware-vstats-8.0.0.10100-10658411 ########################################
Preparing... ########################################
Updating / installing...
VMware-wcp-0.1.1-20841510 ########################################
/usr/sbin/usermod.bk -g lwisRegReader wcp
Preparing... ########################################
Updating / installing...
VMware-wcpovf-3.0.480-20884059 ########################################
Preparing... ########################################
Updating / installing...
VMware-spherelet-2.0.1-10658411 ########################################
Preparing... ########################################
Updating / installing...
VMware-spherelet_1_24-2.0.1-10658411 ########################################
Preparing... ########################################
Updating / installing...
VMware-spherelet_1_23-2.0.0-10658411 ########################################
error: unpacking of archive failed on file /storage/lifecycle/vmware-wcp/spherelet/k8s-1.23/spherelet.vib;63cea63c: cpio: read failed - Inappropriate ioctl for device
error: VMware-spherelet_1_23-2.0.0-10658411.x86_64: install failed

Well, even in /storage/lifecycle/vmware-wcp/spherelet the k8s-1.23 directory was empty.

root@localhost [ / ]# ls -ll -R /storage/lifecycle/vmware-wcp/spherelet/
/storage/lifecycle/vmware-wcp/spherelet/:
total 8
drwxr-xr-x 2 root root 4096 Jan 23 15:22 k8s-1.23
drwxr-xr-x 2 root root 4096 Jan 23 15:22 k8s-1.24

/storage/lifecycle/vmware-wcp/spherelet/k8s-1.23:
total 0

/storage/lifecycle/vmware-wcp/spherelet/k8s-1.24:
total 66708
-rw-r--r-- 1 root root 68304368 Dec 8 08:26 spherelet.vib
-rw-r--r-- 1 root root 65 Dec 8 08:26 version.json
root@localhost [ / ]#

 

Bad Santa quote :

“The world ain't fair. You've gotta take what you need when you can get it. You've gotta learn to stand up for yourself.”

Some weeks ago, I bought this HP 250 G8 laptop. Lowcost, but very useful. Big mistake.

The lowcost laptop' ethernet adapter didn't play well with ESXi7. Hence, I was tinkering with the vmkusb fling, and, success! It worked, I was the superAdmin for my homelab kiddies again. Yeah, then, ESXi 8.0 came along, wohoo. After a couple of hours, I found out that TPM 2.0 is supported, but, for the moment not the VMware supported way. I can live that, free cookies for all, what are you waiting for ? it's Christmas time !

Whaaat, spherelet vibs are failing? Remember ESXi 7.0U3d? Spherelet vib installations fail on vTPM hosts.
No, no, no, imagine that, could secure boot of ESXi with missing TPM 2.0 support be the root cause ? I've updated the bios to the latest greatest firmware. Well, it's January 2023, headache prime time. Why did I choose the wrong laptop model?

---

@Kinnison Thanks for the support 🙂

 

0 Kudos
Kinnison
Expert
Expert

@DCasota, hi,


So, if you want we can try again, but if you get angry (I can understand your frustration) we're going nowhere.
Do you know why I repeatedly deploy multiple VCSA objects? Because I use them to validate those "modifications" which I then applied to the one that manages my small "IT infrastructure" and that the software vendor (surely for his own good reasons) doesn't support (or even consider).


In the meantime, I wish you many good things,


Ferdinando

0 Kudos
DCasota
Expert
Expert

 

“I have a question” by Bill Dobbs

If you haven't read this book, you missed reading a good book! A brave man. I met Bill at an IT company in Switzerland. He gave English lessons for employees. Such a pleasure.
If you haven't read his book, please do it and have a look to page 58/59.

That's how it started, sort of. Now, assuming that the available iso bits of vcsa8 never ever will be installable on the homelab laptop because of no VMware-supported TPM 2.0, how to avoid page 58/59 content? Is it possible? Indeed. Action.

Same environment, same settings, stage 1 done, stage 2 done. Boom.

Nice, a running vcsa7 accomplished. Should I test an upgrade?

Let's do tiny steps. The best update available is 7.0.0.10700. Pre-update checks - passed.

A occurred and then B occurred. Therefore A is the cause of B. Never heard from the Loci method ? Iso, remember? Project Pacific had been announced in 2019 and luckily, I archived at the beginning of the Corona lockdown time the VMware-VCSA-all-7.0.0-15934039.iso bits too. What a fine mess.

0 Kudos
DCasota
Expert
Expert

Dear readers, I apologize that I cannot offer you a better happy-end of this story for now. Meh, a so-called expert with a low-cost laptop ignoring already tested and recommended homelab bills of materials, that's pretentious. Maybe he managed to get the vcsa8 up and running or he has been arrested in peace with 16gb ram, vcsa 7.0.3.01200 and no real TPM 2.0.

A morale boost:
The Skyline Proactive Unit Needs Your Help ! https://blogs.vmware.com/kb/15207-2
Get you reputation points as case agent. Follow e.g. @Kinnison. People overestimate what they can achieve in 1 year und underestimate what the can do for the 2030 goal.

0 Kudos
Kinnison
Expert
Expert

Hi,

No one must follow me or even pay attention to what I say because I'm not an expert at all, but I try to prevent certain problems because then I konw I would have difficulty in curing them. Some requirements have not been "discovered", they are reported written in the product documentation, perhaps it could have been written better but it is the same.
And then, excuse me, the vSphere 7 product line which has now reached appreciable stability, what is wrong with it.
There are many who don't even think about implementing the vSphere 8 line in production contexts (rightly or wrongly).

Your notebook has nothing wrong with it, it's a good machine that was designed to do something else and with completely different software.


Many good things,
Ferdinando

0 Kudos