6 Replies Latest reply on Oct 16, 2020 2:20 AM by kimmotuhkanen

    Errors Containing "The object or item referred to could not be found"

    MattVilla Lurker

      Hi,

       

      I'm noticing numerous errors being raised in the vpxa.log file. Not sure why. Entries are added every minute. Looks like vSphere is raising errors about a datastore. Can someone help with this?

       

      2019-06-25T20:15:10.511Z info vpxa[2105145] [Originator@6876 sub=Default opID=sps-Main-408949-939-a5-a7] [VpxLRO] -- ERROR task-3766 -- catalogS

      --> Result:

      --> (vim.fault.NotFound) {

      -->    faultCause = (vmodl.MethodFault) null,

      -->    faultMessage = <unset>

      -->    msg = "The object or item referred to could not be found."

      --> }

      --> Args:

      -->

      --> Arg catalogChangeSpec:

      --> (vim.vslm.CatalogChangeSpec) {

      -->    datastore = 'vim.Datastore:ds:///vmfs/volumes/5cfee289-1a7e9580-1a6c-08f1ea9882f0/',

      -->    startVClockTime = (vim.vslm.VClockInfo) {

      -->       vClockTime = 0

      -->    },

      -->    fullSync = false

      --> }

        • 2. Re: Errors Containing "The object or item referred to could not be found"
          EdSp Novice

          Did you have any luck with this issue?

          Thanks,Ed

          • 3. Re: Errors Containing "The object or item referred to could not be found"
            marcethz Novice

            Hi Ed

             

            No unfortunatly not. I just discovered this by accident tailing vpxd log while searching for another error. As far as I can see there is nothing really affected by this. Did you observe anything in your environment that is related to this error ?

            • 4. Re: Errors Containing "The object or item referred to could not be found"
              vmhomelab Lurker

              Hello everyone,

               

              I have the same problem as stated in the first post. That’s why I want to push this topic a little bit.

              I´ve already done some investigation because I have this error for a long time.

               

              There are a few requirements for this error to show:

               

              1. ESXi 6.7 u2 (Buildnumber: 13006603) to the latest 6.7 u3b (Buildnumber: 15160138)

              2. ESXi Host is connected to a vCenter/vShere server for management

              3. ESXi Host not in maintenance mode

              4. At least one configured Datastore besides the rest of the boot drive (ssd/hdd)

               

              I first encountered these errors after updating my ESXi host from version 6.7 u1 (EP 07) (Buildnumber: 13004448) to 6.7 u2 (Buildnumber: 13006603)

              After reverting the upgrade everything was fine again.

              I thought it was maybe just a bug in the new version but it is still present in U3b that’s why I think there is maybe something that we (and or I) miss in our configuration?

              I mean the log is somehow clear there is something that the vCenter is searching for (or wants to do) but is not present.

               

              So I created a nested ESXi and vCenter environment (tested different versions from 6.7 u2 to the latest 6.7 u3b (fresh installs and upgrades)) to test things out.

               

              I tried to delete all Datastores and recreated them over the vCenter -> same error

              I tried to configure the different Host cache and swap settings on the datastore -> same error

               

              This error is really annoying because there is almost nothing to find about it and it also shows up in the log field of VMware Workstation (which I use as a convenient way to connect to the vm console):

               

              Capture.PNG

               

              But the error is not connected with the workstation client the log still shows this error.

               

              Here is also my vpxa.log from a few of the error events:

              2019-12-16T22:17:22.755Z info vpxa[2099552] [Originator@6876 sub=vpxLro opID=HB-host-32@105-4f7d333b-6a] [VpxLRO] -- BEGIN lro-73 -- vpxa -- vpxapi.VpxaService.getChanges -- 52e48445-c569-334c-f840-3b6e67a419e4

              2019-12-16T22:17:22.758Z info vpxa[2099552] [Originator@6876 sub=vpxLro opID=HB-host-32@105-4f7d333b-6a] [VpxLRO] -- FINISH lro-73

              2019-12-16T22:17:33.522Z info vpxa[2099177] [Originator@6876 sub=vpxLro opID=sps-Main-532158-159-97-80] [VpxLRO] -- BEGIN task-14 -- catalogSyncManager -- vim.vslm.host.CatalogSyncManager.queryCatalogChange -- 52e48445-c569-334c-f840-3b6e67a419e4

              2019-12-16T22:17:33.532Z info vpxa[2099184] [Originator@6876 sub=vpxLro opID=sps-Main-532158-159-97-80] [VpxLRO] -- FINISH task-14

              2019-12-16T22:17:33.533Z info vpxa[2099184] [Originator@6876 sub=Default opID=sps-Main-532158-159-97-80] [VpxLRO] -- ERROR task-14 -- catalogSyncManager -- vim.vslm.host.CatalogSyncManager.queryCatalogChange: vim.fault.NotFound:

              --> Result:

              --> (vim.fault.NotFound) {

              -->    faultCause = (vmodl.MethodFault) null,

              -->    faultMessage = <unset>

              -->    msg = "The object or item referred to could not be found."

              --> }

              --> Args:

              -->

              --> Arg catalogChangeSpec:

              --> (vim.vslm.CatalogChangeSpec) {

              -->    datastore = 'vim.Datastore:ds:///vmfs/volumes/5ddaae48-11495169-cea0-005056bef362/',

              -->    startVClockTime = (vim.vslm.VClockInfo) {

              -->       vClockTime = 1

              -->    },

              -->    fullSync = false

              --> }

              2019-12-16T22:17:33.680Z info vpxa[2099178] [Originator@6876 sub=vpxLro opID=sps-Main-532158-159-f7-af] [VpxLRO] -- BEGIN task-15 -- catalogSyncManager -- vim.vslm.host.CatalogSyncManager.queryCatalogChange -- 52e48445-c569-334c-f840-3b6e67a419e4

              2019-12-16T22:17:33.723Z info vpxa[2099174] [Originator@6876 sub=vpxLro opID=sps-Main-532158-159-f7-af] [VpxLRO] -- FINISH task-15

              2019-12-16T22:17:33.723Z info vpxa[2099174] [Originator@6876 sub=Default opID=sps-Main-532158-159-f7-af] [VpxLRO] -- ERROR task-15 -- catalogSyncManager -- vim.vslm.host.CatalogSyncManager.queryCatalogChange: vim.fault.NotFound:

              --> Result:

              --> (vim.fault.NotFound) {

              -->    faultCause = (vmodl.MethodFault) null,

              -->    faultMessage = <unset>

              -->    msg = "The object or item referred to could not be found."

              --> }

              --> Args:

              -->

              --> Arg catalogChangeSpec:

              --> (vim.vslm.CatalogChangeSpec) {

              -->    datastore = 'vim.Datastore:ds:///vmfs/volumes/5ddaa89c-0e6f7a0c-0a99-005056bef362/',

              -->    startVClockTime = (vim.vslm.VClockInfo) {

              -->       vClockTime = 1

              -->    },

              -->    fullSync = false

              --> }

              2019-12-16T22:17:37.738Z info vpxa[2099183] [Originator@6876 sub=vpxLro opID=PollQuickStatsLoop-2dfc6514-19] [VpxLRO] -- BEGIN lro-76 -- vpxa -- vpxapi.VpxaService.fetchQuickStats -- 52e48445-c569-334c-f840-3b6e67a419e4

              2019-12-16T22:17:37.738Z info vpxa[2099183] [Originator@6876 sub=vpxLro opID=PollQuickStatsLoop-2dfc6514-19] [VpxLRO] -- FINISH lro-76

              2019-12-16T22:18:34.530Z info vpxa[2099179] [Originator@6876 sub=vpxLro opID=sps-Main-532158-159-78-38] [VpxLRO] -- BEGIN task-16 -- catalogSyncManager -- vim.vslm.host.CatalogSyncManager.queryCatalogChange -- 52e48445-c569-334c-f840-3b6e67a419e4

              2019-12-16T22:18:34.539Z info vpxa[2099195] [Originator@6876 sub=vpxLro opID=sps-Main-532158-159-78-38] [VpxLRO] -- FINISH task-16

              2019-12-16T22:18:34.539Z info vpxa[2099195] [Originator@6876 sub=Default opID=sps-Main-532158-159-78-38] [VpxLRO] -- ERROR task-16 -- catalogSyncManager -- vim.vslm.host.CatalogSyncManager.queryCatalogChange: vim.fault.NotFound:

              --> Result:

              --> (vim.fault.NotFound) {

              -->    faultCause = (vmodl.MethodFault) null,

              -->    faultMessage = <unset>

              -->    msg = "The object or item referred to could not be found."

              --> }

              --> Args:

              -->

              --> Arg catalogChangeSpec:

              --> (vim.vslm.CatalogChangeSpec) {

              -->    datastore = 'vim.Datastore:ds:///vmfs/volumes/5ddaae48-11495169-cea0-005056bef362/',

              -->    startVClockTime = (vim.vslm.VClockInfo) {

              -->       vClockTime = 1

              -->    },

              -->    fullSync = false

              --> }

              2019-12-16T22:18:34.692Z info vpxa[2099169] [Originator@6876 sub=vpxLro opID=sps-Main-532158-159-7f-c3] [VpxLRO] -- BEGIN task-17 -- catalogSyncManager -- vim.vslm.host.CatalogSyncManager.queryCatalogChange -- 52e48445-c569-334c-f840-3b6e67a419e4

              2019-12-16T22:18:34.734Z info vpxa[2099186] [Originator@6876 sub=vpxLro opID=sps-Main-532158-159-7f-c3] [VpxLRO] -- FINISH task-17

              2019-12-16T22:18:34.734Z info vpxa[2099186] [Originator@6876 sub=Default opID=sps-Main-532158-159-7f-c3] [VpxLRO] -- ERROR task-17 -- catalogSyncManager -- vim.vslm.host.CatalogSyncManager.queryCatalogChange: vim.fault.NotFound:

              --> Result:

              --> (vim.fault.NotFound) {

              -->    faultCause = (vmodl.MethodFault) null,

              -->    faultMessage = <unset>

              -->    msg = "The object or item referred to could not be found."

              --> }

              --> Args:

              -->

              --> Arg catalogChangeSpec:

              --> (vim.vslm.CatalogChangeSpec) {

              -->    datastore = 'vim.Datastore:ds:///vmfs/volumes/5ddaa89c-0e6f7a0c-0a99-005056bef362/',

              -->    startVClockTime = (vim.vslm.VClockInfo) {

              -->       vClockTime = 1

              -->    },

              -->    fullSync = false

              --> }

              2019-12-16T22:18:37.738Z info vpxa[2099188] [Originator@6876 sub=vpxLro opID=PollQuickStatsLoop-2dfc6514-c9] [VpxLRO] -- BEGIN lro-79 -- vpxa -- vpxapi.VpxaService.fetchQuickStats -- 52e48445-c569-334c-f840-3b6e67a419e4

              2019-12-16T22:18:37.738Z info vpxa[2099188] [Originator@6876 sub=vpxLro opID=PollQuickStatsLoop-2dfc6514-c9] [VpxLRO] -- FINISH lro-79

              2019-12-16T22:18:50.393Z info vpxa[2099179] [Originator@6876 sub=vpxaInvtHost] Increment master gen. no to (106): Event:VpxaEventHostd::CheckQueuedEvents

              2019-12-16T22:18:52.785Z info vpxa[2099186] [Originator@6876 sub=vpxLro opID=HB-host-32@106-4755e2f-2] [VpxLRO] -- BEGIN lro-80 -- vpxa -- vpxapi.VpxaService.getChanges -- 52e48445-c569-334c-f840-3b6e67a419e4

              2019-12-16T22:18:52.785Z info vpxa[2099186] [Originator@6876 sub=vpxLro opID=HB-host-32@106-4755e2f-2] [VpxLRO] -- FINISH lro-80

               

              Does anyone have a solution for this?

              I am willing to test almost anything in my nested test environment to nail down this error/problem.

              I really want to update the main hosts to the latest version mainly for security concerns.

               

               

              Thank you very much, Johannes

              • 5. Re: Errors Containing "The object or item referred to could not be found"
                Izuku Lurker

                I'm also getting this error every 60 seconds or so.  Not sure what it's looking for, but the folder does exist.

                 

                2020-06-24T15:07:38.049Z info vpxa[5402227] [Originator@6876 sub=vpxLro opID=sps-Main-878893-982-e1-b1] [VpxLRO] -- BEGIN task-1739 -- catalogSyncManager -- vim.vslm.host.CatalogSyncManager.queryCatalogChange -- 52ba6490-a984-c58b-9e6b-8c77a481309f

                2020-06-24T15:07:38.057Z info vpxa[5402223] [Originator@6876 sub=vpxLro opID=sps-Main-878893-982-e1-b1] [VpxLRO] -- FINISH task-1739

                2020-06-24T15:07:38.057Z info vpxa[5402223] [Originator@6876 sub=Default opID=sps-Main-878893-982-e1-b1] [VpxLRO] -- ERROR task-1739 -- catalogSyncManager -- vim.vslm.host.CatalogSyncManager.queryCatalogChange: vim.fault.NotFound:

                --> Result:

                --> (vim.fault.NotFound) {

                -->    faultCause = (vmodl.MethodFault) null,

                -->    faultMessage = <unset>

                -->    msg = "The object or item referred to could not be found."

                --> }

                --> Args:

                -->

                --> Arg catalogChangeSpec:

                --> (vim.vslm.CatalogChangeSpec) {

                -->    datastore = 'vim.Datastore:ds:///vmfs/volumes/5cb4b11a-be3612f4-a14b-a4bf015a5d79/',

                -->    startVClockTime = (vim.vslm.VClockInfo) {

                -->       vClockTime = 1

                -->    },

                -->    fullSync = false

                --> }

                • 6. Re: Errors Containing "The object or item referred to could not be found"
                  kimmotuhkanen Lurker

                  Dunno if this is correct but I spend some time going trough the logs when I noticed the same... but vCenter vmware-sps service runs somekinda datastore poller all the time from all the datastores is sees from all the ESXi hosts it manages. And at least in my case we have hosts in different clusters that sees different datastores.

                   

                  If you tail the vCenter /storage/log/vmware/vmware-sps/sps.log, and as soonest datastore poller starts rolling, in ESXi side you will see same "sps-Main-223474-979"

                  ...

                  2020-10-16T11:41:40.837+03:00 [datastore list poller] INFO  opId=sps-Main-223474-979 com.vmware.vslm.globalcache.FCDDatastore - FCD Datastore Initialized: ds:///vmfs/volumes/5bbcb0eb-d13a17f6-6a34-00215a9b1640/ [ { datastore-50} ]

                  2020-10-16T11:41:40.837+03:00 [datastore list poller] INFO  opId=sps-Main-223474-979 com.vmware.vslm.globalcache.FCDDatastore - FCD Datastore Initialized: ds:///vmfs/volumes/570213cd-c71adbda-05f4-00215a9b14c4/ [ { datastore-33} ]

                  2020-10-16T11:41:40.837+03:00 [datastore list poller] INFO  opId=sps-Main-223474-979 com.vmware.vslm.globalcache.FCDDatastore - FCD Datastore Initialized: ds:///vmfs/volumes/58353bd2-4464af00-e81c-00215a9b1644/ [ { datastore-35} ]

                   

                  ESXi hostd.log

                  --> [context]zKq7AVICAgAAAEpT5wAPaG9zdGQAACxONmxpYnZtYWNvcmUuc28AALC+GwCQnBcBgyJjaG9zdGQAAaB2agGuD8MBq5fIgh84SgFsaWJ2aW0tdHlwZXMuc28AAUXIwgE90cIAteUoAJPpKACrxDYDO30AbGlicHRocmVhZC5zby4wAAR9nw5saWJjLnNvLjYA[/context]

                  2020-10-16T08:43:11.327Z info hostd[2100349] [Originator@6876 sub=Vimsvc.TaskManager opID=sps-Main-223474-979-98-be-f880 user=vpxuser:VSPHERE.LOCAL\vpxd-extension-8f6480ed-b9e0-48c0-99b6-d99a2ff340bb] Task Completed : haTask--vim.vslm.host.CatalogSyncManager.queryCatalogChange-4126283950 Status error

                  2020-10-16T08:43:12.338Z info hostd[2099996] [Originator@6876 sub=Vimsvc.TaskManager opID=sps-Main-223474-979-19-27-f886 user=vpxuser:VSPHERE.LOCAL\vpxd-extension-8f6480ed-b9e0-48c0-99b6-d99a2ff340bb] Task Created : haTask--vim.vslm.host.CatalogSyncManager.queryCatalogChange-4126283951

                  2020-10-16T08:43:12.363Z info hostd[2100346] [Originator@6876 sub=Default opID=sps-Main-223474-979-19-27-f886 user=vpxuser:VSPHERE.LOCAL\vpxd-extension-8f6480ed-b9e0-48c0-99b6-d99a2ff340bb] Transfer to exception eraro code: 403, message:

                  2020-10-16T08:43:12.363Z info hostd[2100346] [Originator@6876 sub=Default opID=sps-Main-223474-979-19-27-f886 user=vpxuser:VSPHERE.LOCAL\vpxd-extension-8f6480ed-b9e0-48c0-99b6-d99a2ff340bb] AdapterServer caught exception: N3Vim5Fault8NotFound9ExceptionE(Fault cause: vim.fault.NotFound

                   

                  is "Transfer to exception eraro code: 403, message:" error code just that it cant find the host/mob/?moid=datastore-number ? 403 html error code is resource is forbitten..

                   

                  # ls /vmfs/volumes/56f3a356-630f5d14-fe08-00215a9b1644/

                  ls: /vmfs/volumes/56f3a356-630f5d14-fe08-00215a9b1644/: No such file or directory

                   

                  Nice vmware error handling?

                   

                   

                  EDIT:

                   

                  yeah and same seen in vpxa.log

                  ..

                  2020-10-16T09:03:28.863Z info vpxa[2100960] [Originator@6876 sub=Default opID=sps-Main-223474-979-c9-d4] [VpxLRO] -- ERROR task-339468 -- catalogSyncManager -- vim.v

                  slm.host.CatalogSyncManager.queryCatalogChange: vim.fault.NotFound:

                  --> Result:

                  --> (vim.fault.NotFound) {

                  -->    faultCause = (vmodl.MethodFault) null,

                  -->    faultMessage = <unset>

                  -->    msg = "The object or item referred to could not be found."

                  --> }

                  --> Args:

                  -->