Configuration element write test

[rkrichevskiy]

I am trying to understand the cause of the problem we have been plaguing our environment. It has to do with writing attributes to configuration element. A bit of background first. We utilize config element as a sort of reference table for workflows that take VM name as an input parameter. In a grand deployment scheme we search for target VM once, populate config element with the attribute reference for VM entry as an object, and then any subsequent workflows will use that config reference where we need an object instead of an incoming VM name string.

This works fairly well for us but once in awhile attributes are missing from the config. We can see in the even log for the config element that it was saved during write attempt but not all expected entries are present. We have tried to use locking mechanism during write calls to rule out race condition, as well we padding adding couple of seconds of sleep after setting the attribute, no dice.

I am still fairly certain that this occurs when two or more write attempts are executed in parallel. I put together a simple mock up of how to reproduce this issue and was hoping to get some feedback for improvement or perhaps a suggestion for another way of doing fast object lookups (vCD or VC).

• Package contains 5 workflows and a config element.
• Config element "write test" has two attributes, populate them with VC:VirtualMachine objects (two or three unique entries per set should be enough).
• Execute workflow "config write test" with boolean set to yes; expect for config "write test" to be populated with however many objects were referenced in each set (all should be there).
• Execute workflow "config write test" with boolean set to no; expect all VM name attributes to be removed from the config "write test".

Running this test in my tiny lab environment exhibits this problem fairly frequently (vro7.0.1 appliance with local postgres). Either on add or delete, only one set is written, yet event log says that element is being saved on each entry.

[iiliev]

Hi,

Thanks for the workflow. I was able to reproduce it in my environment with the latest vRO code.

I have a hypothesis why is this happening (probably some caching of config element content inside foreach element) but need to look at it deeper.

[rkrichevskiy]

Ok thanks Ilian. I have tried to completely get rid of "for each" and call inner workflows directly from nested steps with array of objects, but still getting inconsistent attribute addition result.

If it's caching related, I wonder if there a way to reload config attribute list right before adding a new value. Perhaps not define it in the inner workflow but search for config element. EDIT: tried this and it doesn't work either.

[iiliev · Accepted solution]

There is an issue with your locking logic. You are using LockingSystem.lock() which is probably not what you want - it tries to obtain a lock and immediately returns true if the lock is acquired, or false otherwise.

What you probably wanted to use is LockingSystem.lockAndWait() which tries to obtain a lock and waits until the lock is acquired.

BTW, configuration elements have a method reload() which reloads the content of the element and its attributes.

[rkrichevskiy]

Ok yes, definitely an issue with locking method I used in my test; I think I incorrectly implemented this test as our non-lab implementation actually does use lock and wait method.

Trying it it now with proper method in my lab and test comes back with expected results.

So right now I have inconsistent results between environments. I'll dig around for other differences, my test is just a slimmed down version of what we run. Thanks for reload pointer, missed it.

[rkrichevskiy]

Ok i ported my test to our development and surely it started to fail even with lock and wait method. There are environmental differences that may be in play, in my lab I have postgres vs mssql in our dev and in my lab I don;t have vCD so all tests are against VC plugin.

However, reload() seems to be what is doing the trick for us and it makes sense it terms of caching. Thanks again for the help!

[rkrichevskiy]

Follow up question. I just realized that reload method is not available on older Orchestrators (5.5.3). Is there any type of equivalent that can do a configuration reload there?

[iiliev]

Well, for most other elements you could just get a fresh instance of the element using Server.findForType(type, id);

Unfortunately, configuration element's ID is not exposed to scripting. One option would be to use config element's path and name properties and first locate config element's category object using Server.getConfigurationElementCategoryByPath(path) method, and then iterate over all config elements within this category using categoryObj.configurationElements property and find your element by comparing the names.

[rkrichevskiy]

Yes I see what you're saying and I actually tried that approach before I knew about reload method. I went back to it and tried again this time with proper locking in place. It doesn't seem to be as sturdy. I got a few missed addition or removals in about a dozen of executions.

Anyways, reload() it is. Thanks!