acravens9
Enthusiast
Enthusiast

NSX-T 3.2.0 Advanced Load Balancer virtual service creation fails with custom Application Profile

Jump to solution

Created a custom Application Profile via the Policy REST API.  Proceeded to create a new Virtual Service.  The newly created VIP was created successfully and has a green status.  Next, the new Virtual Service was auto-created and has a red status.  The error is as follows:

 

Jan 18, 2022, 3:55:16 PM: Realization failure, waiting for realization of [{ALBApplicationProfile}]: my-application-profile, Realization will be reattempted in next cycle (max 5 minutes)

 

The "my-application-profile" was successfully created and shows up in the Application Profile drop down box, and I can select it.  However, this error indicates there is still something wrong with it.  I've found posts about other "Realization failure" errors but none in this particular context.  Any tips on this?

Tags (1)
2 Solutions

Accepted Solutions
acravens9
Enthusiast
Enthusiast
0 Kudos
acravens9
Enthusiast
Enthusiast

Yes there is a solution.  It got deleted somehow.  Here is an explanation.  We created a new application profile using the API.  We used the sample application profile listed in the vmware documentation for the AVI Application Profile.  We are using the basic AVI license and the sample application profile has some features in it that are only available with the advanced license.  When you create an application profile that has features you aren't licensed for, the profile gets created on the NSX side but it fails to get created on the AVI side.  NSX does not know if failed to get created on the AVI side (it's in the logs though) and when you try to deploy a virtual service, NSX can't retrieve the application profile from AVI.  

 

We discovered that creating an application profile of type HTTP fails when using the basic AVI license.  So now we are wasting a lot of time trying to figure out through trial and error what options we can and can't use based on license restrictions.  It appears the NSX Advanced Load Balancer integration isn't quite ready for prime time yet.  

 

To sum it up, when you create an application profile using the api, you have to make sure you aren't selecting options that you aren't licensed for.  If you choose the wrong options the profile will be successfully created on the NSX side but never gets created on the AVI side.

 

I suspect some of the application profile options are legit for the basic AVI license, but due to problems with the integration not being bug free,   AVI rejects the options.  

View solution in original post

22 Replies
acravens9
Enthusiast
Enthusiast

.

0 Kudos
jeffersonc47
Enthusiast
Enthusiast

This is marked as a solution, but it appears to be blank. Is there a solution here?

0 Kudos
acravens9
Enthusiast
Enthusiast

Yes there is a solution.  It got deleted somehow.  Here is an explanation.  We created a new application profile using the API.  We used the sample application profile listed in the vmware documentation for the AVI Application Profile.  We are using the basic AVI license and the sample application profile has some features in it that are only available with the advanced license.  When you create an application profile that has features you aren't licensed for, the profile gets created on the NSX side but it fails to get created on the AVI side.  NSX does not know if failed to get created on the AVI side (it's in the logs though) and when you try to deploy a virtual service, NSX can't retrieve the application profile from AVI.  

 

We discovered that creating an application profile of type HTTP fails when using the basic AVI license.  So now we are wasting a lot of time trying to figure out through trial and error what options we can and can't use based on license restrictions.  It appears the NSX Advanced Load Balancer integration isn't quite ready for prime time yet.  

 

To sum it up, when you create an application profile using the api, you have to make sure you aren't selecting options that you aren't licensed for.  If you choose the wrong options the profile will be successfully created on the NSX side but never gets created on the AVI side.

 

I suspect some of the application profile options are legit for the basic AVI license, but due to problems with the integration not being bug free,   AVI rejects the options.  

jeffersonc47
Enthusiast
Enthusiast

Thanks - Do you have an example of a basic application profile that works for the basic NSX-T integrated ALB?

0 Kudos
acravens9
Enthusiast
Enthusiast

I have an application profile but it's not useful.  Here's what we did, we took the sample json code from the documentation and deleted all the configuration parameters except for display name and application profile type.  We changed type from HTTP to L4 and were able to use that bare bones json to successfully create an application profile.  Our next step is to substitute in various options to see which ones work and which ones don't.  The api documentation states that "type" is the only parameter that is a required parameter. 

One thing I may try is to edit the newly created "bare bones" application profile from within the AVI web interface and then see if I can retrieve the json from the AVI web interface.  I'm wondering if I can create the bare bones profile in NSXT and then edit and refine it from within the AVI web interface.  If that doesn't work I we will have to experiment and through trial and error figure how what API options work and which ones don't. Needless to say, we are not happy with the current level of documentation and support for the NSX-T/AVI integration.  We're about ready to bag it and switch to using the AVI web interface or maybe just get a job 7-Eleven.

 

 

jeffersonc47
Enthusiast
Enthusiast

Thanks - I'm wondering if I just want to stick with the legacy NSX-T LB and move to the NSX ALB with 3.2.1/3.3/<whatever the next version is>. I don't need anything more complicated than basic HTTP(s) round-robin LB with a basic health monitor on the backend, so the built-in load balancer does everything I need.

0 Kudos
acravens9
Enthusiast
Enthusiast

I'm going to test a few things today.  I've created a "template" application using the native avi web interface.  I'm going to query the AVI api to get the json for this "template" app profile and use this to create a similar profile using the nsx-t api.  At this point I have the json from my "template" application profile.  Have not had a chance to try creating a profile yet.  Here are the steps I used to get the json for my "template" app profile.

 

Step 1 - Log in to the AVI controller CLI and allow API authentication.  See section titled "Enable via CLI"

REFERENCE: https://avinetworks.com/docs/21.1/http-basic-auth-for-api-queries/

NOTE: This could be a potential security risk in some environments so you may want to turn this off later.

 

Step 2 - From the native AVI web interface, create an application profile to use as a template.

 

Step 3 - Use cURL to get a list of all application profiles

curl -v --insecure --user someuser:somepassword -H "Content-Type: application/json" -X GET https://some_avi_controller/api/applicationprofile 

 

Step 4 - Go to any web site that formats and verifies json data.  I use https://jsonformatter.curiousconcept.com/ 

Paste the output from the command above into the json formatter and click "process" to format and verify the data.  NOTE: Depending on your setup, your terminal may add a line feed in the middle of the json data.  You have to paste the output into a text editor to remove the line feed.  Take the corrected output and paste into the json validator and re-validate.  Manually fix any errors.

 

Step 5 - Look through the formatted json data for the specific application profile you want to use as your template. Copy the "url" for that specific profile as you will need it in the next step. 

Example url:  "https://some_avi_controller/api/applicationprofile/applicationprofile-e0d1c1fe-c5ab-48f8-acf3-b6c56d88098c"

 

Step 6 - Run the same command in step 3 except change the url to the one you got in step 5 above.  Run the json in step 5 through the json formatter in step 4.  Now you have a "template" to create the json for a new application profile.  You could skip steps 5 and 6 and just copy and paste the json section for the profile you are after.  However, if you aren't familiar with json, completing all the steps should get you a nicely formatted json string for your "template" application profile.

 

Again, I haven't gotten to the point where I have tried to use this json to create an application through the nsx-t api. 

 

 

acravens9
Enthusiast
Enthusiast

It looks like the problem may be solved just by allowing basic auth at the avi controller as described in my previous post.  Here is some additional info.   After allowing basic auth you can create a simple application profile using this json:  I had to add back-slashes in front of the curly braces in order to post this.  Don't put those in your code.

\{
  "display_name": "test-application-profile",
  "resource_type": "ALBApplicationProfile",
  "type": "APPLICATION_PROFILE_TYPE_HTTP",
  "http_profile":{
      "secure_cookie_enabled":false
\}

 

This should create test-application-profile and you can verify it's existence by going to the native avi web ui to see if it's there.  If the NSX-T api call was successful it will return the full json for the newly created application profile.  The json result info can be used to craft a more complete application profile.  It will show you what options are available at your current license level.  

acravens9
Enthusiast
Enthusiast

Here is the json for an application profile of type HTTP this is verified working when created through the NSX-T policy API.  This profile shows up in the AVI web UI if the creation is successful.  The options below seem to work for the basic ALB license.

 

{ 
  "display_name" : "test-profile-2",
  "type" : "APPLICATION_PROFILE_TYPE_HTTP",
  "resource_type" : "ALBApplicationProfile",
  "http_profile" : {
    "allow_dots_in_header_name" : false,
    "client_body_timeout" : 30000,
    "client_header_timeout" : 10000,
    "client_max_body_size" : 0,
    "client_max_header_size" : 12,
    "client_max_request_size" : 48,
    "connection_multiplexing_enabled" : true,
    "detect_ntlm_app" : true,
    "disable_keepalive_posts_msie6" : true,
    "disable_sni_hostname_check" : false,
    "enable_chunk_merge" : true,
    "enable_fire_and_forget" : false, 
    "enable_request_body_buffering" : false,
    "enable_request_body_metrics" : false,
    "fwd_close_hdr_for_bound_connections" : true,
    "hsts_enabled" : false,
    "hsts_max_age" : 365,
    "hsts_subdomains_enabled" : false,
    "http_to_https" : false,
    "http_upstream_buffer_size" : 0,
    "httponly_enabled" : false,
    "keepalive_header" : false,
    "keepalive_timeout" : 30000,
    "max_bad_rps_cip" : 0,
    "max_bad_rps_cip_uri" : 0,
    "max_bad_rps_uri" : 0,
    "max_keepalive_requests" : 100,
    "max_response_headers_size" : 48,
    "max_rps_cip" : 0,
    "max_rps_cip_uri" : 0,
    "max_rps_unknown_cip" : 0,
    "max_rps_unknown_uri" : 0,
    "max_rps_uri" : 0,
    "post_accept_timeout" : 30000,
    "reset_conn_http_on_ssl_port" : false,
    "respond_with_100_continue" : true,
    "secure_cookie_enabled" : false,
    "server_side_redirect_to_https" : false,
    "ssl_client_certificate_mode" : "SSL_CLIENT_CERTIFICATE_NONE",
    "use_app_keepalive_timeout" : false,
    "websockets_enabled" : true,
    "x_forwarded_proto_enabled" : false,
    "xff_alternate_name" : "X-Forwarded-For",
    "xff_enabled" : true
  },
  "preserve_client_ip" : false,
  "preserve_client_port" : false,
  "preserve_dest_ip_port" : false
}'

 

jeffersonc47
Enthusiast
Enthusiast

Thanks - I was able to get a basic virtual server working with this profile. I was able to do a pure HTTP and an HTTPS offload (SSL from the client and unencrypted to the BE) working. I'm now going to see about making it work for HTTPS bridging and/or HTTPS passthru.

0 Kudos
acravens9
Enthusiast
Enthusiast

Are you using the basic license?  Can you post the json for the other profiles you created?  I might help me and others.  Please be sure to rename any elements that might be specific to your install. 

0 Kudos
jeffersonc47
Enthusiast
Enthusiast

Yep

0 Kudos
jeffersonc47
Enthusiast
Enthusiast

The only profile I've done through the API is the application profile you posted above. Other than that everything I've done has been via the integrated NSX-T UI. I'm assuming I'm going to need to make another application profile for an HTTPS BE; I'll post that if I get it working.

0 Kudos
jeffersonc47
Enthusiast
Enthusiast

I spent some time working on this today, and I wanted to share some notes on what I found:

  • I was able to use the above application profile to do HTTP->HTTP, HTTP->HTTPS, HTTPS->HTTP, and HTTPS->HTTPS. In order to use a backend HTTPS server, you need to enable SSL in the pool config. (This can be done via the NSX-T UI.) The issue I ran into is none of the 3 default SSL profiles would work against one of the backend pools due to the servers having old/insecure ciphers that weren't enabled in the default profiles. I expect I could have resolved this by making a new SSL profile, but I didn't end up doing that.
  • I decided to move to doing SSL passthru which required a new application profile. See the attached screenshot for the application profile settings. (Unfortunately I couldn't copy/paste it.) In order to do SSL passthru, I needed to do the following
    • Select that application profile
    • Don't enable SSL on the virtual server since we're not terminating SSL
    • Don't enable SSL on the pool since we're not initiating an SSL connection to the backend
    • Make sure the pool health check is doing a basic TCP check rather than an HTTPS check (or make an SSL profile such that it can do the HTTPS check)
0 Kudos
grimsrue
Enthusiast
Enthusiast

acravens9,

Your posts were very helpful with understanding the API info needed to create an application profile, but did the Default Application Profiles, that are pre-built into AVI, get pushed to your NSX-T environment when you installed your controllers through the NSX-T UI?

When I attempt to build a "Virtual Service" through the NSX-T UI I am getting nothing listed under the application profile drop down. It seems to me those should have "at least" been push to the NSX-T during initial setup. Not sure if this is normal for everyone or I had a problem during my initial setup.

grimsrue_0-1647396867637.png

 

0 Kudos
acravens9
Enthusiast
Enthusiast

I have found that ANY application profile you create in the AVI native web interface does not EVER get pushed to NSX-T.  I have also found that if you create an application profile in NSX-T you can go into the AVI web interface and modify it and the modifications do not get pushed to NSX-T.  Modifying the application profile in AVI after it was created in NSX-T seems to break the NSX-T object.  It appears that when you create an application profile in NSX-T there are objects created in NSX-T and they are stored in NSX-T along with the objects that get created and stored in the AVI back end.  The two must remain in-sync.  However, I noticed that I can create a virtual service in NSX-T and I can successfully modify that service in the AVI interface and it works as expected.    

 

The application profile part of NST-T is not fully functional.  This product is not ready for prime time.  My last interaction with support (Feb 2022)  revealed the engineer did not know of a single person who was using the new NSX-T ALB interface in production.  From what I understand, everybody is using the native AVI web interface to manage the load balancer.   

0 Kudos
grimsrue
Enthusiast
Enthusiast

Well, I was thinking that the Application Profiles in NSX-T UI is buggy and not working like it should and it seems I was right. I will probably open a ticket with VMWare and then reach out to my TAM and get this issue escalated a bit. I assume they already know about this problem, but does not hurt to pile on top of other complaints.

 

On a different note:

What curl URL are you using to push the new Application profiles. I keep getting a "Method not Allowed" when i use this URL.

Postmans returning header is telling me that only "GET" is allowed.

https://my-NSX-T-manager/policy/api/v1/infra/alb-application-profiles

0 Kudos
acravens9
Enthusiast
Enthusiast

I'm not using postman to push json to NSX-T.  I'm just using a bash shell script.  I'll post that so you can see what I did.  BTW I think I made an incorrect statement above.  I said that I created a virtual service in the new NSX-T interface and successfully modified it in the AVI interface and all worked as expected.  I think I actually modified a pool and not a virtual service.  I'll have to check my notes.

acravens9
Enthusiast
Enthusiast

Here's the bash script I use to upload the JSON to NSX-T.  It's just a quick and dirty script to test with.  I store the actual JSON in a separate file.  JSON does not allow comments but I added comments to my JSON so I could easily find the parameters to tweak.  My script removed the comments before sending the JSON to NSX-T.

#!/bin/bash

USERNAME="username"
PASSWORD="password"
COOKIEFILE="cookies.txt"
HOST="nsx-hostname.com"

# Your JSON needs to be stored in this separate file
FILE="sample-nsx-app-profile.json"

# Remove the comments from the json file and store in temp file
# JSON does not allow comments but I added them anyway so have
# to remove them before processing.
TEMP_FILE="temporaryfile.json"
cp $FILE $TEMP_FILE
sed -i '/^\s*[#]/ d' $TEMP_FILE

# Get the application profile's display name from the temporary json file.  Need it for the URI.
DISPLAY_NAME=`grep display_name $TEMP_FILE | cut -d"\"" -f4`
URI="/policy/api/v1/infra/alb-application-profiles/$DISPLAY_NAME"

/usr/bin/curl -v\
   --dump-header $COOKIEFILE \
   --insecure \
   -u "$USERNAME:$PASSWORD" \
   -H "Content-Type: application/json" \
   -X PUT https://$HOST$URI \
   -d @$TEMP_FILE

exit

 

Here is the sample JSON file I use:

{
  "http_profile": {
    "allow_dots_in_header_name": false,
    "client_body_timeout": 30000,
    "client_header_timeout": 10000,
    "client_max_body_size": 0,
    "client_max_header_size": 12,
    "client_max_request_size": 48,
    "connection_multiplexing_enabled": true,
    "disable_keepalive_posts_msie6": true,
    "disable_sni_hostname_check": false,
    "enable_chunk_merge": true,
    "enable_fire_and_forget": false,
    "enable_request_body_buffering": false,
    "enable_request_body_metrics": false,
    "fwd_close_hdr_for_bound_connections": true,
    "hsts_enabled": false,
    "hsts_max_age": 365,
    "hsts_subdomains_enabled": false,
    "http_to_https": false,
    "http_upstream_buffer_size": 0,
    "httponly_enabled": false,
    "keepalive_header": false,
    "keepalive_timeout": 30000,
    "max_bad_rps_cip": 0,
    "max_bad_rps_cip_uri": 0,
    "max_bad_rps_uri": 0,
    "max_keepalive_requests": 100,
    "max_response_headers_size": 48,
    "max_rps_cip": 0,
    "max_rps_cip_uri": 0,
    "max_rps_unknown_cip": 0,
    "max_rps_unknown_uri": 0,
    "max_rps_uri": 0,
    "post_accept_timeout": 30000,
    "reset_conn_http_on_ssl_port": false,
    "respond_with_100_continue": true,
    "secure_cookie_enabled": false,
    "server_side_redirect_to_https": false,
    "ssl_client_certificate_mode": "SSL_CLIENT_CERTIFICATE_NONE",
    "use_app_keepalive_timeout": false,
    "websockets_enabled": true,
    "x_forwarded_proto_enabled": false,
# For X-Forwarded-For to work set below two commands
    "xff_alternate_name": "X-Forwarded-For",
    "xff_enabled": true
  },
  "preserve_client_ip": false,
  "preserve_client_port": false,
  "preserve_dest_ip_port": false,
  "type": "APPLICATION_PROFILE_TYPE_HTTP",
  "resource_type" : "ALBApplicationProfile",
# Use display_name below to name the new application profile
  "display_name": "my-nsx-created-profile2"
}

 

You can see I added comments so have to remove them before processing.