Troubleshooting with VMware NSX ALB/Avi Vantage

Get into the OS Shell (all elements)

Controller Log Locations

Note: Everything in is managed by Elasticsearch. I wouldn't mess with it.

Events published to the GUI:

The primary log directory for Avi Vantage Controllers is . As this feeds into Elasticsearch, they have file outputs for every severity level. An easy way to get data on a specific object would be to build a statement like this:

grep {{ regex }} /opt/avi/log/{{ target }}

Troubleshooting Deployment Failures

  • : Presumably for NSX-T integration. further investigation required
  • : Avi's cloud connector is pretty important given their architecture. This is where you can troubleshoot any issues getting a cloud turned up, or any initial provisioning issues.
  • : vCenter write mode activity logs. Look here for SE deployment failures in a traditional vSphere cloud.

Service Engines

Troubleshooting

Checking the Routing Table

NSX ALB / Avi uses FRRouting (7.0 as of release 20.1) over network namespaces to achieve management/data plane separation and VRF-Lite. To access the data plane, you will need to change namespaces! Unlike NSX-T, this doesn’t happen over docker namespaces. This means that the follow commands work in both as root:

  • Show all VRF+Namespaces
  • Send a one-shot command to the namespace: Example:
  • Start a shell in the desired namespace: Example:

After in the shell, all normal commands apply as if there was no namespace/VRF.

For more information on Linux Network Namespaces, here’s a pretty good guide: https://www.opencloudblog.com/?p=42

Logging

All SE logging is contained in . Here are the significant log directories there:

  • IMPORTANT! : This is where all the routing protocol namespace logging from FRRouting lands.
  • : This one's pretty touch to parse and it's better to use Avi's Elasticsearch instead.

Conclusion

Avi Vantage has a pretty solid logging schema, but is very much a growing product. These logs will eventually be exposed more fully to the GUI/API, but for now it’s handy to away. I'll be updating this list as I find more.

Originally published at https://blog.engyak.net.

I am a network engineer based out of Alaska, pursuing various methods of achieving SRE/NRE