Get rid of certificate errors with Avi (NSX-ALB) and Hashicorp Vault!

7 min readDec 1, 2021

Have you ever seen this error before?

This is a really important issue in enterprise infrastructure because unauthenticated TLS connections teach our end users to be complacent and ignore this error.

TLS Authentication

SSL/TLS for internal enterprise administration typically only addresses the confidentiality aspects of an organizational need — yet the integrity aspects are not well realized:

This is an important aspect of our sense of enterprise security, but the level of effort to authenticating information endpoints is high for TLS, so we make do with what we have.

The practice of ignoring authentication errors for decades has promoted complacency

Here’s another error that enterprise systems administrators see all the time:

ssh {{ ip }}
The authenticity of host ‘{{ ip }} ({{ ip }})’ can’t be established.
RSA key fingerprint is SHA256:{{ hash }}.
Are you sure you want to continue connecting (yes/no)?

This probably looks familiar too — Secure Shell (SSH) follows a different method of establishing trust, where the user should verify that hash is correct by some method, and if it changes, it’ll throw an error that we hopefully don’t ignore:

ssh {{ ip }}
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
SHA256:{{ hash }}.
Please contact your system administrator.
Add correct host key in known_hosts to get rid of this message.
Offending ECDSA key in known_hosts
{{ cipher }} host key for {{ ip }} has changed and you have requested strict checking.
Host key verification failed.

SSH is performing something very valuable here — authentication. By default, SSH will record a node’s SSH hash in a file called known_hosts to ensure that the server is in fact the same as the last time you accessed it. In turn, once the server authenticates, you provide some level of authentication (user, key) afterward to ensure that you are who you say you are too. Always ensure that the service you’re giving a secret to (like your password!) is authenticated or validated in some way first!

Web of Trust versus Centralized Identity

Web-of-Trust (WoT)

Web-of-Trust (WoT) is typically the easiest form of authentication scheme to start out with but results in factorial scaling issues later on if executed properly. In this model, it’s on the individual to validate identities from each peer they interact with since WoT neither requires nor wants a centralized authority to validate against.

Typically enterprises use WoT because it’s baked into a product, not specifically due to any particular need. Certain applications work well with it — so generally you should:

Keep your circle small
Replace crypto regularly
Leverage multiple identities for multiple tasks
Easy initial set-up
Doesn’t depend on a third party to establish infrastructure
The user is empowered to make both good and bad decisions, and the vast majority of users don’t care enough about security to maintain vigilance
If you’re in an organization with hundreds of “things to validate”, you have to personally validate a lot of keys
It’s a lot of work to properly validate — Ex. You probably don’t ask for a person’s ID every time you share emails with them
Revocation: If a key is compromised, you’re relying on every single user to revoke it (or renew it, change your crypto folks) in a timely manner. This is a lot of work depending on how much a key is used.

Centralized Identity

Centralized Identity services are the sweetheart of large enterprises. Put your security officers in charge of one of these babies and they’ll make it sing.

In this model, it’s on the Identity Administrator to ensure the integrity of any Identity Store. They’ll typically do quite a bit better than your average WoT user because it’s their job to do so.

Centralized Identity services handle routine changes like ID refreshes/revocations much more easily with dedicated staffing — mostly because the application and maintainer are easy to define. But here’s the rub, you have to be able to afford it. Most of the products that fit in this category are not free and require at least part-time supervision by a capable administrator.

It’s not impossible, though. One can build centralized authentication mechanisms with open source tooling, it just takes work. If you aren’t the person doing this work, you should help them by being a vigilant user — if an identity was compromised, report it quickly, even if it was your fault — the time to respond here is vital. Try to shoulder some of this weight whenever you can — it’s an uphill hike for the people doing it and every little contribution counts.

Back to TLS and Certificates

In the case of an Application Delivery Administrator, an individual is responsible for the integrity and confidentiality of the services they deliver. This role must work hand-in-glove with Identity administrators in principle, and both are security administrators at heart.

This is really just a flowery way to say “get used to renewing and filing Certificate Signing Requests (CSRs)”.

In an ideal world, an Application Delivery Controller (ADC) will validate the integrity of a backend server ( Real Server) before passing traffic to it, in addition to providing the whole “ CIA Triad” to clients. Availability is an ADC’s thing, after all.

Realistically an ADC Administrator will only control one of these two legs — and it’s plenty on its own. Here’s one way to execute this model.

Certificate Management

Enough theory, let’s do some things. First, we’ll build a PKI inside of HashiCorp Vault — this assumes a full Vault installation. Here’s a view of the planned Certificate Hierarchy:

From the HashiCorp Vault GUI — let’s set up a PKI secrets engine for the root CA:

Note: Default duration is 30 days, so I’ve overridden this by setting the default and max-lifetime under each CA labeled as “TTL”
Let’s create the services and user CAs:

This will provide a CSR — we need to sign it under the root CA:

Copy the resulting certificate into your clipboard — these secrets engines are autonomous, and don’t interoperate — so we’ll have to install it into the intermediate CA.

We install the certificate via the “Set signed intermediate” button in Vault:

Now, we have a hierarchical CA!

Mega NB: The root CA should nominally be “offline” and at a minimum part of a separate Vault instance!

For this post, we’ll just be issuing certificates manually. We need to extract the intermediate and root certificates to install in NSX ALB and participating clients. These can be pulled from the root-ca module:

Note: Vault doesn’t come with a certificate reader as of 1.8.3. You can read these certificates with online tools or by performing the following command with OpenSSL:

Once we have the files, let’s upload them to Avi: