How it works: Command chain building

The purpose of this article is to explain how Command builds chains so that when issues appear around chain building, this can be referenced for details and fixes.

How it works

Command chain building is done by combining certs from the Command database and the webserver trust store that Command is running on into one dataset that can be used for chain management. For MSI installations, the trust store would be the Windows Local Machine trust store. Specifics around the containerized behavior can be found under the Containers section.

For HA setups, this applies to all webservers involved in running Command. If only a subset of the webserver trust stores is modified, there will be conflicting chain-building behavior, depending on which webserver handles a given request.

Using the certs from these locations, Command then attempts to build a chain for the end entity. This is done by using the combined set of CA certs to determine the first chain that we can successfully build. In v12.5 and earlier, if a chain cannot be built, we will make round trips to the database looking for the next CA cert in the chain based on issuer and authority key IDs of the highest cert in the chain that we have so far. As of v24.4, this was slightly reworked to pull all CA certs from the database into memory at once to improve performance, and new functionality was added to use any AIA URLs in the cert to load chain contents from publicly hosted locations if the chain still cannot be built from the original CA certs set. We do not, at this time, have a deterministic way to get a specific chain for any end entity cert when it's possible that multiple chains may be available for a cert, given cross-signed CAs are involved.

How to exclude “undesirable” roots from chains

To control which chains are included in things like PEM downloads, the undesirable root CAs need to be removed from both the database and the trust store in order for the desired chain to be built every time.

In the diagram, you have an end entity cert with two possible chains, chaining up to “Root (self signed)” and “Cross Signed Root”. When the end entity and chain is downloaded from Command, it is likely that the chain for “Root (self signed)” is the chain that will get returned, but not guaranteed. If it is desirable to always build chains for the end entity to “Cross signed Root,” then the cert “Root (self signed)” would need to be removed from the Command database and the trust stores. Then, all chains built for end entity will result in a chain to “Cross Signed Root.”

Trusted roots in containers

Containerized Command uses OpenSSL’s default trust paths instead of a Windows cert store. For the UBI based images we run Command on, this would be the /etc/ssl/certs/ca-bundle.crt and /etc/ssl/certs/ca-bundle.trust.crt files. Similar to the Windows chain-building process, these trust files will need to be manipulated to include or exclude roots depending on the chain building use case.