Should you use Let's Encrypt for internal hostnames?
By @edent
· https security ssl · 30 comments · 500 words · read ~34,751 times.
Julien Savoie has written a brilliant post explaining how you can enable https on your intranet.
This is useful for several reasons. It means your employees aren't constantly fighting browser warnings when trying to submit stuff internally. All your http traffic is encrypted. You don't need to install a self-generated root certificate on devices. Lovely!
But there's a downside. Every TLS certificate created by Let's Encrypt is recorded in a Certificate Transparency log. These CT logs are primarily to detect maliciously or mistakenly issued certificates. For example, you can look through them and see that someone unauthorised has created a cert for your domain - or its sub-domains.
But there is a downside. The CT logs are public and can be searched. Here's all the certificates issued for Twitter's sub-domains.
There are a few ways that this can be dangerous for use with internal services.
Firstly, it aides reconnaissance for attackers. Having a "map" of your internal infrastructure is useful. Especially if you have "obviously" named servers like exchange.example.com or customerdata.example.com. Also handy for social engineering - who else but someone internal would know that gandalf.example.com was a valid server?
Secondly, it might expose some vulnerabilities - depending on how you name things. Let's hope you don't have log4j.example.com!
Thirdly, there's the potential for espionage. Do you want your competitors knowing that you've got olympics-campaign.staging.example.com?
I'm sure you can think of a few other ways this could be used for mischief and mayhem.
As I wrote a few years ago, "There's no HTTPS for the Internet of Things". Internal networks which only have IP addresses cannot use TLS certificates. OK, so you decide to have an internal DNS - now the whole world knows you have doorbell-model-xyz.myhome.example.com!
The only real answer to this is to use Wildcard Certificates. You can get a TLS certificate for *.internal.example.com
This requires setting up a DNS-01 Challenge - which can be more difficult to configure and has some non-obvious risks. And, sadly, Wildcard certificates come with their own difficulties.
Recap
I don't think there's a good solution to this.
- Self-signed certificates require something to be installed on all clients. Not always possible with BYOD.
- Named LE certificates expose details of your infrastructure which you may wish to keep private.
- Wildcard certificates require a heightened level of co-ordination and management.
These problems have all been discussed before. But I can't help but wishing that there was something obvious I'm missing.
How would you solve this knotty problem?
Sikander says:
Wow, I didn't know of the Certificate Transparency Log!
That's cool, but also a bit surprising that it's not something most people would be aware of when signing up for Let's Encrypt free certificates.
Alex Chamberlain says:
FWIW, most public CAs post certificates to a CT log. It's not a feature of Let's Encrypt.
Sikander says:
Ah, in that case the heading can be modified to: "Should you use any public CA for internal hostnames?"
Vernay Bruno says:
Indeed, "As of 2021, Certificate Transparency is mandatory for all publicly trusted TLS certificates" (https://en.wikipedia.org/wiki/Certificate_Transparency)
I appreciate the response.
Whilst not a true solution to the certificate transparency thing, I add a little security by obscurity. If the company name was acmeproducts[dot]com for example then our 'internal' tooling might be on something like acmp[dot]cc - visible on these lists but no idea whose it is.
Personally im good with people knowing i have some of these host names. You could find them reversing my IPv6 ptr records anyways.
Patrick Smears says:
Is there such a thing as a "CA certificate restricted to signing subdomains of a certain domain"? That seems like it might be a useful piece in the puzzle: if LE could issue those (with the same validation as for wildcard certificates), then you could make your own (valid) certificates for your devices, without them appearing in the transparency logs. That may seem a bit dangerous, but it's not really any more dangerous than a wildcard certificate in the first place. And there are still downsides - but it might be a good fit in quite a few cases. If it's possible 😉
sthen says:
@Patrick: There is such a thing, name constraints, but there are problems. It relies on the client to check and while it's better supported now than it used to be, I bet there are some important things that don't use it, so nullifying the constraint. Also it only applies to subjectAlternativeName not the names in CN. While a CA can enforce that SAN is used in certs that it signs directly (indeed this is part of the CA/B forum requirements) that's not the case with a subCA. Similarly some other things enforced by the CA (validity is a big one) could no longer be enforced as a subCA.
Also the requirement to publish ct logs is still there.
Basically there needs to be a lot more trust in somebody to operate a subCA (even restricted by name and path length constraints) than any type of host certificate (whether that's wildcard or for an individual server).
Dave Cridland says:
You're quite right that Certificate Transparency has information leakage that can be a security risk in multiple ways.
Wildcard certificates also present the problem that - obviously - they're no longer specific to a particular service or even host, and that means sharing the private keys about, and that in turn means heightened risk. Given the outcome of compromise with a wildcard certificate is worse too, I'd advise avoiding wildcard certificates.
Instead, I'd go for a private CA - not self-signed certificates, but an actual, honest to goodness private X.509 infrastructure. For an organisation, that's not too dramatically painful, and the CA can be limited to only your org's domain (with Name Constraints, now apparently supported in some places). "Bring Your Own Device" doesn't (and shouldn't!) mean "Do Whatever You Want On Your Own Device When Using It For Organisational Work", so installing the trust anchor (CA certificate) shouldn't be seen as an onerous task, as long as Name Constraints preclude abuse.
For purely internal services which can only be reached inside the network, it feels like there should be a way of handling the trust anchor via either the VPN or DHCP, but I'm not sure there is - but there is via MDM software (though those are not ideal with BYOD). Still, even where Name Constraints haven't been used, I've found a private CA a one-time task that's easy to talk people through. Once in place on the client device, it "just works", providing the security of a public CA without the downsides.
In short, I think the focus on addressing this problem should be in making private CAs simpler to manage on client devices, instead of trying to use public CAs (with CT leaking information) or wildcard certs (with heightened risk), and other poorer solutions.
sthen says:
@Dave: remember you can have more than one wildcard cert, each with a different key. So one possibility is to use various certs like *.randomstring.example.org, then ct logs won't have a directly useful name, while avoiding shared keys. If it needs to be typed by a human then for at least https you could have one server trusted with a key for *.example.org issuing http redirects to the "hidden" name. By no means perfect, but it can be useful in some cases where a private CA is just too difficult.
mlatu says:
i haven't looked deep into this but perhaps you could use GNS? https://www.gnunet.org/en/
Should you use Let's Encrypt for internal hostnames? Link: shkspr.mobi/blog/2022/01/s… Comments: news.ycombinator.com/item?id=298082…
Xzenor says:
Why not just use an internal CA? Sure you'll have to deploy your CA to the trusted Certificate Authorities of all your machines but it works and you only have to do that once. Or maybe twice as some browsers have their own store..
Works just great 👍🏻
Randall says:
Some cloud providers (Amazon and Google at least) provide a private CA feature, for a fee. At least Amazon also has their own way to provision LE-like free certs for their own resources (ELBs, etc.) that are valid on the public Internet (and logged to CT).
An independent but also interesting practice I've seen is a totally distinct domain for internal stuff; for example, maybe the public site is acme.tv and internal stuff is at acmeinc.net. Whatever your cert practices are, that separates stuff like webpage origins and email-related DNS entries between prod and internal, which might impede/slow down efforts to use a bug on one side to impact the other.
Anonymous says:
Section 3 of this paper (https://petsymposium.org/2017/papers/issue4/paper69-2017-4-source.pdf) discusses a cryptographic solution to this problem.
Max Truxa says:
The by far best solution to these problems: Deploy a private ACME CA like step-ca (https://smallstep.com/docs/step-ca) and install the private root on all devices. Now you got your own private "LE". As others have already stated, installing a company root cert on all devices should not be a problem, even with BYOD.
Randito says:
Check out https://smallstep.com/ if you want to go the self-signed root CA route.
They have some really nice tools and some good articles for learning.
CW says:
I use a reverse proxy for internal and external access with a wildcard. Internally I point the cosmetic dns names to the reverse proxy host. Those names match the external dns name so whether I'm at home or out and about I only need to use 1 name to access my services.
Anonymous says:
Handshake/DANE is always an option
Liam Ayr says:
Internal hosts are frequently referred to by shortname. so appserver.example.com being an internal host is only referred as 'appserver', not using an FQDN.
The problem then arises that any certificate must contain the short name to match the user typing https://appserver If the certificate does not contain that name but only appserver.example.com then you get a name mismatch error. It will only match if the user always presents a FQDN as the URI. Uncommon in corporate environments and a usability compromise. A public CA such as LE won't accept a cert request using 'appserver' as the name as it's not globally unique. Public CAs only assert the identity of globally unique identifiers.
The right way to do this for internal sites is an a private internal CA. Many many products and solutions available to do this. I like the simplicity and flexibility of openssl in this regard.
Wildcard LetsEncrypt certificates on localhost are the way to go.
DNS-01 challenge is easily solved if you host your domains on an API enabled registrar, like cloudflare.
shkspr.mobi/blog/2022/01/s…
AlisonW says:
I've gone the internal CA route in the past. Because they don't require from-external access the need to accept the internal-root certificate for each machine is a one-time thing.
I put some references to existing articles and existing solutions here: https://brunovernay.blogspot.com/p/integrating-tls-in-constrained-devices.html
The IoT use-case where you want to browse to your device locally - without configuring a domain name - without installing an extra application still has no solution in sight. Even Opportunistic Encryption that would have been a progress has been discarded.
What we need is a Shadow CA service for all reserved domains: .home, .internal, .corp..
What do I mean a "Shadow CA". Similar to a Shadow DOM; it only "really" exists at an edge endpoint, hiding the internal structure.
A Shadow CA is basically a clone of Let's Encrypt, but: - never signs any public internet names - assigned ONE, unique identifier per level - OS/browsers can trust the CA, or at least IT only have to install 1 certificate ever - Chain your self-signed CA at ca.mycompany.corp to the Shadow CA and bada bam bada boom
For now, by running your own root CA now and having your own chained CA, you can at least get down to just installing one certificate.
For installing one certificate, at least this much can also be automated with signed software.
For the beginning developer that is still quite a hurdle.
Garet says:
Ah, some missing text
assigned ONE, unique identifier per level to each organization or individual
In other words; you can't buy multiple domains. you can have one .home, one .corp, one .internal
If you want to authorize further custom craziness, you can chain it but it will only be valid for users of your custom infrastructure.
James says:
You might need to make this a part of DHCP or something, so that I can set up a malicious Cert.