The answer is... A big list. The HTTP Strict Transport Security (HSTS) list is a list of domain names which have told Google that they always want their website served over https. If the user tries to manually request the insecure version, the browser won't let them. This means that a user's connection to, for example, their bank cannot be hijacked. A dodgy WiFi network cannot force the user to visit an insecure and fraudulent version of a site.

After about a decade of use, the list is now 14MB in size, with around 130,000 entries in it. You can view the list online or download it.

The format is relatively straightforward:

{
 "name": "example.com",
 "policy": "bulk-1-year",
 "mode": "force-https",
 "include_subdomains": true 
},

When the list is updated, Chrome creates a trie with Huffman coding compression - so it doesn't have to parse that monster file each time.

A rummage inside

The most popular (over 1,000 entries) TLDs / Public Suffixes are:

After .com, the free .tk domain names absolutely dominate. I wonder how many of them are fraudulent?

There are 2,676 .uk domain names - only 537 of which aren't on .co.uk.

Going a bit further, there are 418 IDNs (which start with xn--).

And about 187 have "porn" in the domain.

You can't really extrapolate much from this as a data set. Lots of the domains seem to have expired or otherwise no longer work. Reading around https://hstspreload.org it notes that because this list is hard-coded into Chrome it can take months before a site is added. Similarly, removal can take a long time as well.

I can't help feeling that there should be a better way to manage all this though.

This is useful for several reasons. It means your employees aren't constantly fighting browser warnings when trying to submit stuff internally. All your http traffic is encrypted. You don't need to install a self-generated root certificate on devices. Lovely!

But there's a downside. Every TLS certificate created by Let's Encrypt is recorded in a Certificate Transparency log. These CT logs are primarily to detect maliciously or mistakenly issued certificates. For example, you can look through them and see that someone unauthorised has created a cert for your domain - or its sub-domains.

But there is a downside. The CT logs are public and can be searched. Here's all the certificates issued for Twitter's sub-domains.

There are a few ways that this can be dangerous for use with internal services.

Firstly, it aides reconnaissance for attackers. Having a "map" of your internal infrastructure is useful. Especially if you have "obviously" named servers like exchange.example.com or customerdata.example.com. Also handy for social engineering - who else but someone internal would know that gandalf.example.com was a valid server?

Secondly, it might expose some vulnerabilities - depending on how you name things. Let's hope you don't have log4j.example.com!

Thirdly, there's the potential for espionage. Do you want your competitors knowing that you've got olympics-campaign.staging.example.com?

I'm sure you can think of a few other ways this could be used for mischief and mayhem.

As I wrote a few years ago, "There's no HTTPS for the Internet of Things". Internal networks which only have IP addresses cannot use TLS certificates. OK, so you decide to have an internal DNS - now the whole world knows you have doorbell-model-xyz.myhome.example.com!

The only real answer to this is to use Wildcard Certificates. You can get a TLS certificate for *.internal.example.com

This requires setting up a DNS-01 Challenge - which can be more difficult to configure and has some non-obvious risks. And, sadly, Wildcard certificates come with their own difficulties.

Recap

I don't think there's a good solution to this.

• Self-signed certificates require something to be installed on all clients. Not always possible with BYOD.
• Named LE certificates expose details of your infrastructure which you may wish to keep private.
• Wildcard certificates require a heightened level of co-ordination and management.

These problems have all been discussed before. But I can't help but wishing that there was something obvious I'm missing.

How would you solve this knotty problem?

Most of the app's communication with the Path servers is over SSL. This means that no-one can see the data you're sending and receiving. If there are snoops on your network, they will only be able to see the encrypted data flowing back and forth. In general, this is a good thing.

Apart from images. If your friends are posting images, they are sent over http. No security. Anyone monitoring your network connection will be able to see all the images you're viewing.

Now, that's bad enough - but it turns out that all the images you send are visible to the the world even if you've set your post to private.

The images are sent over SSL, but as soon as you return to your "Path", a thumbnail is shown of what you've just posted!

Here's a picture of the logs, so you can see what's happening.

So, every image you post or see - including the avatars of your friends - are visible to all. A rather serious security and privacy problem.

Oh, does anyone know what the unencrypted call to "sendgrid.net" is all about?

Twitter's secure API hides the contents of the tweets you are reading. But it doesn't hide the images of those you converse with.

Raised as Issue 2175.

A Bit More Detail

Twitter has a secure (HTTPS) and insecure (HTTP) API.

When calling the secure API, all the content of the returned message (tweets) are encrypted. Eavesdroppers only see the cipher-text - essentially garbage.

However, within that cipher-text are links to insecure resources.

For example, a user requesting my tweets will get an object which contains a link to my avatar image.

Twitter is currently returning the insecure link:

"profile_image_url" :
    "http://a2.twimg.com/profile_images/1283757621/Sketch_Avatar.jpg"

Twitter should be returning the secure link:

"profile_image_url" :
    "https://si0.twimg.com/profile_images/1283757621/Sketch_Avatar.jpg"

Exploiting This Weakness

A user (Anna) will request the encrypted text of my tweets She then requests the unencrypted image. An eavesdropper (Eve) is listening in on the connection between Anna and Twitter.

Anna ---->Eve---->Twitter  (Secure request)
Anna <----Eve<----Twitter  (Secure response)

When Anna makes the initial request to Twitter, the malicious Eve can't see what they're talking about.

• The request "https://example.com/twitter/edent" is itself encrypted. Eve only sees an encrypted request to example.com - not "twitter/edent
• The response containing all the tweets is also encrypted

Anna ---->Eve---->Images  (insecure request)
Anna <----Eve<----Twitter  (insecure response)

Anna then makes the subsequent request for the twitter user's image, a malicious user can see

• The URI of the request.
• The content of the image.

Impact

Truth is, this has a pretty low security impact.

• There is no way to determine a user's name based on the URI for their image. (Unless you already have both).
• An eavesdropper has no way of knowing if the image is from the timeline, a reply, a DM, a search, a retweet, or the public timeline.
• Images may be locally cached by the user's browser - so frequency analysis isn't reliable.
• A malicious user could alter the image in transit.

Worst case scenario is that if a malicious man-in-the-middle knows which images relate to which Twitter users, they know the intercepted user has seen at least one tweet from that user.

Let's say Anna is communicating with Bob. Eve is trying to eavesdrop. If Bob has never tweeted, and Eve sees repeated requests from Anna for Bob's avatar, she may reasonably surmise that they are exchanging DMs.

Overall

This is a pretty low-impact privacy risk. It can be fixed by Twitter's API returning HTTPS URIs where possible. In the meantime, developers can replace "http://a2.twimg.com/" with "https://si0.twimg.com".