These days, encryption is everywhere. On the Internet, what was once used mostly to protect online purchases has become practical enough to secure almost all of our interactions with websites and online services. This is a good thing.

What’s not so good is that encryption and the way we use it are both dangerously fallible. Malicious hackers break into supposedly secure systems all the time, to say nothing of what government agencies like the NSA have accomplished. Almost every day, there is another story about some weakness in our encryption systems being exploited. Why does this keep happening? Why is securing our data and communications so hard?

Two reasons:

  1. Good encryption is extraordinarily difficult to create.
  2. Users of encryption are largely ignorant and/or lazy.

Good Encryption is Hard

Maybe this goes without saying, but it’s true. The reasons why it is so hard are complex and difficult to explain, so I will try to summarize.

The basic concepts of encryption involve plaintext, ciphertext, keys, and algorithms. Plaintext is the source information–the original data. Ciphertext is the encrypted form of the plaintext. A key is a secret code–generally just a large number–that can be used to turn plaintext into ciphertext, and vice versa. An algorithm is a series of logical, mathematical steps. Put a set of plaintext through an algorithm with a particular key, out of it comes ciphertext that can’t be decoded by anyone who doesn’t have the key. That’s the ideal, anyway.

In cryptography, there is what is known as “perfect encryption.” Perfect encryption is impossible to break, and the vast majority of the time it is completely useless. This is because a perfect encryption scheme takes some set of input data and uses a key that is exactly the same length as the input data. This means every single bit of information in the plaintext has its own unique part of the key. In addition, you use this key exactly once. This type of encryption is considered perfect because nothing can be learned about the key or the plaintext, given the resulting ciphertext. Since the key is used only once, an attacker never has the opportunity to learn anything about it, either. Thus, the ciphertext perfectly protects the plaintext–as long as no one gets their hands on the key, that is.

In everyday usage, however, this type of encryption is impractical. Two parties who wish to communicate securely over the Internet–say, you and Amazon.com–instead go through a process that allows (relatively) secure communications over an indefinite period of time:

  1. Initial handshake. Here, both parties advertise what encryption "languages" they speak, and the most secure language they share is chosen.
  2. Key exchange. You can't encrypt data without keys, so both parties generate their own keys, transform them using a particular algorithm, then share them. Crucially, neither party actually possesses the other's key, but just enough information to encrypt data in such a way that the other party can decrypt it.
  3. With a secure means of communications established, both parties will send each other encrypted messages, and decrypt what they are sent and use it accordingly. You encrypt your payment information and send it to Amazon, Amazon decrypts it and uses it to finalize your purchase.

This glosses over a lot of detail, but those are the basics. There are multiple ways for this to go wrong. If the parties agree on a weak form of encryption, the data could be compromised. A malicious party could even force a weak form of encryption, as in the Logjam attack. The encryption algorithm in use could also be weak in some way: it might inadvertently reveal information about the plaintext, or the key. This makes that algorithm “leaky.” Over time, keys that were once strong enough to prevent attacks fall victim to growing computing power–a key that might have taken years to break a decade ago could now take minutes or even seconds. In response, key sizes have grown: doubling the length of the keys used exponentially increases the time required to break them.

In addition to computing power eroding the strength of encryption, mathematics also betrays us. Math is a double-edged sword: great complexity is possible, but in the end everything boils down to numbers and logical patterns. An encryption algorithm is made a bit more mathematically complex, and others seek out patterns in that complexity in order to break it. An ideal, practical encryption algorithm resists every attempt to reduce its complexity to such patterns. At this point in time, the encryption methods available to us are highly resistant to such analysis, but that is unlikely to last forever.

There’s an even greater risk looming on the horizon: should quantum computing become practical, all the existing forms of encryption we use today become hopelessly broken. In response, cryptography that is secure against quantum computing attacks is already being developed.

This is an arms race unlikely to end anytime soon.

Users are Ignorant and/or Lazy

Most people who use encryption don’t know anything about how it works, internally. It is not reasonable to expect everyone to, either. Unfortunately, this lack of knowledge also extends into areas where it shouldn’t: website administrators, corporate IT staff, software vendors, and others. As an example, Google recently demoted a popular algorithm used in encryption because it was found to be weak and prone to attacks. Chrome now puts up warnings when visiting a website using that algorithm, and in 2017 will consider such sites untrustworthy and hide them behind an error page. There are many sites today still using the weak algorithm, and it is likely some will persist all the way until 2017, when Chrome (and potentially other browsers) make it inconvenient to reach those sites. As it stands, website administrators are dragging their feet implementing the newer algorithms.

Weak encryption is commonplace in commercial software, too, not for lack of anything better, but for lack of motivation to upgrade, and sometimes because of difficulty. Much legacy software remains in use, sometimes decades old, and retrofitting it to use modern encryption techniques can be troublesome and expensive. At the very least, people using such software must be informed of the risks they are taking. Insecure software is even illegal in some circumstances, such as if it handles personal health information. And yet, security breaches are commonplace. They aren’t always the result of using inadequate encryption techniques, but they do result from poor and lackadaisical security practices overall. A flaw publicized yesterday, linked at the beginning of this article, results not from a weak algorithm, but the fact that it is commonly used with some default settings that, with some effort, can be broken by parties with access to tremendous computing power, such as the NSA. But oftentimes, part of what permits data to be compromised is that it isn’t encrypted at all.

As we close in on the end of 2015, we can at least take comfort that the worst predictions for this year did not come to pass. From a computer security perspective, this year wasn’t much better or worse than the last. The Internet wasn’t fundamentally broken, but some flaws were found and there were high-profile data hacks, like the now-infamous Ashley Madison breach. Information security is a game of constant attention and diligence. You may not spend much time thinking about how secure your data is, but I promise you that others do: those who want to help protect it, and those who just want to get their hands on it.