A review on the common mistakes found on security assessments and incident responses
When I was in the Navy working as an Engineering Laboratory Technician, we would receive training messages relating mistakes that other ships had made in order to learn how to avoid making those same mistakes ourselves. We called them the “Reactor Funnies.” They were written in military speak, were very dry, and did not pull any punches. While we got a laugh at the matter-of-fact wording, we learned what the other guys should have been doing, but were not, to prevent their mistake. As a result, the same mistake was rarely, if ever, made again. In much the same way, I would like to share some lessons learned by organizations in recent months and help other organizations avoid those same mistakes. I do not intend to be funny as this problem is getting more serious every day. For companies that have experienced breaches, it is an extremely expensive, serious, and existential experience. I am hoping we can save others from that cost as well.
Common Mistakes
One item that has been on the rise lately has been occurrences of Spear Phishing and Whaling. As many of you already know, these are targeted attacks on a company. One data point we are seeing is that threat actors are using data from the Apollo breach in 2018. Like many other IT personnel, I did not think this lost data was very useful to a threat actor. But, if you are spear phishing, this data is a gold mine. Who are the leaders at an organization? Who are the targets that have the most power at the company? Who has their fingers on the purse strings? And who are most likely to be the employees that IT will make exceptions for? All those employees’ names, email addresses, and titles were in the Apollo data breach.
Another item we are seeing is that people are still in the habit of using their work email for personal mail. This makes the likelihood of success for a phishing campaign greater since those in the habit of receiving non-business email at work are desensitized to new and outside emails that are unrelated to work. This permits phishing in general to be more successful.
We are also finding a lot of residual remote access capabilities left in place. For some, they have not completed the migration to the new method. For others, IT departments are being forced by executives, managers, or shareholders to allow the continued use of legacy systems, even when new and more secure methods are available. This can be an old firewall or secure remote access appliance that is not using multi factor authentication (MFA). Some companies still leave port 3389 (RDP) opened directly to internal resources.
The use of MFA is still not universal. If someone is connecting to corporate resources from outside the boundaries of your internal network (and I would say that includes anything outside the data center including SaaS and cloud hosted applications), they should be using multiple factor to ensure they are who they say they are. Since humans are so much easier to hack than computers, the point of attack is frequently the employee. If a username, password, and token (MFA) are required to access your resources, you will have gone a long way toward protecting your company from a breach. Remote access is frequently the point of compromise and while organizations often consider their internal resources to be protected, threat actors have an easy time moving laterally throughout your infrastructure unless additional layers of protection are in place.
Frequently, alerts and warnings are not followed up on well enough. If a protective software package is in place (anti-virus, infrastructure analytics, etc.) and it is disabled for any reason and alerts you, do not rest until you know why it was disabled and why you were not informed it was going to happen. Recent breaches have shown that alerts were not followed up on, and as a result, the threat actor was not detected and acted upon as early as they should have.
Backups and storage devices are generally not protected as much as they should be. If administrator credentials are compromised and your backup appliances, VMWare Virtual Center, SAN control panels, etc. permit AD credentials to login, threat actors, with those credentials, can really wreak havoc. Segment the networks. Require separate credentials for regular network use and administrative functions. If you permit RDP into your server virtual machines, setup multifactor on those as well. And things like snapshots and data immutability should not be optional, you should require them.
Threat actors are taking the time to ensure that they have deleted snapshots and backups before they start encrypting the data. If they can compromise those systems, they know that any ransom is more likely to be paid. Even if you buy the decryption key, decrypting the data can take a very long time. So, make sure you are taking the steps necessary to protect these valuable items.
One other thing that we have noticed as a trend in recent attacks is that not only is the data encrypted, but it is exfiltrated and threatened for publication should the victim company not pay. Having immutable backups and SAN snapshots can go a long way toward speeding up a recovery, but if the threat actor has taken data of value like patents or embarrassing emails etc. the inclination to pay may be increased regardless of the ability to recover data.
How can you quantify the reputational hit on your company from a data breach?
So, what steps should be taken?
Keep no less than 4 snapshots of all of your production data
- Snapshots would permit the immediate rollback of the important systems to pre-encryption state.
Credentials
Storage credentials should not be integrated with Active Directory unless Multi-factor authentication can be applied. If AD credentials can be used to authenticate with no second factor, then compromised credentials can be used to delete those ever important snapshots. Additionally:
- Local administrator passwords should exceed 15 characters in length, and they should be changed regularly (at least every 14 days).
- Every device local account should have a unique password
- Store these accounts and passwords in a secure vault (Think something like Thycotic or CyberArk)
- Did I say MFA everything yet? I did not but you will hear it over and over.
Your backup software/appliance should not be attached to the domain.
Again, use a long, unique, secure password and MFA, MFA, MFA.
Ensure Data Immutability
I am going to take a few lines to discuss this. Many modern backup targets offer Data Immutability which means that once written, your data cannot be modified or deleted for a set period. 2 weeks is usually the default and ensures that whether it is attempted to be encrypted or deleted, the data is still safe. Think of it as a write once, read many situation. If you replicate your backups to a cloud provider, accept the add-on charge to make those backups immutable. If they do not offer that ability, find another provider (Same goes for Encryption at rest).
Personal and work email
It is recommended that people have multiple email accounts now. As you do not want to let email of different security levels mingle with the same login credentials, it is strongly recommended that you not only have separate personal and work email addresses, but that you have different personal addresses for different uses as well. You should consider having the following email addresses for personal use:
Junkmail Newsletter address – Use this one when filling out forms online etc. This is where advertisements are headed. If you choose to do business with them, you can modify the email address later.
Consumer email address – Use this for companies you do business with personally (eg. Amazon, Hulu, Twitter, etc.)
Family address – Use this address for friends and family
Financial address – Use this address for financial institutions. I not only have a separate financial address, but I also have separate addresses for each financial service. For example, I might have a Chase bank account with an email address of fr0d23sju@hotmail.com . Should this email address be compromised, and login credentials discovered, the threat actor would not have access to my Schwab account since it uses s30edlw954@gmail.com. These are just examples, I do not have accounts with either institution, but I think you get the idea.
And for your personal email accounts and financial accounts – MFA, MFA, MFA. If the threat actor can’t access the token, you’re golden!
Multi-Factor Authentication
Did I say MFA? I am going to say it again. Require MFA for any method of connecting from the outside network to your inside network. This includes back doors you have left for yourselves like LogMeIn, TeamViewer, etc. Or better yet, do not permit back doors. If you cannot get in the front door, drive to the data center.
Places you might not have thought of, but you should have MFA applied include:
- vCenter console (do not use AD credentials without it!)
- RDP connections to virtual servers
- Administrative File Shares (Yes, there’s a neat product that will do just that), or just disable them
Password requirements
Recent advice from NIST has changed the recommendations for passwords from Complexity to length being the most important factor. The best thing to do is get your users used to the idea of a passphrase. The Quick Brown Fox is a lengthy and useful passphrase that is hard to guess and complex. But, for the user it is not only easy to remember, it is easier to type than MyS3Cur3P@ssw0rd! They are even reducing or even eliminating the changing of the password. Personally, I am looking forward to FIDO2 and no username/password, but that nirvana has not arrived yet even though I have purchased my Yubikey.
No one is excused from security
I know, this is easier said than done. But, you can use examples like this breech to show the higher ups why it is so important that they, more so than the regular employees, are more secure. And if you explain it to them in dollar amounts, this generally will get their attention
Jump Boxes
I am seeing the value of them because of what I have seen threat actors do once inside the network. The purpose of a jump box is to provide an RDP server (pair for redundancy) that ALL management actions occur from. All management consoles are installed on these machines, and ACLs and Firewalls are configured to only permit these machines to talk to the infrastructure. Switches, firewalls, storage, you name it will not respond to management from anywhere else. This allows you to reduce the number of IP addresses that can manage your devices to 2 instead of any internal address. Then, you guessed it, require MFA on these jump boxes. If the world melts down and everything goes offline for a time, you can drive into the data center and attach a laptop to the network using one of the two hard coded IP addresses and bring everything online.
IP Reputation
Most firewalls now permit blocking IP addresses by country. If you are not a multi-national company, block the countries you do not do business with. At least block those that encourage hacking like China and Russia. Subscribe to and use services like Umbrella or Webroot to reduce the number of IP addresses you have to worry about.
Infrastructure Analytics
Probably the least used protective measure is Infrastructure Analytics. If suddenly my printer is trying to ftp to my firewall, block it and let me know. This is the equivalent to having security cameras inside the bank and not just on the outside. There are many different providers of these but get something in place. If a desktop that does not normally connect to a file sharing site suddenly connects and starts uploading gigs of data, you should know about it. If command and control connections are being made outbound, you need to be alerted. The faster you detect a breach, the better.
No one is excused from security
I know, this is easier said than done. But you can use examples of expensive breaches to show the higher ups why it is so important that they, more so than the regular employees, are more secure. And if you explain it to them in dollar amounts, this generally will get their attention
Zero Trust and SASE
Implementing Zero Trust and Secure Access Service Edge initiatives can help detect unwanted activities as well as reduce the impact of a breach. Most companies do not have visibility into the activities of their employees on Software as a Service or other cloud-based services. Secure Access Service Edge (SASE) brings the employee in to a monitored environment before providing access to internal and external resources for the company. This converts traffic to cloud-based applications into monitored traffic, where it currently is not monitored or tracked. Zero Trust Architecture means that we never automatically trust traffic either inside or outside the network. Following a Zero Trust Architecture stops or at least limits the lateral movement should a threat actor find themselves inside your network.
I hope this article has given you a lot to think about. If it scared you, good. We the protectors of the network must be right all the time. The hackers only need to be right once.
The Château Gaillard was once considered to be an impenetrable fortress. From September 1203 until March 1204 King Philip laid siege on the castle. After searching for a way to gain access, a soldier named Ralph found a latrine chute that the french could clamber up into the chapel. They ambushed a couple of guards and lowered the movable bridge into place allowing the rest of the army in and the castle fell. While not as popular a story as the Trojan Horse, the lesson learned here is ensure even the least likely access like a “poop chute” is guarded and access is controlled.
