ICS Security Is a Team Sport
As we discussed in the first article in this series, there are many Internet-exposed control systems, but they are very different from traditional IT systems and require a different security approach. With these systems being so critical and controlling processes that can potentially lead to loss of life if they fail, what is being done to tackle this issue? In this article I’ll dive into this and more, looking at: the challenges Control Systems Engineers face today, our partnership with Schneider Electric, what manufacturers are doing to tackle industrial control system security and raise security awareness overall, and potential areas for improvement.
It isn’t easy automating everything
Control Systems Engineers have a very tough job. I suspect that, given our customer base, the consumers of this article are primarily from an IT Security background, so some background on the challenges our friends on the Industrial Control Systems (ICS) side experience may be helpful.
For starters, whatever they build has to run and work flawlessly 24/7/365 and often in environments more challenging than a climate controlled server farm, where there may be little to no opportunity for downtime or maintenance for years. They implement complex logic running programs using Ladder Logic, a graphical programming language that will appear foreign to anyone familiar with textual languages, and do it using C primitive types that even confuse people with a Computer Science background. To be fair, there are textual languages you can use, but none of them will really look familiar to someone coming from a traditional development background.
Even something as simple as assigning an IP address is not a trivial task in the ICS world. For some devices you can discover them, if you’re in the same subnet, with the device programming software. For others you’ll have to directly connect to them with a serial or usb cable and configure them. And in some cases you may even have to connect via a serial cable, configure the device to get its address from a BootP server, create a config file for the BootP server, run it, reboot the Programmable Logic Controller (PLC) and hope it actually gets the IP assigned, and finally configure the PLC to no longer get its IP from a BootP server or it’ll lose its IP with a power cycle (such as with the Wago 750-842).
The configuration software isn’t nearly as polished as the IDEs we’re all familiar with, and with rare exceptions they only run on windows. The lack of polish isn’t all that surprising since every manufacturer has to produce and update their own proprietary configuration software, which operates as an IDE for ladder logic. If a Control Systems Engineer makes an error or a device fails, in the best case scenario an assembly line is down until it is fixed, but in the worst case scenario people die (and depending on the system it could be a large number of people). As we discussed in the last article, to make matters worse, additional security measures can potentially undermine the reliability of these systems and add to the complexity of configuring them.
Manufacturers are investing in ICS device security
There’s a common narrative that you’ll encounter when discussing PLCs and other Industrial Control Systems with folks from the IT world that they’re “bad” devices and that their manufacturers don’t care about security. I believe this comes from a fundamental misunderstanding of these devices, the constraints of their underlying Real Time Operating Systems, and their intended purpose. In both cases, I’d argue that the opposite is true.
Kyle McMillan from Siemens delivered a great talk at Defcon 32 about the “ICS Perma-lag,” outlining many of the challenges manufacturers face in keeping their products in line with current trends in security. He mentioned the fragmented ecosystem, which we’ll touch on a bit in the section covering what manufacturers can do better, long device service-life, and determining who owns fixing the technical debt. The last item is particularly interesting. Especially for protocols that are used across the board, you’d need every manufacturer to chip in and update their implementation so that interoperability isn’t broken. Additionally, many of the issues or risks are present because customers use the devices in a way other than is intended by exposing them to the public internet, so some of this falls on the consumer.
Device reliability
A computing device that can reliably run in harsh environments 24/7/365 for 25 years — and do exactly what it is supposed to do while meeting tight deadlines — should be considered a modern engineering marvel. I’ll admit that I was also guilty of this misconception, until I learned to wire, network, and program these devices while building our own ICS lab.
In the process I gained a strong appreciation for them. The challenge here is that a PLC has one purpose in life, to read inputs then write outputs within a tight timeframe. As we discussed in our first post, any security features baked in at the device level may threaten its most important characteristic, reliability.
As for the manufacturers, today they’re actually doing a lot. At the device level, they are adding reasonable controls such as VPNs and firewalls on Remote Terminal Units meant to transmit to and receive information from remote systems. In the engineering software, they’ve added access control and password protection for the running program. Unfortunately, with the longer lifespan of these devices, it may take a while for any controls implemented in newer products to make a difference.
Security investments and partnerships
Schneider Electric, Rockwell, and Siemens have made significant financial investments across multiple funding rounds in Claroty, who produces a platform for anomaly detection and secure remote access for industrial networks. I am sure some of the fantastic research we’ve seen come out of Team 82 over the last several years is a positive side effect of those investments. Rockwell, Emerson, GE, and Schweitzer Engineering Labs have built partnerships with Dragos, who offers an Industrial Control System threat detection and response platform. Rockwell Automation and CrowdStrike have announced a partnership, which I am guessing will lead to better interoperability between EDR agents and their PLC configuration or SCADA software. We’ve built a partnership with Schneider Electric to identify exposures in their install base, which we’ll discuss further in a bit, and are in talks with others as well.
Beyond investment and partnerships, technical experts from the major manufacturers have been contributing informative cybersecurity sessions at conferences such as Defcon, RSA, Hannover Messe, and S4. Helping both Control Systems Engineers and Infosec practitioners understand the ICS security landscape and best practices is absolutely essential to tackling this issue. As we’ll talk about when discussing our Schneider Electric partnership, some manufacturers have even gone so far as to proactively reach out to their customers to notify them of devices in an insecure state. I certainly have not listed all of the efforts and partnerships, but even with just this sampling, it becomes clear that device manufacturers are making significant investments in the security of both their products and customers.
Bitsight and Schneider Electric
Due to limited processing capacity and the risk of elongating the PLC scan cycle, only limited security controls can be implemented at the device level. Manufacturers provide architecture and guidelines for implementation, for customers and partner integrators to apply, follow and control. Ensuring adherence by customers and integrators to manufacturer recommendation is always a challenge. Schneider Electric chose to partner with Bitsight to help tackle the problem by funding research and development of new capabilities. It is a very ambitious project aiming to identify exposed ICS devices within their customer base and to help them remediate the issue. So they aren't stopping at implementing new security controls in their devices and trying to educate their customers. Since technology and education alone have not solved the issue, they're taking it a step further to proactively assist their customers in identifying exposed systems.
Manufacturers (and the ICS community) can do more
Beyond hardware and software, there are definitely some areas where device manufacturers (or the ICS community) could do more to improve the state of industrial security. Many of the most effective Cybersecurity professionals got to where they are by being curious. Being driven to understand how things work, what happens if you do x or y, and often performing the same actions as threat actors in a safe environment. It is important to understand that, in order to defend against an attack, you have to understand the weaknesses of the system, the way the attacker will operate, and what they’re after. In order to do that, you need to get your hands on these devices and explore them, but to do that you’ll need to overcome two major barriers; the difficulty in purchasing an ICS device and the EULA.
Purchasing an ICS device
Once you get beyond the most common PLCs, purchasing an ICS device is similar to buying a car (a lovely experience universally hated by all), due to the fragmented ecosystem that Kyle McMillan mentioned in his talk at Defcon 32. You’ll have to locate a company that offers the device in your region, if it is available at all. If you’re lucky enough to find a company that lists the device on their website, in many cases you’ll have to contact them for pricing and work through a sales rep to complete the purchase. Then if the device isn’t in stock, you’ll have to wait for some unspecified amount of time (sometimes months) while the elves at the PLC factory build it. The one exception to this model is Automation Direct, who sells their products direct-to-consumer. You can find every available product on their website, order it, and it’ll show up within about 2 days. Their devices aren’t as robust as the offerings from the major players, but they’re easy to buy and simple to configure.
For an illustrative example of the nonsensical process to purchase something simple in the industrial world, I’ll walk you through our experience purchasing OPC licenses for some of our PLCs. We were tasked with researching OPC-UA and thankfully two PLCs we already purchased included on-board OPC-UA servers. In order to enable the server on each you have to attest (not apply, prove, etc) that you have a license. Being an upstanding company, we decided we’d do the right thing and purchase them. So of course I reached out to the vendor directly to buy them. Unfortunately, they won’t sell it to you, and told me to find a distributor. After some Google-fu, I found a distributor that had the license available with a 10 week lead time. Keep in mind that the license is just a pdf that says “Hey, you’re a good guy, you paid for the license”. We purchased the licenses and thankfully the distributor did deliver them within a few weeks. I have to wonder though, how many companies just skip this painful process entirely and just enable the server since there is nothing preventing you from just turning it on? For an even more painful example, I’ll leave it to my friend Pedro Umbelino to tell the story of our struggle to buy one of his Automatic Tank Gauges.
EULA challenges
Once you get the device you’ll have to tackle the biggest obstacle you’ve faced so far (especially if your tinkering is company-sponsored), the dreaded EULA (End User License Agreement). In order to install the configuration software you’ll have to accept the EULA, which your legal department is going to want to review beforehand. Nearly every EULA for PLC configuration software either explicitly bans or implicitly restricts the activities that would fall under the category of “Security Research”. The relevant clauses are generally the ones used to protect the company’s intellectual property (restrictions on reversing, decompiling, etc.), which is important. It’d be helpful for them to add in language permitting security research activity. I think it is universally accepted that threat actors don’t read EULAs, so this language really only hampers those with good intentions.
ICS education
If you are a budding Control Systems Engineer or Cybersecurity professional and want to achieve an ICS Security certification, a common practice in the IT world, your primary options are to pay an ungodly sum to the SANS Institute, or go for an entry-level certification from EC-Council. Looking for something through a University, well that isn’t much better with only a handful of Universities offering a single course on the subject, Idaho State University offers a program, and Western Virginia University specifically focusing on ICS security. As far as general education goes, most of the ICS vendors now have cybersecurity offerings within their training catalogs. This is a good step, but I can tell you from my past life in the IT Training industry that vendor produced content about a broadly applicable subject often falls short. This is because these courses are often too tailored to the vendor solution, leaving blindspots in areas that aren’t applicable to that specific solution. CISA has stepped up and put together a large catalog of offerings that are free. A certification tied to these courses would go a long way to add additional value to completing the training and further incentivize participation.
To sum it all up, it isn’t easy being a Control Systems Engineer and to tackle the security debt in the industry, we’re going to need more of them. Manufacturers are doing a lot to improve their product security and to educate or even assist their customers. However, for more folks to become involved in ICS Security, especially those who are not currently in the ICS field, they need to be able to get their hands on the devices, learn without fear of legal repercussions, and have access to training and education.
