Engineering Solutions for Cyber Protection with Andrew Ginter

00:00:00 Speaker 1
This episode of The Energy Pipeline is sponsored by Caterpillar Oil and Gas. Since the 1930s, Caterpillar has manufactured engines for drilling, production, well service, and gas compression. With more than 2, 100 dealer locations worldwide, Caterpillar offers customers a dedicated support team to assist with their premier power solutions.

00:00:28 Speaker 2
Welcome to The Energy Pipeline Podcast with your host, KC Yost. Tune in each week to learn more about industry issues, tools, and resources to streamline and modernize the future of the industry. Whether you work in oil and gas or bring a unique perspective, this podcast is your knowledge transfer hub. Welcome to The Energy Pipeline.

00:00:52 KC Yost
Hello, everyone, and welcome to this episode of The Energy Pipeline Podcast. Today we'll be visiting with Andrew Ginter, Vice President of Industrial Security at Waterfall Security Solutions. And we'll be discussing a topic that I'm very interested in, OT security. Now, right off the bat, Andrew, that's industrial cybersecurity, right?

00:01:15 Andrew Ginter
That's right. And thank you for having me on the podcast.

00:01:17 KC Yost
Yep.

00:01:17 Andrew Ginter
OT is operational technology, sometimes industrial, sometimes SCADA security. There's a lot of different words that mean the same thing.

00:01:26 KC Yost
Sweet. Well, great to have you here. Thanks for being on the podcast. So before we start talking about OT security, if you would please take a few minutes to share your background with our listeners. I know you graduated from the University of Calgary. I know you're an accomplished author, you have your own podcast, you do lots of things. So I'm very, very interested in hearing about your background, please.

00:01:52 Andrew Ginter
Sure. Thank you so much. Very briefly, I'm on the end of my career. I've done a lot of stuff. I started out, the grass was always greener, bouncing around between opportunities. Young people do this. I settled in at Hewlett-Packard. He laughs.

00:02:08 KC Yost
Yep. Yep.

00:02:09 Andrew Ginter
First-hand experience.

00:02:09 KC Yost
Been there, done that.

00:02:11 Andrew Ginter
Yeah.

00:02:11 KC Yost
Yep.

00:02:13 Andrew Ginter
Settled in at Hewlett-Packard developing industrial control system product, so for anyone in the industry, HMIs, historians, communications drivers, protocol drivers. This was the era before OPC. There was a lot of drivers to produce. I moved to Agilent Technology leading a project, doing IT/ OT middleware, connecting those control systems, lots of control systems to SAP because that was the thing to do in the mid-1990s.

00:02:47 KC Yost
You've hit an acronym that I recognize, so good.

00:02:50 Andrew Ginter
There you go.

00:02:50 KC Yost
Good, Good. We're on topic. Thank you.

00:02:52 Andrew Ginter
SAP-

00:02:52 KC Yost
Go ahead.

00:02:52 Andrew Ginter
SAP automates, I don't know, something ridiculous like 950 of Fortune 1000 and everybody else. Yeah.

00:03:04 KC Yost
Yeah.

00:03:05 Andrew Ginter
And in the course of connecting control systems to SAP, we connected a lot of IT and OT networks contributing to the cybersecurity problems that now plague many industries, this industry. I got religion. I wound up the Chief Technology Officer at Industrial Defender developing the world's first industrial SEM. And now on the end of my career, I've been 10, 15 years with Waterfall. I have the opportunity to lead a small group of subject matter experts. We work with the world's most secure industrial sites. I learn from them and I have opportunity to write about what I've learned through the career and especially from the world's most secure industrial sites. They ask different questions, they get different answers, they look at things a different way. It's been eye-opening.

00:03:52 KC Yost
Interesting, very interesting, very interesting. So you said 15 years with Waterfall. Can you give us an elevator speech on Waterfall, please?

00:04:05 Andrew Ginter
Oh, yes, sorry. Waterfall, we do OT security. We do security for industrial operations. We tend to focus on, let's call it high consequence operations. Some people call them boomable industries, where worst-case consequences of compromise are truly unacceptable. Things blow up, people die. So we produce a family of technologies. We're not a services company, we produce technology. The technology, the flagship is called the Unidirectional Gateway. It is deployed most often at the ITOT interface. It is a hardware-based device. There's hardware and software involved, but the hardware physically prevents attacks pivoting, which means using compromised machines to attack other machines, stepping through firewalls, going deeper into the target. That pivoting method of attack is the modern attack method that almost everyone uses. And we're focused on preventing attacks, pivoting into OT, into industrial automations, into the computers that automate our refineries and pipelines and offshore platforms.
                 

00:05:15 KC Yost
Fascinating, fascinating. So if you would please, Andrew, let's drill down into OT. What is OT? What's the background associated with OT?

00:05:26 Andrew Ginter
Yeah, so OT stands for operational technology. It is in a sense a complement to IT, information technology. The Gartner group coined the phrase OT, or operational technology, and they did it back in about 2005. And they did it because as I mentioned in the mid-1990s, we had started connecting industrial control systems to SAP and eventually other business systems. And they saw that happening for a decade and they had to start talking about it. It had become mainstream. And so, they had to invent terminology because at the time the Gartner Group did not have engineers on staff. They had a bunch of IT gurus on staff. And so, they needed a word for all of that computer stuff, all that engineering stuff that we don't understand. They needed a word for that. And they called it operational technology. The IT community jumped on that phrase. They started using that phrase routinely for a good decade. It was a good decade or decade and a half before the engineering community started using the phrase. Why? Well, because engineers know what human machine interfaces are and what historians are and what OPC, they know all that. They didn't need a word that meant all that engineering stuff we don't understand. They did understand it.

00:06:53 KC Yost
So basically, you're telling me that OT is a black box that IT made up for all of this engineering stuff that they don't do.

00:07:04 Andrew Ginter
Now, Gartner will deny that. Don't let me put those words into Gartner's mouth. That is my opinion. Yes.

00:07:09 KC Yost
Okay.

00:07:09 Andrew Ginter
That's my opinion.

00:07:11 KC Yost
Uh-huh. Okay. Uh-huh.

00:07:12 Andrew Ginter
And that's changed in the last half decade. In the last half decade, I've seen engineering teams start to use the phrase. Why? Because it's been used everywhere else so universally. And because finally after 15 years of talking about the problem, we're seeing a majority of engineering teams finally engaging with IT teams saying, "Cybersecurity, we need to solve this problem." And in order to engage, there has to be a common language. And so, they've adopted the phrase operational technology in many industries in order to converse, in order to communicate effectively with it, because that's still the word IT uses.

00:07:52 KC Yost
Right.

00:07:52 Andrew Ginter
Even though a lot of IT people have started learning about, "What is all that engineering stuff?" The black box is no longer a black box because a lot of the IT people are now responsible for securing the black box. You can't secure a black box. It has to be a white box. And so, both teams are learning about each other. This is a good thing, and using that phrase as a means of communication. Especially when we talk about the boundary between the cybersecurity discipline, it's a different approach on IT than in OT. When we talk about that boundary, we had to give that boundary a name. And it's pretty universally today called the IT/ OT interface. Engineering calls it that. IT calls it that. Everyone calls it that.

00:08:38 KC Yost
Gotcha, gotcha. I think I understand. So thank you for digging in deeper there. So we've had a cybersecurity expert on the podcast. Actually, we've done two podcasts with Philippe Flichy, who's very, very good. And we've been talking about emails, linking onto to the wrong email. And I understand that that's something different than what you do. So can you talk about what Philippe was talking about versus what you're discussing?

00:09:12 Andrew Ginter
Very much. Let me start with best practice. A nearly universally accepted practice on OT networks, on the networks that contain computers that control catalytic crackers, that control pumps and compressors on pipelines, nearly universally the firewall or the device at the IT/ OT interface, whether it's a firewall or a unidirectional gateway or what else, forbids any communication with an email server. You cannot pull email onto the OT network. It forbids communication. It forbids a connection to Google. You cannot browse the internet from an OT network. If something in the OT network messes up, literally, you risk things blowing up. At the very least, you risk very expensive physical assets stopping. You trip an emergency shut down of the refinery, what does that cost you? It's $150 million a day in lost opportunity. How long does it take a refinery to go from dead stop to full production again? It takes six and a half days. That's a big hit. So nobody does that IT stuff on OT networks. But, but, the big risk, one of the big risks is what's called a pivoting attack. This is the modern attack. I'm not sure if you're familiar with the word, what it means.

00:10:43 KC Yost
I'm not. I'm not.

00:10:44 Andrew Ginter
There you go. It means the bad guys-

00:10:46 KC Yost
Tell me about it.

00:10:47 Andrew Ginter
The bad guys get a foothold on IT. Somebody on IT clicks a bad link, opens up a bad attachment, and the bad guys activate what's called a remote access Trojan, a RAT. And this is a piece of software that calls out to the internet. You laugh, it's a RAT.

00:11:11 KC Yost
I love your acronyms. That's good. That's good. Go ahead. Sorry.

00:11:16 Andrew Ginter
Calls out to the internet to a command and control center. Imagine it reaches hackersrus. com. The domain name is never that obvious, but gets out to hackersrus. com or some other what's called a command and control center, and the bad guys rendezvous. They log into the command and control center, and they bring up a user interface and they say, "Okay, what have I got? I've got 17 RATs. I have 17 new victims. Which one shall I pick on today?" And they pick one, and they start giving commands to the malware in the compromised machine. And it looks around the network. It tries to find higher value targets. They might download additional attack tools. They're using the compromised machine manually by remote control to learn about their target and attack other devices. This is called pivoting. Pivoting is using a compromised machine to attack another machine. Got you can pivot what's called laterally, sort of within a network. You can pivot what's called vertically, or north/ south sometimes it's called, to go through a firewall to a more heavily defended and presumably more valuable target. This is the modern attack pattern. The big concern with physical operations, with the computers that control the refineries and the pipelines, is that it's possible to pivot through the IT/ OT interface into the computers that control physical operations and work bad stuff there; the simplest being encrypt a bunch of stuff, everything shuts down, the worst being tampering with safety systems. This happened in a Middle Eastern facility in 2017, Google Triton if you want, or stuff that people don't even think about. Bricking the controllers, downloading new firmware into the seven CPUs that are on your pump and erasing those devices. And now they can't boot. They can't even boot far enough to download new firmware.

00:13:24 KC Yost
Wow.

00:13:25 Andrew Ginter
Your device, your pump, your engine is dead and stays dead until you can physically replace all those circuit boards. Again, major downtime we're talking. So this is the modern attack pattern. This is the consequence. When we talk about high consequence networks, it's not what happened. I clicked on an attachment, my laptop was corrupted, three other laptops got corrupted. I had to clean them all out. It cost me $17,000 of lost production and labor and blah, blah, blah. No, we're talking high consequence scenarios.

00:14:03 KC Yost
Colonial pipeline?

00:14:05 Andrew Ginter
Colonial pipeline, absolutely. Think about Colonial.

00:14:10 KC Yost
Is that an example?

00:14:10 Andrew Ginter
Colonial is absolutely an example. So let's talk Colonial for a second.

00:14:16 KC Yost
Okay.

00:14:17 Andrew Ginter
The first is, I will argue that the Colonial incident made the front pages because it was a threat to national security. Really? How did national security come into there? The Colonial Pipeline is critical infrastructure. The legal definition of critical infrastructure is critical to the nation. If you impair critical infrastructure for any length of time, you have a national security issue. This is why it made the front pages. How did Colonial happen? The bad guys got into the IT network. They pivoted around the IT network, they encrypted a bunch of stuff. And the supervisor, now this was testimony under oath, so this is reasonably reliably known in public. It was a congressional inquiry or something like this. The CEO testified that the supervisor in charge of the pipeline was alerted that bad stuff was happening on the IT network. And they said, "We are not confident running this safety-critical resource." And what's the worst that can happen if you misoperate a pipeline? The pipeline ruptures and you have a discharge of, in this case, gasoline into a population area. That's very, very bad. They said, "We are not confident of running a safety-critical asset while that is happening on the IT network," because of the risk of the attack pivoting through the IT/OT interface into the safety critical systems. And so, the buzzword was abundance of caution, in an abundance of caution. They saw no evidence that they'd been compromised on the OT side, but in an abundance of caution, they shut down.
                 

00:16:02 KC Yost
I see.

00:16:02 Andrew Ginter
And it was six days, six and a half days before it was up again.

00:16:05 KC Yost
Sure, sure.

00:16:06 Andrew Ginter
So pivoting was the fear, the worry, because they were not confident of the strength of their IT/ OT interface.

00:16:15 KC Yost
I see, I see. So with that in mind, you mentioned earlier something about unidirectional gateways. Would that have protected Colonial having something like that? And is that part of your protection system between the IT/ OT interface arrangement?

00:16:38 Andrew Ginter
So the short answer is I don't know because I don't have a Colonial network diagram or other security program information in front of me. So anything I say is going to be speculation. What I will say is that the gateway is designed to prevent attacks pivoting from one network to another.

00:16:57 KC Yost
I see.

00:16:58 Andrew Ginter
Really briefly, if you're interested, the gateway usually replaces the firewall at the IT/ OT interface. And really, the word IT/ OT interface is sort of, it's a big catch all. Usually there's several layers of firewalls. There's several intervening networks. But if you can pivot through one firewall, and all ransomware pivots from the internet into it through the IT to internet interface, through that firewall. So clearly, they know how to get through firewalls. If you can pivot through one layer of firewall, you can eventually pivot through all of them. And so, the gateway is a different sort of solution. You pick one of those layers of firewalls, you replace it with the gateway. What is the gateway? It's a combination of hardware and software. The hardware is physically able to send information in only one direction. How does that work? Well, there's a circuit board in the industrial that has a fiber optic laser, a transmitter, not a transceiver, a transmitter. There's a short piece of fiber, usually six, 10 inches long to another circuit board in the same device. And that receiving circuit board has a receiver, a photocell, not a transceiver, a receiver. There's a single fiber connecting sort of the OT side of the device to the IT side. You can send from OT to IT. It's not physically possible to send anything back. There is no laser in the receiving circuit board.

00:18:20 KC Yost
So I'm a pipeliner. So what you're describing is your version of my check valve.

00:18:29 Andrew Ginter
Absolutely.

00:18:30 KC Yost
And that flow only goes one way.

00:18:32 Andrew Ginter
Yes, yes.

00:18:33 KC Yost
Okay.

00:18:33 Andrew Ginter
Absolutely.

00:18:34 KC Yost
All right. Very good.

00:18:36 Andrew Ginter
Both flow one way. And I said there's software involved. The software is designed to simplify integration. All modern internet-based communication uses what's called TCP. TCP is two-way. So you can't do TCP through the one-way hardware. What you do instead, the job is not to connect to OT so that computers on the internet can operate the pipeline. No, that's not the job.

00:19:06 KC Yost
Right.

00:19:06 Andrew Ginter
The job is to get the data from OT out into SAP and the custody transfer system and the predictive maintenance systems at the vendor's cloud site. You need to get the data out without introducing a pivoting path back in. So the one-way hardware lets data out, the software simplifies integration. Imagine there's an Oracle database or if there's anybody industrial on the call, imagine a Pi database. Imagine a database of some sort in the OT network. It's a very common design. You use, it's called a historian. You use the history of what's happened in the pipeline to understand what's currently happening, to predict what's going to happen in the future, et cetera, et cetera. And you want all that data available to the IT business automation so we can run the pipeline efficiently and so we can save a lot of money. The software logs into, call it Oracle, username and password, nothing tricky and asks for everything in the database that is newer since the last time we connected a second ago or whatever's configured. We get a snapshot, five, 10 megabytes of new data. We push that snapshot through the weird one-way hardware, and the software on the other side logs into an enterprise Oracle or an enterprise Pi or an enterprise whatever, and inserts the data. And so, now we have two databases that are synchronized in real time sub-second from OT to IT. And now anybody who needs that data can have it on the IT network. And here's the thing. A, even if the IT network is compromised, if the gateway is the only connection into OT, you cannot pivot through the gateway. No attack information can get in. And B, people ask, "Well, yeah, but all database communications is query, response. You ask for something, you get an answer. The hardware means I can no longer send queries into the OT network." Well, that's right. But the software means you no longer need to send queries into the OT network because all of the data that's allowed to be shared with the enterprise is out there already.

00:21:26 KC Yost
Fascinating. Great. Great. I like this. This is slick stuff. I'm impressed. I'm impressed. Let's talk about another acronym or initials, AV. What does AV stand for?

00:21:44 Andrew Ginter
Oh, wow.

00:21:45 KC Yost
And how does that fit into contract commitments?

00:21:50 Andrew Ginter
And if I may, I suggest let's edit this and rephrase that question. AV is antivirus. The intent of the question was you just said, "We can no longer send questions into the OT network."

00:22:08 KC Yost
Oh, okay.

00:22:08 Andrew Ginter
What if I need to send something into the OT network? I need to send antivirus updates. I need to send new contract commitments from my pipeline so the operators knows what are going on. How do I do that? That's sort of the question.

00:22:21 KC Yost
Okay, say that again please.

00:22:25 Andrew Ginter
Sure. So I just finished with-

00:22:27 KC Yost
AV stands for antivirus. Yeah.

00:22:29 Andrew Ginter
Antivirus. Yeah.

00:22:30 Andrew Ginter
You've got to send signatures.

00:22:30 KC Yost
Antivirus. Uh-huh.

00:22:31 Andrew Ginter
You've got to download new signatures into, do you not use antivirus on the OT network? Well, yeah, you do. Well then, how do you get the signatures in? The gateway won't let them in.

00:22:41 KC Yost
Okay. All right.

00:22:42 Andrew Ginter
And how do you send anything else in? If you can't send signatures in, then with a pipeline, the operator needs to know what are the legal commitments they've made to the customers in terms of getting the product moved. And usually, that is sent to the operator as information that shows up on one of their displays and that can be integrated into the control system.

00:23:09 KC Yost
Okay. So let me-

00:23:11 Andrew Ginter
Those are just two examples of information that happens to the other way.

00:23:15 KC Yost
So if I start with now we have to get commands into OT, how do you prevent antivirus from getting into OT?

00:23:27 Andrew Ginter
Nope. No, sorry.

00:23:29 KC Yost
How do you get the commands in to OT?

00:23:33 Andrew Ginter
No, let's back up. On your laptop, your antivirus, twice a day it calls out to the vendor. I don't care if it's Symantec or McAfee or whoever, calls out to the vendor and asks for new signatures, new ways to recognize the bad stuff. Okay. That's what a signature is. It's a rule that says, "If you see this byte at position 17 of the file, and this string is the next eight bytes, that's a RAT. Quarantine it."

00:24:12 KC Yost
Got it.

00:24:13 Andrew Ginter
Okay.

00:24:13 KC Yost
Got it.

00:24:14 Andrew Ginter
So if this is not common knowledge, you could just ask, "Well, what about updates? Do you not need to send security updates into the network?"

00:24:28 KC Yost
Or security updates or commands into the network? Can I say that?

00:24:32 Andrew Ginter
You could say commands. You could ask that question. It's a different answer. So you've talked about sending questions into the network. What if I need to send commands in? What if I need to send security updates in? How can I keep my network updated?

00:24:48 KC Yost
Okay. All right. Okay, we'll do that then.

00:24:51 Andrew Ginter
Okay, so I finished with queries and you're on.

00:24:55 KC Yost
Yes. Okay. So when we're talking about trying to, we've got this one-way communication, how do we send security updates, or how do we send commands from IT into the OT network? How do we do that?

00:25:18 Andrew Ginter
So that's a good question. In a sense, it's two different questions. Let me deal with commands first and then talk about security updates. Commands, let me answer your question with a question.

00:25:33 KC Yost
Okay.

00:25:35 Andrew Ginter
How many computers on the internet should be able to tell our pipeline, to tell the pumps, to tell the valves what to do?

00:25:51 KC Yost
One.

00:25:54 Andrew Ginter
Most customers I talk to say zero. What are you talking about?

00:25:58 KC Yost
Oh, really?

00:25:59 Andrew Ginter
We do not control the pipeline from the internet.

00:26:02 KC Yost
Oh, really?

00:26:03 Andrew Ginter
Do you know what happens if the pipeline is misoperated? You've got a hydraulic hammer, the pipe ruptures.

00:26:09 KC Yost
Sure.

00:26:09 Andrew Ginter
You don't do that. Zero. Okay. How many computers on the IT network control the pipeline? Again, zero is the answer most people give. We don't do that. And so, that's sort of the knee-jerk, the instinctive reaction among OT security practitioners, industrial security, among engineering teams saying, "Are you crazy? We don't do that." And then, you ask, "Well, what about security updates? Someone has to send these updates in." And so, every kind of update has a different answer. So let's deal with security updates real quickly.

00:26:54 KC Yost
Great.

00:26:54 Andrew Ginter
On my laptop right now talking to you, I've got automatic updates deployed. Why? Because I'm on the internet all day long, and what's the worst that happens if an update fails? My laptop blue screens, I have to restore from backups. That's an acceptable loss. Nobody auto updates industrial control systems. When an IT team says, " What do you mean you haven't installed the latest updates? What's wrong with you people?" The correct answer from engineering is, "I will be happy to install those updates as soon as you can tell me how likely those updates are to kill anyone." What do you mean? Engineers are responsible for the correct operation. If engineers misdesign the pipeline and people die.

00:27:44 KC Yost
Sure. Right.

00:27:45 Andrew Ginter
The engineers personally who signed off on that design could go to jail.

00:27:50 KC Yost
Right. Right.

00:27:51 Andrew Ginter
This isn't about practice.

00:27:52 KC Yost
I understand. Okay.

00:27:53 Andrew Ginter
This is why the engineering profession is a regulated profession like doctors, like lawyers, because public safety is at risk if these professionals-

00:28:01 KC Yost
Of course.

00:28:02 Andrew Ginter
So nobody auto updates. They put the updates on a test bed. They test the updates exhaustively, often for months, sometimes for years before they trust them enough into a safety critical. So we can come back to this, but the correct answer from IT is, I don't know. You need to figure that out. Yes. How long is it going to take you to figure that out? Three years. Okay then. The correct answer from IT is, so I understand that it's not possible to fix these vulnerabilities, but the fact that we cannot fix them does not make the risk go away. How are you going to address the risk? To which most engineering teams go, "What?" Those teams that have a cyber security function have figured this out, but sort of vanilla engineers 10 years ago would go, "What?" So to me, this is a good thing. Once the right questions are asked, once these two very different teams start asking each other questions, this is where we see progress. As long as they're throwing rocks over the fence saying, "Why haven't you? What's wrong with you people?" And coming back the other way going, "What's wrong with you? We can't do this." So that's sort of part of what's called the IT/ OT interface, the ITOT conflict. It's an interface is networks coming together, it's also people coming together. But you had a question about updates. So let me give you a simpler example. Antivirus updates have to come in. What's that? I've got antivirus on my laptop. Most of us listening have antivirus on our laptop.

00:29:49 KC Yost
Right.

00:29:49 Andrew Ginter
A couple of times a day, our antivirus contacts the vendor. I don't care if it's Symantec or McAfee or who knows what, contacts the vendor and gets a file downloaded with what's called signatures. These are rules saying, "Look at every file on your system. If byte number 13 through 79 is this string, that's a RAT. Quarantine the file." So it's a file of sort of things to look for to identify bad stuff. And there's always new bad stuff. This file is updated nearly constantly by the vendors, and so we download new signatures on a regular basis. Does nobody in OT use antivirus? Well, you cannot use antivirus on some systems. They're too sensitive. But you do use antivirus where you can. Well, how do we get updates in there? Well, that's a good question because the gateway will not let it through.

00:30:43 KC Yost
Right.

00:30:44 Andrew Ginter
95% of our customers, they've got a manual process. A person is tasked with doing this every day, going to the internet, downloading the file, checking the crypto checksum to make sure that it was actually signed by the vendor, writing it to a write once CD drive or a worm drive.

00:31:02 KC Yost
Oh, yeah.

00:31:03 Andrew Ginter
They don't trust the person carrying that CD across the office to a computer that's physically wired into the OT network and loading the antivirus onto the OT antivirus server. 5% of the customers do something different. So Waterfall again, our flagship is the Unidirectional Gateway. We have a family of products. One of the family is called a FLIP. It's a variation on the Gateway. It can be one way out or one way in, but never both. And so, most people program it to say, "At 2:00 in the morning, flip." So now it's one way in. "And in 10 minutes, flip back." Now it's one way out again. What good is that? Well, the FLIP is a hardware function. The software cannot control the FLIP. It does what it does. But the software can sense, "Oh, the thing just flipped. It's going in." And the software can say, "Okay, what do I do when it goes in? I go to a file server that has the latest antivirus on it. I pull it, and I pull that file." Okay? I don't pull all the files on the file server. Who knows what kind of nonsense has been deposited? "I pull this file and I check whatever I can check on the file. I push it through." Again, there's checks on the receiving side. "And then, it's dropped into the antivirus server. And then 10 minutes later, it flips again." So here's the thing. Because we're flipping in this example twice a day, the device is designed to make it as impossible as we can to set up a TCP connection through the device, to set up a command response, stimulus response communication through the device. It's designed to defeat that so that again, it is as close to impossible as we can practically make it, as close to impossible as practical to pivot and attack into OT.

00:33:04 KC Yost
I see. I see. So can I relate this process to my pipeline pump stations, compressor station applications?

00:33:17 Andrew Ginter
Yes. And this is a key concept. I've been talking about the IT/ OT interface. People ask us, "Can I put these devices into my compressor stations?" And my answer is, "Well, it depends." Yes, you can certainly do it. The question is, does it make sense? Does it give you any security? And most of the time, sort of in my most recent book, the advice I give people is you put this class of device, I call it network engineering. The Interdirectional Gateway is an example of network engineering. There's about five or six other examples of network engineering in chapter five. The Gateway is the most common kind, but you do network engineering to prevent pivoting attacks. There's lots of different ways to do it. And you use network engineering at what's called a consequence boundary, at a connection between two networks with dramatically different worst-case consequences of compromise. What's the worst that happens on an IT network? We leak all of our employees' private data into the internet. 50,000 employees are now at risk of identity fraud. This is very bad.

00:34:33 KC Yost
Sure. Okay. Yeah.

00:34:35 Andrew Ginter
We encrypt a whole bunch of stuff. We've got to pay experts, gurus to come in, pay through the nose on an emergency basis to clean up the mess. We can't take new orders for six days. These are all business consequences. We have to buy insurance, identity fraud insurance, for all of our employees. What's the worst case? Worst case is, I don't know, 10, 20, $50 million of mess we have to clean up. We can buy insurance for that. The fact that we can buy insurance for it means that the insurer regards it as an acceptable loss.

00:35:13 KC Yost
Right. Right.

00:35:15 Andrew Ginter
What's the very worst that can happen on an OT network? Again, pipelines. You're a pipeline guy. Hydraulic hammer, pipeline ruptures, massive amounts of gasoline are leaked.

00:35:24 KC Yost
Catastrophe, catastrophe.

00:35:26 Andrew Ginter
We're talking, people die. We cannot restore damaged equipment or people's lives from backups. So the nature-

00:35:35 KC Yost
Or the environment, right.

00:35:35 Andrew Ginter
Or the environment, precisely. The nature of the consequence is dramatically different.

00:35:42 KC Yost
Right.

00:35:42 Andrew Ginter
Add a consequence boundary. You put this technology, is there a consequence boundary between the pumping station, the compressor station and the SCADA system? Well, it depends. If we have leased fiber all the way down or forget leased, it's our own fiber all the way down the pipeline, we're renting some of that.

00:36:02 KC Yost
More and more pipelines are having fiber installed along with the pipe in the ditch.

00:36:08 Andrew Ginter
Right.

00:36:08 KC Yost
Yeah. Okay.

00:36:09 Andrew Ginter
Yeah. It's a revenue opportunity. You have the right of way.

00:36:13 KC Yost
Sure.

00:36:14 Andrew Ginter
If it's our own fiber, we can make it really hard to break into. So what happens if someone gets into the compressor station and messes with the settings on the programmable logic controllers on the flow computers controlling the valves badly? What happens? Well, very bad things can happen. What happens if someone breaks into the SCADA system and connects to the flow computers and the programmable logic controllers, connects to them as if they were the SCADA system and tells them to do the wrong thing? Well, pipelines rupture. Bad thing. It's the same consequences. There is no consequence boundary. You don't get a lot of value putting one of these devices in there, much as I'd love to sell you some. Where do we see people putting these devices into the compressor stations? If we connect out to the IT network, if we tunnel our communications from those compressor stations back to the SCADA systems through the internet, I'm sorry, that's a consequence boundary. Now it makes sense to start putting network engineering style devices, Unidirectional Gateways or variations of the Unidirectional Gateways or other network engineering that prevents pivoting at those boundaries.

00:37:45 KC Yost
So back in the 70s when I was at Tennessee Gas Pipeline, I was working in engineering training program, and I would sit in what was called gas control out here north of Houston, northwest of Houston. And every hour on the hour the compressor stations would call in. They were manned compressor stations and they would say, "Inlet pressure is this, outlet pressure is this, flow is this," that type of thing. Now, they have gone to unmanned stations in many locations where they're actually having this data given from the compressor station to gas control where they can make alterations. So this is exactly what you're talking about, right? In that interface, that communication and control?

00:38:40 Andrew Ginter
I fear that we have to redo this because it's a complicated question and I had six messages popping up that I'm dismissing. I lost track of the question. Could you try it again?

00:38:50 KC Yost
Okay. No, that's all right. Let's see. That was it.

00:38:56 Andrew Ginter
All right. Leave the question and just paraphrase for me. So you've got people phoning in from the compressor stations. I don't understand the scenario.

00:39:07 KC Yost
Yeah. So most compressor stations and pump stations are now-

00:39:14 Andrew Ginter
Unstaffed, yeah.

00:39:14 KC Yost
...remotely controlled.

00:39:16 Andrew Ginter
Yep.

00:39:16 KC Yost
Right? So the idea is back in the 70s they used to call in. They were manned stations. They would call in and give that information and then we would say, "Do this, do that, or whatever."

00:39:31 Andrew Ginter
Yep.

00:39:31 KC Yost
Now it's all done over the internet.

00:39:34 Andrew Ginter
Yes.

00:39:34 KC Yost
So is this what we're talking about? Okay?

00:39:37 Andrew Ginter
Yes. Yes.

00:39:38 KC Yost
So you can go ahead and start with that. Yeah. Okay.

00:39:44 Andrew Ginter
Yes, absolutely. The remote control is what we worry about. Yes, we worry about attacks pivoting directly into the compressor station, but nowadays you don't need to pivot all the way into the compressor station. If you've pivoted into the control center, okay, with a pipeline, nine times out of 10, you've got a controlled room with 5, 10, 20 operators sitting in front of banks of screens, looking at the pipeline, monitoring, operating the pipeline. If you break into that room, you can start sending commands out to the compressor stations through these connections to the compressor stations. The stations are unstaffed today, they're operated by remote control from that control room. You get into that control room and it's all over.

00:40:38 KC Yost
I got you. Okay. All right. So perfect, I understand. Andrew, fascinating conversation. I'm totally overwhelmed and I want to do more research and maybe have you come back and visit with us some more about this and maybe drill down on some of the topics. But we are running out of time, so I'm just interested if you can give me an overview and some ways of getting in touch with you, I'd appreciate that.

00:41:09 Andrew Ginter
Absolutely. Thank you for having me. It's a great pleasure talking about the stuff that I do every day. As you can tell, I can go off forever. The short message is that cybersecurity for physical operations, OT security, is evolving. It's changing. We've talked about it for a long time. Okay? There've been sort of leaders in the field doing stuff in this field for a long time. In fact, the leaders, sort of the super majors in the oil and gas industry have been doing this for a long time. Even if there's lots of people around who don't know they've been doing it, they have been doing it for a long time. And very recently, just the last couple of years, things have been changing fairly dramatically. More and more people are becoming aware of the problem. More and more people are becoming involved. More and more stuff is being done, and the way we look at the problem is changing. I'm a big fan of something called cyber-informed engineering, which is something that the Department of Energy is paying the Idaho National Laboratory, a dozen PhDs, to figure out for us. It's a way of... Historically, IT and OT have sort of butted heads over cybersecurity. Today we're starting to see them ask each other questions. Progress is being made. Cyber-informed engineering is in a sense, capturing that progress and saying, OT security is a coin with two sides. One side of the coin is cybersecurity. Teach engineers about cyber threats, teach them about cybersecurity tools. Teach them about the limits of those tools so they know what kind of risk they're still exposed to, even when they have cybersecurity deployed.

00:42:56 KC Yost
And teach them how to ask the questions to the IT guys like you were suggesting earlier.

00:43:01 Andrew Ginter
Absolutely. Absolutely.

00:43:02 KC Yost
Communication. Communication. Okay.

00:43:04 Andrew Ginter
And if the other side of the coin-

00:43:05 KC Yost
Sorry, didn't mean to interrupt you.

00:43:07 Andrew Ginter
The other side of the coin is engineering. We have overpressure relief valves. We have mechanical devices, electromechanical devices that are unhackable that can take some of the risk off the table that IT may not be aware of. You don't have to overdesign the security program if you can take risk off the table by subtle changes to the physical process. And so, this is what cyber-informed engineering is talking about. It's what my book is talking about. My latest book is Engineering Grade OT Security: A Manager's Guide. I love CIE, but I'm sorry, it's written by a dozen PhDs, and you can tell. So if I have a superpower, it's dumbing things down. So the subtitle says it all, it's a manager's guide. I fly high and slow. And the book is available on Amazon, but you don't have to buy it on Amazon. You can go to the Waterfall website, waterfall-security. com/ engineering-grade-OT-security, and you can request a free copy. We ship to most of the world's countries, but not all of them. I'm happy to get feedback on the book. This is how I write my next book, is I see how stuff is being received. I've already had feedback that I've made mistakes in chapter six. Thank you everyone who's pointed that out. I'm going to fix that. But the book is available for free. I'm on LinkedIn. I'm happy to interact with people. Look for me, Andrew Ginter Waterfall on LinkedIn. You'll find me. This has been a great opportunity. Thank you.

00:44:49 KC Yost
Oh, I've enjoyed having you on there. I really appreciate that. Again, if anyone wants to learn more about Waterfall Security Solutions, you can find them on the web at Waterfall-Security. com. That's Waterfall-Security. com, and get a free copy of Andrew's book. I think that's pretty darn cool that you're offering that out. So thanks to all of you for tuning into this episode of The Energy Pipeline Podcast, sponsored by Caterpillar Oil and Gas. If you have any questions, comments, or ideas for podcast topics, feel free to email me at kc.yost@oggn.com. I also want to thank my producer, Anastasia Willison-Duff, and everyone at the Oil and Gas Global Network for making this podcast possible. Find out more about other OGGN podcasts at OGGN. com. This is KC Yost saying goodbye for now. Have a great week and keep that energy flowing through the pipeline.

00:45:50 Speaker 5
Thanks for listening to OGGN, the world's largest and most listened to podcast network for the oil and energy industry. If you liked the show, leave us a review and then go to OGGN. com to learn about all our other shows. Don't forget to sign up for our weekly newsletter. The show has been a production of the Oil and Gas Global Network.