Separating IT and OT Networks Helps Manufacturer Boost Productivity, Revenue and Lead Times
JUMP TO: Customer | Challenge | Discovery | Solution | Results
Customer
As a leading architectural aluminum manufacturer, this company specializes in designing and producing pre-engineered aluminum framing systems, such as commercial and institutional storefronts and entrances.
As customer demands increase and construction schedules become more condensed, the company takes advantage of every opportunity available to advance its manufacturing facilities, production systems and technology.
Challenge
The manufacturer runs between five and seven days a week, 24/7, to produce as many products as possible. Its entire production process runs on a network. For years, however, the IT team had maintained ownership of that OT network.
IT had concerns about separating the OT network from IT and didn’t want to relinquish partial network control to production. As a result, workers on the plant floor depended on IT for everything—from machine operation to the recipes that run the presses.
Nearly three times every week, production struggled with unplanned network downtime. This created major gaps in lead times, productivity and revenue—not to mention an incredibly frustrating work environment.
Without explanation or warning, an entire production line would lose network communications and shut down. “At minimum, this caused at least 20 to 30 minutes of downtime every time it happened,” says the manufacturer’s controls engineer. “IT could never tell us what happened. They would say it was our PLC hardware, but I knew that wasn’t true.”
The problem turned out to be IT’s network switches. While the company did have industrial-grade switches at the cabinet level, they connected to enterprise switches that were managed by IT.
Even if IT were to support network separation, however, the network’s current design and operation made it impossible.
“All the PLCs, three large aluminum extrusion presses, control consoles with HMIs—everything was set up as part of the IT network,” explains the controls engineer. “Every Ethernet address had a PLC, I/O rack or sensor that ran on Ethernet. Each had its own Ethernet cable running between 200 and 250 feet to non-industrial managed switches. We had three setups like this networked together on a plant fiber.”
The OT team wanted to make its production process more “error tolerant,” as the controls engineer described. If the host devices went down, the production line should still be able to operate
Discovery
The manufacturer needed to bring in multiple sources for quotes and had a long-standing relationship with H.H. Barnum, a distributor of factory automation solutions.
After learning about the scope of the project, H.H. Barnum brought Belden to the table:
“Belden was at our plant three or four times to go over things, ask questions and visit the plant floor to look at our setup and architecture. For that reason alone, they stood out,” says the controls engineer. “They showed up, they showed interest and I already knew the quality of the product.”
Solution
The company’s controls engineer knew the time would come when OT had no choice but to establish its own network. To prepare, he mocked up a media redundancy protocol (MRP) network topology to separate the OT and IT networks and avoid single points of failure.
The engineer shared these plans with a Belden solutions engineer, who made recommendations about what it would take to complete the design and deploy and maintain a separate network.
“With respect to the aluminum extrusion presses, we brought nearly everything on Ethernet back to Hirschmann switches in the panels,” says the company’s controls engineer. “If an uplink went down, the press will still run and talk to everything else.”
Those Hirschmann BOBCAT switches were connected to three network control cabinets: one for each aluminum extrusion press. Each network cabinet features a new redundant fiber system that connects the Hirschmann BOBCAT switches to Hirschmann Greyhound switches in network boxes. In each cabinet, one Hirschmann BOBCAT switch is dedicated to the company’s video system, which monitors the production process and presses.
The manufacturer also invested in Industrial HiVision Network Management Software to identify, map and configure network devices; find faults quickly and achieve timely remediation; and view real-time status and performance data 24/7.
When it was time to separate the networks, it wasn’t as simple as a quick disconnection and reconnection. Collaboration and cooperation were required between OT, automation and IT.
To help the company prepare, Belden’s solutions engineer arrived onsite three days prior to the go-live date. His goal was to bring OT and IT together and make sure everyone was on the same page before moving forward.
During these discussions, he discovered that IT and OT were struggling to explain their plans and concerns, and neither group understood what the other was saying. Because production was moving off the IT network, IT wanted complete and immediate separation—but that wasn’t possible. Subsequent suggestions made by IT weren’t feasible, either, due to firewall communication issues. If the manufacturer had followed that action plan, the network separation and cutover would have failed.
To overcome these deployment challenges, Belden’s solutions engineer acted as the liaison to help the groups align priorities, work as a single entity and make sure they understood the roles they would play.
Results
The company’s production floor holds a new record: six months (and counting) with no network problems. This is a significant improvement from the downtime workers experienced multiple times each week before establishing their own OT network. As a result, productivity, revenue and lead times have all improved.
Network visibility is now a reality for the production team. If a device does go down, employees can see exactly what’s happening. For example, without this level of insight, OT leadership wouldn’t have noticed that workers were shutting off panels when they shouldn’t have been. With the network management software, they were able to detect a group of devices going down, coming back online and then going down again.