It’s an understatement to say that something broke in Seattle in February 2017. A major spill hit Seattle’s West Point wastewater facility causing massive damage and danger to human and animal life. Hundreds of millions of gallons of raw sewage were dumped into the Puget Sound over the next three months with the TV news all over it. Network engineers John Geer and Chris Roberts from down the road at Tacoma Wastewater Management watched the disaster unfold, glad it wasn’t them.
Up in Seattle the EPA and Department of Ecology immediately stepped in, with West Point being required to provide detailed sequence of events reports on exactly what happened and why. They knew a big storm tipped things off but it seemed faulty tank float alarms were a culprit along with human error, a lack of training and bad communication. That they were in the process of a systems upgrade wouldn’t help, and John mentions a programming mistake as one of the factors. “In Seattle, somebody’s gonna lose their job”.
Those detailed event reports are critical. They can lead to system improvements - or decide who’s on the hook. The City of Tacoma get sequence of events accuracy with a Tekron NTS-03G running time sync across five physically and logically isolated networks.
Chris (left) and John with their NTS-03G
“I was surprised at how easy it was to configure” says John. A minor issue with antennae placement was quickly resolved through support. Chris says “there was a high level of knowledge whenever we talked to somebody which led to a high level of confidence … it was easy to get a good level of detail without going through a bunch of steps”
Since then it’s been business as usual, with one exception. The system was put to the test last November during a spill – tiny compared to the Seattle event but still needing detailed reports.
John says “I had to create a report that documented what happened in the system based off alarms and statuses, look at the audit logs to see how the operators responded, and then report how long we were overflowing due to mechanical and operational problems.”
The City of Tacoma's North End Wastewater Treatment Facility
Their NTS03-G provides time sync to record the chain of events with extreme precision during a fault, far better than the 5-30 second uncertainty they used to get with windows time. It’s a turnkey solution that needs no offsite servers or separate hardware and software. The audit logs are only as accurate as their timestamps. In this case, their logs helped pinpoint the culprit – a faulty piece of equipment rather than operator error.
“We report on what actions coincided with the incidents that happened, just to make sure that it wasn’t an operator pushing the wrong button. If one log incorrectly showed a time difference it could have pointed towards the operator making a mistake and not the equipment failure”
That could lead to the wrong people being blamed.
Fortunately for the City of Tacoma, when the EPA or Ecology come knocking their reporting system is more than ready.