Command Center

It's quiet in the Project Operations Center. A few bodies stir in their seats as they study the numbers that scroll up on three computer screens. Eyes also scan the huge map board in front. On its surface, lit and flashing lights note status situations and alarms. Decisions must be made quickly; phone calls placed if needed. But still the mood is calm, almost relaxed.

This placid atmosphere, however, belies the significance of this spacious room on the third floor of the Joint Operations Center building in Sacramento. This is the nerve center of one of the largest water and power systems in the world, the State Water Project. The SWP moves an annual average of 3 million acre-feet water from Lake Oroville to areas in southern San Francisco Bay, San Joaquin Valley, and Southern California. The entire system -- over 50 power plants, pumping plants, and reservoirs with 660-plus miles of pipelines and aqueducts -- can be controlled from this central control center.

Once occupying a snug spot on the 16th floor at headquarters, the POC has not only had a makeover in space and looks, its control system has undergone a metamorphosis which will bring SWP operations into the future.

Old Equipment, New Tools
First installed in the late 1960s, the POC's old control system had outlived its usefulness. "It was obsolete and difficult to keep online or even find parts for," says Ed Trevino, Chief of the Control Systems Branch. "Data update rates were 3 to 5 minutes, meaning if there was a power failure or a unit tripped (stopped operating), the POC staff was unaware during that delay."
Different computers drove the control systems for the main center and the five area control centers located in each field division. Field staff normally control Project facilities within their jurisdictions, with the POC acting as backup if any ACC failed. But the dissimilar systems tested the capabilities of the old controls.

For one, if two or more ACCs failed simultaneously, the POC computer was severely overstressed. Another problem arose in data calculations. With each system analyzing numbers for reservoir storage, water flows and levels differently, data values used for operational decisions varied. As time passed and with the databases managed by separate staff, the values grew even more disparate.

The biggest stumbling block, however, was the old system's inability to change as change was needed, particularly where new concepts for operation were envisioned.

So the search began for a new control system--and a new center. The original site for the new POC was at Jibboom Street near Sacramento's Old Town. When toxic waste was found on the grounds and the site was rezoned within the 100-year floodplain, another site was selected on El Camino Boulevard. Since the existing building had to be stripped and rebuilt, Al Cosper (now retired) engineered the POC to fit the specifications of its users. The new building now houses and integrates SWP operations with DWR's Division of Flood Management, the U.S. Bureau of Reclamation's Central Valley Project, and the National Weather Service.

Around the same time, the project to replace the old control systems began. "Various headquarters and field division staff met to discuss design concepts and list system requirements," says Ed Trevino, who was then an associate control engineer working on his first DWR assignment.

Water and power dispatchers (see A Different Breed) who staff the POC and the hydroelectric plant operators and seniors who monitor the ACCs, needed more detailed information on equipment status at the remote facilities, faster data updates of changes as they occurred, and a more fault-tolerant, reliable system. Control engineers required a software system that could be easily modified as operational needs changed. All agreed that, with a database expected to reach 70,000 monitor-and-control points, a fast and easy-to-navigate display system was essential.

"The new system we chose meets all those requirements, as well as takes advantage of recent technological advances such as wide area networking and relational database design,” says Jain Fong, head of the POC Engineering section, which provides application development and engineering support for the POC. "These advances give us tools to face future needs with confidence."

State-of-Art Equipment, Great Improvements
Replacing the old with the new system was a massive undertaking consisting of two major phases-- installing and programming of (1) many new Remote Terminal Units used to monitor and control equipment (see Role of an RTU) and (2) control center computer systems for each of the 10 largest pumping and power plants, the Area Control Centers, and the POC.

For the latter phase, the JC-6000 SCADA (Supervisory Control And Data Acquisition) was selected. Each SCADA system incorporates a redundant backup computer for increased reliability. The POC's system can control the entire SWP as well as allow ACC operators and POC dispatchers to perform their respective duties.

"Now there is more than enough power to handle the entire statewide system and then some," explains Trevino. "Both POC and ACC staff will be looking at the same database and see the same values. SCADA will also coordinate whether the POC or the ACC is in control of Project facilities."

The new SCADA system brings improvements that will radically alter how data is presented and how fast. Data transmission is down to 4 seconds along the aqueduct and 8 seconds from major plants. Displays will present data in tables, charts and graphs unlike the earlier displays with their lines of numbers, which often lacked sufficient detail.

The new hierarchal database provides another pragmatic refinement, by providing the speed needed for rapid display updates, control calculations, and alarm functions. Plus the technology is easier to use and maintain than the older flat file system. Data can be located by a name or ID, instead of remembering its specific location in the database.

"This modification will make a big difference in our ability to program changes in the system," explains Jain Fong.

An added feature especially welcomed by the dispatchers is alarm priority. "Alarms are set off by changing conditions such as a water level drop in the aqueduct or pumping or power units shutting down. Because the system can now bring in more data, all kinds of alarms may go off at once," says Ron McAfee, the POC's Chief Water and Power Dispatcher with 30 years’ DWR experience in his field.

"There is no way a dispatcher can look or even deal with a 1,000 alarms, so the system is programmed to prioritize the alarms and display only those that need immediate attention."

But the improvement that most excites the engineers and users alike is the Information Storage and Retrieval System, which will allow more users to view ongoing SWP operations. "Previously, the only way to access the database was to directly connect to the control system," says Lupe Rodriquez, a Program Water and Power Dispatcher in charge of the POC's day-to-day operations. However, because of security constraints, only operators and dispatchers were allowed connections.

"Through ISR, users can have access to a near real-time, mirror image of the data. This means they can see the numbers but can't change them. With this new security, anyone in the Department who is interested (and has permission) can view Project operations as they’re happening."

Big Map Board, More Displays
Standing 12 feet high and spanning 54 feet across the front of the POC, the map board displays data from specific facilities such as pumping and power plants, electrical interties, reservoirs, and check sites--the 66 radial gates that control water flows along SWP aqueducts. The display panels correspond to the location of the facility along the board's plan view of the Project's alignment against a topographical background of California.

"The map board is made of several hundred 8-inch by 8-inch silk screened metal tiles laid over a special magnetic backing," says Rodriquez, who spearheaded its design.

Lights on the board also alert dispatchers to alarms that may indicate a plant exceeding operational limits, the failure of a unit in a pumping or power plant, or an unexpected change in a check pool’s water elevation. (The pool is the body of water between gates.)

"We also wanted visitors to the POC to be able to clearly see and understand what is happening and where it’s happening," adds Rodriquez. "It's not just impressive to look at but provides a more complete picture of SWP operations, as well as that of USBR reservoirs affecting Project releases."

The board however is not yet complete. Plans are to add data showing earthquakes as they occur in the state and water quality conditions in the Delta.

Fresh Challenges, Outstanding Team Work
To replace the old with the new was a "massive" undertaking for all involved. Initially the POC's and ACCs' systems were scheduled to be completed in 1995, with the plant SCADA systems to follow later. But plans aren't always foolproof, and challenges along the way sometimes seriously delay deadlines. Such was the case with the system's software.

"Our operations are very complex to simulate for factory tests with the vendor," says Ed Trevino. "There were errors in the system and it took a long time to debug the software, which went back to the vendor many times for fixes. That was the main reason for the slip in completion dates."

But the effort was well worth the time. "Because all the ACCs, the POC, and the major plants will use the same software, it's crucial that the software be right and reliable," Trevino explains. "Once the software is right, the rest is easy."

Even more critical to the success of the program was the team effort between headquarters and field division staff. "The field divisions' input into the system's development was essential. Plus their staff did all of the installation at each ACC-- including the wiring and physical mounting of equipment. Without their cooperation and hard work, it wouldn't have been possible to achieve what we did."

The feelings are mutual, says Jeff Said, Chief Operator at Delta Field Division. "They gave us outstanding service," he explains. "The control systems technicians and operators here also did an excellent job. Working together made all the difference. Because our division was the first to operate the aqueduct on the new SCADA system, we helped them get most of the 'bugs' out before the system was expanded to other field divisions."

Headquarters' staff also took their concerns seriously. "If we wanted to see certain displays or change the way we entered data, they tried to accommodate our needs. Of course, not everything could be done, but whatever they could do, they did."

As for the new system itself, Said says basically his operators love it. "It's fast, easy, and pleasing to look at -- that's important when you consider the long hours we work, the amount of water that is moved, and the many sites we have to monitor. But adjusting to it hasn't been easy. We still have work to do."

Chief Operator at San Joaquin Field Division, April Petok and her staff are experiencing a similar learning curve. "Change is hard; it's difficult to let go what you were comfortable with. Plus there are still some 'bugs' to work out in the new system.

"But when it's functioning well, the operators are discovering what the new SCADA can do," she adds. "It is much faster. There is a lot more operators can do and do it easier. At the same time, the system is a much more complicated to use. But it's like anything brand new; you have bumps here and there until you master it."

The Work Continues
Even with the new system now online, the project remains a work in progress.

"We can't sit back and take a deep breath that it's over. By having the ability to make changes, we find we are asked to make changes. We will continue to implement modifications and upgrades, and try to support what's needed in the field," says Trevino.

Jain Fong adds, "It's always an evolutionary process. We have to try to grow with changes in technology, and we will experience growing pains while getting accustomed to these changes. But this system will take us into the future and leave us ample room to improve and expand."


sidebar
A Different Breed of People
It takes a different breed of people to work in the POC. The 24-hour workday is divided into three shifts -- day, swing and graveyard. They rotate shifts, constantly juggling work hours and sleep schedules. The consoles must be continually staffed, even through meal times. While the activity is constant, it's not always exciting. But these hardy souls already know the routine, having paid their dues in the field divisions’ Area Control Centers where the duties are much the same.

Of the three dispatchers, one monitors water operations, another deals with power generation, loads, and transactions; while the third is the senior who oversees all of it and makes decisions in emergencies. Each must be able to fill the other’s shoes, a feat which can take up to two years to master comfortably.


sidebar
Role of an RTU
RTUs, Remote Terminal Units, are microprocessor-based equipment, which serve as a system's local input-output devices. These units are connected to all pumps, generators, aqueduct checks, turnouts, and other SWP facilities, which require frequent monitoring and remote control by POC dispatchers and ACC operators. Each of the system's approximately 250 RTUs is programmed to automatically perform equipment functions such as start-up or shutdown of units, as well as to send updated data on their operations.

The RTUs within a pumping or power plant are part of a local SCADA system, operated from the plant's control room. These units, together with the RTUs along the aqueduct, are controlled by the SCADA control centers serving the POC dispatchers and ACC operators.