It is often said the devil is in the detail. This idiom is generally applied as a warning, most likely from an individual who has experience in the impact of a detail being missed.
Within the commissioning process, we uncover a lot of detail; some foreseen, some not. Commissioning authorities interact with systems and equipment in ways that are often in greater detail and rigor than that system may be subjected to in daily and ongoing operation. Below are a few lessons learned from ESD’s experiences:
Mechanical Rapid Restart
Problem: In many of our data center applications, mechanical rapid restart is required to stabilize the cooling and avoid thermal runaway after a power cycle. In one application, the unit was capable of rapid restart, but it was not outfitted with sufficient capacitance in the control circuits to keep the controls online. Two options were identified to solve the issue: shorten the delays for restoration of input power or add a separate UPS supported control power circuit.
Solution: In the end, pulling a UPS supported control power circuit and throw-over circuit to the mechanical system controls was decided upon as the most reliable option.
Relays Failing
Problem: One of our applications had control power throw-over relays failing due to current spikes during power transfers between separately derived and out-of-synchronization sources. Investigation uncovered that the control power throw-over relays were sized based on sources being closed to in sync and did not account for the current spike when transferring to an out-of-phase source.
Solution: The fix was pretty simple: upsize the relay ampacity.
Testing the Switchgear
Problem: Functional testing on utility-to-generator-transfer switchgear had used simulation of the voltage input dropping out to trigger automated transfers. When testing the facility during the integrated-systems test, it was the first time that we had the ability to have the local utility interrupt power external to the building. We expected to witness the same process we had seen numerous times: Interrupt power, switchgear senses power loss, generators start, switchgear transfers loads to generator.
What actually occurred when we got the utility to interrupt power was quite a bit different: Interrupt power and then… crickets. Investigation uncovered that the voltage-sensing relay responsible for telling the switchgear to start generators had two key details to it: the power for the relay was derived off the incoming utility, and the signal to initiate transfer to generators was a normally open contact. With no input power, the relay shut down and no signal to generator was initiated because the relay was off.
Solution: Change the contact for the generator start signal from normally open to normally closed. A loss of input power was then able to complete the expected transfer to generator.
Tripping Offline
Problem: On one project we repeatedly had instances of the main utility breaker on our switchgear tripping offline. A deep dive on the issue uncovered that the protective relays were tripping offline due to reverse power that exceeded the allowable limits of the utility. The root cause of the reverse power turned out to be a UPS unit going into a self-test mode that generated the reverse power. As a factory default, this mode was enabled.
Solution: Disable the self-test mode. Beware of factory defaults.
it is important to remember, as shown by these examples, that sometimes even the smallest problems within a system can have major impacts on its operation and ability to support a critical load. Often times, the quick and easy fixes are the ones that prove to be the most impactful and the ones that provide the most value after turnover. While many problems can be hard to foresee, sticking to an agreed upon plan and testing procedures that take the level of detail up a notch will uncover where the devil resides.