Transmission Network (MS-OTN/Liquid OTN) Synchronization Architecture for Smart Grid

Chen Xiaozhou

Chief Engineer of Huawei's Optical Product Line in the Energy Industry
Standing Committee Member of the IEEE PLC Standards Committee
IEEE p2893 OTN OSU Series Standard Group Chair
IEEE PES China Cable Working Group Member and Deputy Leader of the F5G Working Group
IEC TC57 wg20 China Member
Senior Member of the CIGRE International Power Grid Conference

Transmission Network Synchronization for Critical Services

Optical transmission is a key process to provide the optimal synchronization network for critical services. For more than 30 years, operators have provided reliable Plesiochronous Digital Hierarchy (PDH) and Synchronous Digital Hierarchy (SDH) for critical services. In the future, however, smart grid critical services will be deployed based on the Multiprotocol Label Switching Transport Profile (MPLS-TP) and Liquid OTN technologies, which are similar to traditional Time Division Multiplexing (TDM) over SDH.

Most transmission network carriers provide one platform (POTN/MS-OTN or Liquid OTN) that includes pulse code modulation (PCM), SDH, MPLS-TP, and Optical Transport Network (OTN) functions on the same device, to cover most medium- and low-voltage services, as well as extend and deploy new communications infrastructure. The proposed system platform must integrate SDH, MPLS-TP, and Dense Wavelength Division Multiplexing (DWDM) technologies to ensure high quality, highly reliable networks for production services — as well as high bandwidth, highly efficient networks for packet services.

SDH networks and Synchronous Optical Networks (SONETs) — often deployed by transmission network carriers — are typically built to carry information between a utility control center and all remote sites. The information is used to manage capacity, monitor and control the system, bill customers, and enable staff communications. Traffic is typically carried over TDM circuits, such as T1/E1, RS-232, X.21, and E&M. These TDM networks — SDH and SONET — ensure high reliability using mechanisms such as a Unidirectional Path-Switched Ring (UPSR), which enables a utility communications network to switch over in less than 50 ms in the event of a failure.

With SDH/SONET, circuits are established using static configuration, a method applied in many utility communications networks. SDH provides the best service latency and jitter features based on TDM architecture. However, traditional Global Positioning System (GPS) clock synchronization is required to provide a synchronous time source for smart grid substations. Technological upgrade is therefore required to protect existing investments and minimize risks.

A POTN/MS-OTN communications network incorporates state-of-the-art technologies to enable a power utility to deploy future-proofed, highly available IP networks, which continue to support existing TDM and legacy applications while providing a smooth migration path to IP, Ethernet, and MPLS-TP services. This new MS-OTN network maximizes the cost effectiveness and efficiency without affecting reliability, as well as enables the deployment of new devices and applications that can improve operations and workflow efficiency. A highly available MS-OTN network is ideal to support both the critical operations that come with an SDH network, and all other corporate communications requirements of an MPLS-TP network.

The figure above shows the network structure of a transmission device that supports IEEE 1588v2.

For IP interfaces (MPLS-TP and IP L2 Ethernet), IEEE 1588v2 requires that all devices on a network support the IEEE 1588v2 protocol.

A transmission device can support IEEE 1588v2 only when an isolated clock channel (such as OSC) and a clock board or device are available on the network. The clock board provides all IP device functions, and the isolated clock channel provides network routing to transmit IEEE 1588v2 messages. Traditional MS-OTN standards support this design through an Optical Supervisory Channel (OSC).

Next-generation Liquid OTN networks support low-rate services, from 2 Mbit/s to 100 Mbit/s, and greatly reduce latency by transforming from multi-level mapping of PCM, SDH, and OTN to direct mapping of Liquid OTN. Like SDH, Liquid OTN provides the best service latency and jitter features based on TDM architecture.

Network Synchronization Architecture Design for a Smart Grid

To provide a synchronous time source for smart grid substations, an entire WDM/OTN transport and IP network must support clock synchronization; in a smart grid communications network, time synchronization must be strictly implemented between substations. There are multiple time synchronization solutions in the industry, including GPS and IEEE 1588v2.

Traditional GPS clock synchronization:

• High cost: A GPS system must be configured or each base station.

• High failure rate: Only one satellite card is configured for each base station, and no protection channel is available.

• Poor maintainability: If the GPS fails, the hardware must be replaced onsite, and remote maintenance cannot be performed.

IEEE 1588v2 clock synchronization:

• Low cost: Only two GPS devices need to be configured to implement clock synchronization over the entire network.

• High availability: End-to-end clock protection is supported.

Although providing obvious benefits, IEEE 1588v2 requires that all devices on a network support the IEEE 1588v2 protocol. Otherwise, when only simple transparent time transmission is implemented, clock performance may not meet the high-precision clock requirements of a wireless network.

IEEE 1588v2 Implementation on IP Devices

The figure below shows the hardware structure of a modular switch that supports IEEE 1588v2. IEEE 1588v2 is supported by the modular switch only when a Main Processing Unit (MPU), which provides a clock daughter card, and a line card that provides a clock function are configured. Cards can be classified into two different types: Application-Specific Integrated Circuit (ASIC), and Ethernet Network Processor (ENP). An ASIC card provides a Field Programmable Gate Array (FPGA) logical module to process IEEE 1588v2 messages; an ENP card directly uses an ENP chip to process IEEE 1588v2 messages.

Hardware structure of a switch that supports IEEE 1588v2.

Note: If the external clock signal is inputted, the device will operate in the master state. The time synchronization process is as follows:

The external time information is sent to the FPGA logic module on the clock subcard through the external clock port. The clock subcard updates its local clock time based on the external time. The clock subcard sends the time information to the FPGA logic module on the ASIC card or to the ENP card. The FPGA logic module or the ENP card updates the local clock time after receiving the time information, implementing time synchronization between the NE clock and the external clock.

The IEEE 1588v2-based time synchronization process is as follows:

1. An IEEE 1588v2 message is sent to the Physical Layer (PHY).

2. The 1588v2 engine of the FPGA logic module on the ASIC card, or the 1588v2 module on the ENP card, identifies the IEEE 1588v2 message and adds a timestamp to the message based on the local time.

3. The timestamp is sent to the FPGA logic module on the clock daughter card, through the channel between the line card and the clock daughter card. The FPGA module parses the timestamp and sends the timestamp to the Phase-Locked Loop (PLL) system.

4. The PLL system calculates the time difference between master and slave devices based on the timestamp information, and sends the time difference information to the local FPGA logic module.

5. The local FPGA logic module calibrates the local time after receiving the time difference information sent by the PLL system.

6. The PLL system sends the time information to the FPGA logic module on the ASIC card or to the ENP card.

7. The FPGA logic module or the ENP card updates the local clock time after receiving the time information, implementing time synchronization between the Network Element (NE) clock and the external clock.

IEEE 1588v2 Implementation by the OSC on Transmission Devices

The figure below shows the network structure of a transmission device that supports IEEE 1588v2, which can be deployed in an SDH/MS-OTN/Liquid OTN network.

MS-OTN asymmetry compensation approach: Single-fiber bidirectional OSC resolves asymmetric delay.

• IEEE 1588v2 packets are transmitted over the same fiber (OSC) to resolve the asymmetric delay problem.

• Single-fiber bidirectional OSC can adapt to multiple protection schemes. Asymmetry compensation is not required after protection switching.

• After the fiber is repaired or changed, asymmetry compensation is not required.

If the ST2 clock board is a hardware module that implements the IEEE 1588v2 protocol, the ST2 clock board transmits IEEE 1588v2 protocol packets to the Fiber Interface Unit (FIU) board, and IEEE 1588v2 synchronization information shares the fiber with service data. The entire process is as follows:

1. The IEEE 1588v2 information of the clock board is sent to the IEEE 1588v2 FIU board.

2. The FIU board transmits these 1588v2 messages to the network fiber through the 1491 nm and 1511 nm channels, which are defined in the MS-OTN standard.

3. IEEE 1588v2 information is transmitted over the same fiber to guarantee asymmetric delay.

4. As shown in the following network, hop-by-hop hardware supports IEEE 1588v2 and implements time synchronization between the transmission structure device clock and the external clock.

IEEE 1588v2 Performance

• Normally, routers and MS-OTN devices can guarantee a time error of less than ±30 ns per device from one Ethernet port to another.

• In some scenarios, MS-OTN subracks are located at the same site and the total time error is less than ±30 ns.

• The time error of microwave devices is less than ±50 ns from one Intermediate Frequency (IF) port to another.

The network itself does not need to implement clock synchronization. To provide clock signals for a PTN/SDH network and implement clock synchronization, the MS-OTN network needs to obtain the active and standby clock sources from the Building Integrated Timing Supply (BITS) system.

The PTN/SDH network can implement clock synchronization by accessing the clock source of the MS-OTN network, and then providing clock information for base stations to achieve synchronization.

Frequency Synchronization

• Synchronization network architecture is specified in ITU-T G.803.

• Synchronization information is transmitted through the network via synchronization network connections, transmitting signals at different synchronization levels. Each synchronization network connection is provided by one or more synchronization link connections, each of which is supported by a synchronized PDH trail, SDH multiplex section trail, OTN OTU trail, OTN OSC trail, or IEEE 802.3 physical media trail.

• Some of these synchronized PDH trail, SDH multiplex section trail, OTN OTU trail, OTN OSC trail, or IEEE 802.3 physical media trails contain a communication channel, Synchronization Status Message (SSM), Timing Marker (TM), OTN Synchronization Message Channel (OSMC), or Ethernet Synchronization Message Channel (ESMC), which transports a quality-level identifier. This quality-level identifier can be used to select the highest synchronization level incoming reference signal from a set of nominated synchronization references available at the NE.

• Synchronization network connections are unidirectional and generally point-to-multipoint. ITU-T G.803 specifies a master-slave synchronization technique for synchronizing SDH and OTN networks, and ITU-T G.8261 specifies it for packet transport networks.

• The Primary Reference Clock (PRC) network connects to the primary synchronization source and backup synchronization source. If the primary synchronization source fails, the backup synchronization source will be selected.

Definitions of Clock Source Quality Levels

• SDH and packet transport networks throughout the world are based on different synchronization philosophies, which ITU-T G.8272 categorizes in sections I, II, and III.

WAN Interconnection Solution for Telecom Services

Telecom networks require MS-OTN and IP networks to implement clock synchronization. Frequency synchronization is a prerequisite for time synchronization and can be implemented through SyncE and IEEE 1588v2. There are two combination solutions for time synchronization:

• SyncE + IEEE 1588v2: SyncE implements frequency synchronization, and IEEE 1588v2 implements time synchronization.

• IEEE 1588v2 + IEEE 1588v2: IEEE 1588v2 implements both frequency and time synchronization.

IEEE 1588v2-based time synchronization can be implemented in boundary clock (BC) and transparent clock (TC) modes. The TC mode, though, has several major disadvantages:

1. Poor reliability:

TC nodes do not use the Best Master Clock (BMC) algorithm. Therefore, IEEE 1588v2 path protection can only be ensured through service path protection. In addition, TC nodes do not support automatic path switching based on link status and quality level.

2. Complex maintenance:

• Protocol fault: TC nodes do not parse messages and cannot detect protocol faults. They only provide transmission channels and calculate residence time.

• Performance fault: TC nodes only calculate residence time. However, residence time changes randomly and as a result, its accuracy cannot be measured. When a fault occurs on a downstream BC or Ordinary Clock (OC) node, the faulty NE cannot be located.

3. High bandwidth load:

Messages sent by TC nodes need to be replicated and transmitted. All Delay_Req/Delay_Resp messages sent by E2E TC nodes are aggregated to the upstream OC or BC node, consuming high bandwidth, which causes service message loss.

The SyncE + IEEE 1588v2 time synchronization solution has the following advantages:

1. Good compatibility and mature SyncE: SyncE evolved from SDH and has been widely deployed on live networks. In addition to base stations, fixed network carriers or integrated carriers need to provide synchronization services for other services or applications. However, these services may only require frequency synchronization, rather than time synchronization. Moreover, the number of SyncE-capable devices is larger than that of IEEE 1588v2-capable devices.

2. Higher reliability: In SyncE + IEEE 1588v2 networking mode, if the IEEE 1588v2 clock source or a 1588v2 link fails, NEs can use SyncE to maintain time precision. In IEEE 1588v2 + IEEE 1588v2 networking mode, if the IEEE 1588v2 clock source fails, NEs have to rely on themselves to maintain time precision. However, the frequency precision of NEs is far inferior to the frequency synchronization precision provided by SyncE.

The IEEE 1588v2 + IEEE 1588v2 time synchronization solution (BC mode) has the following advantages:

1. Easy network planning: IEEE 1588v2 can implement automatic synchronization path calculation through BMC, avoiding loops.

2. Easy maintenance: The SyncE + IEEE 1588v2 solution uses different synchronization protocols to achieve frequency and time synchronization, as the frequency and time synchronization paths are different. Whereas the IEEE 1588v2 + IEEE 1588v2 solution uses the same protocol to achieve frequency and time synchronization, as the frequency and time synchronization paths are the same.

IEEE 1588v2-based Frequency and Time Synchronization (BC Mode)

IEEE 1588v2 implements frequency and time synchronization on a network.

Two Building Integrated Timing Supply (BITS) devices function as OCs to inject the clock source to switches. The BMC algorithm is used to determine the grandmaster clock (GMC), and all devices on the synchronization path must support the IEEE 1588v2 function.

The switches work in BC mode and recover time hop by hop. Base stations function as OCs and obtain time information from the switches. Only the BMC algorithm is used for synchronization; therefore, frequency and time synchronization paths are the same. The minimum rates of sending Sync, Delay_Req/Delay_Resp, and Announce messages are 128/s, 128/s, and 8/s respectively.

BC synchronization on the entire network is advantageous because every site recovers time, and synchronization precision can be measured hop by hop, facilitating maintenance and fault location. In addition, the BMC algorithm is used on the BCs to ensure rapid protection switching between BITS reference sources and tracing paths.

IEEE 1588v2-based Frequency and Time Synchronization (TC Mode)

In the transparent transmission scenario, bearer network devices act as TCs and do not require time recovery, they only calculate the residence time that IEEE 1588v2 messages take to traverse TCs. In TC mode, the time of bearer network devices is not synchronous with that of the BITSs. Base stations that act as slave OCs recover time through IEEE 1588v2 messages and synchronize the time to the BITSs.

In TC mode, bearer network devices can transparently transmit the time of different domains. Two BITSs can belong to different carriers or time domains, and the time of the two paths can be different. Bearer network devices do not need to support the BMC algorithm, which has low requirements for device software.

However, because TC nodes do not recover time and only calculate the variable residence time, it is difficult to determine the accuracy of calculated residence time. Therefore, maintenance and fault location become difficult. As TC nodes do not use the BMC algorithm, they cannot automatically switch to another path when the original TC path on the bearer network is faulty. A service-layer protection protocol other than the IEEE 1588v2 protocol must be used for TC path protection.

SyncE-based Frequency Synchronization and IEEE 1588v2-based Time Synchronization (BC Mode)

SyncE implements frequency synchronization while IEEE 1588v2 implements time synchronization on a network.

The BITSs obtain the clock source from the GPS or BDS. Devices requiring frequency synchronization obtain the reference clock source from the deployed SDH network or SyncE network, while devices requiring time synchronization obtain the reference clock source from the BITSs. As SSM and BMC are different algorithms, the frequency and time synchronization paths are not the exact same. When a BITS fails, the network can use SyncE to maintain time precision.