The Digital Substation is a trend in the energy industry. If you are close to the topic, then you have probably heard that a large amount of data is transmitted in the form of multicast streams. But do you know how to manage these multicast streams? What flow control tools are used? What does the regulation recommend?
Everyone who is interested in understanding this topic is welcome under the cut!
How is data transmitted over the network and why manage multicast streams?
Before proceeding directly to the Digital Substation and the nuances of building a LAN, I offer a brief educational program on the types of data transfer and data transfer protocols for working with multicast streams. We hid the educational program under the spoiler.
Data transfer types
LAN traffic types
There are four types of data transfer:
- Broadcast - broadcast.
- Unicast is the exchange of messages between two devices.
- Multicast - sending messages to a specific group of devices.
- Unknown Unicast - Broadcast to find a single device.
In order not to confuse the cards, let's briefly talk about the other three types of data transmission before moving on to multicast.
First of all, let's remember that within a LAN, addressing between devices is based on MAC addresses. Any transmitted message has SRC MAC and DST MAC fields.
SRC MAC – source MAC – source MAC address.
DST MAC - destination MAC - destination MAC address.
The switch based on these fields transmits messages. It looks at the DST MAC, finds it in its MAC address table, and sends a message to the port specified in the table. He also watches SRC MAC. If there is no such MAC address in the table, then a new pair "MAC address - port" is added.
Now let's talk more about the types of data transfer.
unicast
Unicast is the address transfer of messages between two devices. In fact, this is a point-to-point data transfer. In other words, two devices always use Unicast to communicate with each other.
Sending Unicast traffic
Broadcast
Broadcast is a broadcast. Those. broadcast, when one device sends a message to all other devices on the network.
To send a broadcast message, the sender specifies the DST MAC address as FF:FF:FF:FF:FF:FF.
Broadcast traffic transmission
Unknown Unicast
Unknown Unicast, at first glance, is very similar to Broadcast. But there is a difference between them - the message is sent to all network members, but is intended for only one device. It's like a message at the mall asking you to repark your car. Everyone will hear this message, but only one will respond.
When the switch receives a frame and cannot find the Destination MAC from it in the MAC address table, it simply sends this message to all ports except the one from which it received it. Only one device will respond to such a mailing list.
Sending Unknown Unicast traffic
Multicast
Multicast is sending a message to a group of devices that "want" to receive this data. It's very similar to a webinar. It is broadcast to the entire Internet, but only those people who are interested in this topic connect to it.
This data transfer model is called Publisher-Subscriber. There is one Publisher who sends data and Subscribers who want to receive this data subscribe to them.
When multicasting, the message is sent from a real device. The source MAC in the frame is the sender's MAC. But as a Destination MAC - a virtual address.
A device must connect to a group in order to receive data from it. The switch redirects information flows between devices - it remembers which ports data is transmitted from, and knows which ports this data should be sent to.
Multicast traffic transmission
An important point is that IP addresses are more often used as virtual groups, but since in the context of this article we are talking about energy, then we will talk about MAC addresses. In the IEC 61850 family of protocols that are used for the Digital Substation, grouping is done based on MAC addresses.
Brief educational program about the MAC address
The MAC address is a 48-bit value that uniquely identifies a device. It is broken into 6 octets. The first three octets contain manufacturer information. Octets 4, 5 and 6 are assigned by the manufacturer and are the device number.
MAC address structure
In the first octet, the eighth bit determines whether the given message is unicast or multicast. If the eighth bit is 0, then this MAC address is the address of a real physical device.
And if the eighth bit is 1, then this MAC address is virtual. That is, this MAC address does not belong to a real physical device, but to a virtual group.
A virtual group can be compared to a broadcasting tower. The radio company broadcasts some music to this tower, and those who want to listen to it tune the receivers to the desired frequency.
Also, for example, an IP video camera sends data to a virtual group, and those devices that want to receive this data connect to this group.
Eighth bit of the first octet of the MAC address
If the switch does not support multicast, then it will perceive the multicast stream as a broadcast. Accordingly, if there are many such flows, then we will very quickly clog the network with "garbage" traffic.
What is the essence of multicast?
The main idea of multicast is that only one copy of the traffic is sent from the device. The switch determines which ports the subscribers are on and forwards data from the sender to them. Thus, multicast can significantly reduce the data transmitted over the network.
How does it work in a real LAN?
It is clear that it is not enough to simply send one copy of the traffic to some MAC address, the eighth bit of the first octet is 1. Subscribers must be able to connect to this group. And the switches must understand from which ports the data comes from, and to which ports they need to be transmitted. Only then will multicast optimize networks and manage flows.
To implement this functionality, there are multicast protocols. The most common:
- IGMP.
- PIM.
In this article, we will tell you about the general principle of these protocols in a tangential way.
IGMP
An IGMP-enabled switch remembers which port the multicast stream is on. Subscribers must send an IMGP Join message to join the group. The switch adds the port from which the IGMP Join came from to the list of downstream interfaces and starts sending the multicast stream there. The switch constantly sends IGMP Query messages to downstream ports to check if it needs to continue transmitting data. If an IGMP Leave message is received from the port or there is no response to the IGMP Query message, then broadcasting to it stops.
At the moment, we are already using PIM
The PIM protocol has two implementations:
- PIMDM.
- PIMSM.
The PIM DM protocol works in reverse compared to IGMP. The switch initially sends a multicast stream as a broadcast - to all ports, except for the one from which it was received. Then it disables the stream on those ports from which messages came that it is not needed.
PIM SM is similar in principle to IGMP.
If you very roughly generalize the general principle of multicast operation - the Publisher sends a multicast stream to a specific MAC group, subscribers send connection requests to this group, switches control these flows.
Why did we go over multicast so superficially? Let's talk about the specifics of the Digital Substation LAN to understand this.
What is a Digital Substation and why is multicast needed there?
Before talking about the LAN of the Digital Substation, you need to understand what a Digital Substation is. Then answer the questions:
- Who is involved in the data transfer?
- What data is transferred to the LAN?
- What is the typical LAN architecture?
And after that, discuss multicast ...
What is a Digital Substation?
Digital Substation is a substation, all systems of which have a very high level of automation. All secondary and primary equipment of such a substation is focused on digital data transmission. Data exchange is built in accordance with the transmission protocols described in the IEC 61850 standard.
Accordingly, all data is transmitted here in digital form:
- Measurements.
- diagnostic information.
- Control commands.
This trend has been very developed in the Russian energy sector and is now being implemented everywhere. In 2019 and 2020, a lot of regulatory documents have appeared that regulate the creation of a Digital Substation at all stages of development. For example, STO 34.01-21-004-2019 PJSC "Rosseti" defines the following definition and criteria for the CPS:
Definition:
Digital substation - an automated substation equipped with digital information and control systems interacting in a single time mode and functioning without the presence of permanent duty personnel.
Criteria:
- remote observability of the parameters and modes of operation of equipment and systems necessary for normal operation without the constant presence of duty and maintenance personnel;
- provision of remote control of equipment and systems for the operation of the substation without the constant presence of duty and maintenance personnel;
- high level of equipment and systems control automation using intelligent control systems for equipment and systems operation modes;
- remote control of all technological processes in the single time mode;
- digital data exchange between all technological systems in a single format;
- integration into the power grid and enterprise management system, as well as ensuring digital interaction with the relevant infrastructure organizations (with related facilities);
- functional and information security in the digitalization of technological processes;
- continuous monitoring of the state of the main technological equipment and systems online with the transfer of the required amount of digital data, controlled parameters and signals.
Who is involved in the data transfer?
The Digital Substation includes the following systems:
- Relay protection systems. Relay protection is practically the "heart" of the Digital Substation. Relay terminals take current and voltage values from measuring systems. Based on this data, the terminals work out the internal protection logic. The terminals communicate with each other in order to transmit information about the triggered protections, the positions of switching devices, etc. The terminals also send information about the events that have occurred to the APCS server. In total, there are several types of communication:
▸Horizontal connection – communication between terminals.
▸Vertical connection – communication with the APCS server.
▸Measurement – communication with measuring devices.
- Commercial electricity metering systems.Commercial accounting systems communicate only with measuring devices.
- Dispatch control systems.Partial data must be sent from the APCS server and from the commercial metering server to the control room.
This is a very simplified list of systems that communicate within the Digital Substation. If you are interested in delving deeper into this topic, write in the comments.
Let's talk about it separately
What data is transferred to the LAN?
In order to combine the described systems with each other and organize horizontal and vertical communication, as well as the transfer of measurements, buses are organized. For now, let's agree that each bus is just a separate LAN on industrial Ethernet switches.
Structural diagram of an electric power facility in accordance with IEC 61850
Tires are shown in the block diagram:
- Monitoring/Control.
- Relay signal transmission.
- Transmission of instantaneous values of voltages and currents.
Relay terminals participate in both horizontal and vertical communication and also use measurements, so they are connected to all buses.
Through the bus "Transmission of signals RZA" terminals transmit information between themselves. Those. there is a horizontal connection.
Through the bus "Transfer of instantaneous values of voltages and currents" the transfer of measurements is implemented. Measuring devices are connected to this bus - current and voltage transformers, as well as relay protection terminals.
Also, the ASKUE server is connected to the bus “Transmission of instantaneous values of voltages and currents”, which also takes measurements for accounting.
And the bus "Monitoring / Control" is used for vertical communication. Those. through it, the terminals send various events to the APCS server, and the server also sends control commands to the terminals.
From the APCS server, the data is sent to the control room.
What is the typical LAN architecture?
Let's move on from an abstract and rather conditional block diagram to more mundane and real things.
The diagram below shows a fairly standard LAN architecture for a Digital Substation.
Digital Substation Architecture
At substations of 6 kV or 35 kV, the network will be simpler, but if we are talking about substations of 110 kV, 220 kV and above, as well as about the LAN of power plants, then the architecture will correspond to that shown.
The architecture is divided into three levels:
- Station/substation level.
- Attachment level.
- Process level.
Station/substation level includes workstations and servers.
Connection level includes all technological equipment.
Process level includes measuring equipment.
There are also two buses for combining levels:
- Station/substation bus.
- Process bus.
The station/substation bus combines the functions of the Monitoring/Control bus and the Relay Protection and Protection Bus. And the process bus performs the functions of the bus "Transfer of instantaneous values of voltage and current".
Features of Multicast transmission in Digital Substation
What data is transmitted using multicast?
Horizontal communication and transmission of measurements within the Digital Substation is performed using the Publisher-Subscriber architecture. Those. relay protection terminals use multicast streams to exchange messages among themselves, and measurements are also transmitted using multicast.
Prior to the digital substation in the power industry, horizontal communication was implemented using point-to-point communication between terminals. Either copper or optical cable was used as an interface. The data was transferred using proprietary protocols.
Very high demands were placed on this connection, because. protection operation signals, switching devices positions, etc. were transmitted through these channels. The algorithm for operational blocking of terminals depended on this information.
If the data is transmitted slowly or not guaranteed, there is a high probability that one of the terminals will not receive up-to-date information on the current situation and may give a signal to turn off or turn on the switching device when, for example, some work will be carried out on it. Or the breaker failure will not work in time and the short circuit will spread to the rest of the electrical circuit. All this is fraught with large monetary losses and a threat to human life.
Therefore, the data had to be transmitted:
- Reliably.
- Guaranteed.
- Fast.
Now, instead of point-to-point communication, the station/substation bus is used, i.e. LAN. And data is transmitted using the GOOSE protocol, which is described by the IEC 61850 standard (in IEC 61850-8-1, to be more precise).
GOOSE stands for General Object Oriented Substation Event, but this decoding is no longer very relevant and does not carry a semantic load.
Within the framework of this protocol, relay protection terminals exchange GOOSE messages with each other.
The transition from a point-to-point connection to a LAN did not change the approach. Data still needs to be transferred securely, reliably, and quickly. Therefore, a somewhat unusual data transfer mechanism is used for GOOSE messages. About him a little later.
Measurements, as we have already discussed, are also transmitted using multicast streams. In DSP terminology, these streams are called SV streams (Sampled Value).
SV streams are messages containing a certain set of data and transmitted continuously with a certain period. Each message contains a measurement at a specific point in time. Measurements are taken at a certain frequency - the sampling rate.
Sampling frequency is the sampling frequency of a time-continuous signal during its sampling.
Sampling rate 80 samples per second
The composition of SV streams is described in IEC61850-9-2 LE.
SV streams are transmitted through the process bus.
The process bus is a communication network that provides data exchange between measuring devices and bay level devices. The rules for data exchange (instantaneous current and voltage) are described in the IEC 61850-9-2 standard (currently the IEC 61850-9-2 LE profile is used).
SV streams, like GOOSE messages, must be transmitted quickly. If the measurements are transmitted slowly, then the terminals may not receive the current or voltage value necessary to operate the protection in time, and then the short circuit will spread to a large part of the electrical network and cause great damage.
Why is multicast necessary?
As mentioned above, in order to cover the data transmission requirements for horizontal communications, GOOSEs are transmitted somewhat unusually.
First, they are transmitted at the link level and have their own Ethertype - 0x88b8. This ensures high data transfer rates.
Now it is necessary to close the warranty and reliability requirements.
Obviously, to be sure, it is necessary to understand whether the message was delivered, but we cannot organize the sending of acknowledgments of receipt, as, for example, it is done in TCP. This will greatly reduce the data transfer rate.
Therefore, a Publisher-Subscriber architecture is used for GOOSE transmission.
Publisher-Subscriber Architecture
The device sends a GOOSE message to the bus and the subscribers receive the message. Moreover, the message is sent with a constant time T0. If an event occurs, a new message is generated, regardless of whether the previous period T0 has ended or not. The next message with new data is generated after a very short period of time, then after a slightly longer one, and so on. As a result, the time increases to T0.
The principle of transmission of GOOSE messages
The subscriber knows from whom he receives messages, and if he has not received a message from someone after time T0, then he generates an error message.
SV streams are also transmitted at the link level, have their own Ethertype - 0x88BA and are transmitted according to the Publisher-Subscriber model.
Nuances of multicast transmission in the Digital Substation
But "energy" multicast has its own nuances.
Nuance 1. GOOSE and SV have their own multicast groups
For "energy" multicast, their groups are used for distribution.
In telecom, the range 224.0.0.0/4 is used for multicasting (with rare exceptions, there are reserved addresses). But the IEC 61850 standard itself and the IEC 61850 corporate profile from FGC PJSC define their own multicast ranges.
For SV streams: 01-0C-CD-04-00-00 to 01-0C-CD-04-FF-FF.
For GOOSE messages: 01-0C-CD-04-00-00 to 01-0C-CD-04-FF-FF.
Nuance 2. Terminals do not use multicast protocols
The second nuance is much more significant - relay protection terminals do not support IGMP or PIM. Then how do they work with multicast? They just wait for the required information to be sent to the port. Those. if they know that they are subscribed to a specific MAC address, then they accept all incoming frames, but process only the necessary ones. The rest are simply discarded.
In other words, all hope is placed on the switches. But how will IGMP or PIM work if the terminals do not send Join messages? The answer is simple - no way.
And SV streams are rather heavy data. One stream weighs about 5 Mbps. And if everything is left as it is, it turns out that each stream will be broadcast. In other words, we will pull only 20 streams per 100 Mbps LAN. And the number of SV-flows at a large substation is measured in hundreds.
What is the solution then?
Simple - use the old verified VLANs.
Moreover, IGMP in the LAN of the Digital Substation can play a cruel joke, and vice versa, nothing will work. After all, switches without a request will not start transmitting flows.
Therefore, a simple commissioning rule can be distinguished - “The network is not working? – Disable IGMP!”
Normative base
But maybe it is still possible to somehow organize a LAN of a Digital Substation based on multicast? Let's try to turn now to the regulatory documentation on the LAN. In particular, I will cite excerpts from the following SRTs:
- STO 34.01-21-004-2019 - DIGITAL SUPPLY CENTER. REQUIREMENTS FOR TECHNOLOGICAL DESIGN OF DIGITAL SUBSTATIONS WITH VOLTAGE 110-220 kV AND NODAL DIGITAL SUBSTATIONS WITH VOLTAGE 35 kV.
- STO 34.01-6-005-2019 - SWITCHES OF POWER OBJECTS. General technical requirements.
- STO 56947007-29.240.10.302-2020 - Typical technical requirements for the organization and performance of technological LANs in the APCS of the UNEG Substation.
Let's first see what can be found in these SRTs about multicast? The mention is only in the latest service station from PJSC FGC UES. The service station asks during acceptance tests of the LAN to check whether the VLANs are configured correctly and to check the absence of multicast traffic in the ports of the switches that are not specified in the working documentation.
Well, the service station also prescribes that the service personnel should know what multicast is.
That's all about multicast ...
Now let's see what can be found in these SRTs about VLANs.
Here, all three STOs agree that switches should support VLANs based on IEEE 802.1Q.
STO 34.01-21-004-2019 says that VLANs should be used for flow control, and with the help of VLANs, traffic should be divided into relay protection, process control systems, AIIS KUE, video surveillance, communications, etc.
STO 56947007-29.240.10.302-2020, in addition, still requires the preparation of a VLAN distribution map during design. At the same time, the service station offers its ranges of IP addresses and VLANs for DSP equipment.
The CTO also provides a table of recommended priorities for different VLANs.
Table of recommended VLAN priorities from STO 56947007-29.240.10.302-2020
From a flow management point of view, that's it. Although there is still a lot to discuss in these stations - from various architectures to L3 settings - we will definitely do this, but next time.
Now let's summarize the flow control in the Digital Substation LAN.
Conclusion
In the Digital Substation, despite the fact that a lot of multicast streams are transmitted, in fact, standard multicast traffic management mechanisms (IGMP, PIM) are not used. This is due to the fact that end devices do not support any multicast protocols.
For flow control, the good old VLANs are used. At the same time, the use of VLAN is regulated by regulatory documentation, which offers sufficiently elaborated recommendations.
Useful links:
.
Source: habr.com