-
Notifications
You must be signed in to change notification settings - Fork 20
PlaygroundPassthroughQuickstart
This is a quick introduction to inserting a protocol stack in Playground.
Just like in the OSI model, you can have a stack of protocols work together in Playground. The "bottom" layer is the Playground Wire Protocol (PWP). It provides a mix of layers 1, 2, 3, and even 4. Much like UDP, it has ports for multiplexing. It also provides addressing using Playground addresses as well as data link management.
[ Application Layer ] [ Application Layer ]
| |
[ PWP ] <----------> [ PWP ]
But PWP does NOT provide any reliable delivery. You have to insert that layer into the stack.
[ Application Layer ] [ Application Layer ]
| |
[ Reliable ] [ Reliable ]
| |
[ PWP ] <----------> [ PWP ]
Similarly, there is no secure layer. You will also have to add that into the stack.
[ Application Layer ] [ Application Layer ]
| |
[ Secure ] [ Secure ]
| |
[ Reliable ] [ Reliable ]
| |
[ PWP ] <----------> [ PWP ]
There are three basic elements to this process:
- Create a "stacking" protocol, transport, and factory
- Insert the stacking factory (with associated protocol and transport) into playground as a named connector
- Modify a client and server to use this named connector
A network stack protocol is slightly different from the application protocols that you have been creating so far. Consider the protocol class you created for the HTTP server or the escape room. It's not a network stack protocol. It's an application protocol that sits at the top of the stack. It is not expected to communicate with another stack higher.
But your reliable and secure layers have to communicate with protocols above and below.
The first thing to do is create a stacking protocol that can communicate with a protocol above it. For an example, we will define a Passthrough protocol. It is very simple. It does nothing more than pass data through it. It literally does nothing except pass data up and down the stack.
[ Application Layer ] [ Application Layer ]
| |
[ Passthrough ] [ Passthrough ]
| |
[ PWP ] <----------> [ PWP ]
Just to be clear, this layer is worthless and wasteful. You would never really do this. It's just to teach you how to think about network stacking protocols and how to insert them.
The data_received method for this protocol is going to be very simple. It just passes the data up to the higher layer.
from playground.network.common import StackingProtocol
class PassthroughProtocol(StackingProtocol):
def data_received(self, data):
self.higherProtocol().data_received(data)
The only real difference for a class that inherits from StackingProtocol is that it has the higherProtocol() method for allowing access to the higher protocol in the stack.
But not only must your protocol pass data to the higher layer, it must also tell the higher protocol when it has connected and when it has disconnected. The higher protocol is completely disconnected from those lower signals and will only get them when your protocol gives it to them. This is done by signaling higherProtocol().connection_made() and higherProtocol().connection_lost().
When should you call these signals? It depends on your protocol. For our passthrough example, the protocol calls the higher protocol's connection_made and connection_lost when those same methods are called on the passthrough.
class PassthroughProtocol(StackingProtocol):
def connection_made(self, transport):
self.higherProtocol().connection_made(<what goes here?>)
def data_received(self, data):
self.higherProtocol().data_received(data)
def connection_lost(self, exc):
self.higherProtocol().connection_lost(exc)
Again, this is pretty simple. If you think back to the stack:
[ Application Layer ]
/ \
| Pasthrough call's Application's connection_made
[ Passthrough ]
/ \
| PWP calls Passthrough's connection_made
[ PWP ]
But, as was hinted at in the code, we need to pass a parameter to the connection_made method. If you look at the method signature:
def connection_made(self, transport):
It is obvious that what we need to pass is a transport. But what transport? UNDER ALMOST ALL CIRCUMSTANCES, YOU DO NOT PASS THE LOWER LAYER'S TRANSPORT.
class PassthroughProtocol(StackingProtocol):
def connection_made(self, transport):
self.higherProtocol().connection_made(transport) ## THIS IS WRONG!!!!
The reason why this is wrong is because this is the transport from the lower layer. The next layer up should write to the passthrough layer, not the layer below the passthrough layer. Here is a visualization of why this is wrong:
[ Application Layer ] (PWP Transport) --+
|
|
[ Passthrough ] |
(PWP Transport) |
| |
\ / |
[ PWP ] <----------------+
Instead, we need to give a Passthrough transport to the next layer up. The way to do this is create a StackingTransport. A StackingTransport, has a lower transport the same reason a StackingProtocol has a higher protocol. The StackingTransport class is fully functional by itself as a passthrough layer. That is, StackingTransport.write just calls the lower transport's write method. So, for a Passthrough layer, we can use this unmodified.
from playground.network.common import StackingProtocol
from playground.network.common import StackingTransport
class PassthroughProtocol(StackingProtocol):
def connection_made(self, transport):
passthroughTransport = StackingTransport(transport)
self.higherProtocol().connection_made(passthroughTransport)
Of course, for pretty much every other protocol, you will need to modify the StackingTransport to do something more than just pass data through. For example, what if our Passthrough layer wrapped all the data in a packet before sending it on. The transport class would need to add the wrapping packet and the protocol class would need to remove it.
from playground.network.common import StackingProtocol
from playground.network.common import StackingTransport
class PassthroughPacket(PacketType):
DEFINITION_IDENTIFIER="passthroughpacket"
DEFINITION_VERSION="1.0"
BODY=[
("data",BUFFER)
]
class PassthroughTransport(StackingTransport):
def write(self, data):
passthrough_packet = PassthroughPacket()
passthrough_packet.data=data
self.lowerTransport().write(passthrough_packet.__serialize__())
class PassthroughProtocol(StackingProtocol):
def __init__(self):
super().__init__()
self.buffer = PassthroughPacket.Deserializer()
def connection_made(self, transport):
passthrough_transport = PassthroughTransport(transport)
self.higherProtocol().connection_made(passthrough_transport)
def data_received(self, data):
self.buffer.update(data)
for passthrough_packet in self.buffer.getNextPackets():
self.higherProtocol().data_received(passthrough_packet.data)
def connection_lost(self, exc):
self.higherProtocol().connection_lost(exc)
It should be fairly clear what is happening. The upper layer will get the PassthroughTransport. When it calls transport.write() the write method is wrapping the data in the PassthroughPacket before sending it on.
On the receiving side, the data_received method de-encapsulates the data from the packet before passing
it up.
The very last thing that needs to be done is to create a stacking factory. A stacking factory is responsible for creating all the protocols in a stack and linking them together. The StackingProtocolFactory has a simple CreateFactoryType method that creates a new type for a set of protocol factories that are to be called together starting from the bottom to the top.
from playground.network.common import StackingProtocolFactory
PassthroughFactory = StackingProtocolFactory.CreateFactoryType(PassthroughProtocol)
factory1 = PassthroughFactory()
This may not seem very helpful. It's more useful when there are two or more protocols in the stack. In our example, there's nothing that prevents us from having two passthrough layers:
from playground.network.common import StackingProtocolFactory
DoublePassthroughFactory = StackingProtocolFactory.CreateFactoryType(
PassthroughProtocol, # This is the first layer passthrough factory
PassthroughProtocol # This is the second layer passhtrough factory
)
factory1 = DoublePassthroughFactory()
When factory1 is called it produces an instance of each factory in its list and links them together (e.g., the higherProtocol() of the first is set to the second, and the higherProtocol() of the second is set to the application layer).
But you don't need to really think about all these details. For the most part, they are going to be hidden from you. All that is required is the factory. Once you have it, you can insert it into your playground installation.
Some day, I will have an automated importer for Playground network stacks. For now, you need to insert it manually.
The first thing to do is to create a Python module. A module is a directory that contains a __init__.py file and any other files required by the module. The __init__.py file is automatically executed when the module is imported by another Python script. If you are not clear about how Python modules work, you should look up the relevant documentation.
For your module, I recommend have a file that defines the protocol, transport, and factory. So, suppose that we are creating the Passthrough protocol described above. We would put all the code above for PassthroughProtocol, PassthroughTransport, and PassthroughStackFactory into a file. For simplicity, let's name it protocol.py and we will put it in a directory called passthrough. So, our directory structure looks like this:
passthrough/
__init__.py
protocol.py
The protocol.py file has all the code described above. In the __init__.py file we will put just a few instructions that tell Playground how to insert this module in. Basically, we need to register it with a unique name that can be used to identify it.
import playground
from .protocol import PassthroughFactory
passthroughConnector = playground.Connector(protocolStack=(
PassthroughFactory(),
PassthroughFactory()))
playground.setConnector("passthrough", passthroughConnector)
playground.setConnector("any_other_name", passthroughConnector)
The second import is from .protocol. That tells it to import from the protocol.py file within the module. We import the PassthroughFactory type that can be used for creating stacking factories.
Next, we create a Playground Connector. This is an object that says how to create a stack for clients and servers (e.g., using the create_connection and create_server methods). In our case, our Passthrough protocol is the same whether it's a client or server so we create the client and server factories from the same type. But if we had a protocol like TCP that had different behavior for client versus server connections (e.g., the TCP client sends the SYN, etc), we would need to define different factories for each one.
Once we have created the connector, we register it with Playground using setConnector. The first parameter is the name of the stack. It can be any legal string and you can have aliases. In the example above, the stack can be accessed under either the name passthrough or any_other_name. Note: these can be over-written, so it's your responsibility that you actually use a unique name.
Once the module is finished, you insert into Playground by finding your .playground directory (check pnetworking status if you're not sure). Copy (or sym-link) your module (the entire directory) into .playground/connectors/. In our example, when you're done, your directory structure should look like this:
.playground
connectors/
passthrough/
__init__.py
protocol.py
Your layer is now ready to be used by Playground programs.
There are two typical ways to employ a network stack. The first is to use the family parameter in the create_connection or create_server methods. The name family comes from the original asyncio methods. The family parameter referred to the family of socket (TCP, UDP, etc). For consistency, Playground re-uses that name.
playground.create_connection(factory, host, port, family="passthrough")
playground.create_server(factory, host, port, family="passthrough")
Note that the factory in the example above is the factory for producing the application layer protocol. It is not related in any way to the factory that is inside your module.
The family parameter is useful for when you want to control the network stack with some kind of configuration parameter (e.g., a command line parameter).
Another way is to include the stack name directly in the Playground address.
playground.create_connection(factory, "passthrough://20191.1.2.3", 101)
This is convenient for many client situations. However, because the host is often not specified for a server, it's not always a great solution for servers.
Within the class github repository, you can find a version of the passthrough network stack in src/samples/passthrough. You should be able to symlink the passthrough directory directly into your Playground installation.
You can test out the passthrough stack with the echo test source code that is within the Playground repo's src/test directory. The echo test script takes a -stack argument at the commandline. To start the server:
python echotest.py server -stack=passthrough
To start the client:
python echotest.py localhost 101 -stack=passthrough
Obviously, this won't work until you've installed the passthrough module into your .playground/connectors directory.