Add PacketStream protocol. #85

rowanG077 · 2024-07-03T14:07:38Z

Todos:

Test haddock and make nicer
Copy over tests
Make top-level PacketStream module that exports all sub-modules
Top-level module should hide inner modules
Add forceResetSanity to async fifo
Remove code duplication in the tests

src/Protocols/PacketStream/AsyncFifo.hs

src/Protocols/PacketStream/Base.hs

src/Protocols/PacketStream/AsyncFifo.hs

src/Protocols/PacketStream/Base.hs

bgamari · 2024-08-02T14:22:44Z

The motivation here is a bit unclear to me. The documentation says

Provides the PacketStream protocol, a simple streaming protocol for transferring packets of data between components

However the "data" in question appears to be only bytes. Presumably there is a specific use-case in mind here but it is not clear to me what it is; network packets, perhaps?

It seems to me like ideally protocols in clash-protocols should be polymorphic in the type of the data that they transfer. Afterall, I wouldn't expect packets of bytes to be terribly prevalent in the "interior" of most designs. Unless you are building actual networking hardware, I would imagine you would typically want to parse the packet into a more structured form somewhere near the "edge" of the design.

t-wallet · 2024-08-02T14:51:08Z

The motivation here is a bit unclear to me. The documentation says

Provides the PacketStream protocol, a simple streaming protocol for transferring packets of data between components

However the "data" in question appears to be only bytes. Presumably there is a specific use-case in mind here but it is not clear to me what it is; network packets, perhaps?

It seems to me like ideally protocols in clash-protocols should be polymorphic in the type of the data that they transfer. Afterall, I wouldn't expect packets of bytes to be terribly prevalent in the "interior" of most designs. Unless you are building actual networking hardware, I would imagine you would typically want to parse the packet into a more structured form somewhere near the "edge" of the design.

This protocol was indeed specifically created for a hardware-based MAC/IP/UDP stack (which will soon be coming to clash-cores). However, it is just a simpler version of AXI Stream with stricter metadata invariants. You can use it for any other purpose that needs high-speed data streaming, like video streaming. This PR also provides several circuits which are able to read these unstructured bytes into arbitrary types.

rowanG077 · 2024-08-02T15:10:49Z

@bgamari It was discussed with @lmbollen whether to make the payload polymorphic. In theory there isn't anything that currently really looks at the payload so it would not be super hard to do add a type variable for the payload.

But in my experience working with packetized streams there usually is not a sane way to have an interior type. You are almost certainly only dealing with fragments, these will generally not have a clear structure at the scope of a single cycle. The structure comes from the fragments together.

Do you have some concrete idea how you see a packet stream packet with a non-byte oriented payload? I'm not against the idea but just don't see the advantage.

martijnbastiaan · 2024-08-24T11:02:13Z

But in my experience working with packetized streams there usually is not a sane way to have an interior type.

This is my experience too. In particular, any protocol that has byte enables (including the interface of block RAMs) are a pain to deal with if they're not having bits/bytes/bitvector as an interior type.

Still, ADTs are the killer feature of Clash IMO, so perhaps we should also offer API compatible versions without byte enables.

…ream.Hedgehog

* Support zero-byte transfers * Add utility that merges trailing zero-byte transfers * Make PacketStream test framework more flexible Now allows users to: - Set the distribution of abort generation themselves; - Choose whether to generate zero-byte packets; - Choose whether to generate trailing zero-byte transfers. * Converters: remove expensive operations `mod` and `*` are removed.

Reverted back to the old version where all M2S outputs are registered, but now also handles arbitrary metadata and output data widths bigger than 1 that divide the input data width. Zero-byte transfers are always preserved. This new version reduces the LUT resource usage by half compared to the old registered version, because of a mux with 1 selector bit (optimal) being inferred by synthesizers for the next state computation, instead of a mux with 2 selector bits.

…source files

Now correctly drops packets that are too big to fit in the content FIFO in 'Backpressure' mode.

Forcing bytes in PacketStream _data to be 0x00 turned out to be too restrictive. Allowing them to be undefined makes a lot of components more efficient: depacketizers, packetizers, dropTail, and upConverter. It also strengthens the test framework as we can now force un-enabled bytes to throw an error when evaluated.

Adds a generic component which enforces packets to be a certain length, which is extracted from the metadata. Packets that are too small are aborted. This is useful for higher-level network protocols such as IP and UDP.

lmbollen

lgtm modulo some minor remarks.

lmbollen · 2024-11-22T13:49:12Z

clash-protocols/src/Protocols/PacketStream/Base.hs

+    -- Bitmask used for data equality. If the index of a data byte is larger
+    -- than or equal to the size of `_data`, it is a null byte and must be
+    -- disregarded in the equality check.
+    mask = case _last t1 of
+      Nothing -> repeat False
+      Just size -> imap (\i _ -> resize i >= size) (_data t1)
+
+    dataEq = case compareSNat (SNat @dataWidth) d0 of
+      SNatLE -> True
+      SNatGT ->
+        leToPlus @1 @dataWidth
+          $ fold (&&)
+          $ zipWith3 (\b1 b2 isNull -> isNull || b1 == b2) (_data t1) (_data t2) mask


Why is the mask inverted?

lmbollen · 2024-12-02T13:48:27Z

clash-protocols/src/Protocols/PacketStream/Depacketizers.hs

+__NB__: assumes that packets are longer than @ceil(n / dataWidth)@ transfers.
+If this is not the case, the behaviour of this component is undefined.
+-}


Behavior when this assumption is false should not affect other packets and at the least abort the packet.

clash-protocols/src/Data/Constraint/Nat/Extra.hs

clash-protocols/src/Protocols/PacketStream/Base.hs

clash-protocols/src/Protocols/PacketStream/Converters.hs

lmbollen · 2024-12-02T14:21:09Z

clash-protocols/src/Protocols/PacketStream/Converters.hs

+  forall (dwIn :: Nat) (dwOut :: Nat) (meta :: Type) (dom :: Domain) (n :: Nat).
+  (HiddenClockResetEnable dom) =>
+  (1 <= dwIn) =>
+  (1 <= dwOut) =>


Replace with dwIn * n

clash-protocols/src/Protocols/PacketStream/Delay.hs

lmbollen · 2024-12-02T14:36:10Z

clash-protocols/tests/Tests/Protocols/PacketStream/Depacketizers.hs

+{- | Test the depacketizer with varying datawidth and number of bytes in the header,
+  with metaIn = () and toMetaOut = const.
+-}
+depacketizerPropertyGenerator ::
+  forall
+    (dataWidth :: Nat)
+    (headerBytes :: Nat).
+  (1 <= dataWidth) =>
+  (1 <= headerBytes) =>
+  SNat dataWidth ->
+  SNat headerBytes ->
+  Property
+depacketizerPropertyGenerator SNat SNat =
+  idWithModelSingleDomain
+    @System
+    defExpectOptions{eoSampleMax = 1000, eoStopAfterEmpty = 1000}
+    (genPackets 1 4 (genValidPacket defPacketOptions (pure ()) (Range.linear 0 30)))
+    (exposeClockResetEnable (depacketizerModel const))
+    (exposeClockResetEnable ckt)
+ where
+  ckt ::
+    (HiddenClockResetEnable System) =>
+    Circuit
+      (PacketStream System dataWidth ())
+      (PacketStream System dataWidth (Vec headerBytes (BitVector 8)))
+  ckt = depacketizerC const


Can you also test the behavior of metadata fields?

lmbollen · 2024-12-02T14:58:01Z

clash-protocols/src/Protocols/PacketStream/Routing.hs

+-- | Collect mode for `packetArbiterC`
+data ArbiterMode
+  = -- | Collect in a round-robin fashion. Fair and cheaper than `Parallel`.
+    RoundRobin
+  | -- | Check components in parallel. This mode has a higher throughput, but is
+    -- biased towards the last source and also slightly more expensive.
+    Parallel


Reuse existing type from Df

Just checked it, it's not quite the same. roundrobinCollect also has a NoSkip mode:

Collect in a roundrobin fashion. If a component does not produce data, wait until it does.

We could add this mode to the arbiter as well.

That would be nice, let's try to make it uniform

- Removed dangerous leZeroIsZero (could cause type checker to break) - Added source argument to nullByte - Improved documentation of unsafeAbortOnBackpressureC - Remove use of bitCoerce if only pack or unpack is required - Removed M2SNoMeta in favour of PacketStreamM2S n ()

rowanG077 force-pushed the packetstream branch 3 times, most recently from 0669c0c to ee9e485 Compare July 3, 2024 14:35

clash-lang-builder reviewed Jul 4, 2024

View reviewed changes

src/Protocols/PacketStream/AsyncFifo.hs Outdated Show resolved Hide resolved

christiaanb reviewed Jul 4, 2024

View reviewed changes

src/Protocols/PacketStream/Base.hs Outdated Show resolved Hide resolved

christiaanb reviewed Jul 4, 2024

View reviewed changes

src/Protocols/PacketStream/AsyncFifo.hs Outdated Show resolved Hide resolved

leonschoorl reviewed Jul 8, 2024

View reviewed changes

src/Protocols/PacketStream/Base.hs Outdated Show resolved Hide resolved

rowanG077 force-pushed the packetstream branch 5 times, most recently from e11179f to 919bb9b Compare July 17, 2024 13:03

t-wallet force-pushed the packetstream branch from f70eb2c to a7e1678 Compare July 19, 2024 11:42

t-wallet marked this pull request as ready for review July 22, 2024 14:01

t-wallet force-pushed the packetstream branch from 1ae3a4f to 7c96795 Compare July 31, 2024 13:10

This was referenced Aug 2, 2024

Add metadata support to packetstream up/down converters #102

Closed

Make PacketStream up/down converter less strict on the required width. #100

Closed

t-wallet mentioned this pull request Aug 20, 2024

PacketStream depacketizer remove redundant buffer #105

Closed

t-wallet force-pushed the packetstream branch from 581ec7a to 30d979c Compare August 20, 2024 14:16

t-wallet force-pushed the packetstream branch 2 times, most recently from 4d8758d to a38eadc Compare August 30, 2024 16:21

t-wallet mentioned this pull request Sep 3, 2024

Add PacketStream Delay circuit #101

Closed

t-wallet force-pushed the packetstream branch from 56ab0c0 to dac1bc4 Compare September 16, 2024 10:17

t-wallet force-pushed the packetstream branch from 2e0e8df to bfe1baf Compare October 10, 2024 15:24

t-wallet force-pushed the packetstream branch 2 times, most recently from 97b5f56 to a90007a Compare October 18, 2024 14:11

t-wallet added 25 commits November 14, 2024 12:18

Formatting

0243d20

Improve documentation

f14c73a

Export useful testing functions and generators via Protocols.PacketSt…

7fb3fba

…ream.Hedgehog

Rename packetizer model functions

ffbfa9a

Add sensible DfConv functions

605767c

Export chopBy

7088bcc

Allow user to pass custom packet generator to genPackets

7a0065e

Optimize delay circuit

91fb7f8

Add delayStream id test

6ec7a6c

Add dropTailC

2664979

Fix typos in dropTailC tests

77d3bb5

Implement more basic operations

9d8a6a9

Add autoreg instances

92c7355

Slap unsafe on abortOnBackpressureC

7436a3e

Add unsafe version of upConverterC

3ba2619

Adapt to new repo structure

aaf5e87

Prefer use of deepErrorX over undefined

9ad34c7

Converters: patch up documentation

640a948

Add copyright, license and maintainer attributes to all PacketStream …

048510f

…source files

Fix PacketFifo bug

ed59f06

Now correctly drops packets that are too big to fit in the content FIFO in 'Backpressure' mode.

Fix CSignal simulation functions

da6da16

Remove global constraint solver iterations and unused code

9120a9b

t-wallet force-pushed the packetstream branch from 6c02bde to 2e9557d Compare November 14, 2024 11:18

Add generic padding stripper

fd1d06c

Adds a generic component which enforces packets to be a certain length, which is extracted from the metadata. Packets that are too small are aborted. This is useful for higher-level network protocols such as IP and UDP.

lmbollen approved these changes Dec 2, 2024

View reviewed changes

t-wallet force-pushed the packetstream branch from f17fbcf to 1d2f936 Compare December 2, 2024 16:36

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add PacketStream protocol. #85

Add PacketStream protocol. #85

rowanG077 commented Jul 3, 2024 •

edited by t-wallet

Loading

bgamari commented Aug 2, 2024 •

edited

Loading

t-wallet commented Aug 2, 2024

rowanG077 commented Aug 2, 2024 •

edited

Loading

martijnbastiaan commented Aug 24, 2024

lmbollen left a comment

lmbollen Nov 22, 2024

lmbollen Dec 2, 2024

lmbollen Dec 2, 2024

lmbollen Dec 2, 2024

lmbollen Dec 2, 2024

t-wallet Dec 2, 2024 •

edited

Loading

lmbollen Dec 3, 2024

Add PacketStream protocol. #85

Are you sure you want to change the base?

Add PacketStream protocol. #85

Conversation

rowanG077 commented Jul 3, 2024 • edited by t-wallet Loading

bgamari commented Aug 2, 2024 • edited Loading

t-wallet commented Aug 2, 2024

rowanG077 commented Aug 2, 2024 • edited Loading

martijnbastiaan commented Aug 24, 2024

lmbollen left a comment

Choose a reason for hiding this comment

lmbollen Nov 22, 2024

Choose a reason for hiding this comment

lmbollen Dec 2, 2024

Choose a reason for hiding this comment

lmbollen Dec 2, 2024

Choose a reason for hiding this comment

lmbollen Dec 2, 2024

Choose a reason for hiding this comment

lmbollen Dec 2, 2024

Choose a reason for hiding this comment

t-wallet Dec 2, 2024 • edited Loading

Choose a reason for hiding this comment

lmbollen Dec 3, 2024

Choose a reason for hiding this comment

rowanG077 commented Jul 3, 2024 •

edited by t-wallet

Loading

bgamari commented Aug 2, 2024 •

edited

Loading

rowanG077 commented Aug 2, 2024 •

edited

Loading

t-wallet Dec 2, 2024 •

edited

Loading