[WIP] DF/60/69/10 -- EOM/EOP/EOB markers. #242
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
EOProcess: -E|--eop EOMessage: -e|--eom *No default EOM/EOP values specified *EOM/EOP variables ($EOM, $EOP) are not applied now in this .sh script
About $EOP output: While we have only 1 message (so-called JSON-array) EOM and EOP should be just sent after that message. It's possible to write both of them into $tmp-file, (or) but it would be better to leave only EOM in $tmp and to output EOP explicitly (just to be prepared for a case when we'll divide somehow "data channel" and "command channel")
If -p or -f specified -- it's "file mode" Otherwise -- "stream mode". $EOM and $EOP variables are internal and keep data from command line. $EOMarkers and $EOProcess variables are for output.
The previous solution doesn't allow to pass '' as the EOM/EOP marker by user. $EOM_set (Y/N) -- a variable which keeps the info if EOM is set by user $EOP_set (Y/N) -- a variable which keeps the info if EOP is set by user
Let's assume using command line options in case statements the following way just because to write the same way in each script: -short-option|--long-option)
In case of handling escape sequences for custom markers
and because we should handle somehow whitespace symbols
like '\0', '\n', etc., we should choose one proper approach
and stick to it.
The most suitable way is to put escape sequences to variables and then
interpret them when marker is used.
Like this:
{{{
EOProcess='\x06'
<...>
while read -d $(echo -ne "$DELIMITER") line ; do
<...>
echo -ne "$EOProcess"
done
}}}
This solution also works with whitespace symbols.
|
@anastasiakaida, although it isn`t a brand new PR and I know that we have discussed this and that questions about this task... yet I can not remember everything.
Thank you! |
|
@mgolosova |
mgolosova
reviewed
Apr 10, 2019
anastasiakaida
force-pushed
the
df-connectors-markers
branch
from
9943205 to
f503c27
Compare
EOP -- End-of-process -E|--eop Default: "" -- for file mode '\0' -- for stream mode $EOP_set (Y/N) -- a variable which keeps the info if EOP is set by user
-b|--batch (e)enabled|(d)isabled
Default: (d)isabled
This option means batch-mode support.
If there is no arguments (e|d) with the option, batch-mode will
be switched on automatically.
-B|--eob
Dafault: "NOT_SPECIFIED" -- just a "foolproof" decision for cases
where EOB marker is not passed via the command line.
EOB -- End-of-batch (old: $DELIMITER)
BATCHMODE -- an option to support batch-mode processing.
Combinations of markers lead to the following EOB markers ('\x11'
means here any custom marker):
-b || X | X | X | X || 'e' | 'e' | 'e' | 'e' |
------- || ---- | ---- | ------ | ------ || ------ | ---- | ------ | ------ |
-B || X | '' | '\x17' | '\x11' || X | '' | '\x17' | '\x11' |
======= || ==== | ==== | ====== | ====== || ====== | ==== | ====== | ====== |
EOBatch || '\n' | '\n' | '\x17' | '\x11' || '\x17' | '\n' | '\x17' | '\x11' |
'\x17' - End Of Transmission Block (ASCII)
Note:
$EOB_set (Y/N) -- a variable which keeps the info if EOB is set by user
Unfortunately, the previous solution for 'read' command doesn't work with some ASCII control characters as EOB like '\n' or '\0'. "$(echo -ne $EOBatch)" is empty, so a delimiter can't be use. For that case it's better to use 'eval'. The 'eval' statement tells the shell to take eval's arguments and run them through the command-line processing steps all over again.
EOP -- End-of-process
-E|--eop
Default: '\n' ("") -- for file mode, '\0' -- for stream mode
$EOP_set (Y/N) -- a variable which keeps the info if EOP is set by user
-b|--batch (e)enabled|(d)isabled
Default: (d)isabled
This option means batch-mode support.
If there is no arguments (e|d) with the option, batch-mode will
be switched on automatically.
-B|--eob
Dafault: "NOT_SPECIFIED" -- just a "foolproof" decision for cases
where EOB marker is not passed via the command line.
EOB -- End-of-batch (old: $DELIMITER)
BATCHMODE -- an option to support batch-mode processing.
Combinations of markers lead to the following EOB markers ('\x11'
means here any custom marker):
-b || X | X | X | X || 'e' | 'e' | 'e' | 'e' |
------- || ---- | ---- | ------ | ------ || ------ | ---- | ------ | ------ |
-B || X | '' | '\x17' | '\x11' || X | '' | '\x17' | '\x11' |
======= || ==== | ==== | ====== | ====== || ====== | ==== | ====== | ====== |
EOBatch || '\n' | '\n' | '\x17' | '\x11' || '\x17' | '\n' | '\x17' | '\x11' |
'\x17' - End Of Transmission Block (ASCII)
Note:
1. $EOB_set (Y/N) -- a variable which keeps the info if EOB is set by user
2. Unfortunately, the previous solution for EOP 'read' command doesn't work
with some ASCII control characters as EOB like '\n' or '\0'.
"$(echo -ne $EOBatch)" is empty, so a delimiter can't be use.
For that case it's better to use 'eval'. As it's used for 069.
The 'eval' statement tells the shell to take eval's arguments
and run them through the command-line processing steps all over again.
-e|--eom Dafault: '\n' We suppose that EOM in a stream mode is '\n' by default, bacause this is one and only acceptable EOM signal. To prevent errors during upload process it's better to change custom EOM delimiter to the standard one in line (if it was custom).
-e|--eom Dafault: '\n'
anastasiakaida
force-pushed
the
df-connectors-markers
branch
from
1fee438 to
9ed37e5
Compare
anastasiakaida
force-pushed
the
df-connectors-markers
branch
from
9ed37e5 to
f4f3769
Compare
|
@anastasiakaida, |
anastasiakaida
changed the title
|
@mgolosova |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Because of a need for communication between "worker" and "supervisor" processes the following markers (signals) were implemented for Sink/Source connectors:
Stage 010 (Source-connector):
*End-of-message
*End-of-process
Note: For some reason, there is no need to implement EOB.
Stage 069 (Sink-connector):
*End-of-message
*End-of-batch
*End-of-process
Stage 060 (Sink-connector):
*End-of-message
*End-of-batch
*End-of-process
NOTE: There still some logical collisions in a reading of EOB markers (now pre-sink stage is defined to solve this issue -- see #278 ), so it should be fixed here also according to the new convention.