sg_stream_ctl: add --hex, --inhex=, --json= and --raw options; JSON: …

…make output more consistent so most command responses have a *_paramter_data

git-svn-id: https://svn.bingwo.ca/repos/sg3_utils/trunk@1057 6180dd3e-e324-4e3e-922d-17de1ae2f315

COVERAGE

@@ -23,7 +23,7 @@ EXTENDED COPY(LID1)    sg_xcopy, ddpt
 GET CONFIGURATION   sg_get_config
 GET LBA STATUS      sg_get_lba_status <J>
 GET PHYSICAL ELEMENT STATUS      sg_get_elem_status <J>
-GET STREAM STATUS   sg_stream_ctl
+GET STREAM STATUS   sg_stream_ctl <J>
 INQUIRY             sg_dd, sg_format, sg_inq <J>, sginfo, sg_logs,
                     sg_luns <J>, sg_map('-i'), sg_modes, sg_opcodes <J>,
                     sg_persist, sg_scan, sg_ses, sg_vpd <J>
@@ -189,4 +189,4 @@ THIRD PARTY COPY IN (0x83).
 
 
 Douglas Gilbert
-14 May 2023
+19 October 2023

ChangeLog

@@ -2,7 +2,9 @@ Each utility has its own version number, date of last change and
 some description at the top of its ".c" file. All utilities in the main
 directory have their own "man" pages. There is also a sg3_utils man page.
 
-Changelog for pre-release sg3_utils-1.49 [20231017] [svn: r1056]
+Changelog for pre-release sg3_utils-1.49 [20231020] [svn: r1057]
+  - JSON: make output more consistent so most command
+    responses have a *_paramter_data or similar sub-object
   - apply https://github.com/doug-gilbert/sg3_utils/pull/39
     and its revision [20230807] mainly for Android
   - sg_inq: update version descriptors to T10 table 20230814
@@ -15,6 +17,8 @@ Changelog for pre-release sg3_utils-1.49 [20231017] [svn: r1056]
   - sg_turs: fix missing CDB on several codepaths
   - sg_requests: improve buffer truncation
   - sg_get_elem_status: various improvements
+  - sg_stream_ctl: add --hex, --inhex=, --json= and --raw
+    options; additions in sbc5r05
   - rescan-scsi-bus.sh: fix multipath resize without update 
     https://github.com/doug-gilbert/sg3_utils/pull/43.diff
     and 44.diff

debian/changelog

@@ -2,7 +2,7 @@ sg3-utils (1.49-0.1) unstable; urgency=low
 
   * New upstream version
 
- -- Douglas Gilbert <[email protected]>  Mon, 16 Oct 2023 11:00:00 -0400
+ -- Douglas Gilbert <[email protected]>  Fri, 20 Oct 2023 13:00:00 -0400
 
 sg3-utils (1.48-0.1) unstable; urgency=low
 

doc/sg3_utils_json.8

@@ -23,9 +23,10 @@ data types: string, integer, boolean and null. Its other data types are JSON
 object and JSON array.
 .PP
 This project uses the 'snake_case' convention for JSON object names: all in
-lower case letters or numerals with individual words joined with a single
-underscore (e.g. "starting_lba"). There should be no leading or trailing
-underscore characters. The json-builder library uses the
+lower case (ASCII) letters or numerals with individual words joined with a
+single underscore (e.g. "starting_lba"). There should be no leading or
+trailing underscore characters. Repeated underscores are abridged to a
+single underscore. The json-builder library uses the
 SPDX\-License\-Identifier: BSD\-2\-Clause which is the same license as the
 bulk of the utilities in the sg3_utils package.
 .PP
@@ -278,6 +279,46 @@ JSON if the \fI\-\-json[=JO]\fR option is also used.
 There is typically only one form of JSON output so options like
 \fI\-\-brief\fR and \fI\-\-quiet\fR are ignored in the JSON output. In some
 cases (i.e 'sg_inq \-\-descriptors') the JSON output is expanded.
+.SH NAMING: SCSI, JSON, LINUX PSEUDO FILE SYSTEM
+There is no perfect match for the way commands, fields, objects and files
+across these systems are named. SCSI dates from a period when there were
+still 6 bit character sets with no lower case letters in use! So SCSI
+commands and field names are all upper case with space used as a separator.
+JSON prefers "CamelCase" with the first letter of each word capitalized to
+avoid the need for space as a separator. Many JSON representations (like
+this one) ignore JSON's preference and opt for "snake_case". Why do that?
+Because most Unix based pseudo file systems use lower case. Further, spaces
+within file names creates extra complexity when using basic Unix commands
+and scripting so they are best avoided.
+.PP
+In Linux, sysfs is a very important pseudo file system for communicating
+real time information between hardware drivers and the user space. Utilities
+such as lsblk and lsscsi do data\-mining within sysfs. The vast majority of
+filenames in sysfs use "snake_case" but there are a few exceptions where ':'
+and even embedded spaces appear. Also of note is that Unix user commands,
+system commands and libraries have no emdedded spaces in their names
+and upper cases letters are extremely rare.
+.PP
+Another issue is case sensitivity: "CamelCase" and "camelcase" are confusing
+filenames that refer to different files in most Unix file systems but refer
+to the
+.B same
+file in Windows file systems such as FAT and NTFS. Add in the possibility of
+non\-ASCII Unicocde characters and searching in a JSON object for a name
+match could become a nightmare.
+.PP
+So choosing JSON object names as being made up of a string of characters
+restricted to lower case ASCII alphabetical characters, the digits and
+underscore (only 26+10+1 different characters allowed) is a compromise.
+It is a compromise that favours simpilicity of searching while restricting
+the freedom of object naming. It follows Unix (Linux) file naming
+conventions.
+.PP
+Also worth mentioning is that all these utilities in this package use
+UTF-8 . If data from another system is in UTF-16 or UTF-32 format then
+it can be converted to UTF-8 with the
+.B iconv
+utility.
 .SH ERRORS
 No attempts have been made to translate errors into JSON form, apart from the
 final "exit_status" JSON object where a value of 0 means 'no errors'. Exit
@@ -313,4 +354,5 @@ This software is distributed under the GPL version 2 or the BSD\-2\-Clause
 license. There is NO warranty; not even for MERCHANTABILITY or
 FITNESS FOR A PARTICULAR PURPOSE.
 .SH "SEE ALSO"
-.B sg3_utils(sg3_utils), smartctl(smartmontools)
+.B iconv(1), lsblk(8), lsscsi(lsscsi), sg3_utils(sg3_utils),
+.B smartctl(smartmontools)

doc/sg_get_lba_status.8

@@ -1,14 +1,14 @@
-.TH SG_GET_LBA_STATUS "8" "June 2023" "sg3_utils\-1.48" SG3_UTILS
+.TH SG_GET_LBA_STATUS "8" "October 2023" "sg3_utils\-1.49" SG3_UTILS
 .SH NAME
 sg_get_lba_status \- send SCSI GET LBA STATUS(16 or 32) command
 .SH SYNOPSIS
 .B sg_get_lba_status
 [\fI\-\-16\fR] [\fI\-\-32\fR] [\fI\-\-blockhex\fR] [\fI\-\-brief\fR]
 [\fI\-\-element-id=EI\fR] [\fI\-\-help\fR] [\fI\-\-hex\fR]
-[\fI\-\-inhex=FN\fR] [\fI\-\-json[=JO\fR]] [\fI\-\-lba=LBA\fR]
-[\fI\-\-maxlen=LEN\fR] [\fI\-\-raw\fR] [\fI\-\-readonly\fR]
-[\fI\-\-report\-type=RT\fR] [\fI\-\-scan-len=SL\fR] [\fI\-\-verbose\fR]
-[\fI\-\-version\fR] \fIDEVICE\fR
+[\fI\-\-inhex=FN\fR] [\fI\-\-json[=JO\fR]] [\fI\-\-js\-file=JFN\fR]
+[\fI\-\-lba=LBA\fR] [\fI\-\-maxlen=LEN\fR] [\fI\-\-raw\fR]
+[\fI\-\-readonly\fR] [\fI\-\-report\-type=RT\fR] [\fI\-\-scan-len=SL\fR]
+[\fI\-\-verbose\fR] [\fI\-\-version\fR] \fIDEVICE\fR
 .SH DESCRIPTION
 .\" Add any additional description here
 Send the SCSI GET LBA STATUS(16) or GET LBA STATUS(32) command to the

doc/sg_opcodes.8

@@ -1,4 +1,4 @@
-.TH SG_OPCODES "8" "June 2023" "sg3_utils\-1.48" SG3_UTILS
+.TH SG_OPCODES "8" "October 2023" "sg3_utils\-1.49" SG3_UTILS
 .SH NAME
 sg_opcodes \- report supported SCSI commands or task management functions
 .SH SYNOPSIS
@@ -7,9 +7,9 @@ sg_opcodes \- report supported SCSI commands or task management functions
 [\fI\-\-hex\fR] [\fI\-\-inhex=FN\fR] [\fI\-\-json[=JO]\fR]
 [\fI\-\-js\-file=JFN\fR] [\fI\-\-mask\fR] [\fI\-\-mlu\fR]
 [\fI\-\-no-inquiry\fR] [\fI\-\-opcode=OP[,SA]\fR] [\fI\-\-pdt=DT\fR]
-[\fI\-\-raw\fR] [\fI\-\-rctd\fR] [\fI\-\-repd\fR] [\fI\-\-sa=SA\fR]
-[\fI\-\-tmf\fR] [\fI\-\-unsorted\fR] [\fI\-\-verbose\fR] [\fI\-\-version\fR]
-\fIDEVICE\fR
+[\fI\-\-raw\fR] [\fI\-\-rctd\fR] [\fI\-\-repd\fR] [\fI\-\-rep\-opts=RO\fR]
+[\fI\-\-sa=SA\fR] [\fI\-\-tmf\fR] [\fI\-\-unsorted\fR] [\fI\-\-verbose\fR]
+[\fI\-\-version\fR] \fIDEVICE\fR
 .PP
 .B sg_opcodes
 [\fI\-a\fR] [\fI\-c\fR] [\fI\-e\fR] [\fI\-H\fR] [\fI\-i=FN\fR] [\fI\-j\fR]
@@ -18,24 +18,24 @@ sg_opcodes \- report supported SCSI commands or task management functions
 [\fI\-?\fR] \fIDEVICE\fR
 .SH DESCRIPTION
 .\" Add any additional description here
-This utility sends a SCSI REPORT SUPPORTED OPERATION CODES or a REPORT
-SUPPORTED TASK MANAGEMENT FUNCTIONS command to the \fIDEVICE\fR and then
-outputs the response. The default action is to report supported operation
-codes. In this mode it will either list all supported commands or give
-detailed information on a specific command identified by the
+This utility sends a SCSI REPORT SUPPORTED OPERATION CODES (RSOC) or a REPORT
+SUPPORTED TASK MANAGEMENT FUNCTIONS (RSTMF) command to the \fIDEVICE\fR and
+then outputs the response. The default action is to report supported
+operation codes. In this mode it will either list all supported commands or
+give detailed information on a specific command identified by the
 \fI\-\-opcode=OP\fR option (perhaps with additional information from the
 \fI\-\-sa=SA\fR option).
 .PP
 The name of a SCSI command depends on its peripheral device type (e.g. a
-disk). The REPORT SUPPORTED OPERATION CODES and REPORT SUPPORTED TASK
-MANAGEMENT FUNCTIONS commands are not supported in the MMC command set for
-CD and DVD devices. This utility does an INQUIRY to obtain the peripheral
-device type and prints out the vendor, product and revision strings.
+disk). The RSOC and RSTMF commands are not supported in the MMC command set
+for CD and DVD devices. This utility does an INQUIRY to obtain the
+peripheral device type and prints out the vendor, product and revision
+strings.
 .PP
 A similar facility to query supported operation codes previously was available
 via the CmdDt bit in the SCSI INQUIRY command (see sg_inq(8)). However that
-facility was made obsolete and replaced by the REPORT SUPPORTED OPERATION
-CODES command in SPC\-3 (revision 4) during February 2002.
+facility was made obsolete and replaced by the RSOC command in
+SPC\-3 (revision 4) during February 2002.
 .PP
 This utility supports two command line syntaxes, the preferred one is
 shown first in the synopsis and explained in this section. A later section
@@ -121,14 +121,12 @@ sense key of "illegal request".
 \fB\-M\fR, \fB\-\-mlu\fR
 additionally prints out an indication (0 or 1) whether the command
 effects all logical units in the containing target. MLU (Multiple Logical
-Units) is a bit in the REPORT SUPPORTED OPERATION CODES response
-introduced by proposal 18\-045r1 (and possibly in spc5r20). Without
-the option, the default output format which lists all opcodes, does
-not include a MLU indication.
+Units) is a bit in the RSOC response introduced by proposal 18\-045r1 (and
+possibly in spc5r20). Without the option, the default output format which
+lists all opcodes, does not include a MLU indication.
 .TP
 \fB\-n\fR, \fB\-\-no-inquiry\fR
-Prior to calling a SCSI REPORT SUPPORTED OPERATION CODES or a REPORT
-SUPPORTED TASK MANAGEMENT FUNCTIONS command, a SCSI INQUIRY command
+Prior to calling a SCSI RSOC or a RSTMF command, a SCSI INQUIRY command
 is performed. The reason is to determine the peripheral device type (pdt)
 of the \fIDEVICE\fR as this is helpful in translating operation codes
 to the command names. By default this utility prints a summary of INQUIRY
@@ -148,7 +146,10 @@ and \fISA\fR are decimal unless prefixed by "0x" or they have a
 trailing "h". \fIOP\fR should be in the range 0 to 255 (0xff) inclusive.
 \fISA\fR should be in the range 0 to 65535 (0xffff) inclusive. When this
 option is not given then all available SCSI commands supported by the
-\fIDEVICE\fR are listed.
+.br
+The \fI\-\-rep\-opts=RO\fR option, if given, overrides any decisions implied
+in the previous paragraph about the setting of REPORTING OPTIONS field in
+the RSOC command.
 .TP
 \fB\-p\fR, \fB\-\-pdt\fR=\fIDT\fR
 where \fIDT\fR is the peripheral device type. This is used together with
@@ -180,6 +181,12 @@ set read extended parameter data (REPD) bit in the report task management
 functions cdb. 16 bytes rather than the default 4 bytes expected in the
 response. This was added in SPC\-4 (revision 26).
 .TP
+\fB\-Q\fR, \fB\-\-rep\-opts\fR=\fIRO\fR
+\fIRO\fR is the reporting options field value that is placed in RSOC
+command (cdb). If this option is not given and the \fI\-\-opcode=OP[,SA]\fR
+option is also not given then \fIRO\fR default to 0. All supported commands
+are listed when \fIRO\fR is 0 .
+.TP
 \fB\-s\fR, \fB\-\-sa\fR=\fISA\fR
 the \fIDEVICE\fR will be queried for a command with the given service
 action (i.e. the \fISA\fR value). Used in conjunction with the

doc/sg_stream_ctl.8

@@ -1,19 +1,27 @@
-.TH SG_STREAM_CTL "8" "March 2018" "sg3_utils\-1.43" SG3_UTILS
+.TH SG_STREAM_CTL "8" "October 2023" "sg3_utils\-1.49" SG3_UTILS
 .SH NAME
-sg_stream_ctl \- send SCSI STREAM CONTROL or GET STREAM STATUS command
+sg_stream_ctl \- send SCSI GET STREAM STATUS or STREAM CONTROL command
 .SH SYNOPSIS
 .B sg_stream_ctl
 [\fI\-\-brief\fR] [\fI\-\-close\fR] [\fI\-\-ctl=CTL\fR] [\fI\-\-get\fR]
-[\fI\-\-help\fR] [\fI\-\-id=SID\fR] [\fI\-\-maxlen=LEN\fR] [\fI\-\-open\fR]
-[\fI\-\-readonly\fR] [\fI\-\-verbose\fR] [\fI\-\-version\fR] \fIDEVICE\fR
+[\fI\-\-help\fR] [\fI\-\-hex\fR] [\fI\-\-id=SID\fR] [\fI\-\-inhex=FN\fR]
+[\fI\-\-json[=JO\fR]] [\fI\-\-js\-file=JFN\fR] [\fI\-\-maxlen=LEN\fR]
+[\fI\-\-open\fR] [\fI\-\-raw\fR] [\fI\-\-readonly\fR] [\fI\-\-verbose\fR]
+[\fI\-\-version\fR] \fIDEVICE\fR
 .SH DESCRIPTION
 .\" Add any additional description here
-Sends a SCSI STREAM CONTROL or GET STREAM STATUS command to the \fIDEVICE\fR.
+Sends a SCSI GET STREAM STATUS or STREAM CONTROL command to the \fIDEVICE\fR.
 These commands, together with WRITE STREAM(16 and 32) and several fields in
 the Block Limits Extension VPD page [0xb7] support the streams concept.
-The stream commands were added in SBC\-4 draft 8 (September 2015).
+There is also the IO advice hints grouping mode page [0xa,0x4] . Amongst
+other things that mode page indicates whether a stream identifier do double
+duty as a group number found in most SCSI media access commands (e.g. READ,
+WRITE and VERIFY).
 .PP
-Both STREAM CONTROL and GET STREAM STATUS commands expect data from the
+The stream commands were added in SBC\-4 draft 8 (September 2015). The stream
+commands were extended in SBC\-5 draft 5 (October 2023).
+.PP
+Both the GET STREAM STATUS and STREAM CONTROL commands expect data from the
 \fIDEVICE\fR (referred to as 'data\-in'). In the case of STREAM CONTROL
 only the 'open' (STR_CTL<\-\-0x1) actually needs the data\-in as it contains
 the "Assigned stream id" if the open was successful. The assigned stream
@@ -54,12 +62,53 @@ are given.
 \fB\-h\fR, \fB\-\-help\fR
 output the usage message then exit.
 .TP
+\fB\-H\fR, \fB\-\-hex\fR
+.TP
+\fB\-H\fR, \fB\-\-hex\fR
+output response to this command in ASCII hex. Each line of 16 bytes is
+preceded by an address or index, starting at 0 and the address is also in
+hex. If given twice then an ASCII rendering of each byte is appended to the
+line output. If given three or more times then only the ASCII hex of each
+byte is output, 16 bytes per line (i.e. so no leading address nor trailing
+ASCII rendering). This latter form is suitable for placing in a file and
+being used with the \fI\-\-inhex=FN\fR option in a later invocation.
+.TP
 \fB\-i\fR, \fB\-\-id\fR=\fISID\fR
 \fISID\fR is a stream id, a value between 1 and 65535. It is used by STREAM
 CONTROL (close) to identify the stream to close. It is used by the GET
 STREAM STATUS command as the starting stream id (from and including); so
 stream ids that are less than \fISID\fR will not appear in the response.
 .TP
+\fB\-i\fR, \fB\-\-inhex\fR=\fIFN\fR
+where \fIFN\fR is a file name whose contents are assumed to be ASCII
+hexadecimal. If \fIDEVICE\fR is also given then \fIDEVICE\fR is ignored,
+a warning is issued and the utility continues, decoding the file named
+\fIFN\fR. See the "HEX, BINARY AND JSON FORMATS" section in the sg3_utils
+manpage for more information. If the \fI\-\-raw\fR option is also given then
+the contents of \fIFN\fR are treated as binary.
+.br
+\fIFN\fR may either contain data representing the response to a SCSI GET
+STREAM STATUS or a STREAM CONTROL command. This utility defaults to
+expecting the former. When \fIFN\fR contains data from a STREAM CONTROL
+command response then an additional option (other than this option) needs
+to be given: either \fI\-\-close\fR, \fI\-\-ctl=CTL\fR, or \fI\-\-open\fR.
+.TP
+\fB\-j\fR[=\fIJO\fR], \fB\-\-json\fR[=\fIJO\fR]
+output is in JSON format instead of plain text form. Note that arguments
+to the short and long form are themselves optional and if present start
+with "=" and no whitespace is permitted around that "=".
+.br
+See sg3_utils_json manpage or use '?' for \fIJO\fR to get a summary.
+.TP
+\fB\-J\fR, \fB\-\-js\-file\fR=\fIJFN\fR
+output is in JSON format and it is sent to a file named \fIJFN\fR. If that
+file exists then it is truncated. By default, the JSON output is sent to
+stdout.
+.br
+When this option is given, the \fI\-\-json[=JO]\fR option is implied and
+need not be given. The \fI\-\-json[=JO]\fR option may still be needed to
+set the \fIJO\fR parameter to non-default values.
+.TP
 \fB\-m\fR, \fB\-\-maxlen\fR=\fILEN\fR
 \fILEN\fR is the maximum length the response can be. It becomes the
 ALLOCATION LENGTH field in both commands. The default (in the absence of
@@ -83,6 +132,16 @@ write operation on the media (or its metadata) so they take a risk averse
 stance and require read\-write type permissions on the \fIDEVICE\fR open
 irrespective of what is performed by the pass\-through.
 .TP
+\fB\-R\fR, \fB\-\-raw\fR
+output response in binary (to stdout) unless the \fI\-\-inhex=FN\fR option
+is also given. In that case the input file name (\fIFN\fR) is decoded as
+binary (and the output is _not_ in binary).
+.br
+To relax that final restriction this option can be given multiple times. If
+this option is given once or three times, \fIFN\fR is assumed to be binary.
+If this option is given two or three times, the (undecoded) output is in
+binary. These actions only apply when the \fI\-\-inhex=FN\fR option ia given.
+.TP
 \fB\-v\fR, \fB\-\-verbose\fR
 increase the level of verbosity, (i.e. debug output).
 .TP
@@ -108,7 +167,7 @@ Written by Douglas Gilbert.
 .SH "REPORTING BUGS"
 Report bugs to <dgilbert at interlog dot com>.
 .SH COPYRIGHT
-Copyright \(co 2018 Douglas Gilbert
+Copyright \(co 2018\-2023 Douglas Gilbert
 .br
 This software is distributed under a BSD\-2\-Clause license. There is NO
 warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

inhex/get_elem_status.hex

@@ -1,5 +1,5 @@
 #
-# The is a real response to the SCSI GET PHYSICAL ELEMENT STATUS command.
+# This is a real response to the SCSI GET PHYSICAL ELEMENT STATUS command.
 # The storage elements are associated with heads on this hard disk and
 # one of them (element_id: 10) is "out of spec". The same output was
 # obtained with --report-type=1 (report only storage elements) as this

inhex/stream_ctl_get.hex

@@ -0,0 +1,21 @@
+#
+# Get stream status. Note that the sg_stream_ctl utility issues either
+# a SCSI GET STREAM STATUS or STREAM CONTROL command. This is an example
+# of the SCSI GET STREAM STATUS command.
+#
+# This can be tested with:
+#    sg_stream_ctl --inhex=<this_file>
+#
+#
+# For a STREAM CONTROL command example see the stream_ctl_open.hex file.
+
+0 0 0 30
+0 0 0 6
+
+80 0 0 1  0 0 0 0
+80 0 0 2  1 0 0 0
+80 0 0 3  2 0 0 0
+80 0 0 4  3 0 0 0
+
+0 0 0 5  3e 0 0 0
+0 0 0 6  3f 0 0 0

inhex/stream_ctl_open.hex

@@ -0,0 +1,17 @@
+#
+# The SCSI STREAM CONTROL command yields parameter data which in the
+# case of STR_CTL=1 (open) contains the newly opened Stream Identifier
+# ASSIGNED_STR_ID. That field is 16 bits long and should not be 0.
+# In the case of STR_CTL=2 (close) it seems 8 bytes of parameter data
+# is also returned with the value of ASSIGNED_STR_ID is "reserved".
+# In practice that should be zero.
+#
+# This can be tested with:
+#    sg_stream_ctl --open --inhex=<this_file>
+# and the ASSIGNED_STR_ID field should be 3 .
+# Note the the '--open' option is needed to flag that the response data
+# is from a SCSI STREAM CONTROL command.
+#
+# For a GET STREAM STATUS command example see the stream_ctl_get.hex file .
+
+7 0 0 0  0 3 0 0