Translating data from a binary file produced by the Data Storage application

[mc-bane]

Context

This is mostly a part 3 to my questions in #278 and #282.

I would like to write data from instruments to a readable .csv file. I have developed an application with the assistance/framework of the sample app that I have modified to interface with my sensors and publishes said data to the telemetry output application which can also be viewed with the ground station GUI.

Now I am trying to write my data into a file in a readable format. I was interested in using the Data Storage application to accomplish this, and I am able to populate what I think is my data packed into a binary file but I am a bit confused on translating it into the readable csv file I would like.

In #282 I was confused as to why I couldn't read the file header from my binary output file but I needed to have code to translate the file out of binary into readable ASCII characters.

Question

For printing out the rest of the data past the header in a file, would we need another struct to read and print from representing the rest of that message? If so what type?

It's not clear to me what abstract data type is mapped to each message ID I want in my output file. For example, in my ds_filter_tbl.c file I have 3 MessageIDs I would like to send to my output file called SAMPLE_APP_SEND_HK_MID, SAMPLE_APP_HK_TLM_MID, and SAMPLE_APP_CMD_MID. Each messageID, I am assuming, corresponds to some data structure that is packed into the binary file that I need to translate.

I am also assuming that one of those messageIDs specifically corresponds with the values I put into the SAMPLE_APP_Data_t global data structure in my sample app or the telemetry packet but I'm not quite sure which.

//from ds_filter_tbl.c
DS_FilterTable_t DS_FilterTable =
{
   .Descriptor =  "Sample filter table data",
   .Packet     = 
  {
   /* Packet Index 015 */
    {
         .MessageID =SAMPLE_APP_SEND_HK_MID,
         .Filter    =
      {
        /* File table index, filter type, N, X, O */
        { 6, DS_BY_COUNT, 1, 1, 0 },
        { DS_UNUSED, DS_UNUSED, DS_UNUSED, DS_UNUSED, DS_UNUSED },
        { DS_UNUSED, DS_UNUSED, DS_UNUSED, DS_UNUSED, DS_UNUSED },
        { DS_UNUSED, DS_UNUSED, DS_UNUSED, DS_UNUSED, DS_UNUSED }
      }
    },
    /* Packet Index 016 */
    {
         .MessageID = SAMPLE_APP_HK_TLM_MID,
         .Filter    =
      {
        /* File table index, filter type, N, X, O */
        { 6, DS_BY_COUNT, 1, 1, 0 },
        { DS_UNUSED, DS_UNUSED, DS_UNUSED, DS_UNUSED, DS_UNUSED },
        { DS_UNUSED, DS_UNUSED, DS_UNUSED, DS_UNUSED, DS_UNUSED },
        { DS_UNUSED, DS_UNUSED, DS_UNUSED, DS_UNUSED, DS_UNUSED }
      }
    },
    /* Packet Index 017 */
    {
         .MessageID = SAMPLE_APP_CMD_MID,
         .Filter    =
      {
        /* File table index, filter type, N, X, O */
        { 6, DS_BY_COUNT, 1, 1, 0 },
        { DS_UNUSED, DS_UNUSED, DS_UNUSED, DS_UNUSED, DS_UNUSED },
        { DS_UNUSED, DS_UNUSED, DS_UNUSED, DS_UNUSED, DS_UNUSED },
        { DS_UNUSED, DS_UNUSED, DS_UNUSED, DS_UNUSED, DS_UNUSED }
      }
    },

Thanks again for answering my previous questions.

[dmccomas]

Yes you will need to know all of the binary structures of the data being stored in the DS file. Here's a picture showing the structure of a DS file that is storing event messages. After the standard cFS file header DS has its own file header. After this header DS stores packets as defined in its tables. When processing the packets in the file I would recommend using the information in the CCSDS primary header of each packet as opposed to making any assumptions about packet order. Use the message ID to identify which packet structure needs to be used to interpret the binary data and use the packet length to find the start of the next packet in the file.

[mc-bane]

Thanks for the quick response! Could you elaborate on what you mean by using the primary header of the packet that I want translated? Do you mean, when I am translating the binary, my logic would go:

1. read the cfe fs header
2. read the ds file header
3. (in a loop after 1&2) read the packet whose message in my case would be something like

read(file, &SAMPLE_APP_Data, sizeof(SAMPLE_APP_Data_t))); 
printf("%u, "), SAMPLE_APP_Data.HkTlm.Payload.PayloadValue1);

[skliper]

3. Read in the Telemetry packet header to determine which packet it is (can also confirm expected packet length matches)

CFE_MSG_TelemetryHeader_t TlmHeader;
read(file, &TlmHeader, sizeof(TlmHeader));
... whatever logic you want here to examine header and process appropriately

4. Read in the rest of the Telemetry packet (aka payload) for that packet, for example if it's the SAMPLE App HK packet:

SAMPLE_APP_HkTlm_Payload_t hk;
read(file, &hk, sizeof(hk));
print("%u, ", hk.CommandErrorCounter);

[dmccomas]

Thanks skliper, I got busy. I'll still follow up to complete the EVS message example. Steps 1 and 2 are good. In the case of EVS (cFE 6.7.1) the packet structure is

typedef struct {
uint8 TlmHeader[CFE_SB_TLM_HDR_SIZE];
CFE_EVS_LongEventTlm_Payload_t Payload;
} CFE_EVS_LongEventTlm_t;

So if the EVS packet is the only packet being logged by DS you could do something like this for step 3 to print the event message text:

CFE_EVS_LongEventTlm_t EvsTlm;
Loop through file
OS_read(FileHandle, &EvsTlm, sizeof(CFE_EVS_LongEventTlm_t);
printf("%s\n", EvsTlm.Payload.Message);

If your packet is not the only packet being logged then you should process the packet headers as shown above

[mc-bane]

thanks all!