Struggling to get additional options to work

[hardysabs2]

  filterType: "to",
  filterCRS: "<CRSCode>",

These additional options do not seem to be working. Are they being passed down to the underlying interface?

Additionally, numRows over ten seems to be mostly ignored. Is the common strategy to retain the std of the latest service and use that for another request?

Many thanks

[magnatronus]

I would need to know which API call you are trying to access and exactly how you are trying to do this - could you provide example code as to what you are trying to do?

[hardysabs2]

"ldbs-json": "^1.2.1",

// snippet crs set to LIV, date time now - returns data just does not filter and only 10 rows    

const LiveDepartureBoardService = require("ldbs-json");

const options = {
  crs: context.params,
  filterType: "to",
  filterCRS: "CTR",
  // also tried filterList: ["CTR"]
  time: departDate,
  timeWindow: 120,
  numRows: 20
};

const api = new LiveDepartureBoardService(
  process.env.NAT_EXPRESS_TRAINS_TOKEN,
  true
);


let trainResData = await api.call("GetDepBoardWithDetails", options);

[magnatronus]

Great will take a look and get back to you as soon as I can

[magnatronus]

The reason you only get 10 back is that is the max according to the API docs. I have tried and and any number between 1 and 10 work fine.

the filter issue I am still looking into

[hardysabs2]

Appreciate that @magnatronus . I guess one runs another search from a later time depending the last time of the returned service. Thanks for looking at the filtering, it would be good if I can get that working since filtering after the event client side adds to complexity if returned rows are limited.

[magnatronus]

So it looks like it all comes down to the exact case of the options and making sure all options are included. I have tried and verified this:

// Set up the options for the call
const options = {
  numRows: 5,
  crs:"KGX",
  filterCrs: "CBG",
  filterType: "to",
  timeOffset: 0,
  timeWindow: 120
};

So where you have filterCRS it should be filterCrs. Also there is not 'time' param for this call but could probably use the timeOffset instead. Just take a look a the API for the call here https://lite.realtime.nationalrail.co.uk/OpenLDBWS/

Hope this helps.

[hardysabs2]

Thank you so much @magnatronus , really appreciate this.

Just to clarify then, it looks like "time" option does not exist for any of the call types. This was pre-existing code, I have no idea where that comes from.

From what I can see it is simply not possible to retrieve board details for any date (past or future) but only for a time offset from the current time of -120 to +120 and then in a window of up to 120. So the latest/earliest 'potential' board detail would be within 4 hours of now. Unless you have spotted something I have not.

Many thanks,

Andrew

[magnatronus]

Nope, to do that you would need a real time system that processed the Darwin Push Port Real Time feed (that would be a whole new and quite complex project).

This is why this data is just a snapshot as the Darwin Push Port is very data intensive and would require a big data store to keep track of all the millions of records per day (I have also worked on a system using this data as well).

[hardysabs2]

const options = {
  crs: context.params, // LIV
  // time: departDate, // not available
  timeWindow: 120,
  filterCrs: "EUS", // new in this code
  filterType: "to", // new in this code
  timeOffset: 0, // new in this code
  numRows: 5 // new in this code
};

const api = new LiveDepartureBoardService(
  process.env.NAT_EXPRESS_TRAINS_TOKEN,
  true
);

let trainResData = await api.call("GetDepBoardWithDetails", options);

if (!trainResData) {

Just trying this now. This is currently returning false (not empty service array), but the public site returns data. I can only assume this is a transitory unrelated issue. Are you aware of a common strategy to identify these cases as distinct from your search criteria did not lead to any results.

[magnatronus]

I just ran this and get back some results, all for Avanti West Coast trains, so not sure why you get no results - if the call does not fail then it should return OK but with no data if no services are found.

edit: - but only returned 2 results

[magnatronus]

Also it is possible that within the time window there are no services found when the query is done, I have a published mobile app that uses this backend code and it does occasionally show no results.

[hardysabs2]

I'm really sorry @magnatronus I'm getting really confused now. I have the latest version of this package.

"ldbs-json": "^1.2.1",

I have uninstalled and reinstalled.

time option is not in the docs, but if I do not include the 'time' option then I get this...

Error: API Error 500: Internal Server Error
at LiveDepartureBoardService.call (C:\Users\ahardy\dev\...\node_modules\ldbs-json\index.js:56:15)
at processTicksAndRejections (internal/process/task_queues.js:95:5)

...and the api.call returns false.

If I add the 'time' option back in then I get the results I expect, but if I add these additional options...

 numRows: 5,
 filterCrs: "EUS",
 filterType: "to",
 timeOffset: 0,

...they appear to be of no effect.

Is it possible I am somehow using the wring version of the package or some other dependency is interfering? Or is it possible that the token in use could have such an impact on the interface and it's use?

So sorry to message again, I'm just a bit stumped.

Andrew

[magnatronus]

Hi,

this is the example code I am using (modified from the examples dir)

/**
 * Simple example code for running the user version of the DepartureBoard Info
 * NB: You will need a valid Token to access the normal user version of the api
 */
const settings = require('../settings.json');
const LiveDepartureBoardService = require('../index');


const options = {
  crs: "LIV", // LIV
  timeWindow: 120,
  filterCrs: "EUS", // new in this code
  filterType: "to", // new in this code
  timeOffset: 0, // new in this code
  numRows: 5 // new in this code
};

const api = new LiveDepartureBoardService(settings.tokens.user, false);
api.call("GetDepBoardWithDetails", options)
  .then(board => {
    console.log(board);
    console.log(board.GetStationBoardResult.trainServices.service);    
    console.log(board.GetStationBoardResult.trainServices.service.length);
  })
  .catch(error => console.error(error));

The TOKEN will make a difference - I am using a USER token not a STAFF token. I will check using a staff token if the API call is available there.

[magnatronus]

So I can also run the staff version, though it is slightly different and you are correct you need the time attribute, although the API doc does not say this. My code below:

/**
 * Simple example code for running the user version of the DepartureBoard Info
 * NB: You will need a valid Token to access the normal user version of the api
 */
const settings = require('../settings.json');
const LiveDepartureBoardService = require('../index');


const options = {
  crs: "LIV", // LIV
  timeWindow: 120,
  filterCrs: "EUS", // new in this code
  filterType: "to", // new in this code
  time: "2022-04-28T10:15:00",
  timeOffset: 0, // new in this code
  numRows: 1 // new in this code
};

const api = new LiveDepartureBoardService(settings.tokens.staff, true);
api.call("GetDepBoardWithDetails", options)
  .then(board => {
    console.log(board);
    console.log(board.GetBoardWithDetailsResult.trainServices.service);    
  })
  .catch(error => console.error(error));

[hardysabs2]

I understand the token in use was a staff token. I have tried another token with the email that supplied it describes as a "developer" (neither user not staff) token. This has the same issue. I have then visited here http://realtime.nationalrail.co.uk/OpenLDBWSRegistration/Registration
to try to make sure I get the right type of token, but there appears to be no explicit selection between staff and user token but rather free text description, usage type personal/company and user type other/toc/public body.

Do you happen to know how I can be sure of obtaining a user token? I'm a little confused as to how the token may impact the use of option time when both https://lite.realtime.nationalrail.co.uk/OpenLDBWS/ and https://lite.realtime.nationalrail.co.uk/OpenLDBSVWS/ do not include it.

I shall try the example

[magnatronus]

I agree both the user and the staff versions do not have the time, but it seems that the staff version does require it.

The difference between the staff and user versions is the data returned, the data available and the available API calls. The staff version has more data in and is broader. Also the data returned in the staff version is sometimes in a different format.

When calling the staff version the end point is also different.

You have to apply for a USER token here - http://realtime.nationalrail.co.uk/OpenLDBWSRegistration

But your staff token should work as long as the params are correct , I would use that as the returned data is a lot more comprehensive (and a bit more complex to decode). The USER API is simpler and some of the data is pre-processed to be in a more 'friendly' format.

[hardysabs2]

I have tried to follow your example as closely as I can in my context:

/**
 * Simple example code for running the user version of the DepartureBoard Info
 * NB: You will need a valid Token to access the normal user version of the api
 */
// const settings = require("../settings.json");
const LiveDepartureBoardService = require("../node_modules/ldbs-json/index");

const options = {
  crs: "LIV", // LIV
  timeWindow: 120,
  // filterCrs: "EUS", // new in this code
  // filterType: "to", // new in this code
  time: "2022-04-28T10:38:00"
  // timeOffset: 0, // new in this code
  // numRows: 5 // new in this code
};

console.log("~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~");
console.log("~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~");

console.log(
  `Calling ldbs with token: ${
    process.env.NAT_EXPRESS_TRAINS_TOKEN
  } options: ${JSON.stringify(options, null, 4)}`
);

// const api = new LiveDepartureBoardService(settings.tokens.staff, true);
const api = new LiveDepartureBoardService(
  process.env.NAT_EXPRESS_TRAINS_TOKEN,
  true
);
api
  .call("GetDepBoardWithDetails", options)
  .then((board) => {
    console.log(board);

    console.log("~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~");

    console.log(
      JSON.stringify(
        board.GetBoardWithDetailsResult.trainServices.service,
        null,
        4
      )
    );

    // saveAsJSON({
    //   services: board.GetBoardWithDetailsResult.trainServices.service
    // });

    console.log("~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~");
    console.log("~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~");
  })
  .catch((error) => {
    console.error(error);

    console.log("~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~");
    console.log("~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~");
  });

I still have the issue that the filter option appears to make no difference. I do not get destination EUS alone with or without. THs makes me wonder if the option is being passed down. I would not have expected the token to be relevant here, given the idea of the staff token is wider data.

My issue is that filtering client side is more awkward because of the limited number in the response. One would be faced with making repeat calls for later times (assuming time option is actually of any effect) and concatenating to the service array to ensure one can include the item you want to filter on.

I have created a "USER" token following the link for it above, but 2 hours after activating it still returning 401. Once it works I shall see if filtering for some reason only works with USER token and "not" STAFF token for some odd reason.

[magnatronus]

OK I will also run complete checks using both tokens - but probably not until later this afternoon as I am not around until then.

[hardysabs2]

Thanks appreciate that. I want to test the same with a user token to see that filterCRs is also ineffective with user tokens, but I have so far created two different user tokens (following the link you recommended above) with different email addresses and user names and on activation , given the token string and told it will be operational within ten minutes. However after over 12 hours using either token I get 401. Not sure what to do since there is no support available.

[hardysabs2]

OK - Apologies, the 401 was a coding error on my part.

For me - now With a user token filterCrs is effective, but with staff token filterCrs is NOT effective. I guess it just remains to understand whether this is related to ldbs-json or is an idiosyncrasy of the Darwin Soap interface.

I also noticed that results with a user token are not just a clean subset of results with a staff token for example

const api = new LiveDepartureBoardService(
  process.env.NAT_EXPRESS_TRAINS_TOKEN, // staff token
  true
);
api
  .call("GetDepBoardWithDetails", options)

brings back board.GetBoardWithDetailsResult

const api = new LiveDepartureBoardService(
  process.env.NAT_EXPRESS_TRAINS_TOKEN, // user token
  false
);
api
  .call("GetDepBoardWithDetails", options) 

brings back board.GetStationBoardResult

Is this Darwin soap response behavior or ldbs-json ?

[magnatronus]

Staff has a more complex/ complete result set than the user API.

This is the the SOAP response, as they are both different.

[hardysabs2]

Any ideas why with a user token filterCrs is effective, but with staff token filterCrs is NOT effective. You think this is a Soap Darwin thing?

[magnatronus]

OK, so I think this is down to poor documentation of the APIs , this is what I have found , the below example is working for me:

/**
 * Simple example code for running the user version of the DepartureBoard Info
 * NB: You will need a valid Token to access the normal user version of the api
 */
const settings = require('../settings.json');
const LiveDepartureBoardService = require('../index');

const options = {
  crs: "KGX", 
  timeWindow: 120,
  filtercrs: "CBG", 
  filterType: "to", 
  time: "2022-04-30T07:40:00",
  timeOffset: 0, 
  numRows: 6 
};

const api = new LiveDepartureBoardService(settings.tokens.staff, true);
api.call("GetDepBoardWithDetails", options)
  .then(board => {
    
    console.log(board);
    console.log(board.GetBoardWithDetailsResult.trainServices.service);  
    console.log(board.GetBoardWithDetailsResult.trainServices.service.length); 
         
  })
  .catch(error => console.error(error));

Issues:

• Staff API is not well documented (or just wrong)
• For the Staff API filterCrs needs to be filtercrs (case)

If the crs and filtercrs values are NOT a direct route( or not a direct route on the day i.e. at weekends the calling patterns sometimes differ) then the calls will error (all these APIs only work for direct routes)

There also seems to be another doc here that may help - https://wiki.openraildata.com/images/4/4f/LDBSVWS_-_Guidance_Note_v1.doc.gz , last updated in 2014 but seems to have some useful info (this is where I found the case diff for the filtercrs attribute)

[hardysabs2]

Thank you so much @magnatronus , appreciate that, I shall take a look.