Massive Warning spikes in Proposal Backend

[TCMeldrum]

We have been seeing massive warning spikes in our logs every couple of days:

The issue seems to be when an Instrument Sci or UO is getting all the proposals it is calling the getUser endpoint on the UOWS for every principal investigator and if the UOWS isn't available at that moment it causes all these warnings. We should try and batch these requests to the UOWS through a batch loader.

[isisbusapps-ci]

There are a few batch methods on UOWS we could look at - Scheduler does something similar GetBasicPeopleDetailsFromUserNumbers might be a good start.

[Bhaswati1148]

• The spike in the log was presumably due to the issues with loading too many proposals which is already tackled against Proposals fail to load as an experiment scientist #872
• The fix was released in Release 2023-08-24 and no spikes have been observed in the logs since then
• Keeping this issue open for a few more days just to confirm the errors don't re-appear, otherwise I hope it can be closed as fixed

[mutambaraf]

We have limited the amount of proposals to 200 at a time because users was not able to keep up loading 15k proposals. The best solution should be to do a batch request to users if possible as suggested.

[Bhaswati1148]

Thanks for your comments , I was not sure if we need to do any further changes. I will look into this.

[Bhaswati1148]

As per my analysis so far, User-office-backend is making N+1 round trips to database/UOWS when fetching principal investigator details, while resloving the Proposal query.

Best approach to solve this would be using graphql/dataloader for batching requests to UOWS.

However, type-graphql integration with graphql/dataloader is not yet live (please see: MichalLytek/type-graphql#51), so we can't use it at this moment.

There is another option type-graphql-dataloader, which I believe we can use? I have done a small poc with this lib and it does the batching, but it has a dependency on apollo-server-express. At this moment I am trying to figure out how it can be used in our set up.

If there are any other potential options which we can consider, I'm open to them.

[Bhaswati1148]

I have tried using type-graphql-dataloader in our project set up, but it is not compatible with apollo server 4.9.3, that we are currently using. Might have to look for other options

[srconway]

@Bhaswati1148 In case it's useful at all I've sent you a joining link for apollostudio

[Bhaswati1148]

@Bhaswati1148 In case it's useful at all I've sent you a joining link for apollostudio

Thank you @srconway , I shall have a look into this.