Manual Data uploads not working

[Dacrane91]

Good afternoon,

Thank you for providing this solution. We are currently working to get it implemented at our site, but have run into a couple of issues that may or may not be related.

I have reviewed a similar issue here.

But this appears to have been a different cause, however, we are getting the same errors in the httpd logs and seeing similar problems.

Whenever we try to do a manual data upload we get to step 2 and after selecting the report type and file, it goes to Check before data import and just keeps spinning. Eventually I get an error that preflight failed with error 500. If I try to import this same file in Celus One it works immediately.

Error loading preflight data: Error: Request failed with status code 500

I have verified that celery is running.

Attached is the file that I am trying to import:

Manual_Import_Test.csv

The other issue that we are seeing is, the Interest section of the Dashboard is not populating, even though we do have data imported from SUSHI sources.

I also noticed that Django has several report types listed, but the manual data uploads page only allows us to choose between 2, "Counter 5 - Title report" or "Counter 4 - Journal Report 1"

[Dacrane91]

After doing some additional troubleshooting I have been able to confirm that the application is able to communicate with the database and is not giving a credential error. These are my troubleshooting notes so far if they are of any use.
10-26-21-Celus-Notes.txt

[Dacrane91]

We have also found that the files are being uploaded to the server during this process. It appears to be saving the files even though the import is not completing in the UI.

[root@celus 1]# pwd
/var/www/celus/media/custom/1
[root@celus 1]# ll
total 460
-rw-r--r-- 1 apache apache   1304 Oct 26 11:33 JR1-ScienceDirect_20211026-163318.860475.csv
-rw-r--r-- 1 apache apache   4393 Oct 26 11:35 JR1-ScienceDirect_20211026-163521.926678.csv
-rw-r--r-- 1 apache apache   1304 Oct 26 10:35 TR-ScienceDirect_20211026-153546.463673.csv
-rw-r--r-- 1 apache apache   1304 Oct 26 10:42 TR-ScienceDirect_20211026-154257.178140.csv
-rw-r--r-- 1 apache apache   1304 Oct 26 10:43 TR-ScienceDirect_20211026-154340.677717.csv
-rw-r--r-- 1 apache apache   1032 Oct 26 11:34 TR-ScienceDirect_20211026-163446.836328.csv
-rw-r--r-- 1 apache apache   4393 Oct 26 11:35 TR-ScienceDirect_20211026-163504.536999.csv
-rw-r--r-- 1 apache apache   1304 Oct 26 11:45 TR-ScienceDirect_20211026-164504.676088.csv
-rw-r--r-- 1 apache apache 389385 Oct 26 11:58 TR-ScienceDirect_20211026-165811.759045.csv
-rw-r--r-- 1 apache apache   1304 Oct 26 12:02 TR-ScienceDirect_20211026-170203.757149.csv
-rw-r--r-- 1 apache apache   1304 Oct 26 13:37 TR-ScienceDirect_20211026-183757.820173.csv
-rw-r--r-- 1 apache apache   1304 Oct 26 14:13 TR-ScienceDirect_20211026-191347.386101.csv
-rw-r--r-- 1 apache apache   1304 Oct 26 14:23 TR-ScienceDirect_20211026-192329.394489.csv
-rw-r--r-- 1 apache apache   1304 Oct 26 14:32 TR-ScienceDirect_20211026-193255.876977.csv
-rw-r--r-- 1 apache apache   1304 Oct 26 15:19 TR-ScienceDirect_20211026-201937.566921.csv
-rw-r--r-- 1 apache apache   1304 Oct 26 15:21 TR-ScienceDirect_20211026-202129.954656.csv
-rw-r--r-- 1 apache apache   1304 Oct 26 16:14 TR-ScienceDirect_20211026-211425.035856.csv
-rw-r--r-- 1 apache apache   1304 Oct 27 09:01 TR-ScienceDirect_20211027-140154.800772.csv

[clintbellanger]

David located this thread where people are having similar kinds of symptoms that we're seeing:
encode/django-rest-framework#5616

One suggestion we find there is changing permission_classes from SuperuserOrAdminPermission to IsAuthenticated

We're seeing this would affect many files under /opt/celus/apps/ and we're not confident whether this is pointing in the right direction. Wanted to check in with your team (Celus devs) to see if any of this rings a bell. Thanks!

[beda42]

Hello,

Error loading preflight data: Error: Request failed with status code 500

In general, when debugging such issues, I would recommend turning DEBUG to True in the Django configuration (most probably in the config/settings/production.py file) and then looking into the debug console of your browser to see the content of the reply which fails with 500. There should be some indication why it is failing - you can either deduce the cause of the problem from it or post it here for us to review.

The other issue that we are seeing is, the Interest section of the Dashboard is not populating, even though we do have data imported from SUSHI sources.

Interest is computed from the usage data in the background. For it to work, both celery and celerybeat services have to be running. You can also verify if interest is computed at all by looking at platform detail page to see if "Interest" is among the offered reports. If not, it means it was not computed for some reason.

Another reason why you are not seeing any Interest might be that for each platform Interest is only computed from a particular report type. This is usually the title report for journals and database report for databases. So if you for example only harvest platform reports, these will not be used to compute interest without reconfiguration.

I also noticed that Django has several report types listed, but the manual data uploads page only allows us to choose between 2, "Counter 5 - Title report" or "Counter 4 - Journal Report 1"

Well, this might be the cause of the first problem above - you are trying to upload a COUNTER 5 PR report, which is not supported in Celus version 1 for several reasons:

1/ COUNTER 5 table reports are not supported in general, just the JSON version
2/ Only those reports which are marked as defining interest for a particular platform are offered for manual data upload.

As COUNTER 5 requires SUSHI to be available for a platform to be COUNTER compliant, it is highly recommended to use SUSHI rather than manual uploads for C5 data.

[beda42]

David located this thread where people are having similar kinds of symptoms that we're seeing: encode/django-rest-framework#5616

One suggestion we find there is changing permission_classes from SuperuserOrAdminPermission to IsAuthenticated

I do not think that this issue is caused by permission problems - the 500 HTTP error signals something crashing rather than a permission issue (which would return 401 or 403).

In general, if Celus offers you some feature in the UI, you should have the corresponding privileges. At least we are not aware of any cases where this would be a problem.