Replies: 10 comments 11 replies
-
|
@bjarman, what process did you use to create your index? This error seems to suggest that the MARC data is missing from the Solr index, so understanding your ingest process will help with troubleshooting. If you could share your import configuration and/or some sample records that are causing problems, that might also be helpful. |
Beta Was this translation helpful? Give feedback.
-
|
Hi! I used the bundled import/index script in VuFind. I have made some further analysis and have found, perhaps, a clue. All imported/indexed marc records that have the field 010 are working fine. All records, that I have tried so far, that does not work do not have the field 010. My suspicion is that the field 010 must be present in the marc records? Here is a working record: =LDR 01528cam a22004097a 4500 Here is a not working record: =LDR 01895cam a22004337i 4500 Do I perhaps need to make my own local marc config - omitting field 010 in some way? |
Beta Was this translation helpful? Give feedback.
-
|
@bjarman, I compiled the provided sample records into binary format and loaded them with SolrMarc in my test environment: Both records display correctly:
I don't think the issue is likely to be related to the 010 field. When SolrMarc indexes a record, it not only maps selected data to specialized Solr fields, it also is supposed to store a representation of the full record in a field called |
Beta Was this translation helpful? Give feedback.
-
|
I will check the logs from my import tomorrow and get back. Thank you so
much so far!
ons 9 okt. 2024 kl. 16:52 skrev Demian Katz ***@***.***>:
… @bjarman <https://github.com/bjarman>, I compiled the provided sample
records into binary format and loaded them with SolrMarc in my test
environment:
***@***.***:/usr/local/vufind$ ./import-marc.sh /code/test.mrc
Now Importing /code/test.mrc ...
Oct 09, 10:48:32 /usr/lib/jvm/default-java/bin/java -Xms512m -Xmx512m -DentityExpansionLimit=0 -Duser.timezone=UTC -Dlog4j.configuration=file:///usr/local/vufind/import/log4j.properties -jar /usr/local/vufind/import/solrmarc_core_3.5.jar /usr/local/vufind/local/import/import.properties -solrj /usr/local/vufind/solr/vendor/.solrj -lib_local /usr/local/vufind/import/lib_local;/usr/local/vufind/solr/vendor/modules/analysis-extras/lib /code/test.mrc
INFO [main] (Boot.java:47) - Starting SolrMarc boot loader shim
INFO [main] (LoggerDelegator.java:272) - Command line:
INFO [main] (LoggerDelegator.java:272) - java -Xms512m
INFO [main] (LoggerDelegator.java:272) - -Xmx512m
INFO [main] (LoggerDelegator.java:272) - -DentityExpansionLimit=0
INFO [main] (LoggerDelegator.java:272) - -Duser.timezone=UTC
INFO [main] (LoggerDelegator.java:272) - -Dlog4j.configuration=file:///usr/local/vufind/import/log4j.properties
INFO [main] (LoggerDelegator.java:272) - -classpath /usr/local/vufind/import/solrmarc_core_3.5.jar
INFO [main] (LoggerDelegator.java:272) - /usr/local/vufind/import/solrmarc_core_3.5.jar
INFO [main] (LoggerDelegator.java:272) - /usr/local/vufind/local/import/import.properties
INFO [main] (LoggerDelegator.java:272) - -solrj /usr/local/vufind/solr/vendor/.solrj
INFO [main] (LoggerDelegator.java:272) - -lib_local /usr/local/vufind/import/lib_local;/usr/local/vufind/solr/vendor/modules/analysis-extras/lib /code/test.mrc
INFO [main] (LoggerDelegator.java:272) - Invoking main program:
INFO [main] (LoggerDelegator.java:272) - java -classpath <CLASSPATH>
INFO [main] (LoggerDelegator.java:272) - org.solrmarc.driver.ConfigDriver
INFO [main] (LoggerDelegator.java:272) - /usr/local/vufind/local/import/import.properties
INFO [main] (LoggerDelegator.java:272) - -solrj /usr/local/vufind/solr/vendor/.solrj
INFO [main] (LoggerDelegator.java:272) - -lib_local /usr/local/vufind/import/lib_local;/usr/local/vufind/solr/vendor/modules/analysis-extras/lib
INFO [main] (LoggerDelegator.java:272) - /code/test.mrc
DEBUG [main] (ConfigDriver.java:25) - Using config /usr/local/vufind/local/import/import.properties to initialize SolrMarc
INFO [main] (ConfigDriver.java:133) - Effective Command Line is:
INFO [main] (ConfigDriver.java:134) - java -jar solrmarc_core.jar IndexDriver -reader_opts import.properties -dir /usr/local/vufind/local/import|/usr/local/vufind/import;local/import -config "marc.properties, marc_local.properties" -solrURL http://localhost:8983/solr/biblio/update -solrj /usr/local/vufind/solr/vendor/.solrj -lib_local /usr/local/vufind/import/lib_local;/usr/local/vufind/solr/vendor/modules/analysis-extras/lib /code/test.mrc
INFO [main] (LoggerDelegator.java:272) - Invoking main program:
INFO [main] (LoggerDelegator.java:272) - java -classpath <CLASSPATH>
INFO [main] (LoggerDelegator.java:272) - org.solrmarc.driver.IndexDriver
INFO [main] (LoggerDelegator.java:272) - -reader_opts import.properties
INFO [main] (LoggerDelegator.java:272) - -dir /usr/local/vufind/local/import|/usr/local/vufind/import;local/import
INFO [main] (LoggerDelegator.java:272) - -config marc.properties, marc_local.properties
INFO [main] (LoggerDelegator.java:272) - -solrURL http://localhost:8983/solr/biblio/update
INFO [main] (LoggerDelegator.java:272) - -solrj /usr/local/vufind/solr/vendor/.solrj
INFO [main] (LoggerDelegator.java:272) - -lib_local /usr/local/vufind/import/lib_local;/usr/local/vufind/solr/vendor/modules/analysis-extras/lib
INFO [main] (LoggerDelegator.java:272) - /code/test.mrc
DEBUG [main] (BootableMain.java:119) - Adding directory: /usr/local/vufind/local/import
DEBUG [main] (BootableMain.java:119) - Adding directory: /usr/local/vufind/import
DEBUG [main] (Boot.java:713) - Adding jars files in directory: /usr/local/vufind/import/lib_local
DEBUG [main] (Boot.java:713) - Adding jars files in directory: /usr/local/vufind/solr/vendor/modules/analysis-extras/lib
INFO [main] (ValueIndexerFactory.java:119) - Using directory: /usr/local/vufind/import/index_java as location of java sources
INFO [main] (PropertyUtils.java:337) - Opening file (instead of 1 other options): /usr/local/vufind/local/import/import.properties
DEBUG [main] (SolrCoreLoader.java:48) - Pinging Solr at URL: http://localhost:8983/solr/biblio/admin/ping
DEBUG [main] (SolrCoreLoader.java:65) - ...
DEBUG [main] (SolrCoreLoader.java:54) - "status":"OK"
DEBUG [main] (SolrCoreLoader.java:96) - Found Solrj class org.apache.solr.client.solrj.impl.HttpSolrClient$Builder
INFO [main] (IndexDriver.java:166) - Reading and compiling index specifications: marc.properties, marc_local.properties
INFO [main] (IndexDriver.java:257) - Opening index spec file: /usr/local/vufind/import/marc.properties
INFO [main] (IndexDriver.java:257) - Opening index spec file: /usr/local/vufind/import/marc_local.properties
INFO [main] (IndexDriver.java:100) - Opening input files: [/code/test.mrc]
DEBUG [main] (IndexDriver.java:331) - System Class Path = /usr/local/vufind/import/solrmarc_core_3.5.jar
DEBUG [MarcReader-Thread] (MarcReaderThread.java:35) - record read : 325568
DEBUG [MarcReader-Thread] (MarcReaderThread.java:35) - record read : ncptnw32mqktrx9
INFO [main] (ThreadedIndexer.java:259) - Done with all indexing, finishing writing records to solr
INFO [main] (ThreadedIndexer.java:272) - Done writing records to solr
INFO [main] (Indexer.java:595) - Commmiting updates to Solr
INFO [main] (IndexDriver.java:391) - 2 records read
INFO [main] (IndexDriver.java:392) - 2 records indexed and
INFO [main] (IndexDriver.java:399) - 2 records sent to Solr in 1.12 seconds
Both records display correctly:
image.png (view on web)
<https://github.com/user-attachments/assets/a60313a1-a04d-497f-a663-33ae93f6ef9f>
image.png (view on web)
<https://github.com/user-attachments/assets/8a783642-49a5-404d-8703-f68f93d7c93b>
I don't think the issue is likely to be related to the 010 field. When
SolrMarc indexes a record, it not only maps selected data to specialized
Solr fields, it also is supposed to store a representation of the full
record in a field called fullrecord. Your problem seems to be caused by
this field being unpopulated. That should never happen under normal
circumstances, so either there's a misconfiguration, or there's something
about the records that is preventing them from encoding correctly. I'd be
interested to know whether there are any errors in the output of your
import command, for example. As you can see above, everything worked fine
for me, but since I've rebuilt the records, it's possible if there's an
encoding problem that it got worked out through all the conversion steps.
:-)
—
Reply to this email directly, view it on GitHub
<#3999 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAFOBOJFO7EOR77CPCI25X3Z2U7MJAVCNFSM6AAAAABPURH7N2VHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTAOBZGM4DGNQ>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
|
Ok, I have found some warnings in the import log. There are a lot of warnings relating to FullRecordAsJson. Does this shine a light on what I might be doing wrong? |
Beta Was this translation helpful? Give feedback.
-
|
Okay, that's very helpful -- we have a step that formats the input MARC into MARC-in-JSON format for storage in the index, and for some reason, some of your records are causing that conversion to fail. It would be very helpful if you could share a problem record in the original format so we can analyze what is going wrong, since it seems that whatever the problem is, it gets worked out if we convert the record into different intermediate formats. In the meantime, a simple workaround is to store the MARC in a different format, since VuFind supports MARC-in-JSON, MARCXML and binary MARC (we choose the JSON option by default because it's a good compromise between the verbosity of XML and the baked-in length limits of the binary format). To choose a different option, just copy $VUFIND_HOME/import/marc_local.properties into $VUFIND_LOCAL_DIR/import/ (if you haven't already) and then uncomment one of the alternate formatting options here: https://github.com/vufind-org/vufind/blob/dev/import/marc_local.properties#L12-L17 You can simply reindex the same file, and the bad records should get overwritten with the corrected versions. If you ever need to fully erase and rebuild your index, there is a Re-Indexing wiki page with details... but when you index a record, Solr just replaces any existing record with the same ID, so as long as you're re-indexing exactly the same set of records (as you are in this case), there's no need to take extra steps. Wiping the index is generally only necessary if the wrong IDs were indexed in the first place. I hope this helps! |
Beta Was this translation helpful? Give feedback.
-
|
The file I used is attached here. |
Beta Was this translation helpful? Give feedback.
-
|
It is exported from the national library in Sweden. From what I can see when looking at the file in Visual Studio Code it is UTF-8? |
Beta Was this translation helpful? Give feedback.
-
|
Aha! I will search for MarcEdit and try to compile it! |
Beta Was this translation helpful? Give feedback.
-
|
What export tool did you use? Does it support other MARC carrier formats? The "=LDR ..." format you are using here is commonly used for human-readable transmission, but VuFind's tools expect one of the machine-readable formats. As mentioned above, the only tool I'm personally aware of that supports this format is MarcEdit, which you can use to compile the text into binary MARC that VuFind can understand. |
Beta Was this translation helpful? Give feedback.
-
|
You can find MarcEdit here: https://marcedit.reeset.net/downloads |
Beta Was this translation helpful? Give feedback.
-
|
Also, for the record, I noted this situation in an existing open issue on the SolrMarc project describing possible problems with the JSON2 conversion: solrmarc/solrmarc#103 |
Beta Was this translation helpful? Give feedback.
-
|
Ok, we exported to marc xml and I redid the import. The problem seems to persist. This is the log from the import: |
Beta Was this translation helpful? Give feedback.
-
|
And here is the xml-file I used |
Beta Was this translation helpful? Give feedback.
-
|
Okay, very interesting! It looks like I'm falsely blaming the carrier format for the problem. It seems to go back to my earlier comment -- the problem is that there's something in the records that the very strict JSON2 conversion doesn't like, but if you switch to a different storage format for the Solr index, the process works correctly. For example, in my test environment, I switched to use XML storage by uncommenting The issue is likely related to the open SolrMarc issue (solrmarc/solrmarc#103) that I cited earlier. In the meantime, simply using a different storage format in your index should be a suitable workaround. The only consequence of MARC XML storage is that your index size will be slightly larger because it's a less efficient encoding... but that's unlikely to cause any issues for you. |
Beta Was this translation helpful? Give feedback.
-
|
Ok! Thank you so much for investigating this! I will switch storage format
to XML and hopefully it will work then!
I am unable to do this right now due to travel. I will let you know as soon
as it is done though.
Have a great weekend!
tors 10 okt. 2024 kl. 15:34 skrev Demian Katz ***@***.***>:
… Okay, very interesting! It looks like I'm falsely blaming the carrier
format for the problem. It seems to go back to my earlier comment
<#3999 (reply in thread)>
-- the problem is that there's something in the records that the very
strict JSON2 conversion doesn't like, but if you switch to a different
storage format for the Solr index, the process works correctly. For
example, in my test environment, I switched to use XML storage by
uncommenting fullrecord = FullRecordAsXML in my marc_local.properties
file, and then the ingest succeeded (even using your original text file).
The issue is likely related to the open SolrMarc issue (
solrmarc/solrmarc#103 <solrmarc/solrmarc#103>)
that I cited earlier. In the meantime, simply using a different storage
format in your index should be a suitable workaround. The only consequence
of MARC XML storage is that your index size will be slightly larger because
it's a less efficient encoding... but that's unlikely to cause any issues
for you.
—
Reply to this email directly, view it on GitHub
<#3999 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAFOBOOBLHEA3UBUDKN7VKTZ2Z66JAVCNFSM6AAAAABPURH7N2VHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTAOJQGQ2DKOA>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
|
You too, and safe travels. :-) |
Beta Was this translation helpful? Give feedback.
-
|
Hi! I have no idea why this worked but thank you either way :) Unless you want me to test/try to get the import using the marc-xml file working I guess we can close this. |
Beta Was this translation helpful? Give feedback.
-
|
Hmm, it's strange that the XML version didn't work, but maybe MarcEdit did something wrong in terms of encoding, etc. If the original file works for you, that's easier anyway, so I don't think it's worth spending more time troubleshooting what happened with the XML version. It's generally safe to ignore those "unknown relator" messages -- this is just informing you that the relator code is not found in author-classification.ini. It will still be displayed and indexed, but if you were trying to standardize everything for translation purposes, this code is not on the list of recognized values and will likely not be internationalized by default. I think it's safe to close this if you're satisfied that we've solved all of your outstanding problems! |
Beta Was this translation helpful? Give feedback.
-
Hi!
I am evaluating VuFind and have recently made a small marc import. The import seemed to have worked. However when searching for some posts I get this error when clicking the suggestion below the search field: "Missing MARC data in record 4ncptnw32mqktrx9". The error is in file MarcReaderTrait.php line 85:
if (empty($this->fields[$preferredMarcField])) {
throw new \Exception('Missing MARC data in record ' . $this->getUniqueId());
}
Some imported posts works but some does not.
The imported marc posts come from the National Library in Sweden, Libris and all have a unique id in field 001.
So, my question is, has anyone else experienced this issue?
Posts exists and are found by the auto suggest but generates error when clicking the suggestion below the search input.
Beta Was this translation helpful? Give feedback.
All reactions