Problems with exporting org-ref citations

[Perangelot]

Actual Behavior

org-ref syntax is not correctly exported to markdown. For example

#+title: Example
#+hugo_base_dir: ../
#+hugo_section: post
#+hugo_type: post
#+hugo_custom_front_matter: :link-citations true
#+csl-style: apa.csl
#+csl-locale: en-US

* First Heading
This should be a title citation: [[citetitle:&anellis2008]].
** Second Heading 

This should be a usual citation without parentheses: [[cite:&anellis2008]].

bibliography:minimal.bib

is exported to

+++
title = "Example"
type = "post"
draft = false
link-citations = true
+++

## First Heading {#first-heading}

This should be a title citation: <&anellis2008>.


### Second Heading {#second-heading}

This should be a usual citation without parentheses: <&anellis2008>.

<minimal.bib>

Expected Behavior

Exporting an org-mode file containing org-ref syntax with ox-hugo results in a .md file with actual in-text citations + a bibliography. To stay with the example:

+++
type = "post"
draft = false
link-citations = true
+++

## First Heading {#first-heading}

This should be a title citation: [2008](#org-target--citeproc_bib_item_1).


### Second Heading {#second-heading}

This should be a usual citation without parentheses: [Anellis, 2008](#org-target--citeproc_bib_item_1).

## References {#references}
<span class="org-target" id="org-target--citeproc_bib_item_1"></span>Anellis, I. H. (2008). The First Russell Paradox. In T. Drucker (Ed.), _Perspectives on the history of mathematical logic_. Birkhäuser.

How to Reproduce the Issue

Load org-ref

(use-package org-ref
:ensure t
:init
(require 'org-ref-utils))

Create a file minimal.bib:

@incollection{anellis2008,
 title = {The {{First Russell Paradox}}},
 booktitle = {Perspectives on the History of Mathematical Logic},
 author = {Anellis, Irving H.},
 editor = {Drucker, Thomas},
 date = {2008},
 series = {Modern {{Birkhäuser}} Classics},
 publisher = {{Birkhäuser}},
 location = {{Boston}},
 isbn = {978-0-8176-4768-1},
 keywords = {history of logic,history of mathematics,history of set theory,paradoxes},
 annotation = {OCLC: ocn213354737}
}

Put the file in the same folder as the org-mode file. Also put the apa.csl file in there. Export to hugo.

[kaushalmodi]

#+hugo_pandoc_citations: t

The pandoc citations are enabled, so the ox-hugo exported Markdown is made a second pass through Pandoc's Markdown exporter (essentially a md-to-md pandoc conversion happens).

I don't use org-ref or pandoc frequently. So I have a few questions:

1. If you are using org-ref, shouldn't #+hugo_pandoc_citations: t be removed?
2. If you take out that line, does the org-ref citation export as expected?
3. Can you also paste what you expected the exported Markdown to look like?

Thanks!

[kaushalmodi]

I just added this documention (it may or may not help in your case; it is using the inbuilt oc.el in Org 9.5+ instead of org-ref): https://ox-hugo.scripter.co/doc/org-cite-citations/.

[Perangelot]

Thank for your fast response! I have just looked into the issue and know why exporting out of the box does not work with org-ref. To answer your questions first: Yes, #+hugo_pandoc_citations: t should be removed, but no, it does not make a difference. .md file below. Now to what I have found out:

1). citeproc-org is deprecated and should not be used anymore. org-ref has taken over the job.
2). You get very nice citations with org-ref and ox-hugo by creating a usual org-file with all the org-ref stuff in it, and then run org-ref-export-as-org. This gives you a temp buffer with org-ref syntax translated to usual org-syntax. Now, you can use the ox-hugo functions and get a .md file which hugo translates very nicely. Citaiton styles are handled by org-mode headers, the bibliography file by bibliography:name.bib. I have no idea to which extent pandoc is involved, though.

This is how an example .md file looks like following 2):

+++
title = "Example"
type = "post"
draft = false
link-citations = true
+++

## First Heading {#first-heading}

This is some text.

### Second Heading {#second-heading}

This should be a citation ([Anellis, 2008](#org-target--citeproc_bib_item_1))

<span class="org-target" id="org-target--citeproc_bib_item_1"></span>Anellis, I. H. (2008). The First Russell Paradox. In T. Drucker (Ed.), _Perspectives on the history of mathematical logic_. Birkhäuser.

At first glance, it looks like only the heading References is missing, which should be added by ox-hugo.

I think the first make-shift solution would be a function which automates 2). You could then put that function in the .dir-locals.el file. I don't know much elisp, so if you could write that function (which will probably only be a few lines lol), that would be great. If the references heading problem is fixed and you like, then, I would write a neat article on how to set everything up, with explanations, example files and all that. If you need any more info, just reach out to me :)

[kaushalmodi]

citeproc-org is deprecated and should not be used anymore.

Right, that's why I just updated this page: https://ox-hugo.scripter.co/doc/org-ref-citations/

I have no idea to which extent pandoc is involved, though.

If you are using org-ref, you should not involve Pandoc. So removing #+hugo_pandoc_citations: t is the right thing to do. Now we have 3 citation parsing solutions:

1. pandoc based
2. org-ref
3. oc.el (integrated as part of Org 9.5): https://ox-hugo.scripter.co/doc/org-cite-citations/

As far as I know, you need to pick one of the three. May be @jkitchin can correct me here.

I think the first make-shift solution would be a function which automates 2)

Correct. org-ref is a really popular package and I believe that manual work shouldn't be needed. Does org-ref based citations export correctly when using ox-html? If so, I don't think ox-html would have added special support for org-ref.

@jkitchin Can you please help out here? Is there something that needs to be added to ox-hugo to support this?

[kaushalmodi]

@Perangelot Can you please update the main post in this issue by removing references to #+hugo_pandoc_citations: t and showing just your org-ref setup?

With just the org-ref setup (and citeproc-org setup removed), how does your minimal Org file with org-ref citations export to Markdown? And what would you expect it to export to instead?

[Perangelot]

@kaushalmodi Thanks for the clarifications! I have updated my question; hope it is more clear now. I have by the way found out that the solution I propose seems to have one minor problem: All citations are put in parentheses, even years. For example, (1908) instead of 1908 and (Zermelo) instead of Zermelo. But this is something where @jkitchin might be able to help!

[jkitchin]

I believe the solution here is to use one of the pre-processing hooks in org-ref with a csl file for styling. The cite links really only export to LaTeX, and org-ref doesn't really have a "markdown" export like it used to have. Instead, there is a pre-processing function (org-ref-csl-preprocess-buffer) that replaces the cites and bibliography links in the buffer with the output of citeproc before proceeding with the export on the rest of the org file. I typically would do it like this in a build section marked noexport, but you could also make your own function, or set the hook variable in an init file.

#+BEGIN_SRC emacs-lisp
(let ((org-export-before-parsing-hook '(org-ref-csl-preprocess-buffer)))
  (ox-hugo-export-command))
#+END_SRC

[Perangelot]

Thanks for your quick response, @jkitchin! Using the code you've send comes near to the desired result, but not quite. I've run this code:

(let ((org-export-before-parsing-hook '(org-ref-csl-preprocess-buffer)))
 (org-hugo-export-to-md))

I see three problems here:
1). Links and italics are not exported. This is different to running org-ref-export-as-org manually and on that buffer org-hugo-export-to-md, where links and italics are preserved.
2). Every citation is put in parentheses as explained in in this post. I have tried it with different .csl files and always had the same result, and of apa.csl I know it should produce different results, so it seems to be independent of that.
3). The References heading is missing.

1. and 2) are things you might be able to help out with, @jkitchin. About 3), I do not know where to implement it best. @kaushalmodi, any idea?

[jkitchin]

On 1. citeproc doesn't support a markdown output. There is plain (which is used in org-ref) or org format (which is what happens when you use org-ref-export-as-org.

The problem is you need a mixed backend export to achieve what you want. You can get this in the following way probably:

(let ((org-export-before-parsing-hook '(lambda (_)
					 (org-ref-process-buffer 'org))))
  (org-hugo-export-to-md))

On 2, I guess that parenthetical is the default style with apa, and to get a cite without parens you can use a citet link. The parentheses are only suppressed on citet, citet*, citenum at the moment. I pushed citetitle as a non-parenthetical type. I am less sure it is the right thing to make cite non-parenthetical as a default.

On 3, if the CSL does not provide the name of the reference section, you should just manually add a References heading in your org file where the bibliography link is.

[kaushalmodi]

@jkitchin

Regarding exporting to Markdown

Hugo allows HTML embedded in Markdown. When I export oc.el citations using ox-hugo.el, it (or the citeproc.el) seems to do the Right Thing..

For example,

[cite:@..] citations export as:

 See Nim Manual <a href="#citeproc_bib_item_2">2022</a> _Generics_.

..

<a href="#citeproc_bib_item_1">2018, sec. 8.25</a>

#+print_bibliography: exports as:

<style>.csl-entry{text-indent: -1.5em; margin-left: 1.5em;}</style><div class="csl-bib-body">
  <div class="csl-entry"><a id="citeproc_bib_item_1"></a>“IEEE Standard for Systemverilog–Unified Hardware Design, Specification, and Verification Language.” 2018. <i>IEEE Std 1800-2017 (Revision of IEEE Std 1800-2012)</i>, 1–1315. <a href="https://doi.org/10.1109/IEEESTD.2018.8299595">https://doi.org/10.1109/IEEESTD.2018.8299595</a>.</div>
  <div class="csl-entry"><a id="citeproc_bib_item_2"></a>Nim contributors. 2022. “Nim Manual.” Website. <i>Nim</i>. <a href="https://nim-lang.org/docs/manual.html">https://nim-lang.org/docs/manual.html</a>.</div>
</div>

Regarding the parentheses

Reading https://blog.tecosaur.com/tmio/2021-07-31-citations.html#cite-styles, I don't know if it's citeproc.el or oc-csl.el that's providing those styles, but "bare" style exports to citations without parentheses if I use the inbuilt oc.el + citeproc.el (yes, I just started looking at citations a bit more closely after this issue :) ).

So this:

[cite/na/b:Nim Manual @nimManual /Generics/]

exports to:

See Nim Manual <a href="#citeproc_bib_item_2">2022</a> _Generics_.

[kaushalmodi]

@jkitchin I see that citations.el wraps the bibliography section in

<div class="csl-bib-body">

..

</div>

Would org-ref also do that, or something similar? If we can document a way to have org-ref export citations and bibliography export in HTML with div blocks with specific classes like above, I can have ox-hugo inject "References" or "Bibliography" (will make it a defcustom) heading just above those div blocks.

[kaushalmodi]

@Perangelot @jkitchin I have moved the "References" heading auto-injection discussion to #558

[jkitchin]

citeproc supports html output, and if you use an html backend you get different behavior than markdown. The html would support links/markup. the plain output doesn't, and the org output does.

It looks like oc-csl treats a bare citation as one that should suppress the affixes (parentheses). I have made a defcustom where you can add cite to it if you want. The default in org-ref-export-suppress-affix-types doesn't currently have cite in it, but you could put it there if you want that default. I am not sure it is the right thing to do. The right thing is certainly use citet/citep to be explicit. cite is ambiguous, and subject to preference I think. I could be convinced to default to what cite would do with latex, but I guess it could be style specific too.

If you use an html backend for pre-processing, you would get that

wrapping. org-ref only uses what citeproc outputs, it does not auto-inject anything. I think if you do this, you will get that.

(let ((org-export-before-parsing-hook '(lambda (_)
					 (org-ref-process-buffer 'html))))
  (org-hugo-export-to-md))

[kaushalmodi]

@Perangelot I just tried this out and it works!

Add this to the bottom of your Org file, save it and export using ox-hugo as usual (e.g. C-c C-e H H).

#+bind: org-export-before-parsing-hook (lambda (_) (org-ref-process-buffer 'html))
# Local Variables:
# org-export-allow-bind-keywords: t
# End:

---

Make sure you update ox-hugo from Melpa so that you get the auto-insertion of the "Bibliography" heading as well. See https://ox-hugo.scripter.co/doc/org-cite-citations/#auto-injecting-bibliography-heading to learn more about it, how to customize that heading, etc.

[Perangelot]

I've just tried it out and everything is working smoothly! Thanks for all the effort you put in, @kaushalmodi @jkitchin! I really appreciate it. I would have loved to help you with the org-file but just came to read the posts. Instead of cluttering the org-file with more content than needed, I have decided to set the hook in the ox-hugo configuration:

(use-package ox-hugo
  :ensure t
  :config 
  (setq org-hugo-use-code-for-kbd t)
  (setq org-export-before-parsing-hook (lambda (_) (org-ref-process-buffer 'html)))
)

I have also adjusted the list of non-parenthetical citations, which now works perfectly for the popular apa 7th style:

(setq org-ref-export-suppress-affix-types
  '("citet"
    "textcite"
    "citet*"
    "citetitle"
    "citenum"
    "cite"
    "citeyear"
    "citeauthor"
))

One feature request I have, @kaushalmodi, is to define a variable which allows toggling automatic bibliography insertion on and off. The advantage of this is that if you use one single .bib file for your blog - which I suppose quite some people do because of Zotero Better Bibtex's function to keep it up-to-date - you can just set bibtex-completion-bibliography to that file, in your .dir-locals.el, e.g. to ("../static/blog.bib") , and set the variable to non-nil. This way, if you call an org-ref command, it will show you all and only those bibliography entries contained in your blog.bib. If you want to add an entry, you just add it to Zotero and you can cite it. And you wouldn't need to worry about setting #+bibliography or bibliography:, plus your org-file wouldn't be that cluttered. Of course, if no org-ref-cite-link is used, no Bibliography should be inserted. If you want to, for some reason, have your bibliography somewhere else than at the end of the file, you can just set the variable to nil and use org-ref's bibliography: link.

[kaushalmodi]

I have decided to set the hook in the ox-hugo configuration:

Note that while you add that, you are modifying that hook for any future Org export using any other backend as well; not just ox-hugo.

A better approach would be to selectively update that hook by using the backend arg of that lambda instead of discarding it using _ and use (org-export-derived-backend-p 'html) which will eval non-nil for HTML as well as Hugo exports.

Or, you may do an :around advice for the ox-hugo wim function so that it affects only ox-hugo exports.

.. Or set that hook in a .dir-locals in your Hugo site directory :)

One feature request I have, @kaushalmodi, is to define a variable which allows toggling automatic bibliography insertion on and off.

The bibliography insertion is not done by ox-hugo. It's done by the citeproc.el library that org-ref uses (oc.el or org-cite also uses the same library).

Check if org-ref has such a global variable. If not, you may open an issue on the org-ref repo requesting this.

---

Suggestion:

(use-package org-ref
  :ensure t
  :init
  (with-eval-after-load 'ox
    (add-to-list 'org-export-before-parsing-hook
                 (lambda (backend)
                   ;; ox-hugo <- ox-blackfriday <- ox-md <- ox-html
                   (when (org-export-derived-backend-p backend 'html)
                     (org-ref-process-buffer 'html))))))

[kaushalmodi]

@Perangelot I've converted this to a discussion as this didn't need fixes in ox-hugo and also so that it gets more visibility for future org-ref users who might have same question as yours.

[kaushalmodi]

@Perangelot The ox-hugo doc page for org-ref is updated at https://ox-hugo.scripter.co/doc/org-ref-citations/ .

[jkitchin]

I am glad to see this is working out!

[Perangelot]

Thanks for your efforts! Quick glitch I found: titlecite is exported like citeyear. Could you check if there's something wrong with the export code?

[kaushalmodi]

Could you check if there's something wrong with the export code?

Not sure who the question was for, but ox-hugo doesn't control that. @jkitchin?

[jkitchin]

I don't think you can support all the cite links with csl. They are based on LaTeX/bib(la)tex citation types and are only fully supported in that environment. I am not sure, but I don't think that csl supports something like citetitle, although it does support something like citeyear and citeauthor. To get that, I think there would have to be a new mode defined for https://github.com/andras-simonyi/citeproc-el#citeproc-citation-create-key-cites-note-index-mode-suppress-affixes-capitalize-first-ignore-et-al- that was title-only for example. I don't think you can do this with org-cite either (at least there doesn't appear to be a style that would be title only).