edn ⟷ opml or edn ⟷ json ⟷ opml #2
Replies: 29 comments 4 replies
-
Thanks for starting this repo and thanks for getting things rolling. ;-) What I'd like to see first is a translation of an EDN file into properly indented JSON. I'd like to see what's in the file in a human-readable way, and then it might be clearer as to how it translates to OPML. So I think that's the smallest thing we can try to build, to get started. |
Beta Was this translation helpful? Give feedback.
-
BTW, here's a utility that converts an OPML file from XML to JSON. Exactly the same outline. http://scripting.com/code/opmltojson/index.html?url=http://this.how/cunyClassNotes/?format=opml If people prefer to use JSON to XML for outlines, this is how to do it. I've written a lot of JavaScript code that assumes exactly this structure. |
Beta Was this translation helpful? Give feedback.
-
Ok great thanks @scripting, that sounds good to me. Have you taken a look at https://github.com/wilkerlucio/edn-json ? I think building a frontend around that makes sense, but I don't really know anything about edn/clojure. From that page's README:
I don't know anything about clojure or clj, so I think starting with this library makes sense. I'm not sure about the "arbitrary decisions" they've had to make, but hopefully we can work within them or make changes if needed. |
Beta Was this translation helpful? Give feedback.
-
@scottrblock -- that doesn't look too good. I don't know anything about EDN, but if it can't be translated directly to JSON then it certainly can't be translated to OPML. It may be that Roam is a well-defended silo and there's no hope of interop. I did get some interop going with LogSeq, even got it so that you could author an Old School blog in LogSeq (Old School is the blogging system that Drummer uses). I didn't manage to get anyone who was familiar enough with LS to get the integration better connected. https://github.com/scripting/myLogseqBlog How much do you know about how LogSeq works? I got Drummer working with it by implementing their Markdown format. It's baked into Drummer now. Maybe that's a good place to pick up from? |
Beta Was this translation helpful? Give feedback.
-
I don't know much about logseq's internals, but I did build a logseq plugin for phonetonote. they have some docs and starter stuff: I think we could probably get |
Beta Was this translation helpful? Give feedback.
-
YES! That's exactly what I want. Exactly. My goal is interop. To get all these things working together so we can build an incredible ecosystem around outline editing. That would be a huge step. |
Beta Was this translation helpful? Give feedback.
-
BTW, here's the thread where the project was discussed. We didn't, in the end, connect with a developer who knew how to do logseq plugins, and to maintain my sanity, i can't stretch that far. I'm already juggling a lot of details here. But Drummer has a great blogging CMS. I've been using it for Scripting News since 2017. It would fit really well with LogSeq, I got far enough into it to be sure of that. Now it's a matter of getting the code that links LogSeq to the CMS into LogSeq. You'll see there is not much to it. Basically Old School takes all its input from a single OPML file. If you want to do this right, probably the first thing to do is to start up Drummer, and go through the blogging docs and start a blog in Drummer where all the integration is already in and functional. http://drummer.scripting.com/ and http://docserver.scripting.com/drummer/blogging.opml If you have questions, there's a support forum here on GitHub. |
Beta Was this translation helpful? Give feedback.
-
ok cool, i think i can port this to a logseq plugin, which seems like a great place to start since they have a relatively stable plugin ecosystem and everything. can you clarify
i'm going to get started wrapping my head around Drummer and the existing code you wrote. |
Beta Was this translation helpful? Give feedback.
-
exactly. |
Beta Was this translation helpful? Give feedback.
-
just to keep a single paper trail (until we make github discussions interoperable, but alas), i'm asking the logseq team about file read/write access in their plugin ecosystem because I think doing this without the opml files would be a bit of a bummer to the goals of making this interoperable — https://discuss.logseq.com/t/writing-to-a-file-for-ics-export/1453/8?u=srb. that was the first blocker i thought of reading it's also not immediately clear to me if https://github.com/eligrey/FileSaver.js/ would work, i'll ask a bit more in the logseq discuss |
Beta Was this translation helpful? Give feedback.
-
BTW, I'm pretty sure the writing of the OPML file to the local disk is not necessary. I just did it so people would have something to refer to if they wanted to see how it works. The OPML has to be transmitted to the server. |
Beta Was this translation helpful? Give feedback.
-
Maybe we should have a voice chat to go over this. Might help to refresh my memory and help show you the big picture. Want to set up a Twitter space? |
Beta Was this translation helpful? Give feedback.
-
I don't think I have twitter spaces, but sure happy to do a voice chat at some point. reading/writing files is possible with the logseq api now, so i think this should be straightforward. i started to push the skeleton of a plugin into this repo. can you setup a twitter space? we can also use discord or something. i'm relatively free this evening and tomorrow morning (EST). |
Beta Was this translation helpful? Give feedback.
-
Let's do it tomorrow morning. Yes I can set up a Twitter space. If not that, I like Facebook message voice. Very high quality but I'd prefer to use Twitter, because it's public and maybe we can do it again if something develops here. BTW -- I see you're in Brooklyn. I live upstate, west of Kingston in the mountains, about 2 hours north of the city. |
Beta Was this translation helpful? Give feedback.
-
cool, does 9am work? I'm free until 9:30! |
Beta Was this translation helpful? Give feedback.
-
That's way too early for me. ;-) Can you talk now? |
Beta Was this translation helpful? Give feedback.
-
i'll DM you on twitter |
Beta Was this translation helpful? Give feedback.
-
hey @scripting, I'm working on the port now, and realizing that I can write an OPML file with the logseq API, but I cannot read existing markdown files, to build the OPML from journals (like i believe you are in |
Beta Was this translation helpful? Give feedback.
-
Also if there's any way I can help let me know. I saw you're tweet yesterday about how they're adding an object model to LS. That's good -- it'll be useful Drummer has built-in scripting, and that can be hooked up to their interface, with perhaps some interesting high level ways to interop. |
Beta Was this translation helpful? Give feedback.
-
hey yea @scripting, I just opened a PR in opmlToJs - scripting/opmlToJs#1 explanation in there, but need that to build a logseq plugin that imports |
Beta Was this translation helpful? Give feedback.
-
I don’t accept pull requests, this is why… http://scripting.com/2020/05/26/194558.html?title=bugReportsNotPullRequests Instead, post a bug report as an issue. |
Beta Was this translation helpful? Give feedback.
-
ah ok, issue opened! |
Beta Was this translation helpful? Give feedback.
-
hey @scripting things are generally going well. what i'm currently thinking through is... since i'm fetching the journal data from logseq's query API, I receive data in a slightly different format than the nested markdown files while you converted the indented markdown files into OPML using I can group the blocks by their journal page, and then I think I need to write something similar to your Alternatively, I could try to fetch the combined/indented markdown of each journal page using a different part of the logseq API, but I don't see a way to do that. |
Beta Was this translation helpful? Give feedback.
-
If you want I can have a voice chat to go over this stuff. I have to ask some questions to understand what's going on. I'm around today. |
Beta Was this translation helpful? Give feedback.
-
i think i have a good handle on things at the moment, i'll get it into a buildable state and then we can go from there. i'll def hit you up if i get stuck or have specific opml/markdownToOPML questions. |
Beta Was this translation helpful? Give feedback.
-
Scott, I think using tags to determine which items go to the blog is a good idea. |
Beta Was this translation helpful? Give feedback.
-
Hey @scripting, i haven't had a ton of bandwidth this week, but was able to make some good progress tonight! i'm successfully:
I have two questions:
In light of not knowing where the blog's opml feed is, here's a link of the generated OPML - https://codebeautify.org/opmlviewer/y22226cb1. It's valid XML, and the dates look correct at first glance. Definitely getting close! I was able to work though mangling the JS dates correctly, and making them unique for the OPML. I'm moving this weekend, but will try to look at this with fresh eyes soon since it's so close! |
Beta Was this translation helpful? Give feedback.
-
You are getting close. About to break through it seems to me. ;-)
|
Beta Was this translation helpful? Give feedback.
-
woo, got it working! http://oldschool.scripting.com/phonetonote/?tab=blog thanks for the help @scripting, and for making rss/opml to begin with! i'll plan to submit this to the logseq plugin store sunday or early next week! |
Beta Was this translation helpful? Give feedback.
-
background
the purpose of this repo is to house code that will make tools for thought more interoperable. opml is a logical choice to convert various tools' data structures into because it's an open, established spec that's designed for tree-like structures.
a lot of popular tools for thought are based loosely on some flavor of datascript. datahike is a popular variant. my understanding (clojure folks please chime in here) is a superset of edn is used as a data transfer and storage format for datascript and clojure programs in general. this includes how tools for thought stores a user's graph, and how filters are formatted.
it seems to follow that a tool that converted between edn and opml would allow tools for thought to become more interoperable.
main question
my initial "smallest thing we can try to build" would be something that takes a roam or logseq edn export, and converts it to an opml file, which should already be readable by existing opml based outliners such as drummer.
if we were to build such a tool, my question is should we first convert the edn to json (such tools already exist, and in fact you can export directly to json), and then convert that to opml, or should we try to create something that converts an edn file directly to opml. one consideration is if we are building a web-based tool, we are likely serializing the edn as json at some point anyways.
other considerations/why edn?
a more broad question is will tools for thought continue to use edn/datascript/clojure forever. something like dgraph that can represent a graph well, but be much closer to web technologies like graphql might be better suited for tools for thought that wish to integrate into more third-party data sources. i think targeting edn is a necessary first step given a lot of current tools use them.
please chime in with any thoughts!
Beta Was this translation helpful? Give feedback.
All reactions