How manage date with vulpea

[Cletip]

Hello,
I would like to know how to manage dates with vulpea, let me explain.

The purpose of this question is to know how you manage the dates in your notes @d12frosted , but everyone can have a good answer !
Manage dates means several things: 

• standardizing dates for the purpose of
• be able to make queries and get an answer to the query "have all notes after a certain date", or "all notes created on this day".

Just a little precision before the beginning, I manage the creation date of my notes not thanks to a metadata, but thanks to the identifier, which corresponds to this
"%Y%m%d%H%M%S%6N"
Which could be translated as
20220709190343324727
and which allows me, not only to have a stable identifier, but also the creation date of the note.

Let's say I want to be able to store the following information:

• creation date (let's call it problem A)
• date "at the time of the action done" (if it's a note about a visit, date of the visit. If it's a note with a course, date at the time of the course). This corresponds in fact not to the creation of the note, but when the "action" took place (problem B)

Another problem: the format of the date. There are several formats

• [a date], for org (let's call it format C)
• , also for org (format D)
• or all the derivations of the syntax of the variable "org-id-ts-format", for example "%Y%m%d%H%M%S%6N", or "%Y-%m-%d-%H" etc. (format F)

For me, there are several solutions:

• Put the date(s) in the title (solution X)
• Put metadata (solution Y)
• use the note identifier (solution Z) or the properties drawer (solution Z')

Finally, I'd like to add that the properties drawer is made to have "hidden" things and doesn't support links, while metadata are made to be seen and to host links. I assume that I don't want to see the creation date, but I want to see the date when the content of the note was made (more details on what it is after).

To solve A, as I mentioned above, we could apply my method, and thus solution Z.

We could also apply solution Y, which seems to be intuitive: a metadata with the name "DateOfCreation" which would have the value of a date formated like C, D, or F. The choice also seems, to me, intuitive: F. Why? Because it allows to render as a number, and thus to make queries "all notes whose date is greater than "20220709190343324727".
But could there be another usable format? Does the org mode timestamp format allow to do queries? Is it usable with vulpea ?

Now, for the problem B. The application to problem A can be done, and seems practical. The only problem is how to manage the intervals. For example, when reading a book, it can take several days, and therefore have a value such as
20220709190343324727-20220809190343324727 (here, a reading lasting one month).

Still in problem B, I consider, for example something that is repeated, e.g. a journal, a course over several hours, etc.
This would result in the title:
Journal of the day x (with a particular format).

(In this case, there is a need to differentiate the Journal x notes. So you need either 1 or a date, 2 or a number. But the number must be a variable that increments, so it must be kept in a place, it is complicated etc.. Do you use another system to differentiate these notes?)

So, in this example, if we use the date to differentiate, the date is going to be in the title, AND elsewhere according to previously (in a metadata). There is redundancy in that, because I can't have a request like this "give me all the title that have a date after the date x). Here, the answer is X and Y.
So, one more question :) How do you manage the dates of repeating events like this.

A solution for the interval can be to separate the intervals in two metadata :

• StartValue
• EndValue
  We then apply the same reasoning as for problem A. The metadata "DateOfCreation" would be named "StartValue" to simplify things when there is no end date

I'm only giving some hints, but I would like to know your method and your point of view on this.

Thanks in advance for reading ^^

[d12frosted]

Hey, sorry for a delay. Crazy month, almost no time for open source.

I would like to know how to manage dates with vulpea, let me explain.

In short, I don't.

be able to make queries and get an answer to the query "have all notes after a certain date", or "all notes created on this day".

Because I don't ask these questions. I mean, I don't care when was something created.

But still, give me some time to read it and answer. I might have some ideas. No ETA, but bear with me :)

P.S. I remember about another conversation we have, I will update there as well.

[Cletip]

Hello, Bad week for me too, sorry

Thanks for your answer

I don't only want to know when it was created (by the way it's not my problem here I realize), but especially when it was done ^^

Don't rush, take your time. Just don't forget me ;)

[d12frosted]

There is no solution out of box, but as you said, there are several directions to solution.

Format

I would really advice you to stick to some standard time format. For example, RFC 822 (or later) or ISO 8601. Why? Because various utilities for time (built-in and org) do not support other formats while decoding time. In general, it's not a big deal, but having one format is nice. More over, org-parse-time-string is a nice thing that can parse [...] and <...> dates (with or without time):

ELISP> (org-parse-time-string "[2022-09-25 Sun]")
(0 0 0 25 9 2022 nil nil nil)

ELISP> (org-parse-time-string "<2022-09-25 Sun>")
(0 0 0 25 9 2022 nil nil nil)

ELISP> (org-parse-time-string "2022-09-25")
(0 0 0 25 9 2022 nil nil nil)

You get some nice goodies out of box. For example, you can compare dates. In case you need it.

ELISP> (org-time< "2022-09-25" "2022-09-26")
t
ELISP> (org-time< "[2022-09-25]" "<2022-09-26>")
t

There are more functions like this.

Extra note, since you've mentioned org-id-ts-format. Don't confuse date and id. I mean, even if content-wise org-id-ts-format gives you a date, semantically it's an ordered and (practically) unique identifier.

Creation date

As you said, there are several places you can put this data.

File-level

1. Filename
2. Buffer option (e.g. commonly recognized #+date: XXX or alike)
3. Properties drawer (e.g. :CREATED_AT: XXX)
4. Metadata (as defined by by vulpea).

Org Roam stores in the DB properties, metadata is stored by Vulpea.

Insertion of creation date can be easily automated by using org roam templates.

Heading level

At the moment you have only one option if you want the value to be persisted in DB - properties drawer. Metadata is not yet supported (will add it some day, see #75).

Not sure how to automate unless you use capture mechanism. Vulpea doesn't support creation of heading-level nodes, and AFAIK Org Roam doesn't support it yet (?).

Completion date

File-level

Basically, it's the same as with creation date. You have four options, and only two are persisted in the db. Though semantically I don't understand what it means to complete a file-level node. In Org mode TODO states exist only on a heading level.

Automation really depends on your definition of done and how you mark it. Theoretically, it can be hooked, but without knowing specifics, I have little to say.

Heading level

Same as with creation date. But since TODO are defined on heading level, Org Mode comes with some helpers to log the progress. It nicely supports repeated tasks. The latter is no synced in the database, but I guess it can be done. It's on my roadmap to make it easy to extend Org Roam database with extra data. Absolutely no ETA though.

Some random comments to your message

Because it allows to render as a number, and thus to make queries "all notes whose date is greater than "20220709190343324727".

1. As far as I remember you can use order operators with strings as well.
2. 20220709190343324727 doesn't fit into integer :) You have to store time and dates either as a string or a Lisp time object - (HIGH LOW USEC PSEC). Maybe there is another approach, didn't spend too much time investigating.

Just a small example showing you can't fit such a big number:

ELISP> (defvar db (emacsql-sqlite "~/test.db"))
db
ELISP> (emacsql db [:create-table people ([name id salary])])
nil
ELISP> (emacsql db [:insert :into people :values (["Jeff" 1000 60000] ["Susan" 1001 20220709190343324727])])
nil
ELISP> (emacsql db [:select * :from people :where (> salary 62000)])
(("Susan" 1001 2.0220709190343324e+19))

ELISP> (emacsql db [:select * :from people :where (> salary 20220709190343324726)])
nil

In the last example I would expect Susan to be part of response. Besides, equality is ill-defined.

Journal of the day x (with a particular format).

Honestly, I am completely lost in these few paragraphs. I don't understand what you are trying to achieve.

The purpose of this question is to know how you manage the dates in your notes

As I already stated, I don't :) I don't care at all when something was created or completed. When I used to care about completion date, I used built-in solutions for progress logging. I really advice you to look into it before building something overly complex 😸

Conclusion?

It really depends on what you want to achieve. If you want to solve a generic problem of tracking creation date and progress date, you need to take into consideration many derivations users can make with stock Org Mode. On the other hand, if you want to solve your specific use-case, you can always come up with some ad-hoc solution :)

Not sure if my comment is helpful at all, as I basically reiterated what you said with some extra input. My advise would be - try built-in solutions. Org agenda is a powerful tool.

[Cletip]

Why? I use agenda for tasks, and todo state is defined only for headings. I know that agenda can be used for more stuff, but it shines with tasks.

I want to do the same thing. I would just need a way to know when a note X was "made". For example, for "the louvre museum", I would like to have a simple data to tell me that I visited the museum at some time.

Not sure if it's correct, but I assumed that in this question you are talking about tasks. And somewhere in my comment I asked - what is the definition todo state for file level note?
Or in other words, in my sense note != task. That's abusing org and asking for troubles :)

Yes I agree with that : note != task. a task is actionable, so it can have a state (a heading). a note is made to have information. (sorry if I was not precise enough in my definition of what is a task).

On the other hand, I want to know WHEN I visited / did something. That's information, so I want to be able to store it. And if I could store it, I might as well use the built-in functionality of org-mode. Tell me if it's not very clear ^^

---

I was also wondering: what is the real interest of the Buffer option? I know that it is useful when exporting org-mode document for some instructions. It is also used to change org-mode variables for some files... but do you use it for anything else? Something different?

[d12frosted]

I want to do the same thing. I would just need a way to know when a note X was "made". For example, for "the louvre museum", I would like to have a simple data to tell me that I visited the museum at some time.

I use active timestamps for something to appear in agenda. See timestamps documentation.

That being said, as far as I know agenda works only with headings. Plus, agenda works the best with smaller set of files.

On the other hand, I want to know WHEN I visited / did something. That's information, so I want to be able to store it. And if I could store it, I might as well use the built-in functionality of org-mode. Tell me if it's not very clear ^^

It's clear. But I guess there is no 'common' solution. At least I never searched for this kind of information, because I don't have this task at hand. For visiting places I would just create a separate file and use a capture template to quickly log me visiting them.

* Place A

<2022-09-26>

* Place B

<2022-09-26>

* Place A

<2022-09-27>

This way it will appear in the agenda. UX wise I would make this capture template to present me a list of available places (like I do with my meeting notes).

[d12frosted]

I was also wondering: what is the real interest of the Buffer option? I know that it is useful when exporting org-mode document for some instructions. It is also used to change org-mode variables for some files... but do you use it for anything else? Something different?

I am not an Org mode developer, so this is my speculation, but there are two reasons. First is historical: properties drawers were not available on a file level till 9.X version of Org mode. Second is semantical: some values meant to be visible (like date, author, etc.). It's common for some libraries/applications (especially exporting ones) to adhere these values.

[Cletip]

Ok, thanks for your responses, I will try to write my reasoning down for the main question and put it as a solution to this discussion. I appreciate your opinions, feel free to comment when I post it ;)
Thanks you for your point of view ^^

[Cletip]

I realize that I have lost one of my answers. I had written it, I don't know where it is. I'm sorry :(
It was the full answer to your more detailed post.

To summarize (sorry to be honest I'm lazy to rewrite it)

• about the format: I totally agree, we need a standard for the date format.
• comment on the "org-time" style functions: This is exactly what I was looking for, I didn't know that all these functions existed, it's quite amazing. Thanks for this.
• On "org-id-ts-format" style functions:
  Yes, its purpose is to be a unique identifier. But why not use it so that org-attach can directly sort my files by date ;)
  But yes, there should be an explicit "creation date" property which would be in fact only a formatting of the identifier.
• I thank you for your relevant comments on the various proposals for headings and notes
• I thank you for the example of salaries in the database
• I was saying that I already know the "https://orgmode.org/org.html#Progress-Logging", but the problem here is that 1 I do not want notes, 2 I do not want particular states for notes

[Cletip]

Hi!
I finally finished my answer! Maybe I went too far, I don't know. Anyway, take your time to read, I'm doing this "for posterity" we'll say. Is it a kind of demonstration ? Anyway, don't bother to answer me if you don't want to read everything ^^

Table of Contents of my answer x)

1. Introduction: 
2. Development
   1. The conceptual mandatory rules
   2. Possible solutions
      1. Normalize dates
      2. Where to put the dates and make them queryable
      3. How to manage "recurring" notes
      4. Solving the "path of knowledge
3. Conclusion
   1. Possible workflow for the "knowledge path

Introduction: 

The purpose of this text is to help answer the following question: 
How to manage dates in vulpea / org-roam / org-mode?

The sub-questions will be the following:

1. How to normalize dates
2. Where to put the dates ?
3. Make the dates "queryable" (To be able to make queries such as "what are the new knowledge I learned between X and Y"?)
4. Know how to manage "recurring" notes (a journal, a course done every day, etc)
5. How to solve "the path of knowledge" (see after) which consists of:
   Task - Event - Notes. OR: know when a certain "event" has been done (example: visit to the Louvre)

Development

The conceptual mandatory rules

These rules are conceptual. But, in practice, everyone is free to do as he wishes for the implementation of these conceptual rules. Examples of practical/technical implementation will be given to each. In addition, a quick demonstration of the rules will be given.

1. If a group of information has to be referenced in one place, then a new note has to be created.
   Example: If A contains information x, information y and information z, and B refers to information y, then note Y must be created with information y, A becomes information x, (link to Y), information z, and B contains a (link to Y).
   This principle is not applicable all the time, but we will assume that we will always do it.
   Demonstration: if we want to refer to information y, we cannot make a link to A, the link would be "false", unless we want to explicitly link with the container and not the content.

2. Tasks are headings of org-mode. They can be found "anywhere". (no need to demonstrate, it is very practical and org-mode is very well done for that). A task = an actionable "information". Done, or not done.

3. Tasks can have all the properties of org-mode headings (schedulded, clock etc.)(no need to demonstrate this either)

4. The titles of the notes should respect the following things as much as possible:

   • to know quickly and naturally what is in the note
   • answer the question "how do I want this note to be seen in other notes?" to be able to exploit the property of "unlinked references" (which consists in finding where a note is located, but the link is not created. More information on the org-roam manual)
   • answer the question "when I insert this note, what do I want to be inserted".

   Example of title:
   "Louvre Museum", so the associated note will contain information about the "Louvre Museum".

5. The titles are not necessarily unique.

6. We can use "vulpea-select-annotate-fn" to allow us to put more details on a note when inserting or searching (like tags).

7. "vulpea-select-describe-fn" will only be used to display the title, as well as the "disambiguation", which represents the heart of the note. Indeed, as the title is not unique, it is possible that the title is not enough, so it is necessary to introduce a disambiguation notion (Example with this on wikipedia). The disambiguation could be stored in metadata for example. It could be put in the title between brackets.
   Note: the disambiguation is only for the user.

8. We must distinguish several things:

   • Action = task (example: "read a book")
   • Event = scheduled (+ task sometimes) + note (example: "physics chemistry course")
   • knowledge = notes (Louvre museum)

9. We must distinguish several "types" of date

   1. The date of creation of the note
   2. The date when the note was made (you can create the note and write 3 weeks later)
   3. A particular date interval (for a certain duration)
   4. The date of modification of a note

Possible solutions

Normalize dates

To solve the "Normalize dates" problem, several formats are available.
But you need a format that is: 

1. Understood by emacs (where you can do operations like "greater than", whether a date is between two dates etc)
2. Understood by the user easily
3. Be a classical format

One can thus choose the format used by org-mode which is very close to the ISO 8601 standard (example standard Iso8601 "2022-09-25", "2019-09-07T20:15:00". Example org-mode : 2019-09-07 20:15).

This type of date can be found thanks to a regular expression, which can be practical.

The precision of the date will be chosen according to the case. For example, it is useless to put the seconds for an annual appointment.

Where to put the dates and make them queryable

This solution will be discussed in several points.
In any case, it will be necessary to store the dates in a place that will allow to put them in the database. So we have the following possibilities:

• Filename
  Not very relevant, because only one date can be used (except if we parse but it becomes complicated).
• Buffer option (e.g. commonly recognized #+date: XXX or alike)
  Impossible, because they are not stored in the database.
• Properties drawer (e.g. :CREATED_AT: XXX)
  If you don't want to see them, hide them in the properties.
• Metadata (as defined by by vulpea)
  Visible to the user at all times.

In the end, it doesn't matter where, it's a matter of preference.

Reminder: dates at task level are managed by org-agenda. So, no need to worry about it.

1. How do I know when a knowledge has been put in the notes ?

   After thinking about it, it is impossible to do this with tools such as vulpea or org-roam. Indeed, by definition, a note is a set of knowledge.
   The dates must be in the database, so if I add a knowledge, I will have to change the date for a whole note. So, this is not relevant.

   On the other hand, a tool that can be useful: git, with the use of magit.

   To see when a knowledge has been put in the notes, you just have to do a "magit-blame" on the note.

   Moreover, to have more details, you can do the command "magit-log-buffer-file", which allows to know exactly all the history of a file

   To select a note and see its history:

   (defun cp/history-of-a-note (&optional file)
     (interactive (list (vulpea-db-get-file-by-id (vulpea-note-id (vulpea-select-from "Chose a note: " (vulpea-db-query))))))
     (find-file file)
     (magit-log-buffer-file)
     ;; (delete-other-windows)
     )
   

   Conclusion : we can't make a query like "all knowledge after X", but we can use the history of git (or "magit-log-buffer-file" and "magit-blame") to visit a previous version of the notes, and thus know if a knowledge was acquired or not at a time T (we can also know its evolution).

How to manage "recurring" notes

With a combination of org-agenda and vulpea.

Sometimes, for example when maintaining a diary, the names of the notes are very close, even the same: only the date differs for the user. It will therefore be necessary to make sure that the date appears to distinguish the different notes for the user. For example, we could consider these examples: 
Title of the note (Disambiguation:theWord) [date]
or
Title of the note (Disambiguation) [date] #listeOfTag
or
Title of the note - Date (Disambiguation) #listeOfTag
or
etc
Whatever the presentation, the user must have access to it when searching or inserting a note.

So we have to distinguish two cases.
But first, a little clarification:

1. Why titles like "Text - date" are not recommended
   This has 2 effects
   • it cancels the unlinked reference
   • it creates an "exception". Let's imagine that X corresponds to a knowledge, and that X Y (with Y a date) corresponds to something related to this knowledge at a date. Then, the user must remember that each time X Y is written, it is this something that is used.
     It is better to be explicit and have the note "X the Y" for example.
2. Why titles like "Text - date" are recommended
   Well for the following reason: the syntax of the language. Indeed, in French for example, several options are available to say a date. On the other hand, if we always put "Text text - date", it standardizes the thing, and create "only one exception" in all notes, which is acceptable.

No perfect answer :(

1. When the date is put in the title

   In this case, the date corresponds to a precision for the content of the note itself (because the title is made for that)
   We can take for example a historical event, like
   "Victory of France in 1998".
   Or a personal event marked by a date, like "Discussion with X after the sport on Y", because here the date allows to give more information to the note.

2. When the date is annotated

   In this case, the date does not correspond to a key element of the note. It doesn't matter what the date is, what matters is just to differentiate two notes during a search. One can think for example of a daily newspaper: putting the date in the title of the note will not help to give more information about the note itself.
   You can therefore put the date elsewhere, as in a metadata or in a property, and use the function defined by "vulpea-select-annotate-fn" to display it after the title.
   On the other hand, aliases can be created to give details about the day, for example : 
   Original title : page of my quotient journal [date in annotation].
   Alias : page of my quotient diary : Ski day [date in annotation]

   Moreover, when the dates are not in the title, they are queryable. That is to say that we can ask "all the notes having a date superior to X". But what are the notes that have dates?

Solving the "path of knowledge

As said before, we need to distinguish several things:

• Action = task (example: "read a book")
• Event = scheduled (+ task sometimes) + note (example: "physics chemistry class")
• knowledge = notes (Louvre museum)

The first and the last case are perfectly managed (thanks to org-mode and org-roam), but how to do for the second case? How do we make the link between tasks and notes?
We can notice this:

1. A task can evolve into an event
2. An event can be "transformed" into knowledge (by taking notes during the event)

Case 1 is managed, you just have to call the "org-schedule" command on a task. For case 2, on the other hand, this is more complicated.
Finally, we can establish a "knowledge path": a task can be done and accumulate knowledge by passing through an event. We must therefore find a technical path to this.

To solve this path, we must succeed in transforming a task into a note, sometimes by passing through an event. For more details, see at the end how to integrate a "workflow", a way of doing things.

As far as the structure of the note is concerned, it is necessary to distinguish the event itself from the notes taken on it. Indeed, to keep track of which event it was, you would have to keep a node each time "Event X" and "Y on event X" or Y would be something that would bring precision.
Example with "Meeting on Y" and "Conclusion of the meeting on Y".

This allows you to do things several times.
Example with a book: let's imagine that the book "X" exists. You read it once, so we can put "Reading of book X". Then, during the second reading, you can rewrite "Reading of book X" with another date, which is very practical

Conclusion

Managing dates is not easy, especially when you have to reconcile task and knowledge.
In the end, we have seen that putting dates on notes makes little sense, because dates are in fact linked not to the note, but to the concept contained in the note.
Moreover, it is also necessary to take into account the evolution of a task in event and an event in note.
Finally, the use of git can allow to know when a knowledge has been added in the knowledge system.

Possible workflow for the "knowledge path

A possible workflow would be with X = a task/something (this works with a book for example)

1. task (heading)
2. event (heading)
3. note taking during the event in the heading
4. when the event is done, then we put it in "done" (logic)
5. archive the event, but before doing so, ask the user if a note with the name of the event should be created.
6. if no, then archive
7. if yes, then it is proposed to create two notes with

   1. A note with a name like "Y on event X" (where Y is a precision on the event)

      • take the text of the event and put it in this note
      • add a "sort this fleeting note" task at the bottom of the note
        As you can see, "sort this fleeting note" will not need a note, so we'll go back to case 6.

   2. A note with the event "X", which will be the event itself

      • gives a start and end date (those of the event's timestamp). To get the timestamp : [[https://www.reddit.com/r/emacs/comments/s5b29a/orgmode_function_to_determine_time_elapsed_since/][look here?]
      • the tag "event" (variable)
      • the title which is the basic title of the event

      Will contain the factual content of the event (date, what it is about etc)

8. add some metadata to say that it comes from where it comes from: a link to the archived event (in hard copy ?), in fact not, because it is implicit, because it is an event, so the task will necessarily be in its archive!

This allows us to have a workflow in several steps:

1. setting up a task or an event (heading)
2. taking a fleeting note
3. end of the task
4. creation of the note associated with the task
5. clean up of fleeting notes and creation of zettels