Improve Handling of 'Set due date'

[brishtibheja]

I'll start with the basic problem. In Anki, you might encounter cards that you have already learned before. This can happen if you learned the content outside Anki or even if you have lost parts of the collection/revlogs. This doesn't happen regularly for a large amount of cards and thus is usually dealt by using 'Set due date'.

In FSRS, however, there is no way for the user to handle this situation. The problem a user wants to solve is unnecessary reviews. But that cannot be done in FSRS if the card is in new state.

This have been bought up before in the forums here and here.

I believe FSRS being able to better deal with 'Set due date' is also essential for it to become the default. LMSherlock, kindly look into the matter.

(pinging @Expertium)

[Expertium]

From Discord:

So the issue is that some people use "Set Due Date" without being familiar with the card, while others use it once they have already reviewed the card outside of Anki

Since we have no way of differentiating between those two, we have to choose how to treat such cards. Whatever solution we choose, it will upset one of those two types of people. If we knew which one was more common, that would help a lot. But we don't. I could make a survey, but I doubt that "Set Due Date" is used by many people, so the sample size will be abysmal.

[brishtibheja]

Another thing to keep in mind is: the first group can choose an alternative workflow whereas the second group can't.

[Expertium]

I think the only way to settle this is to measure how many people are in each group. But I'm afraid I will get a double digit number of responses to a question like "Have you ever used "Set Due Date"?", and a single digit number of responses to a question like "Do you use for cards that you are already familiar with?".
But it's not like I have any better ideas anyway. I'll make a survey and post it on r/Anki and on Discord tomorrow.

[brishtibheja]

People who use 'Set due date' without being familiar with a new card, how their workflow worked before FSRS? As far as I understand, SM2 will simply modify the intervals with it's multipliers. This worked for a long time, why do this new workflow has to be supported? (i.e. using 'Set due date' to distribute your new cards over a period of days).

@dae I invite you to comment on this.

(edit: the manual is vague on this:

New cards will have their interval set to the same delay, but reviews will be rescheduled without changing their current interval, unless '!' is included at the end of the range.
)

[Expertium]

https://forms.gle/GEa59PWEjApNXEKe7
Here's the survey. Just 2 simple questions

[L-M-Sherlock]

I highly recommend Anki implementing something like Execute repetition (Ctrl+Shift+R) of SuperMemo. It allows user to grade the card and set the next interval in the same time. It's very important for FSRS to know that the user reviews the card when they use Set due date. Otherwise, FSRS cannot deal with this case correctly in theory.

https://supermemopedia.com/wiki/Ctrl%2BJ_vs._Ctrl%2BShift%2BR

[user1823]

the first group can choose an alternative workflow

The problem is that FSRS needs to deal with new revlogs as well as the old ones. Even if the first group chooses an alternative workflow, their old cards will still be scheduled incorrectly.

[brishtibheja]

@user1823 I didn't get it. The first group consists of people who use Set due date to distribute their cards over a period of time, as I understand it. What do you mean by new and old revlogs and why the latter will be scheduled wrong?

(edit: closed issue by mistake)

[Expertium]

But I'm afraid I will get a double digit number of responses to a question like "Have you ever used "Set Due Date"?", and a single digit number of responses to a question like "Do you use for cards that you are already familiar with?".

I was more pessimistic than it turned out.

While 22 isn't a lot, we can draw a conclusion even with such a small sample size.
(btw, let's call people who use Set Due Date without knowing the card group 1, and those who use it while knowing the card group 2)
To do that, I'm going to use what's called Clopper–Pearson exact method to calculate a binomial proportion confidence interval. Sometimes math really does sound like magical spells. Thankfully, it's easy to calculate since scipy does everything for me:

from scipy.stats import binomtest

a = 0.99  # confidence
k = 20  # successes
n = 22  # total trials

result = binomtest(k=k, n=n)
ci_clopper = result.proportion_ci(confidence_level=a, method='exact')

I count users in group 2 as "successes".
This gives me the following confidence interval for the % users in of group 2: [0.642, 0.995]. The confidence interval doesn't cover 0.5, so we can be sure that the true value of the proportion is >0.5. Which is all we need to make the decision. We don't need to know whether 95% or 90% or 80% of users are in group 2, we just need to know that they are the majority.

EDIT: forgot to mention that this is an interval for 99% confidence.

@L-M-Sherlock the conclusion is that those who use Set Due Date while knowing the card are the majority.

[L-M-Sherlock]

So can I treat Set Due Date as good?

[Expertium]

I suppose yes

[user1823]

So can I treat Set Due Date as good?

No. If Set Due Date is treated as Good, we will get incorrect estimate of initial stability because the revlog will then also be used for optimization.

If we need a change, it should be a revert of ankitects/anki@db93939
But, it will reintroduce the problem solved by this commit. (the problem faced by users who use set due date on new cards without knowing the card)

[Expertium]

But, it will reintroduce the problem solved by this commit. (the problem faced by users who use set due date on new cards without knowing the card)

Well, screw them. I know this is rude, but since we can't make Set Due Dates work for both group 1 and group 2, we have to make it work for the majority, which is group 2.

[user1823]

But, there is one problem with your poll. The poll says: "I use it on cards that I am not familiar with"
People who say that "I am not familiar with the material" will always be a minority.

For an example, let's say that I am using a premade deck and I studied one topic from a book/lecture today. Now, I am familiar with the material but not in the sense that people in group 2 are. Now, if belonged to group 1, I would have used Set Due Date on new cards so that I can review the cards from the topic in the next, say, 5 days.

Frankly speaking, these people are in group 1 but in your poll, they will likely choose "I use it on cards that I am already familiar with", making us think that they are in group 2.

IMO, better choices for the poll will be

• I use it to schedule new cards evenly over the next few days.
• I use it to assign large intervals to "new" cards that I have already learnt outside Anki.

If you want, you can add an "Other" option too so that we can get to know about the other purposes for which people use Set Due Date on new cards (if there are any other purposes).

[user1823]

What do you mean by new and old revlogs and why the latter will be scheduled wrong?

@brishtibheja

At best, people in group 1 can STOP using Set Due Date on new cards. They can't undo what they did in the past. So, if we make a change, the old cards of people in group 1 will start getting scheduled incorrectly (currently Anki is designed to favour group 1).

[Expertium]

IMO, better choices for the poll will be

• I use it to schedule new cards evenly over the next few days.
• I use it to assign large intervals to "new" cards that I have already learnt outside Anki.

I can make another survey, if you really, genuinely think that we can't use the current results (I think we can), and if you will precisely specify the wording, to ensure that you will have no further objections.

[L-M-Sherlock]

No. If Set Due Date is treated as Good, we will get incorrect estimate of initial stability

But it's also incorrect to use the interval set by users as the initial stability.

The ideally solution is to let user rate the card when they use set due date. To be honest, I still don't know why the user will use set due date. Do they think the initial interval of all four ratings are too short?

[Expertium]

The ideally solution is to let user rate the card when they use set due date

That would still leave us with cards that have been set-due-date'ed in the past, so we would have to deal with that anyway.

[user1823]

Do they think the initial interval of all four ratings are too short?

Yes (for users in group 2) because they have already learnt the information well outside Anki and encountered it in Anki as a new card just now

[Expertium]

@brishtibheja @user1823 https://forms.gle/KPKtLKt24gDb18Fj9

Btw, user1823, do you have a Discord account, or Reddit account, or...something?

[brishtibheja]

At best, people in group 1 can STOP using Set Due Date on new cards. They can't undo what they did in the past. So, if we make a change, the old cards of people in group 1 will start getting scheduled incorrectly (currently Anki is designed to favour group 1).

@user1823 I sympathise with them, but more so with the people in the second group. If there's a way to solve the problems of everyone, then good. Otherwise, considering all the future users of Anki, let us not allow this absurdity to go on longer.

LMSherlock's idea too wouldn't prove useful for everyone and would unnecessarily add extra complexity.

[L-M-Sherlock]

Maybe treating set due date as easy is better than status quo. At least these cards will have a large initial stability, and the second review will increase it more due to delay.

[Expertium]

We'll see how the second survey goes. I do like the idea of treating Set Due Date as Easy though.

[Expertium]

@L-M-Sherlock @user1823
https://docs.google.com/forms/d/1grBGTiHRzV5kcjSM1AKrber0KxDgyLGug7pJ3Io5z1s/viewanalytics
This survey got fewer responses than the last one.

In any case, I don't think the results will be much different.

@brishtibheja you were wrong

For question 2, I'll have to discard users who answered "Other", since I can only calculate the confidence interval of a proportion if there are 2 variables/groups/outcomes/whatever. Let's count people who chose "I use it to assign large intervals..." as "successes". That's 7/13. The 99% likelihood interval* is [0.227, 0.830]; it covers 0.5, so we cannot tell which group of users is the majority.
​*​ I switched to a new method described in this paper instead of the Clopper-Pearson method, which gives intervals that are too wide for small sample sizes (on large sample sizes all methods give pretty much the same results), though in this particular case the conclusion is the same anyway.

For question 3, let's count people who answered "Undesirable" as "successes". That's 11/21. The 99% likelihood interval is [0.267, 0.772]; it covers 0.5, so we cannot tell which group of users is the majority.

If these results are to be believed, then the number of people that we will screw up by changing Set Due Date is about the same, 50/50. So now we have even less reason to do anything.
My personal philosophy is like this:

1. Find an option that satisfies everyone (barring a very small number of absolute lunatics)
2. If 1 is not possible, find an option that satisfies >50% of users, in other words, the majority
3. If 2 is not possible...idk, lie down and put a paper bag over your head or something

[brishtibheja]

I think we should've added a Idk option in the last question because some people didn't even understand it for sure. So what should be done from here? Lie down with a paper bag?

My problem is with the assymtry of the situation. On one hand, you have people who have been robbed off of a useful feature that worked in SM2 in favour of another feature that "allows equally distributing new cards" something that could be already done.

If 1 is not possible, find an option that satisfies >50% of users, in other words, the majority.

If that also includes all the future users of Anki, you will satisfy more people by forcing the first group to use "Filtered decks" or change their workflow.

@dae what do you think about this? the problem is that an existing feature has disappeared with FSRS. I believe the feature is important on itself and there is no need for reimplenting it, just change how Set due date works (although it will create problems for some users).

[Expertium]

@dae Sherlock proposed making it so that Set Due Date also asks the user to grade the card

[brishtibheja]

I think that should be done as a seperate feature, if at all.

[dae]

Intervals going backwards after a set-due-date card is reviewed for the first time does sound problematic. I've skimmed the conversation above - apologies if I have missed something that's already been spoken about.

I'll restate things to confirm my assumptions/understanding. If something is wrong, please let me know.

1. If we set the initial stability at set-due-date time, it would presumably prevent intervals going backwards when subsequently reviewing the card, but it would not capture the initial difficulty.
2. Some users use set due date and expect the cards to be treated like new when subsequently reviewed

The second one surprises me. When we were SM2-only, this wasn't possible, was it? Why is it suddenly a use case we need to support?

For the initial difficulty, what if we set it to some typical value for cards rated good? Or it could be set to some sentinel value that tells FSRS to set it based on the first review?

[brishtibheja]

1. The interval a user inputs in Set due date might not be a reliable indicator of initial stability. The suggestion here was to treat those cards as cards graded Good.
2. This was never supported before but we've got some people now who use this workflow. My whole argument is why support this now if we never did before in SM-2? So I agree with you.

[user1823]

If we set the initial stability at set-due-date time

As brishtibheja said, the interval put by the user is not a reliable estimate of the initial stability. Also, treating the set due date as a Good entry won't solve the problem of intervals going backward on subsequent review because the S0(3) would likely be less than the interval put by the user in Set Due Date.

But anyway, there is no better option than using the interval entered by the user to estimate the initial stability (using memory_state_from_sm2). So, I think we need a revert of ankitects/anki@db93939 (It might be a good idea to retain the change from found_learning to revlogs_complete though.)

The problem:
Reverting the commit will reintroduce the problem faced by users who use Set Due Date on new cards without knowing the card. This brings me to the following point.

The second one surprises me. When we were SM2-only, this wasn't possible, was it? Why is it suddenly a use case we need to support?

In SM-2, Anki simply multiplies the current interval with the ease factor (on pressing Good). However, in FSRS, we need the complete review history (or use a hack if it's unavailable) to calculate the memory states.

Because these people used small intervals with Set Due Date (see examples below), they didn't have problems when using SM-2 even if Anki didn't treat the card as a new card.

However, before ankitects/anki@db93939, FSRS used memory_state_from_sm2 on these cards, causing these cards to get a high difficulty.

In #557 (comment), I was probably mistaken to believe that this high difficulty is not a problem. Surely, FSRS has a function to decrease the difficulty when the card is answered correctly multiple times. However, the decrease in difficulty is slower than desired because the parameters are trained on "normal" revlogs, which don't have this problem of high initial estimates of difficulty.

I don't have bright ideas to solve this problem. So, suggestions are welcome.

So, in summary, my suggestion is to revert ankitects/anki@db93939 while retaining the change of name from found_learning to revlogs_complete. Although this reintroduces the problem faced by some users, we don't have a better option yet.

PS:
I do have a solution for a second case, but it's not ideal. We can add an option to the Helper add-on that allows changing the old revlogs so that they are interpreted correctly by FSRS. Specifically, the option will change the review entry following the first Manual entry (only if it is the very first revlog entry) into a Learn entry. After the user "corrects" their old revlogs, they will need to change their habits and avoid using Set Due Date on new cards.

[brishtibheja]

I like your solution, the survey Expertium did also shows us that Set due date don't have a lot of use cases for the average user. Expecting people in the 1st group to be able to use the Helper add-on to solve this issue isn't too demanding in my opinion.

[L-M-Sherlock]

Also, treating the set due date as a Good entry won't solve the problem of intervals going backward on subsequent review because the S0(3) would likely be less than the interval put by the user in Set Due Date.

Is it a necessary problem to solve? If the user overestimates their memory, their true retention will be lower than desired retention significantly. If the next interval doesn't go backwards, the true retention will never reach the desired retention.

However, it's also a problem if we treat set due date as a Good entry when the user actually forgot the card and use set due date to shorten the interval.

[user1823]

Is it a necessary problem to solve?

Imo, yes. We have to consider why the user is using Set Due Date. In this case, they have already learnt the card outside Anki and don't want to waste time reviewing it in Anki at short intervals. That's why they are assigning large intervals to the card using Set Due Date.

If the user overestimates their memory

It is possible that the user may overestimate their memory, but I think calculating the stability from interval entered by the user is still a better option than making the card go through several short intervals, which will make Set Due Date useless.

You might say that the optimizer can learn the patterns and cause FSRS to schedule longer intervals. But, I think that these cards should not be used for optimization because the optimizer will get spuriously high estimates for S0 in that case. (These cards have a high initial stability but ALL cards of the user don't have such high initial stabilities.)

it's also a problem if we treat set due date as a Good entry when the user actually forgot the card

This problem can be avoided if give such treatment to only those Set Due Date entries that are the first revlog entry for a particular card. (But I still don't recommend treating SDD as Good as I have explained above.)

[dae]

Haven't had a chance to follow the latest posts closely yet, but just a brief comment: remember 'set due date' supports 'n' and 'n!' modes, where the latter both changes the due date and the interval. For new cards, it is always the latter - setting an initial interval means the card should receive future scheduling based on that interval. For the 'forgot the card and manually sets a lower interval' case (which IMHO users should be doing anyway), the card won't be new, so by default the interval would not change.

[brishtibheja]

I mostly agree with user1823, but we recently had a user who consistently use SDD on part of their deck but not the other. In this case, should we really not optimise params on probably what is half of their deck?

If you grade these cards when using SDD, you can avoid these problems in the long run. I think you can do that by always clicking Easy on the SDD cards and graduate your normal cards with Good. By the bye, we can also a add a new parameter that you can call SDD initial stability, which in the long run should work fine.

[user1823]

we recently had a user who consistently use SDD on part of their deck but not the other. In this case, should we really not optimise params on probably what is half of their deck?

Well, there can be two kinds of people who use SDD on new cards to assign them large initial intervals.

1. They use Easy on new cards that are slightly easier than normal cards and SDD on new cards that are much easier (because they learnt them outside Anki) than normal cards.
2. They don't use Easy but use SDD on easy cards (presumably because the Easy interval wasn't long enough).

For people in group 2, we can make FSRS treat SDD on new cards as Easy (instead of using memory_state_from_sm2). This would work fine because the optimizer would learn from the past data and produce a high value of S0(4) as the user desires.

However, this would not work well for people in group 1 because FSRS won't be able to distinguish between slightly easier and much easier cards.

If we want to be conservative, we should use memory_state_from_sm2. This works reasonably well for both the groups.

(To clarify, memory_state_from_sm2 is not good enough for people who use SDD to spread their new cards over the next few days. We need a better solution for them or tell them to convert their first Review entries into Learn entries as I said in the last part of #675 (comment))

If people in group 2 want better results, they can convert their SDD revlog entries into Easy either by using the Debug Console or by using an add-on.

we can also a add a new parameter that you can call SDD initial stability

This seems to be an overkill because only a few people use SDD on new cards.

[bqv]

Alternative solution from discord, if adding a grade to set due date isn't viable; split it into two options, one for each group's use case. That would surely alleviate the situation?

[Expertium]

This debate has been going on for months, with no conclusion. Frankly, this is probably the last time I'll comment on this issue.
Just use memory_state_from_sm2. It's clear that we won't be able to make SDD work for everyone, as people use it in completely different ways. memory_state_from_sm2 seems like the most sane choice.
Adding more add-on options is not good, because the number of users who will use them is lower than the number of users who use FSRS Helper, which, in turn, is lower than the number of people who use FSRS. So only a tiny minority will use a new add-on option.

[brishtibheja]

I think there's agreement on what to do, apart from some issues that may not get solved. For those, I think the documentation on SDD should change a bit and we just expect people to make behavioural changes.