What is the "external update" meant to be for moving Pending -> available/scheduled?

[travisby]

Hey folks,

I have a question about some nitty gritty wording in the documentation, and what a potential way forward for my architecture might be.

tl;dr: InsertOpts.Pending says "... scheduled by an external update", what is that?

---

For my usecase, I would like to "stage" a job (set the args, get a job ID). And then on user interaction (a webhook, containing the job ID), move the job from pending to available (or possibly even scheduled).

The actual architecture I'm hoping to achieve here looks a bit like this:

sequenceDiagram
    river->>+Worker[JobA]: Work(ctx, JobA{})
    Worker[JobA]->>river : Insert(ctx, JobB{}, InsertOpts{Pending: true})
    river->>Worker[JobA] : JobB{ID: 43, State: "pending"}
   Worker[JobA]->>externalService : Do some work, and then call back to me with "Job ID 43" 
   Worker[JobA]->>-river: JobComplete
   externalService->>internalService : I'm complete, this was for Job ID 43
   internalService->>river : Schedule JobB{ID:43}

Loading

(There's some more nuanced situations where that wouldn't even necessarily be "run JobB{ID:43} right now", but could be "run JobB{ID:43} in ten minutes", but that's minor in comparison I think)

---

From the go docs, the comment I'm particularly highlighting:

 go doc github.com/riverqueue/river InsertOpts
package river // import "github.com/riverqueue/river"

type InsertOpts struct {
.....
	// Pending indicates that the job should be inserted in the `pending` state.
	// Pending jobs are not immediately available to be worked and are never
	// deleted, but they can be used to indicate work which should be performed in
	// the future once they are made available (or scheduled) by some external
	// update.
	Pending bool
....

What is intended for that "external update?" Is that something that can/should be done with the river API, or is that running UPDATE river_job SET state="available" WHERE id=??

Would running queries like that be "relatively stable" of an API? What about scheduling? Do I need to make sure to set both scheduled_at and state, is there any other fields, etc?

---

Some prior dicussions that have had workaround answers, but I think haven't specifically answered this one :)

• Make pending jobs scheduled #393 --> the wording of the question matches pretty exactly, but part of the first answer is the same wording I'm asking about :). The full answer/workaround talks more about ordering of sub-jobs, which I think matters less in my case (and also sounds like a great usecase for riverpro!)
• Updating job metadata while running #164 --> talks about using queries to manipulate river for "immutable" properties.
• REST API for Scheduling Jobs and HTTP Request Jobs #115 --> this mentions a similar architecture, and that HTTP breaks the transaction consistency. That's true, but unfortunately in this case it's an externalSystem that I can't bring into the fold of our river DB :).

--

What I haven't tried:

• Client.JobRetry -- does "Retrying" a pending job move it into the available state? Reading https://github.com/riverqueue/river/blob/master/riverdriver/riverpgxv5/internal/dbsqlc/river_job.sql.go#L865-L894 it looks like it, but I'm not sure it's part of the API contract that it will always work.
• Creating a separate table and storing the "to-be" job args along with a callback ID (that is not a JobID), and instead creating the job on callback with Pending: false --> this is an option, but one of last resort if I can shoehorn everything into river ;P.

---

I'd love if there's something I'm missing! If not, a way to expose JobSnooze() outside of the Work() function would be lit. My first guess was actually to see if Client.JobCancel supported passing an error, and if either river.JobCancel or river.JobSnooze would work in that.

[bgentry]

Hi @travisby, I think you’ve mostly grasped the pending state based on your prior comments. It’s a place to stage jobs that will never be worked, not until some piece of logic advances those jobs into the available or scheduled state.

As for what that logic is, it’s really up to you! Your simple query to move the job into available is indeed a way to do this given you already know the job’s ID, though I would caution you to add a WHERE state = ‘pending’ clause to it just to make sure you don’t accidentally mess with a job that already ran or is running.

A more advanced usage would be for something like “I don’t want this job to run before an external event happens, and I also don’t want it to run before X time”. In that case you would want to be sure that your update query puts jobs into the scheduled state if their scheduled_at is still in the future.

I hope that helps confirm your findings!

[travisby]

Thank you, it does!

I wanted to cover:

1. That the two fields were all I needed to adjust (state + scheduled_at for scheduled (nit: I also know we can use available if the timeline is shorter than the schedule latency), or just state for available)
2. That the schema is relatively stable that using outside queries are 👍 , and that there's no other hidden methods that river expects to be called in-go, everything relying on just postgres for signaling is great :D

And your answer covered those!

[nexovec]

Are you considering an API for doing stuff like this?

[bgentry]

@nexovec not currently, I’m not sure how that could be generalized. It’s an advanced use case and most of the time you will probably want to write your own queries for it.

[travisby]

I started reaching for my own queries (especially after seeing sqlc and how river used it... neat tool!) but by reading river source, I opted for what river does.

There is... sorta an API available. It's technically a public method but I imagine with the same guarantees as rivershared ("DO NOT USE THIS API CAN CHANGE AT ANY TIME" type warning)

	t := time.Now()
	if _, err := riverpgxv5.New(nil).UnwrapExecutor(tx).JobUpdate(ctx, &riverdriver.JobUpdateParams{
		ID:                  jobrow.ID,
		FinalizedAtDoUpdate: true,
		FinalizedAt:         &t,
		StateDoUpdate:       true,
		State:               rivertype.JobStateCompleted,
	}); err != nil {
		return nil, fmt.Errorf("%w while trying to set status to complete", err)
	}

worked like a charm. I imagine it would also work for scheduled at.

This, however does miss out on the WHERE state="pending" suggestion, so it's not quite as safe as a more dedicated query, or a first-class API from river, but it works (at a use at your own risk level, I think).

[bgentry]

I would highly recommend against using that one for the stated reason: it can and will change out from under you without warning and is not considered part of our API guarantee.

Again you will really want to include the pending clause too, otherwise you risk breaking your client in unpredictable ways.