Add `counts` and `add_new_item` methods, without tests #4

Oxygen588 · 2023-07-03T11:31:24Z

Close #3 , Close #2 ;
Modified the addItem method to not allow duplicate jobs to be added.
Created a new method that gets the length atomically.

Implemented no duplicate items based on id (A update would be creating the id based on the data hash, as well as currently within some languages add item method getting the list of item could be done using pipeline) Implemented tests for each one of the languages for the no duplicate items. Fixed some issues related to run bash file. Within Job spawner modified the way shared jobs work so its not within the main list as well as fixed the way the host and port is being set. Python test fixed the way the host and port is being set.

AddAtomicItem method was fixed by using transactions instead of pipeline. The normal AddItem was added back.

GetQueueLengthsAtomic function as well as the AddAtomicItem function has been created for the atomic way of adding items, and the normal AddItem function has been added back.

Change in name for the atomic tests function call

Moved the start of job spawner before all the tests so there is no conflict between the new tests.

Old Add_Item function has been added back, the atomic add_item is now add_atomic_item and has been fixed to run atomically as well as the get_queue_lengths in now atomic fixed as well.

The add_atomic_item has been fixed now, the add_item function has been added back, The tests for duplicate items has been fixed.

Small update to tests, forgot to uncomment the add of the second item.

go/WorkQueue.go

node/src/WorkQueue.ts

Removed non Atomic way of getting queue length and renamed the GetQueueLengthsAtomic to GetQueueLengths

Watch atomicity implemented within go-lang.

AddAtomicItem Fix! Added warning message.

The atomic function fully works and tests have been modified to support locking!

Small fix.

go/WorkQueue.go

JOT85 · 2023-07-04T21:10:10Z

Sorry I didn't have your latest commit when writing that last review. I'll update my comments.

go/WorkQueue.go

TS addAtomicItem fix using watch.

'AddAtomicItem Rename' to 'AddItemAtomically'. Re-written the comment for AddItemAtomically.

Better Watch for loop.

AddItemAtomically: removed unnecessary pipe. Improved the return of the lengths function.

Removed the limit of tries until it stops trying.

The new fix implements the redis watch. This should make it fully atomic.

go/WorkQueue.go

rust/src/lib.rs

go/WorkQueue.go

node/src/WorkQueue.ts

python/redis_work_queue/workqueue.py

… fixed return at Lengths function Go Unexecuted pipe fix. addAtomicItem Pipes fixed and executed properly as well as necessary checks for transaction errors or generic errors has been fixed

Fixed unexecuted pipe and properly checked the pipe for items already in list.

node/src/WorkQueue.ts

go/WorkQueue.go

rust/src/lib.rs

tests/rust/src/main.rs

tests/run-test.sh

…ntations: - Go: return boolean from `WorkQueue.AddNewItem`

…k-queue into Atomic-length-method

tests/rust/src/main.rs

JOT85 · 2023-09-26T10:14:56Z

tests/run-test.sh

@@ -42,6 +42,7 @@ while [[ $# -gt 0 ]]; do
  esac
 done

+python3 job-spawner-and-cleaner.py "$host" "$tests" &


Is this intentional?

How come this moved from line 108 to line 45?

rust/src/lib.rs

python/redis_work_queue/workqueue.py

Python: Fixed Add_new_item, fixed exception handling. Rust: Made `get_queue_lengths` not async, now returns future. Test: There was no need for `python3 job-spawner-and-cleaner.py "$host" "$tests"` it was already running at line 45.

tests/rust/Cargo.toml

tests/rust/src/main.rs

JOT85 · 2023-09-27T08:50:57Z

tests/run-test.sh

@@ -42,6 +42,7 @@ while [[ $# -gt 0 ]]; do
  esac
 done

+python3 job-spawner-and-cleaner.py "$host" "$tests" &


How come this moved from line 108 to line 45?

tests/job-spawner-and-cleaner.py

tests/go/main.go

Re-written the way tests are done for `addNewItem` now 3 functions that spawn 10 `addNewItem` calls for the item we are trying to add and runned concurrently. TODO: Currently because of all added logic the difference in speed in between go and node & python, the difference in speed is much higher because of that the difference in number of shared-jobs done is much higher.

go/WorkQueue.go

node/src/WorkQueue.ts

python/redis_work_queue/workqueue.py

rust/src/lib.rs

Moved `addNewItem` with sleep for tests, from the lib to the tests.

…atting According to the docs at https://pkg.go.dev/github.com/redis/go-redis/v9#Client.Watch, we don't need to `Exec` the pipeline. Furthermore, the redis library ensures the database is safe to use multiple transactions concurrently, so I've removed that from the doc comment.

python/redis_work_queue/workqueue.py

JOT85 · 2023-10-05T10:30:52Z

go/WorkQueue.go

+				workQueue.AddItemToPipeline(ctx, pipe, item)
+				return nil
+			})
+			return err
+		} else {
+			return nil
+		}
+	}
+
+	for {
+		err := db.Watch(ctx, txf, workQueue.processingKey, workQueue.mainQueueKey)
+		if err == nil {
+			return true, nil
+		}


This returns true even if the item isn't added!!

JOT85 · 2023-10-05T10:35:42Z

node/src/WorkQueue.ts

+
+        const transaction = db.multi();
+        this.addItemToPipeline(transaction, item);
+        const results = await transaction.exec();


@Oxygen588 what does the return value look like in the success and failure case? I'm not sure what it would be, so it would be nice to have a comment!

Sorry, to clarify what I mean, I don't know what the value of results if the transaction succeeds or fails.

JOT85 · 2023-10-11T08:17:44Z

go/WorkQueue.go

@@ -151,7 +151,7 @@ func (workQueue *WorkQueue) AddNewItem(
 			})
 			return err
 		} else {
-			return nil
+			return false


I'm not sure what's going on here? 😂

…wItem Also format

JOT85

Still needs tests, but we'll merge it into atomic-methods and add tests on that branch.

Oxygen588 and others added 10 commits June 30, 2023 11:16

Remove tests/go/go binary

f4bd9c2

Merge branch 'JavaScript-Implementation' into Atomic-length-method

28cd1c9

Update WorkQueue.ts

b101df6

AddAtomicItem method was fixed by using transactions instead of pipeline. The normal AddItem was added back.

Atomic functions fix.

19989b1

GetQueueLengthsAtomic function as well as the AddAtomicItem function has been created for the atomic way of adding items, and the normal AddItem function has been added back.

Update index.ts

50069e9

Change in name for the atomic tests function call

Update run-test.sh

8abf3ed

Moved the start of job spawner before all the tests so there is no conflict between the new tests.

Atomic lengths Python Fix.

1aa44b3

Old Add_Item function has been added back, the atomic add_item is now add_atomic_item and has been fixed to run atomically as well as the get_queue_lengths in now atomic fixed as well.

Rust-lang Atomicity functions fix

61ac4a2

The add_atomic_item has been fixed now, the add_item function has been added back, The tests for duplicate items has been fixed.

Update main.rs

67b5c70

Small update to tests, forgot to uncomment the add of the second item.

JOT85 requested changes Jul 4, 2023

View reviewed changes

go/WorkQueue.go Outdated Show resolved Hide resolved

go/WorkQueue.go Outdated Show resolved Hide resolved

go/WorkQueue.go Outdated Show resolved Hide resolved

node/src/WorkQueue.ts Outdated Show resolved Hide resolved

Oxygen588 added 5 commits July 4, 2023 13:33

Update WorkQueue.go

0255224

Removed non Atomic way of getting queue length and renamed the GetQueueLengthsAtomic to GetQueueLengths

Update WorkQueue.go

1d8fe42

Watch atomicity implemented within go-lang.

Update WorkQueue.go

dc99a48

AddAtomicItem Fix! Added warning message.

Atomicity fully fixed + tests fixed.

08c47ca

The atomic function fully works and tests have been modified to support locking!

Update WorkQueue.go

5bf622e

Small fix.

JOT85 requested changes Jul 4, 2023

View reviewed changes

go/WorkQueue.go Outdated Show resolved Hide resolved

go/WorkQueue.go Outdated Show resolved Hide resolved

go/WorkQueue.go Outdated Show resolved Hide resolved

JOT85 reviewed Jul 4, 2023

View reviewed changes

go/WorkQueue.go Outdated Show resolved Hide resolved

Oxygen588 added 6 commits July 4, 2023 22:31

Update WorkQueue.ts

d139f55

TS addAtomicItem fix using watch.

AddAtomicItem Rename + Better Doc Comment

7c5fc9b

'AddAtomicItem Rename' to 'AddItemAtomically'. Re-written the comment for AddItemAtomically.

Update WorkQueue.go

e6db6b1

Better Watch for loop.

AddItemAtomically, Lengths small changes

0dfd47a

AddItemAtomically: removed unnecessary pipe. Improved the return of the lengths function.

Update WorkQueue.ts

292f428

Removed the limit of tries until it stops trying.

Fixed add_item_atomically.

2529278

The new fix implements the redis watch. This should make it fully atomic.

JOT85 requested changes Jul 5, 2023

View reviewed changes

go/WorkQueue.go Outdated Show resolved Hide resolved

go/WorkQueue.go Outdated Show resolved Hide resolved

rust/src/lib.rs Outdated Show resolved Hide resolved

rust/src/lib.rs Outdated Show resolved Hide resolved

rust/src/lib.rs Outdated Show resolved Hide resolved

JOT85 requested changes Jul 5, 2023

View reviewed changes

Oxygen588 added 2 commits July 5, 2023 13:59

Unexecuted pipe fix Ts, necessary fixes done within addAtomicItem Ts,…

f07d603

… fixed return at Lengths function Go Unexecuted pipe fix. addAtomicItem Pipes fixed and executed properly as well as necessary checks for transaction errors or generic errors has been fixed

Update workqueue.py

d804ee0

Fixed unexecuted pipe and properly checked the pipe for items already in list.

JOT85 requested changes Jul 5, 2023

View reviewed changes

node/src/WorkQueue.ts Outdated Show resolved Hide resolved

node/src/WorkQueue.ts Outdated Show resolved Hide resolved

Delete redis-work-queue.sln

8523175

JOT85 requested changes Sep 25, 2023

View reviewed changes

Oxygen588 and others added 3 commits September 26, 2023 10:29

Necessary fixes.

61a5052

Update docs, rename Lengths -> Counts and small tweaks to impleme…

612b776

…ntations: - Go: return boolean from `WorkQueue.AddNewItem`

Merge branch 'Atomic-length-method' of github.com:Oxygen588/redis-wor…

0212ee8

…k-queue into Atomic-length-method

JOT85 requested changes Sep 26, 2023

View reviewed changes

Fixes.

14d9392

Python: Fixed Add_new_item, fixed exception handling. Rust: Made `get_queue_lengths` not async, now returns future. Test: There was no need for `python3 job-spawner-and-cleaner.py "$host" "$tests"` it was already running at line 45.

JOT85 requested changes Sep 27, 2023

View reviewed changes

JOT85 requested changes Sep 28, 2023

View reviewed changes

Oxygen588 and others added 5 commits September 29, 2023 10:10

Fixes.

522c5a6

Moved `addNewItem` with sleep for tests, from the lib to the tests.

rust: format and remove unused dependencies and imports

ccb6f6e

go: AddNewItem: remove retry limit

a953fce

go: Counts: format

9b4dfc5

JOT85 reviewed Oct 5, 2023

View reviewed changes

python/redis_work_queue/workqueue.py Outdated Show resolved Hide resolved

JOT85 added 4 commits October 5, 2023 11:29

python: format

07e11a3

node: AddNewItem: correct retry logic and remove logs

37478c6

node: format

5cf7f41

rust: rename get_queue_lengths -> counts and add docs

6d41f55

JOT85 requested changes Oct 5, 2023

View reviewed changes

Small fixes

4f5349d

JOT85 requested changes Oct 11, 2023

View reviewed changes

Restore tests to upstream

2b81138

JOT85 changed the title ~~Atomic length method & no duplicate Jobs~~ Add counts and add_new_item methods, without tests Oct 27, 2023

JOT85 changed the base branch from main to atomic-methods October 27, 2023 12:40

JOT85 added 2 commits October 27, 2023 13:40

Readd tests/node/package-lock.json

7a7294b

go: correctly return boolean indicating if an item was added in AddNe…

994b09f

…wItem Also format

JOT85 approved these changes Oct 27, 2023

View reviewed changes

JOT85 merged commit 11abffe into MeVitae:atomic-methods Oct 27, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add `counts` and `add_new_item` methods, without tests #4

Add `counts` and `add_new_item` methods, without tests #4

Oxygen588 commented Jul 3, 2023 •

edited

Loading

JOT85 commented Jul 4, 2023

JOT85 Sep 26, 2023

JOT85 Sep 27, 2023

JOT85 Sep 27, 2023

JOT85 Oct 5, 2023

JOT85 Oct 5, 2023

JOT85 Oct 11, 2023

JOT85 Oct 11, 2023

JOT85 left a comment

Add counts and add_new_item methods, without tests #4

Add counts and add_new_item methods, without tests #4

Conversation

Oxygen588 commented Jul 3, 2023 • edited Loading

JOT85 commented Jul 4, 2023

JOT85 Sep 26, 2023

Choose a reason for hiding this comment

JOT85 Sep 27, 2023

Choose a reason for hiding this comment

JOT85 Sep 27, 2023

Choose a reason for hiding this comment

JOT85 Oct 5, 2023

Choose a reason for hiding this comment

JOT85 Oct 5, 2023

Choose a reason for hiding this comment

JOT85 Oct 11, 2023

Choose a reason for hiding this comment

JOT85 Oct 11, 2023

Choose a reason for hiding this comment

JOT85 left a comment

Choose a reason for hiding this comment

Add `counts` and `add_new_item` methods, without tests #4

Add `counts` and `add_new_item` methods, without tests #4

Oxygen588 commented Jul 3, 2023 •

edited

Loading