Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Missing Commands in StackExchange.Redis #2055

Open
slorello89 opened this issue Mar 25, 2022 · 25 comments
Open

Missing Commands in StackExchange.Redis #2055

slorello89 opened this issue Mar 25, 2022 · 25 comments

Comments

@slorello89
Copy link
Collaborator

slorello89 commented Mar 25, 2022

Hi @NickCraver @mgravell , as we discussed a few weeks ago (with @chayim and @gkorland) I went through the library and compiled a list of what I believe are the missing commands of StackExchange.Redis.

Methodology

I basically diffed RedisCommand.cs with the main Redis Project's commands.json, which resulted in ~180 commands, I went through and purged most of the commands that felt like they were admin/config (except ACL) to preserve the more interactive commands, this left ~70. Then I went through the library to see if the sub-commands that came up (a lot of stuff from XGROUP XINFO and a few others) were already covered by a combination of the command Enum + a literal to filter those out as they are already supported. After that, and collapsing the many sub-commands of FUNCTION and ACL, we're left with 47. I think what follows is mostly correct (in that the thing is ACTUALLY missing, and that it's something that you would probably want to add to the library if it were feasible). I marked the blocking commands, and then applied a bit of my own reason to determine if I thought the command would be addable without a major overhaul of the library. I also added the Redis Version the command was added in (about 40% of the missing commands are 7.0, I don't know what your policy is on adding Release candidate commands, I'm guessing you wouldn't want to :) )

As it stands, there are about 21 commands which are in stable versions of Redis, interactive, and shouldn't require any major architectural changes, call them low-hanging fruit. Seems like there are a few commands that were either just missed, or there's some story behind why they never made it in (BITFIELD, some OBJECT subcommand, SCRIPT KILL, LOLWUT (ok maybe a bit silly, but can always use a bit of levity)). And then I'm guessing there hasn't been a command refresh since 6.0

This is meant as a starting point, for my part, I'm happy to start opening PRs for any commands you'd approve of adding. I understand the reviews can be burdensome and do not want to overwhelm you, so just let me know what you want to be added and I'm happy to get started.

Full List

All the missing commands, including 7.0 commands and things that are going to be a bigger headache to add.

Command Name PRs Interface Proposed names Notes
ACL (+sub-commands) 6 (TBD) Skip: Broader discussion
BITFIELD #2107 3.2 IDatabase StringBitFieldSet/StringBitFieldGet/StringBitFieldIncrement/StringBitField This API might be tricky, the command is variadic, so an array of sub-commands may make sense, or the three sub-commands split out, or both could work. Also, might want a clever return-type.
BITFIELD_RO #2107 6.2 IDatabase see notes Consider intelligently interrogating the commands executed to see if they are RO.
BLMOVE 6.2 IDatabase Skip: Blocking
BLMPOP 7 IDatabase Skip: Blocking
BZMPOP 7 IDatabase Skip: Blocking
BZPOPMAX 5 IDatabase Skip: Blocking
BZPOPMIN 5 IDatabase Skip: Blocking
COPY #2064 6.2 IDatabase KeyCopy
EVALSHA_RO 7 IDatabase ScriptEvaluateReadOnly
EVAL_RO 7 IDatabase ScriptEvaluateReadOnly
EXPIRETIME #2083 7 IDatabase KeyExpireTime
FCALL 7 IDatabase Skip: Pending functions design
FCALL_RO 7 IDatabase Skip: Pending functions design
FUNCTION (+sub-commands) 7 IDatabase Skip: Pending functions design
GEOSEARCH #2089 6.2 IDatabase GeoSearch
GEOSEARCHSTORE #2089 6.2 IDatabase GeoSearchAndStore
GETEX #1743 IDatabase StringGetSetExpiry
HRANDFIELD #2090 6.2 IDatabase HashRandomField
LCS #2104 7 IDatabase LongestCommonSubsequence
LMOVE #2065 6.2 IDatabase ListMove
LMPOP #2094 7 IDatabase Overload ListLeftPop and ListRightPop
LPOS #2080 6 IDatabase ListPosition/ListPositions
OBJECT ENCODING #2088 2.2 IDatabase KeyEncoding
OBJECT FREQ #2105 4 IDatabase KeyFrequency This can only be used if maxmemory-policy is set to an LFU policy, might be worth some command-map manipulation to interrogate this policy.
OBJECT REFCOUNT #2087 2.2 IDatabase KeyRefCount
PEXPIRETIME #2083 7 IDatabase KeyExpireTime Overload that specifies to get the expire-time in milliseconds
PUBSUB SHARDCHANNELS 7 IServer SubscriptionShardChannels
PUBSUB SHARDNUMSUB 7 IServer SubscriptionShardSubscriberCount
SCRIPT KILL 2.6 IServer ScriptKill
SINTERCARD #2078 7 IDatabase SetIntersectionLength This could also be SetCombineLength, I worry though because there's no corresponding union/diff commands yet that this can lead to some confusion.
SMISMEMBER #2077 6.2 IDatabase Overload SetContains
SORT_RO #2111 7 IDatabase SortReadOnly Another option is to just change the Sort method to use the SORT_RO, but I fear that's a break (and there would need to be some other logic to determine if SORT_RO is available on the Redis Server
SPUBLISH 7 (TBD) Skip: Pending sharded pub-sub design
SSUBSCRIBE 7 (TBD) Skip: Pending sharded pub-sub design
SUNSUBSCRIBE 7 (TBD) Skip: Pending sharded pub-sub design
WAIT 3 (TBD) Skip: Blocking
XAUTOCLAIM #2095 6.2 IDatabase StreamAutoClaim
ZDIFF #2075 6.2 IDatabase SortedSetCombine
ZDIFFSTORE #2075 6.2 IDatabase SortedSetCombine
ZINTER #2075 6.2 IDatabase SortedSetCombine
ZINTERCARD #2075 7 IDatabase SortedSetIntersectionLength Same comment as with SINTERLENGTH
ZMPOP #2094 7 IDatabase Overload SortedSetPop
ZMSCORE #2082 6.2 IDatabase Overload SortedSetScore
ZRANDMEMBER #2076 6.2 IDatabase SortedSetRandomMember
ZUNION #2075 6.2 IDatabase SortedSetCombine
@NickCraver
Copy link
Collaborator

Hey Steve! This is awesome. My thoughts are:

  1. Yes, I'd like to add these and would welcome PRs.
  2. I'd like to ideally get Nullable Reference Types for main library #2041 in first because it gives us a lot of checks in the compiler for free and in applying it, I recognize a lot of missteps we allowed in result processors e.g. with streams that would have been caught.
  3. Working cross-fork is messy, I'd rather add you as a contributor here (main branch is protected, so no one can mess it up without a PR/check already).

I don't think ground rules for these are an issue, your first PR was fantastic. I propose that we pick a branch naming scheme (e.g. feature/<command>), and ideally lay all of these our as method names since we have the list (to ensure they make sense as a whole). In the past, we added as Redis did and it's a bit organic and not everything matches - IMO it'd be a good exercise to lay these out as the return type and method name(s). This helps because when the return types match for 2 very similar commands, sometimes it's better as an argument. If the return types are different (e.g. long for count vs. actual results), it's obvious they'll never be the same method, and both help inform the naming.

I'm happy if we want to setup a call and go through these or whatever you think works to get those down. IMO the naming is worth an hour to go through ahead of time so we're not figuring it out and conflicts/etc. each PR - much less time overall :)

Thoughts?

@NickCraver
Copy link
Collaborator

Updating #1743 to get GETEX off the list here :)

@slorello89
Copy link
Collaborator Author

Hey @NickCraver , totally agree, let's come to an agreement on what the API will look like before we get underway. Can set up a call Next week or early (Mon/Tues) the week after.

@slorello89
Copy link
Collaborator Author

@NickCraver, Striking GETEX given #1743 has been merged.

@NickCraver
Copy link
Collaborator

@slorello89 hmm, at random I went to https://redis.io/commands/zunion/ - the results...aren't what I expect there in the console. General feedback though: the command interface on the new site is nice looking and has good info, but is far less usable. I'm having to go back and re-search when trying to find the command(s) I want, something that was much better in the previous interface. I'm not sure where to put that feedback, but wanted to pass it along!

@slorello89
Copy link
Collaborator Author

Hey @NickCraver, yeah, seeing some of the same with some other commands.

I believe redis-clinterwebz is the home of the app now. @itamarhaber would be the definitive voice on that. Will look at it tomorrow.

@slorello89
Copy link
Collaborator Author

slorello89 commented Apr 4, 2022

Hey @NickCraver RE laying out what the API will look like, I put together the following table (have a copy in sheets as well) as basically a jumping-off point to kick off this discussion. Happy to go back and forth asynchronously, happy to get on a call too (though I'm going to be out-of-pocket Wed-Fri this week.

EDIT

Moved updated table up to the full table in the first comment.

@ttingen
Copy link
Contributor

ttingen commented Apr 9, 2022

@slorello89 I'm happy to help with this list if you're interested. @NickCraver sorry for all the nulls you had to fix in streams for #2041, my fault... :/

@Avital-Fine
Copy link
Contributor

@slorello89 Did you mean SortedSetCombineAndStore for ZDIFFSTORE command?

@NickCraver
Copy link
Collaborator

@slorello89 I'm happy to help with this list if you're interested. @NickCraver sorry for all the nulls you had to fix in streams for #2041, my fault... :/

No worries at all! That was me being irked at me for not catching a few things in PR, we'll keep reving this thing better and better and I'm very, very appreciative of the help :) Now we've inboarded the tooling to help us all better as we add more commands and processors so yay. I'm super happy to find some badness in various places with the tooling lit up...give me decent confidence in the wins.

@NickCraver
Copy link
Collaborator

@slorello89 I went through and updated the list with PR pointers - recommend we collapse your later comment with naming into the original because it's either method names or blocking/punt so that can be 1 column so we can more easily update things to track here - thoughts? Happy to collapse that if agreeable.

@slorello89
Copy link
Collaborator Author

Hey @NickCraver, I collapsed everything as requested, and noted the commands that need more consideration:

  1. ACLs
  2. Functions - the Redis 7 function feature
  3. Shard pub/sub - the Redis 7 Shard Pub-sub feature.

I think probably any command that I've named (with the exception of PUBSUB SHARDCHANNELS and PUBSUB SHARDSUBSCRIBERS) is probably fair game at this point @NickCraver?

I'd also note that we should be careful to separate anything going out in Redis 7, There's a case to be made for having those PRs ready to go (those features are increasingly set), but all the same I don't think any tests written against them are worth anything until Redis 7 is GA.

@Avital-Fine, @ttingen or anyone else working this, I'd say just shout out here what commands you want to work on before getting underway, that way there's no duplication of effort.

I'm going to start on the following today:

  • GEOSEARCH
  • GEOSEARCHSTORE
  • HRANDFIELD
  • LPOS

PS I met @ttingen at Codestock last week - was nice catching up, he was super excited to hear about the work happening with the library :)

@Avital-Fine
Copy link
Contributor

I will start now with EXPIRETIME and PEXPIRETIME commands.
Also, I know that there are new features to EXPIRE, PEXPIRE, EXPIREAT, PEXPIREAT so I'll check it out :)

@Avital-Fine
Copy link
Contributor

Btw, after we finish with the commands we need to check for missing features.
For example, I notice that SortedSetAdd (ZADD) doesn't support GT and LT options which were added in 6.2.0

@NickCraver
Copy link
Collaborator

Heads up: it occurred to me we're not actually testing 7.0-rc bits because of the Docker container version. I remedied that in #2079 which should be followed by a merge of main into any PRs with 7.0 features so we're properly exercising those paths :)

NickCraver added a commit that referenced this issue Apr 11, 2022
@ttingen
Copy link
Contributor

ttingen commented Apr 12, 2022

I will start with the following:
XAUTOCLAIM
ZRANDMEMBER

@NickCraver
Copy link
Collaborator

@ttingen ZRANDMEMBER is up in #2076 ;)

@ttingen
Copy link
Contributor

ttingen commented Apr 12, 2022

@ttingen ZRANDMEMBER is up in #2076 ;)

Oops, let's make that XAUTOCLAIM and ZMSCORE.

NickCraver added a commit that referenced this issue Apr 14, 2022
Implements [`HRANDFIELD`](https://redis.io/commands/hrandfield/) as a part of #2055.

Co-authored-by: Nick Craver <[email protected]>
NickCraver added a commit that referenced this issue Apr 14, 2022
@Avital-Fine
Copy link
Contributor

Starting working on LCS

@ttingen
Copy link
Contributor

ttingen commented Apr 14, 2022

I have XAUTOCLAIM working, just need to write more tests. Should be able to get it out for review tonight.

NickCraver added a commit that referenced this issue Apr 16, 2022
This PR implements [`GEOSEARCH`](https://redis.io/commands/geosearch/) and [`GEOSEARCHSTORE`](https://redis.io/commands/geosearchstore/) for #2055 

To abstract the box/circle sub-options from GEOSEARCH I added a new abstract class `GeoSearchShape` who's children are responsible for maintaining the bounding shape, its unit of measurement, the number of arguments required for the sub-option, and of course the sub-option name. Rather than casting/extracting the arguments, I have it use an IEnumerable state-machine with yield/return. Wasn't sure which was the better option, the IEnumerable seemed cleaner, open to whichever you want.

I changed the `GEORADIUS` pattern of having a `GeoSearch(key, member, args. . .)` and a `GeoSearch(key, lon, lat, args. . .)`, and instead have `GeoSearchByMember` and `GeoSearchByCoordinates`. If I'm honest, it was because my IDE was complaining about breaking compatibility rules by having more than 1 override with optional parameters, not sure if you have a strong feeling on this.

Co-authored-by: Nick Craver <[email protected]>
This was referenced Apr 19, 2022
NickCraver added a commit that referenced this issue Apr 19, 2022
Added [XAUTOCLAIM](https://redis.io/commands/xautoclaim/) support as part of #2055 

The XCLAIM command has two methods, StreamClaim & StreamClaimIdsOnly. Since the XAUTOCLAIM result is a bit more complex than the XCLAIM command, I opted to consolidate to a single method and single result type. 

To return just the message IDs in the result, the `idsOnly` parameter should be set to `true`. The caveat here is that the `StreamEntry` instances will only have `Id` populated when `idsOnly` is `true`, the `Values` array will be empty. This behavior is called out in the XML docs for the method. 

I wasn't sure if this is the proper "feel" you want for this command or not. Let me know if you want to break it up like `StreamClaim`.

Co-authored-by: Nick Craver <[email protected]>
NickCraver added a commit that referenced this issue Apr 19, 2022
Implementing [`ZMPOP`](https://redis.io/commands/zmpop/) and [`LMPOP`](https://redis.io/commands/lmpop/) as part of #2055 

Co-authored-by: Nick Craver <[email protected]>
NickCraver added a commit that referenced this issue Apr 19, 2022
NickCraver pushed a commit that referenced this issue Apr 20, 2022
Introducing [`SORT_RO`](https://redis.io/commands/sort_ro/) as part of #2055 

@NickCraver - I modified the path for message creation for the Sort command to point to using `SORT_RO` when possible, this again had to do a version-check against the multiplexer, which I'm not certain is the correct way - thoughts?

Also, SORT is considered a write command and will be rejected out of hand by a replica (so I'm moving that as well)

```text
127.0.0.1:6378> SORT test
(error) READONLY You can't write against a read only replica.
```
Co-authored-by: Nick Craver <[email protected]>
NickCraver added a commit that referenced this issue May 16, 2022
Adds support for https://redis.io/commands/lcs/ (#2055)

Co-authored-by: slorello89 <[email protected]>
Co-authored-by: Nick Craver <[email protected]>
NickCraver added a commit that referenced this issue May 24, 2022
Add support for `COMMAND` commands (part of #2055):
COMMAND COUNT - https://redis.io/commands/command-count/
COMMAND GETKEYS - https://redis.io/commands/command-getkeys/
COMMAND LIST - https://redis.io/commands/command-list/


Co-authored-by: Nick Craver <[email protected]>
@tyler-boyd
Copy link

re: sharded pub/sub, is it as simple as changing PUBLISH commands to SPUBLISH etc? Or is there some drawback to the sharded version that requires more thought?

@mgravell
Copy link
Collaborator

IIRC we already have equivalent routing code, so it shouldn't be a huge change - however, there are non-trivial validations needed here, to ensure that we route correctly for both initial and reconnect scenarios, and to check the routing isn't impacted for non-routed subscriptions.

@caoyang1024
Copy link

Hi, can I check the plan on commands (SPUBLISH, SSUBSCRIBE, SUNSUBSCRIBE) for sharded pub-sub.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

7 participants