Unable to catch error encountered when blocked by Yahoo for making too many requests #81
Comments
|
That's a helpful error! Hah! Should be a fairly error to catch. I can try to take a look tonight at this. |
|
Had a few minutes before calls this morning. The error is clearly |
|
I've been trying to get around this error too. Does anyone know what a) the rate limit is? and b) what to do when the limit is reached? I tried writing logic such that it attempted to refresh the token when it hit the limit, but it still kept giving me the request denied message. |
|
@ptoninato Here are my notes from a while ago very roughly determining the rate limit. Its very rough because I was making 120 calls at a time and did this twice, with failure happening on the second batch of calls. You could test this more exactly by making individual calls until you hit the limit and recording how many calls you made.
I did some tests and determined the API is blocked at an app registration level (you are identified and blocked based on the API keys you get from Yahoo after registering the app, rather than specific user's being blocked based on their access or refresh token) To get around this you could register multiple times to get extra sets of credentials, and switch between the credentials whenever you are blocked. This is against Yahoo ToS however. |
|
Sorry, it's been a tough month-plus for me here and I've been largely avoiding the machine at night. I did put a request in with my contact at Yahoo! to get a more definitive answer about the actual rates, but it's been radio silent. I will ping him again here now that football season is underway. I just pushed a potential fix to a branch called "over-limit-fix" -- can you test it out and let me know if that helps? If it does, I will merge it into the main branch this week. |
|
It sounds ominous when you call it "The Machine"! I will give it a test later tonight and get back to you. Hope your connect comes back from AWOL soon |
|
Hey Luke, I ran a test using over-limit-fix and am receiving the same result as before (no change). Let me know if I can help you test this again in the future. I can also send you my code to make bulk parallel requests to the players endpoint (easy to hit API limit this way) if you feel that would be helpful. |
|
Damn... If you want to send it to me I will take a peek! |
|
Here you go! This will collect all available players in a league. You may need to run it 2-3 times to hit the request limit. |
|
Thanks, I'll check this out. In the meantime, I got a response yesterday! "As far as rate limits, nothing published about it; we are kind of loose about app-based rate limits and instead enforce some general abuse-oriented rate limits. It's on the order of a couple thousand requests per hour, but don't have it handy now." So if that's something different than what you're seeing we could be running into a different issue... |
|
Interesting! Perhaps I am blocked not by making too many requests in general, but by making too many request simultaneously through my batching method. |
|
Hey @Liam-OShea - what was the time period for your 200 calls estimation? 1 second? 60 seconds? Something else? |
|
Probably a couple seconds, I would make a request and as soon as I received
the result make the next request. There may be different limits for
situations when you are making calls over a greater interval of time
…On Mon., Jul. 18, 2022, 12:21 p.m. Tyler Scott Williams, < ***@***.***> wrote:
Hey @Liam-OShea <https://github.com/Liam-OShea> - what was the time
period for your 200 calls estimation? 1 second? 60 seconds? Something else?
—
Reply to this email directly, view it on GitHub
<#81 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AG5E7BC6E3PVY3LE5SDZX33VUVZALANCNFSM5BEKBIQA>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
Hello everyone! So I am too hitting the rate limiting they have in place. Does anyone know if 1, this rate limiting can be increased, if at all? Cuz uh what I am trying to do would mean that I probably can't do it lol And two, how long does the rate limiting block last for? |
|
I found this in the ToS: How exactly do I "register the application" with them 🤔 |
|
Haha I found that too back in 2021, I couldn't find a way to reach out unfortunately :(
…On Wed, Jul 10, 2024, 9:08 p.m. Daniel Reguero ***@***.***> wrote:
I found this in the ToS:
If you wish to confirm that your application constitutes an acceptable use of the Yahoo Fantasy Sports APIs or wish to inquire about rate limit increases, register your application with us.
How exactly do I "register the application" with them 🤔
—
Reply to this email directly, view it on GitHub
<#81 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AG5E7BEY65PWLIZNVEUH2I3ZLXEHXAVCNFSM6AAAAABKV3F2H6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMRRG42DGNBYGI>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
I'm curious to know how other third party apps get around this limitation if at all if they're using the API. |
|
We basically rate-limit ourselves to 1 request/second or so. We still get rate-limited from time to time, probably due to their opaque anti-abuse measures, but overall 1 req/second seems to be pretty reliable. |
|
I see, I am batching about 200 requests. But if I have to send one request a second that now becomes a 3+ minutes to finish all the requests :( I suppose I could batch like 25 at a time and wait... |
Best I've found is reaching out on Reddit. They have a sub for YahooFantasy where they've answered questions before, but normally the answer is that they don't publicly support the API. @HappyZombies -- what are you doing, exactly? I've found a lot of the time I've queried resources multiple times and I can do it with a single call to collections... |
|
I'm trying to create an app / script that will give "awards" to players on our football fantasy league. My first test award is: “Awarded to the manager with the most points from the QB position during the season” So basically I am going through each team's roster for 17 weeks to get each QB that played each week and their total score, and accumulating it. This is a 12 person league so 12*17 = 204 requests I'm not near my computer but essentially each request url becomes something like this. |
|
Fun! 2 things: 1, are you using the API wrapper here? Or making the calls directly to the Yahoo! API? Based on you providing the URL, I am expecting the latter, but the wrapper is here to help :D 2, 204 requests really isn't that much... I don't understand why you'd be rate limited for that... I certainly make a lot of calls to the API and have only ever hit a rate limit when I'm making a lot of calls that I've eventually optimized. Based on what you're trying to do though, I would think that's probably the easiest way to accomplish it... maybe you could use the players collection to get all owned players with their stats and ownership for a given week, but that feels like more trouble than it's worth. If you want to share the code, I can take a look and run it against one of my test leagues to see if I can figure out where the issue might be? |
|
Sure I'll share my code later today, I should add that in developing this is when I hit the rate limiting, so I'd develop, add changes, make a run and after a second run I would get rate limiting and blocked (which, btw, seems to block me for 5 minutes) I'm using the wrapper but I actually had to manually modify the endpoint in the wrwapper to include weeks on the roster since I wasn't able to add the "week" property to both the roster and stats sub-resource for the wrapper method you have. Overall I have my my script working to get the "award" I want -- but I'm afraid that if say I want to run the award generator for different leagues, I'm gonna be in trouble since I'll hit the rate limiting. But anyways let me clean up some stuff and I'll share a trimmed down version of what I'm essentially doing. |
feel free to submit a PR :) Still feels like you're well within the bounds of the API limits, but I have run into the same problem in the past while developing things. |
|
Alright I am back with an example -- see here that I had to manually use the API directly to send the requests I wanted and also manually import the helpers to parse out the requests I got. The string "CHANGE_ME" is for things you need to add yoursef. import YahooFantasy from 'yahoo-fantasy';
import { mapRoster, mapTeam } from "./node_modules/yahoo-fantasy/helpers/teamHelper.mjs";
const CONSUMER_KEY = "CHANGE_ME";
const CONSUMER_SECRET = "CHANGE_ME";
const REDIRECT_URI = "CHANGE_ME";
const yf = new YahooFantasy(CONSUMER_KEY, CONSUMER_SECRET, () => { }, REDIRECT_URI);
const USER_TOKEN = "CHANGE_ME";
yf.setUserToken(USER_TOKEN);
(async () => {
try {
// UPDATE The values below :D
const SEASON = "2023";
const GAME_CODE = "nfl";
const LEAGUE_ID = "CHANGE_ME"
// NOTE: talk to API method directly since I can't run this request with the wrappers -- I want the NFL game key from last season (or season specified).
const data = await yf.api(yf.GET, `https://fantasysports.yahooapis.com/fantasy/v2/games;game_codes=${GAME_CODE};seasons=${SEASON}`)
const game = data.fantasy_content.games[0].game[0];
const GAME_KEY = game.game_key;
const LEAGUE_KEY = `${GAME_KEY}.l.${LEAGUE_ID}`;
// grab all the teams in this league_key
// in theory if you know the team size you can just generate this instead of sending an API request...but yeah if you want the team name & other metadata then you'll need this
const teams = await yf.league.teams(LEAGUE_KEY);
// generate an array of the team keys, we need this for our next request
const teamKeys = teams.teams.map(t => t.team_key)
console.log({ teamKeys })
// iterate over each team, generate an api request to get their roster given the teamKey AND week
const promiseArray = [];
for (let i = 0; i < teamKeys.length; i++) {
const teamKey = teamKeys[i];
for (let week = 1; week <= 17; week++) {
// NOTE: talk to API method directly since I can't run this request with the wrappers
promiseArray.push(yf.api(yf.GET, `https://fantasysports.yahooapis.com/fantasy/v2/team/${teamKey}/roster;week=${week}/players/stats;type=week;week=${week}`));
}
}
console.log({ apiRequests: promiseArray.length })
// yolo batch all the requests
const result = await Promise.all(promiseArray);
const allQBs = {};
// iterate over the result set and do the math to determine which team has the highest QB
for (let index = 0; index < result.length; index++) {
// use the helpers to make this how the wrapper would return it.
const team = mapTeam(result[index].fantasy_content.team[0]);
const mappedRoster = mapRoster(result[index].fantasy_content.team[1].roster);
team.roster = mappedRoster;
const teamKey = team.team_key;
if (!allQBs[teamKey]) {
allQBs[teamKey] = {
totalQBPoints: 0
};
}
for (let j = 0; j < team.roster.length; j++) {
const player = team.roster[j];
// if this player in the roster has the starting position of QB, they started the player, else it would be bench -- change this to RB, WR, etc. for other position scores too
if (player.selected_position === "QB") {
const playerStats = parseFloat(player.player_points.total, 10);
allQBs[teamKey].totalQBPoints = playerStats + allQBs[teamKey].totalQBPoints;
continue;
}
}
}
// The end
console.log({ allQBs })
} catch (e) {
console.log(e)
}
})()Output: |
|
I would bet dollars to donuts that you're hitting a 1-minute rate limit on this, here's why: You're adding all of these calls to a Promise arrayat the same time. If I was an API developer and saw 204 requests coming in the span of a second I would rate limit you as well. Thinking about this being in active development, you're probably like me and running things a bunch of times in the span of a minute, which is setting off more red flags? Thinking about this in how someone might do it in the real world, you'd likely do this every week, not all weeks at once, and could cache the data? You could also, for development purposes, mock the response so you don't overwhelm the API? Although maybe right now you're done and you're just worried about going forward? |
|
Thanks @whatadewitt yeah I definitely understand why I am getting rate limited, I guess overall I'd like to fully understand what those rate limiting rules are (which is just not documented so I will have to play around with it) so that I can build a proper system in place; maybe queue and/or caching layer of course 😄 . I know right now, I have a script/proof of concept, but I'm hoping to develop a web app where users can connect their Yahoo Fantasy League and generate an award report. So if for example, five users log in simultaneously to generate an award report, I'll quickly hit the rate limits and be blocked. So yeah, I just want to understand these limitations and learn how others have tackled this issue at scale. I may be thinking ahead, but knowing the exact rate limiting rules and existing solutions is just something I want to keep in mind when I want to scale this. 😃 |
|
@HappyZombies - from my own experience, where we connect users to Yahoo via OAuth, I mostly see the rate limit is applied on a per-user basis, not per-app. I had a similar experience: during development, hit rate limits because I was doing a lot of refreshes and probably should have been caching. But my app connects a huge number of Yahoo users essentially simultaneously, and we do see some rate limits hit, but I think that's individual user behavior, and not applied across our entire registered application. I agree it's a bummer the rate limits are opaque. I understand Yahoo's position here, but it's pretty unhelpful, especially since they're so difficult to get in touch with to coordinate raised limits (we still have yet to get a good contact). Still, if you do the right caching and make efficient queries, your users should mostly be fine. |
|
If the rate limiting is per-user basis then I think I am fine then! This actually makes sense and is good to know, thanks for the clarification @coolsoftwaretyler |
|
I did some fun tests with the API to discover how the rate limiting. This is nothing conclusive but should give us a hint... Sending requests back to back like in my script below: import YahooFantasy from 'yahoo-fantasy';
import { performance } from 'perf_hooks';
const CONSUMER_KEY = "REPLACE_ME";
const CONSUMER_SECRET = "REPLACE_ME";
const REDIRECT_URI = "REPLACE_ME";
const SEASON = "2023";
const GAME_CODE = "nfl";
const delay = ms => new Promise(resolve => setTimeout(resolve, ms));
const yf = new YahooFantasy(CONSUMER_KEY, CONSUMER_SECRET, () => { }, REDIRECT_URI);
const USER_TOKEN = "REPLACE_ME";
yf.setUserToken(USER_TOKEN);
const endpoint = `https://fantasysports.yahooapis.com/fantasy/v2/games;game_codes=${GAME_CODE};seasons=${SEASON}`;
(async () => {
let requestCount = 0;
let successCount = 0;
let startTime = performance.now();
while (true) {
try {
await yf.api(yf.GET, endpoint);
// await delay(1000); // Delay of 1 second between requests
successCount++;
console.log(`Request ${successCount} successful.`);
} catch (e) {
console.log({ e })
console.log(`Request ${requestCount + 1} failed: ${e.message}`);
if (e.response && e.response.status === 999) {
console.log("Rate limited.");
break;
}
break;
}
requestCount++;
}
let endTime = performance.now();
let elapsedTime = ((endTime - startTime) / 1000).toFixed(2);
console.log(`Total requests sent: ${requestCount}`);
console.log(`Total successful requests: ${successCount}`);
console.log(`Total time elapsed: ${elapsedTime} seconds`);
console.log(`Time right now: ${new Date}`);
})();Will trip the rate limiting at or before 500 requests. If I add the one second delay, it trips the rate limiting at just over 1000 requests (but that's after waiting for almost twenty minutes 😢 ) Once you are rate limited, you are blocked for 5 minutes, however this number seems to increase or will just totally block you if you keep tripping the rate limit (or it could be the number just increases a lot that you might as well just get a new token) Another fun note, if you send the requests after your 5 minute block is over, the amount of requests you can receive decreases -- on my experiment I got the following results: It appears to be that after the 5 minute block, you get 35% less requests? 325/500 = 0.65, which turns out to be a 35% difference. 325 * .35 is 114 rounded up, which is really close to the third run result: 325 - 114 = 211. The 3rd run was ran just a few seconds after 5 minute was up, suggesting that after that fact, a timer is ran to slowly increase your requests limit back. This is perhaps backup by my 4th run, I ran it after it was closer to ten minutes instead of 5. When I ran that request, I got rate limited at 318 requests. So whatever the case it's definitely some backoff formula that they are using that will "give you requests" back after seconds/minutes go by. Either the case this was just a small test I ran, nothing concrete on how exactly it works but these were my finds non the less :) |
Hey Luke,
When making too many requests to Yahoo, Yahoo blocks us for a short period of time based on the app ID we have registered on their developer portal.
I am looking to catch this error so that I can handle it properly and send an informative error message to the front end of the application. However I cannot catch the error using try catch or .then.catch syntax, and when the error occurs the program crashes. The error is occuring in YahooFantasy.mjs and occurs when parsing the result from Yahoo.
The issue seems to be occurring when trying to use JSON.parse to parse the "Request denied" response from Yahoo (possibly line 287 in YahooFantasy.mjs?) but I am not sure why this error is uncatchable.
Here are some examples of how I am trying to handle the error:
yf.players.leagues(sandlot, { start: 0 }, ['stats', 'ownership', 'percent_owned', 'draft_analysis']).then((result) => { console.log(result) }).catch((error) => { console.log(error) // Never reached })try { const result = yf.players.leagues(sandlot, { start: 0 }, ['stats', 'ownership', 'percent_owned', 'draft_analysis']) } catch (e) { console.log(e) // Never reached }The text was updated successfully, but these errors were encountered: