Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Theory on how to alleviate the token max issue. #28

Open
xbwtyz opened this issue Apr 8, 2023 · 3 comments
Open

Theory on how to alleviate the token max issue. #28

xbwtyz opened this issue Apr 8, 2023 · 3 comments

Comments

@xbwtyz
Copy link

xbwtyz commented Apr 8, 2023

I see you are already working on a method using viewport to cutdown on token amount, but what if apply the following as well

  • Take the simplified DOM and cut it into segments that fit
  • Send it to the LLM with a modified version of the user's instructions such as "what portions of the following message are relevant to [user's instructions]: " etc.,
  • store the return in a "master message"
  • repeat steps 2-3 until all parts of the DOM are sent
  • send the master message alongside the user's instructions so that only relevant portions of the DOM are sent.

This would of course introduce unnecessary space for error and ultimately slow down the extension, but maybe it would function in niche cases?

@CryptoMitch
Copy link

`async function getRelevantDomSegments(dom, instructions, llm) {
const segmentSize = 50; // or any other reasonable size
let masterMessage = "";
let startIndex = 0;
let endIndex = segmentSize;

while (startIndex < dom.length) {
const segment = dom.slice(startIndex, endIndex);
const modifiedInstructions = What portions of the following message are relevant to ${instructions}: ${segment};

const relevantPortion = await llm(modifiedInstructions); // assuming llm is an async function that sends the text to LLM and returns the result
masterMessage += relevantPortion;

startIndex += segmentSize;
endIndex += segmentSize;

}

const finalInstructions = ${instructions} ${masterMessage};
const result = await llm(finalInstructions);

return result;
}`

@ibrahimsk1
Copy link

@xbwtyz cool solution!

My suggestion is after simplifying the html creating indexes and keeping these. Afterwards, asking questions about the user interactions and getting the relavent parts.

@tluyben
Copy link
Contributor

tluyben commented Apr 18, 2023

It seems that keeping this in html format to send to gpt is taking up a lot of space (tokens)? Wouldn't it be better to just generate a flat list of [id, type, text] and send that over? So like [1833, 'c' 'Read More'] where you can have types like 'c' for clickable, 'i' for inputable etc.

Edit; doing some testing here https://github.com/tluyben/browser-extension/tree/feature/token-limit-attempt ; seems to work better for sites i have tested.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants