(refactor) improve comment prompt

[jaykayudo]

Resolves #147

[0x4007]

Please post QA tests comparing results

[jaykayudo]

Please post QA tests comparing results

@0x4007 could you please clarify this for me.
Do you have any tips on how to approach testing this prompt? or am i test another thing?
Because from the use case in project, i am finding it hard to wrap my head around it as there was no initial test for the module.

[gentlementlegen]

@jaykayudo You should prove that this prompt improves the results. There are not tests to avoid using the OpenAi api unnecessarily, so everything is mocked. The best way would be to show two results, one with the old prompt and one with the new.

[jaykayudo]

Okay.. On it

[jaykayudo]

@jaykayudo You should prove that this prompt improves the results. There are not tests to avoid using the OpenAi api unnecessarily, so everything is mocked. The best way would be to show two results, one with the old prompt and one with the new.

@gentlementlegen I ran the new prompt and the old prompt on OpenAi api and i have the result which are basically the same. one thing i am still confused about is that should i post the sample request data and the result i got from it on this PR comments or should it be part of a test case?

[gentlementlegen]

@0x4007 Might know best, but ideally I guess what we expect is to see a full run.

[0x4007]

Let's see scoring before and after

[jaykayudo]

Let's see scoring before and after

Okay.. i will post the sample data and response here for you the view.. it might be very lengthy.

[jaykayudo]

Old Prompt

Instruction: 
    Go through all the comments first keep them in memory, then start with the following prompt

    OUTPUT FORMAT:
    {ID: CONNECTION SCORE} For Each record in the EVALUATING SECTION, based on the average value from the CONNECTION SCORE from ALL COMMENTS, TITLE and BODY, one for each comment under evaluation
    Global Context:
    Specification
    "In the v1 of the Ubiquibot, when a result gets evaluated, a recap is posted to the related issue with a summary and details of the rewards as well as the link to the rewards themselves Example: https://github.com/ubiquity/cloudflare-deploy-action/issues/9#issuecomment-2028623754\r\n\r\nThe same logic should be applied in the v2 version by creating a new Module responsible to post that comment. This module will receive a similar input than the one mentioned [here](https://github.com/ubiquity/cloudflare-deploy-action/issues/9#issuecomment-2028623754)\r\n\r\nThe module should be:\r\n- possible to enable / disable\r\n- eventually configurable (what data to show / hide)\r\n- coming with tests"
    ALL COMMENTS:
    [
    {
        "id": 2030164289,
        "comment": "@whilefoo rfc on how we can deal with comment outputs. Perhaps we can have a standard recognized property on the output interface? Then the kernel can decide whether to pass it around or something. \r\n\r\n```ts\r\ninterface PluginOutput {\r\n  comment: string; // html comment\r\n  rewards: Rewards; // { \"whilefoo\": \"500\", \"token\": \"0x0\" } etc\r\n}\r\n```",
        "author": "0x4007"
    },
    {
        "id": 2033404518,
        "comment": "This needs https://github.com/ubiquibot/conversation-rewards/pull/7 to be merged first. Also probably needs https://github.com/ubiquibot/permit-generation/pull/2 to be able to generate the permits properly.",
        "author":"gentlementlegen"
    },
    {
        "id": 2033488255,
        "comment": " @whilefoo rfc on how we can deal with comment outputs. Perhaps we can have a standard recognized property on the output interface? Then the kernel can decide whether to pass it around or something.\r\n> \r\n> ```ts\r\n> interface PluginOutput {\r\n>   comment: string; // html comment\r\n>   rewards: Rewards; // { \"whilefoo\": \"500\", \"token\": \"0x0\" } etc\r\n> }\r\n> ```\r\n\r\nthere are a couple of options:\r\n1. we let the conversation-rewards plugin generate and post the comment\r\n2. we put comment as output and then another module is responsible for posting it or let conversation-rewards generate rewards and permit-generation generate permits and a third module that uses output from previous plugins to make a comment and post it\r\n3. we let the conversation-rewards plugin generate the comment and pass it as a standard property like you suggested.\r\n\r\nIn theory 2. option sounds good to separate concerns but it's another plugin which means another call to github actions thus more latency, so for the sake of speed it'd go with option 1 or 3, but going with these 2 options would mean there will 1 comment for rewards summary and ",
        "author": "whilefoo"
    },
    {
        "id": 2036174312,
        "comment":"To me 1 is the most straightforward to do for few reasons:\r\n- the comment reward plugin has all the needed data already\r\n- it can import the https://github.com/ubiquibot/permit-generation to generate permits itself\r\n- if it is done this way it can be used as a complete standalone without the kernel\r\n\r\n3 might make more sense in terms of architecture however. In such case the kernel should pass down results. It is more of an architecture question. Although, if we ever have other plugins in the flow that have influence on the total incentives, it would make sense to go through the kernel to aggregate the total result.",
        "author":"gentlementlegen"
    }

]
    IMPORTANT CONTEXT:
    You have now seen all the comments made by other users, keeping the comments in mind think in what ways comments to be evaluated be connected. The comments that were related to the comment under evaluation might come after or before them in the list of all comments but they would be there in ALL COMMENTS. COULD BE BEFORE OR AFTER, you have diligently search through all the comments in ALL COMMENTS.
    START EVALUATING:
    {
        "id": 2036367126,
        "comment":"I think each plugin should output JSON not html as it is not reliable to parse nor manipulate and requires `window` instance to be instantiated which is annoying on `node` based projects. \r\n\r\nHaving a plugin handling commenting seems quite weird as commenting is done by calling Octokit and the REST api which is already a library by itself, so no need to encapsulate it within another one to do the same thing.\r\n\r\nMy view on this, is to finalize https://github.com/ubiquibot/permit-generation/issues/5 to import it withing the `conversation-reward` that will post the comment itself as well, otherwise the architecture will be quite convoluted doing ping pong with everything."
    },
    {
        "id": 2036385985,
        "comment": "If you want to manipulate and convey data, HTML really is not made for this. If you want something formatted similarly but made for data we can use XML format.\r\nThe new comment reward actually does instantiate a DOM through [JSDOM](https://github.com/ubiquibot/conversation-rewards/blob/ba434761281446a23566cd02c68bd3b0e79d4eb1/src/parser/formatting-evaluator-module.ts#L80) to make things way simpler instead of using Regex everywhere which is highly unreliable. But there it makes sense because we are parsing comments from an HTML page content.\r\n\r\nBiggest advantage from this is to have the comment reward fully standalone, while easy to integrate with the kernel.\r\n\r\nIf we do something that handles the comment it means each and every module has to send it there and that module should understand every different content / format we send which would be way easier if the module itself handled its own comments, formatting wise."
    }

    POST EVALUATION:
    THE RESULT FROM THIS SHOULD BE ONLY THE SCORES BASED ON THE FLOATING POINT VALUE CONNECTING HOW CLOSE THE COMMENT IS FROM ALL COMMENTS AND TITLE AND BODY.

    Now Assign them scores a float value ranging from 0 to 1, where 0 is spam (lowest value), and 1 is something that's very relevant (Highest Value), here relevance should mean a variety of things, it could be a fix to the issue, it could be a bug in solution, it could a SPAM message, it could be comment, that on its own does not carry weight, but when CHECKED IN ALL COMMENTS, may be a crucial piece of information for debugging and solving the ticket. If YOU THINK ITS NOT RELATED to ALL COMMENTS or TITLE OR ISSUE SPEC, then give it a 0 SCORE.

    OUTPUT:
    RETURN ONLY A JSON with the ID and the connection score (FLOATING POINT VALUE) with ALL COMMENTS TITLE AND BODY for each comment under evaluation.  RETURN ONLY ONE CONNECTION SCORE VALUE for each comment. Total number of properties in your JSON response should equal exactly 2

Old Response

{
    "2036367126": 0.7,
    "2036385985": 0.8
}

New Prompt

Instruction: 
    Start by thoroughly reading all comments and retaining their content for the evaluation.

    OUTPUT FORMAT:
    Provide a JSON object with the format: {ID: CONNECTION_SCORE} for each record in the evaluation section. 
    The CONNECTION_SCORE should reflect the average relevance based on all comments, title, and body.

    GLOBAL CONTEXT:
    Specification
    "In the v1 of the Ubiquibot, when a result gets evaluated, a recap is posted to the related issue with a summary and details of the rewards as well as the link to the rewards themselves Example: https://github.com/ubiquity/cloudflare-deploy-action/issues/9#issuecomment-2028623754\r\n\r\nThe same logic should be applied in the v2 version by creating a new Module responsible to post that comment. This module will receive a similar input than the one mentioned [here](https://github.com/ubiquity/cloudflare-deploy-action/issues/9#issuecomment-2028623754)\r\n\r\nThe module should be:\r\n- possible to enable / disable\r\n- eventually configurable (what data to show / hide)\r\n- coming with tests"
    
    ALL COMMENTS:
    {
        "id": 2030164289,
        "comment": "@whilefoo rfc on how we can deal with comment outputs. Perhaps we can have a standard recognized property on the output interface? Then the kernel can decide whether to pass it around or something. \r\n\r\n```ts\r\ninterface PluginOutput {\r\n  comment: string; // html comment\r\n  rewards: Rewards; // { \"whilefoo\": \"500\", \"token\": \"0x0\" } etc\r\n}\r\n```",
        "author": "0x4007"
    },
    {
        "id": 2033404518,
        "comment": "This needs https://github.com/ubiquibot/conversation-rewards/pull/7 to be merged first. Also probably needs https://github.com/ubiquibot/permit-generation/pull/2 to be able to generate the permits properly.",
        "author":"gentlementlegen"
    },
    {
        "id": 2033488255,
        "comment": " @whilefoo rfc on how we can deal with comment outputs. Perhaps we can have a standard recognized property on the output interface? Then the kernel can decide whether to pass it around or something.\r\n> \r\n> ```ts\r\n> interface PluginOutput {\r\n>   comment: string; // html comment\r\n>   rewards: Rewards; // { \"whilefoo\": \"500\", \"token\": \"0x0\" } etc\r\n> }\r\n> ```\r\n\r\nthere are a couple of options:\r\n1. we let the conversation-rewards plugin generate and post the comment\r\n2. we put comment as output and then another module is responsible for posting it or let conversation-rewards generate rewards and permit-generation generate permits and a third module that uses output from previous plugins to make a comment and post it\r\n3. we let the conversation-rewards plugin generate the comment and pass it as a standard property like you suggested.\r\n\r\nIn theory 2. option sounds good to separate concerns but it's another plugin which means another call to github actions thus more latency, so for the sake of speed it'd go with option 1 or 3, but going with these 2 options would mean there will 1 comment for rewards summary and ",
        "author": "whilefoo"
    },
    {
        "id": 2036174312,
        "comment":"To me 1 is the most straightforward to do for few reasons:\r\n- the comment reward plugin has all the needed data already\r\n- it can import the https://github.com/ubiquibot/permit-generation to generate permits itself\r\n- if it is done this way it can be used as a complete standalone without the kernel\r\n\r\n3 might make more sense in terms of architecture however. In such case the kernel should pass down results. It is more of an architecture question. Although, if we ever have other plugins in the flow that have influence on the total incentives, it would make sense to go through the kernel to aggregate the total result.",
        "author":"gentlementlegen"
    },
    
    IMPORTANT CONTEXT:
    Consider all comments when evaluating connections. Relevant comments may appear before or after the comment being evaluated, so examine all of them closely.
    
    START EVALUATING:
    {
        "id": 2036367126,
        "comment":"I think each plugin should output JSON not html as it is not reliable to parse nor manipulate and requires `window` instance to be instantiated which is annoying on `node` based projects. \r\n\r\nHaving a plugin handling commenting seems quite weird as commenting is done by calling Octokit and the REST api which is already a library by itself, so no need to encapsulate it within another one to do the same thing.\r\n\r\nMy view on this, is to finalize https://github.com/ubiquibot/permit-generation/issues/5 to import it withing the `conversation-reward` that will post the comment itself as well, otherwise the architecture will be quite convoluted doing ping pong with everything."
    },
    {
        "id": 2036385985,
        "comment": "If you want to manipulate and convey data, HTML really is not made for this. If you want something formatted similarly but made for data we can use XML format.\r\nThe new comment reward actually does instantiate a DOM through [JSDOM](https://github.com/ubiquibot/conversation-rewards/blob/ba434761281446a23566cd02c68bd3b0e79d4eb1/src/parser/formatting-evaluator-module.ts#L80) to make things way simpler instead of using Regex everywhere which is highly unreliable. But there it makes sense because we are parsing comments from an HTML page content.\r\n\r\nBiggest advantage from this is to have the comment reward fully standalone, while easy to integrate with the kernel.\r\n\r\nIf we do something that handles the comment it means each and every module has to send it there and that module should understand every different content / format we send which would be way easier if the module itself handled its own comments, formatting wise."
    }

    POST EVALUATION:
    Provide only the connection scores as floating-point values indicating the relevance of each comment based on its connection to the overall context.

    SCORING CRITERIA:
    Assign scores from 0 to 1, 0: Not related (e.g., spam), 1: Highly relevant (e.g., solutions, bugs)
    Consider the context of all comments; even minor details may be significant for resolving the issue. If a comment is unrelated to all comments, title, or issue specification, assign a score of 0.

    OUTPUT:
    Return a JSON object containing the ID and the connection score for each evaluated comment. The number of entries in the JSON response must match exactly 2.

New Response

{
    "2036367126": 0.7,
    "2036385985": 0.8
}

@gentlementlegen @0x4007 Here it is

[0x4007]

Thanks for this. Could you do a more complex example? This doesnt have enough variety to show any difference

[jaykayudo]

@0x4007 Please could you refer me to an issue that i could use that you think would be complex enough

[jaykayudo]

Here is a more complex example @gentlementlegen @0x4007

Old Prompt

Instruction: 
    Go through all the comments first keep them in memory, then start with the following prompt

    OUTPUT FORMAT:
    {ID: CONNECTION SCORE} For Each record in the EVALUATING SECTION, based on the average value from the CONNECTION SCORE from ALL COMMENTS, TITLE and BODY, one for each comment under evaluation
    Global Context:
    Specification
    "In the v1 of the Ubiquibot, when a result gets evaluated, a recap is posted to the related issue with a summary and details of the rewards as well as the link to the rewards themselves Example: https://github.com/ubiquity/cloudflare-deploy-action/issues/9#issuecomment-2028623754

The same logic should be applied in the v2 version by creating a new Module responsible to post that comment. This module will receive a similar input than the one mentioned [here](https://github.com/ubiquity/cloudflare-deploy-action/issues/9#issuecomment-2028623754)

The module should be:
- possible to enable / disable
- eventually configurable (what data to show / hide)
- coming with tests"
    ALL COMMENTS:
    [{"id": 2030164289, "comment": "@whilefoo rfc on how we can deal with comment outputs. Perhaps we can have a standard recognized property on the output interface? Then the kernel can decide whether to pass it around or something. \r\n\r\n```ts\r\ninterface PluginOutput {\r\n  comment: string; // html comment\r\n  rewards: Rewards; // { \"whilefoo\": \"500\", \"token\": \"0x0\" } etc\r\n}\r\n```", "author": "0x4007"}, {"id": 2033404518, "comment": "This needs https://github.com/ubiquibot/conversation-rewards/pull/7 to be merged first. Also probably needs https://github.com/ubiquibot/permit-generation/pull/2 to be able to generate the permits properly.", "author": "gentlementlegen"}, {"id": 2033488255, "comment": "> This needs https://github.com/ubiquibot/conversation-rewards/pull/7 to be merged first. Also probably needs https://github.com/ubiquibot/permit-generation/pull/2 to be able to generate the permits properly.\n\nI think you should fork from and overtake that second pull due to us being behind schedule ", "author": "0x4007"}, {"id": 2035427134, "comment": "> @whilefoo rfc on how we can deal with comment outputs. Perhaps we can have a standard recognized property on the output interface? Then the kernel can decide whether to pass it around or something.\r\n> \r\n> ```ts\r\n> interface PluginOutput {\r\n>   comment: string; // html comment\r\n>   rewards: Rewards; // { \"whilefoo\": \"500\", \"token\": \"0x0\" } etc\r\n> }\r\n> ```\r\n\r\nthere are a couple of options:\r\n1. we let the conversation-rewards plugin generate and post the comment\r\n2. we put comment as output and then another module is responsible for posting it or let conversation-rewards generate rewards and permit-generation generate permits and a third module that uses output from previous plugins to make a comment and post it\r\n3. we let the conversation-rewards plugin generate the comment and pass it as a standard property like you suggested.\r\n\r\nIn theory 2. option sounds good to separate concerns but it's another plugin which means another call to github actions thus more latency, so for the sake of speed it'd go with option 1 or 3, but going with these 2 options would mean there will 1 comment for rewards summary and 1 comment for permits.\r\nI'm not sure if option 3 is any better than option 1 because the plugin already has a token that has permissions to post comments so passing it to the kernel doesn't make much difference.\r\n", "author": "whilefoo"}, {"id": 2036174312, "comment": "To me 1 is the most straightforward to do for few reasons:\r\n- the comment reward plugin has all the needed data already\r\n- it can import the https://github.com/ubiquibot/permit-generation to generate permits itself\r\n- if it is done this way it can be used as a complete standalone without the kernel\r\n\r\n3 might make more sense in terms of architecture however. In such case the kernel should pass down results. It is more of an architecture question. Although, if we ever have other plugins in the flow that have influence on the total incentives, it would make sense to go through the kernel to aggregate the total result.", "author": "gentlementlegen"}, {"id": 2036355445, "comment": "> \r\n> there are a couple of options:\r\n> \r\n> 1. we let the conversation-rewards plugin generate and post the comment\r\n\r\nI think the most pure architecture would be that plugins can NOT inherit (write) authentication (only read if possible) of the kernel. As a consequence, no plugin should be able to post directly any issue. Ideally it should only be the kernel with a direct interface to issues. Plugins should just output comment HTML and the kernel can post it all in a single comment at the end of the webhook event invocation chain.\r\n\r\n> 2. we put comment as output and then another module is responsible for posting it or let conversation-rewards generate rewards and permit-generation generate permits and a third module that uses output from previous plugins to make a comment and post it\r\n\r\nCould be interesting to have a dedicated plugin to handle commenting, but because this seems like such an essential capability, I would more carefully consider the pros/cons of including this within the kernel. \r\n\r\n- On one hand, I really like the idea of making the kernel as pure and lean as possible\r\n- On the other hand there are practical considerations that if every plugin standard output interface includes `comment:` and then we don't have the `comment` capability available (the comment plugin is not included in the config) then why is this a standard output property required on every plugin?\r\n\r\n> 3. we let the conversation-rewards plugin generate the comment and pass it as a standard property like you suggested.\r\n> \r\n> In theory 2. option sounds good to separate concerns but it's another plugin which means another call to github actions thus more latency, so for the sake of speed it'd go with option 1 or 3, but going with these 2 options would mean there will 1 comment for rewards summary and 1 comment for permits. I'm not sure if option 3 is any better than option 1 because the plugin already has a token that has permissions to post comments so passing it to the kernel doesn't make much difference.\r\n\r\nI'm not concerned about latency now. Besides we can port any plugin to Cloudflare Workers (or combine several inside a single Worker) when we are ready to fix latency issues. To be honest though, except for setting labels, I have no problem with latency (like with permit generation) so there's no need to overengineer yet. I also love the GitHub Actions logs being available for easy debugging, and the fact that they are super generous with the compute. \r\n\r\nGiven the entire conversation and all the considerations, I definitely think we should do option 3. \r\n\r\nThis separates concerns the best, which will allow our plugin ecosystem to grow the fastest. ", "author": "0x4007"}, {"id": 2036367126, "comment": "I think each plugin should output JSON not html as it is not reliable to parse nor manipulate and requires `window` instance to be instantiated which is annoying on `node` based projects. \r\n\r\nHaving a plugin handling commenting seems quite weird as commenting is done by calling Octokit and the REST api which is already a library by itself, so no need to encapsulate it within another one to do the same thing.\r\n\r\nMy view on this, is to finalize https://github.com/ubiquibot/permit-generation/issues/5 to import it withing the `conversation-reward` that will post the comment itself as well, otherwise the architecture will be quite convoluted doing ping pong with everything.", "author": "gentlementlegen"}, {"id": 2036370459, "comment": "> I think each plugin should output JSON not html as it is not reliable to parse nor manipulate\r\n\r\nI know JSON makes things more complicated than it needs to be with serialization/transport. Then we have problems like this https://github.com/ubiquibot/conversation-rewards/issues/18. If the comment property is ONLY meant to be used for GitHub comments, then transporting HTML only is the most sensible. Besides, there's no other metadata thats necessary within the `comment` property, which is the main point of sending a JSON object (it has multiple properties.) \r\n\r\nAs a somewhat side note: there should also not be any variables inside of the HTML. We could look it as like server-side-rendering (or in this case, plugin-side-rendering) handle producing the final HTML output and then output it as a single string of html entities. \r\n\r\n> requires window instance to be instantiated which is annoying on node based projects.\r\n\r\nWe are currently using `mdast` in `@ubiquibot/comment-incentives` to generate virtual DOMs. No `window` needed.\r\n\r\n> My view on this, is to finalize https://github.com/ubiquibot/permit-generation/issues/5 to import it withing the conversation-reward that will post the comment itself as well, otherwise the architecture will be quite convoluted doing ping pong with everything.\r\n\r\nIf we agree that this is considered as technical debt, and that we are accruing this so that we can get back on schedule, ok.", "author": "0x4007"}, {"id": 2036385985, "comment": "If you want to manipulate and convey data, HTML really is not made for this. If you want something formatted similarly but made for data we can use XML format.\r\nThe new comment reward actually does instantiate a DOM through [JSDOM](https://github.com/ubiquibot/conversation-rewards/blob/ba434761281446a23566cd02c68bd3b0e79d4eb1/src/parser/formatting-evaluator-module.ts#L80) to make things way simpler instead of using Regex everywhere which is highly unreliable. But there it makes sense because we are parsing comments from an HTML page content.\r\n\r\nBiggest advantage from this is to have the comment reward fully standalone, while easy to integrate with the kernel.\r\n\r\nIf we do something that handles the comment it means each and every module has to send it there and that module should understand every different content / format we send which would be way easier if the module itself handled its own comments, formatting wise.", "author": "gentlementlegen"}, {"id": 2036393020, "comment": "> If you want to manipulate and convey data, HTML really is not made for this.\r\n\r\nGoing back to my \"plugin-side-rendering\" mention, the data manipulation happens inside of the logic of the plugin. Then when the plugin is finally done with all of its compute, it emits a single string in the `comment` output property. This string is the final, rendered HTML. \r\n\r\nThe `comment` output is not intended to be consumed by other plugins. In most cases, the kernel will concatenate all the `comment` outputs from every plugin and post a single comment at the end. Plugins will consume the `metadata` property output which will include raw values to work with. I forget where I originally proposed this, but imagine something like:\r\n```typescript\r\n\r\ntype HtmlString = string;\r\ntype RequestedPermits = { username: string; amount: string; token: string; }[]\r\n \r\ninterface PluginOutput {\r\n   metadata: Record<string, any>;\r\n   comment: HtmlString;\r\n   rewards: RequestedPermits;\r\n}\r\n```", "author": "0x4007"}, {"id": 2036411811, "comment": "But then how do we consider the formatting of that output?\r\n\r\nPractical case: we want to post a comment when a user queries a wallet. That comment is 'user name': 'wallet 0x0'\r\nKernel calls the comment plugin, saying that it wants a comment to be posted. Should the Kernel send the rendering it wants, should the comment plugin transform the data to HTML?\r\n\r\nThen, comment-reward wants to post the results. Should ask the Kernel to call the comment plugin, but then formatting is different. Should the Kernel notify the comment plugin that it wants a different output formatting? Should the Kernel compute beforehand the HTML and send it to the comment plugin?\r\n\r\nMight be something I don't grasp there. Because I do understand your use case but it seems to be very deterministic on what is the purpose of the plugins which kinda defeats the purpose of having plugins, looks more like a monolithic approach to me", "author": "gentlementlegen"}, {"id": 2036433646, "comment": "> But then how do we consider the formatting of that output?\r\n\r\nThe proposed `comment` output is intended for ease of composability for MOST situations, basically that the output from one plugin would just be appended to the final rendered comment. For simple plugin configurations (plugins not modifying the output results of others) this seems like straightforward architecture. \r\n\r\nHowever, as we know, there will be situations where a subsequent plugin will consider the results of a previous plugin, which means it would need to change the comment that is rendered. \r\n\r\nIn this situation the subsequent plugin should clobber the output of the previous plugin. It is now clear to be that this will be a new challenge to express to the kernel that it should ignore the comment output of a previous plugin. I suppose it would be straightforward in the metadata using the plugin ID i.e. `{ \"silences\": [\"ubiquibot/conversation-rewards\"] }`.\r\n\r\n> Kernel calls the comment plugin,\r\n\r\nI think that comments should be handled from within the kernel. There should not be a separate comment plugin. Read my explanation [here](https://github.com/ubiquibot/conversation-rewards/issues/5#issuecomment-2036355445). \r\n\r\n\r\n### Inputs\r\n\r\nFor completeness of my [previous comment](https://github.com/ubiquibot/conversation-rewards/issues/5#issuecomment-2036393020):\r\n\r\n```typescript\r\n\r\ntype PluginId = string; // i.e. `ubiquibot/conversation-rewards` `ubiquibot/assistive-pricing`\r\n \r\ninterface PluginInput {\r\n   metadata: Record<PluginId, any>;\r\n   context: GitHubEventContext;\r\n}\r\n```\r\n\r\nNotice that we should not pass in the HTML of other plugins. Instead, just the metadata (\"computed\") values from the previous plugins.", "author": "0x4007"}, {"id": 2036458775, "comment": "This can work, but we skyrocket coupling and to me defeat purpose of plugins that should be unaware of each other. If any plugin has to understand the result of a previous plugin, it means these plugins have necessarily to co-exist so basically they become a single plugin with no purpose to split them.", "author": "gentlementlegen"}]
    IMPORTANT CONTEXT:
    You have now seen all the comments made by other users, keeping the comments in mind think in what ways comments to be evaluated be connected. The comments that were related to the comment under evaluation might come after or before them in the list of all comments but they would be there in ALL COMMENTS. COULD BE BEFORE OR AFTER, you have diligently search through all the comments in ALL COMMENTS.
    START EVALUATING:
    [{"id": 2036516869, "comment": "> so basically they become a single plugin with no purpose to split them.\r\n\r\nI understand your concern and I would need to put more thought into composability. Maybe the `comment` proposal is bad after all due to your point. It's only useful with simple plugins, but for more complex ones your point is valid. \r\n\r\nI need help to think through how any partner in the future can create new plugins that modify \r\n1. permit reward amounts \r\n2. XP reward amounts\r\n\r\nI don't like the idea of having a single monolithic rewards generation plugin that wraps all the ways possible to compute rewards (i.e. time estimation, comment rewards, review rewards etc)"}, {"id": 2036535332, "comment": "@pavlovcik To mitigate that that's why inside the comment reward itself I also integrated that `Module` principle so code is not coupled tightly and easy to build on. There is as usual pros and cons to both approaches (splitting or not) but biggest pro is that comments get evaluated once in the same spot, so we save calls to OpenAPI and speed up the process. Also makes it only one configuration file in one location. We need to think about our best options there."}, {"id": 2051094255, "comment": "I realized that to carry this task properly we need to handle flags for comment more delicately as they only indicate if the comment is `ISSUE | REVIEW` with the level `MEMBER | CONTRIBUTOR` etc. but doesn't specify if it is from a task, a specification and so on. Tags should be added to the config properly as well."}, {"id": 2053332029, "comment": "> I realized that to carry this task properly we need to handle flags for comment more delicately as they only indicate if the comment is `ISSUE | REVIEW` with the level `MEMBER | CONTRIBUTOR` etc. but doesn't specify if it is from a task, a specification and so on. Tags should be added to the config properly as well.\n\nI see, so you're suggesting that we must annotate each comment as well in order to properly handle the scoring at the end?\n\nOff hand I think there's:\n\n1. Specification \n2. Issue comment\n3. Pull request comment (a normal comment on a pull request, not related to posting a \"review\"\n4. Pull request review comment (associated with a \"review state\" I.e. approved, left comments, requested changes)\n\nI suppose we need a better naming convention for the pull related ones. They are considered as separate entities according to the GitHub api. They require different methods to obtain both types. "}, {"id": 2054424028, "comment": "Agreed, I think currently there are 3 possible things to annotate on the comments:\r\n- `SPECIFICATION` for the issue itself, `TASK` for the related PR fixing it, or simply `COMMENT`\r\n- `MEMBER` or `CONTRIBUTOR` for the status of the member\r\n- `REVIEW` or `ISSUE` for where the comment was added\r\n\r\nI think this shall cover all cases."}, {"id": 2055783331, "comment": "Consider calling it \"contributor\" and \"collaborator\" as that is how it is presented on the GitHub APIs as I recall. \n\nAlso I think you forgot about the \"review comments\""}]
    POST EVALUATION:
    THE RESULT FROM THIS SHOULD BE ONLY THE SCORES BASED ON THE FLOATING POINT VALUE CONNECTING HOW CLOSE THE COMMENT IS FROM ALL COMMENTS AND TITLE AND BODY.

    Now Assign them scores a float value ranging from 0 to 1, where 0 is spam (lowest value), and 1 is something that's very relevant (Highest Value), here relevance should mean a variety of things, it could be a fix to the issue, it could be a bug in solution, it could a SPAM message, it could be comment, that on its own does not carry weight, but when CHECKED IN ALL COMMENTS, may be a crucial piece of information for debugging and solving the ticket. If YOU THINK ITS NOT RELATED to ALL COMMENTS or TITLE OR ISSUE SPEC, then give it a 0 SCORE.

    OUTPUT:
    RETURN ONLY A JSON with the ID and the connection score (FLOATING POINT VALUE) with ALL COMMENTS TITLE AND BODY for each comment under evaluation.  RETURN ONLY ONE CONNECTION SCORE VALUE for each comment. Total number of properties in your JSON response should equal exactly 6

Old Prompt Response

{
  "2036516869": 0.8,
  "2036535332": 0.7,
  "2051094255": 0.6,
  "2053332029": 0.7,
  "2054424028": 0.6,
  "2055783331": 0.5
}

New Prompt

    Instruction: 
    Start by thoroughly reading all comments and retaining their content for the evaluation.
    OUTPUT FORMAT:
    Provide a JSON object with the format: {ID: CONNECTION_SCORE} for each record in the evaluation section. 
    The CONNECTION_SCORE should reflect the average relevance based on all comments, title, and body.
    GLOBAL CONTEXT:
    Specification
    "In the v1 of the Ubiquibot, when a result gets evaluated, a recap is posted to the related issue with a summary and details of the rewards as well as the link to the rewards themselves Example: https://github.com/ubiquity/cloudflare-deploy-action/issues/9#issuecomment-2028623754

The same logic should be applied in the v2 version by creating a new Module responsible to post that comment. This module will receive a similar input than the one mentioned [here](https://github.com/ubiquity/cloudflare-deploy-action/issues/9#issuecomment-2028623754)

The module should be:
- possible to enable / disable
- eventually configurable (what data to show / hide)
- coming with tests"
    
    ALL COMMENTS:
    [{"id": 2030164289, "comment": "@whilefoo rfc on how we can deal with comment outputs. Perhaps we can have a standard recognized property on the output interface? Then the kernel can decide whether to pass it around or something. \r\n\r\n```ts\r\ninterface PluginOutput {\r\n  comment: string; // html comment\r\n  rewards: Rewards; // { \"whilefoo\": \"500\", \"token\": \"0x0\" } etc\r\n}\r\n```", "author": "0x4007"}, {"id": 2033404518, "comment": "This needs https://github.com/ubiquibot/conversation-rewards/pull/7 to be merged first. Also probably needs https://github.com/ubiquibot/permit-generation/pull/2 to be able to generate the permits properly.", "author": "gentlementlegen"}, {"id": 2033488255, "comment": "> This needs https://github.com/ubiquibot/conversation-rewards/pull/7 to be merged first. Also probably needs https://github.com/ubiquibot/permit-generation/pull/2 to be able to generate the permits properly.\n\nI think you should fork from and overtake that second pull due to us being behind schedule ", "author": "0x4007"}, {"id": 2035427134, "comment": "> @whilefoo rfc on how we can deal with comment outputs. Perhaps we can have a standard recognized property on the output interface? Then the kernel can decide whether to pass it around or something.\r\n> \r\n> ```ts\r\n> interface PluginOutput {\r\n>   comment: string; // html comment\r\n>   rewards: Rewards; // { \"whilefoo\": \"500\", \"token\": \"0x0\" } etc\r\n> }\r\n> ```\r\n\r\nthere are a couple of options:\r\n1. we let the conversation-rewards plugin generate and post the comment\r\n2. we put comment as output and then another module is responsible for posting it or let conversation-rewards generate rewards and permit-generation generate permits and a third module that uses output from previous plugins to make a comment and post it\r\n3. we let the conversation-rewards plugin generate the comment and pass it as a standard property like you suggested.\r\n\r\nIn theory 2. option sounds good to separate concerns but it's another plugin which means another call to github actions thus more latency, so for the sake of speed it'd go with option 1 or 3, but going with these 2 options would mean there will 1 comment for rewards summary and 1 comment for permits.\r\nI'm not sure if option 3 is any better than option 1 because the plugin already has a token that has permissions to post comments so passing it to the kernel doesn't make much difference.\r\n", "author": "whilefoo"}, {"id": 2036174312, "comment": "To me 1 is the most straightforward to do for few reasons:\r\n- the comment reward plugin has all the needed data already\r\n- it can import the https://github.com/ubiquibot/permit-generation to generate permits itself\r\n- if it is done this way it can be used as a complete standalone without the kernel\r\n\r\n3 might make more sense in terms of architecture however. In such case the kernel should pass down results. It is more of an architecture question. Although, if we ever have other plugins in the flow that have influence on the total incentives, it would make sense to go through the kernel to aggregate the total result.", "author": "gentlementlegen"}, {"id": 2036355445, "comment": "> \r\n> there are a couple of options:\r\n> \r\n> 1. we let the conversation-rewards plugin generate and post the comment\r\n\r\nI think the most pure architecture would be that plugins can NOT inherit (write) authentication (only read if possible) of the kernel. As a consequence, no plugin should be able to post directly any issue. Ideally it should only be the kernel with a direct interface to issues. Plugins should just output comment HTML and the kernel can post it all in a single comment at the end of the webhook event invocation chain.\r\n\r\n> 2. we put comment as output and then another module is responsible for posting it or let conversation-rewards generate rewards and permit-generation generate permits and a third module that uses output from previous plugins to make a comment and post it\r\n\r\nCould be interesting to have a dedicated plugin to handle commenting, but because this seems like such an essential capability, I would more carefully consider the pros/cons of including this within the kernel. \r\n\r\n- On one hand, I really like the idea of making the kernel as pure and lean as possible\r\n- On the other hand there are practical considerations that if every plugin standard output interface includes `comment:` and then we don't have the `comment` capability available (the comment plugin is not included in the config) then why is this a standard output property required on every plugin?\r\n\r\n> 3. we let the conversation-rewards plugin generate the comment and pass it as a standard property like you suggested.\r\n> \r\n> In theory 2. option sounds good to separate concerns but it's another plugin which means another call to github actions thus more latency, so for the sake of speed it'd go with option 1 or 3, but going with these 2 options would mean there will 1 comment for rewards summary and 1 comment for permits. I'm not sure if option 3 is any better than option 1 because the plugin already has a token that has permissions to post comments so passing it to the kernel doesn't make much difference.\r\n\r\nI'm not concerned about latency now. Besides we can port any plugin to Cloudflare Workers (or combine several inside a single Worker) when we are ready to fix latency issues. To be honest though, except for setting labels, I have no problem with latency (like with permit generation) so there's no need to overengineer yet. I also love the GitHub Actions logs being available for easy debugging, and the fact that they are super generous with the compute. \r\n\r\nGiven the entire conversation and all the considerations, I definitely think we should do option 3. \r\n\r\nThis separates concerns the best, which will allow our plugin ecosystem to grow the fastest. ", "author": "0x4007"}, {"id": 2036367126, "comment": "I think each plugin should output JSON not html as it is not reliable to parse nor manipulate and requires `window` instance to be instantiated which is annoying on `node` based projects. \r\n\r\nHaving a plugin handling commenting seems quite weird as commenting is done by calling Octokit and the REST api which is already a library by itself, so no need to encapsulate it within another one to do the same thing.\r\n\r\nMy view on this, is to finalize https://github.com/ubiquibot/permit-generation/issues/5 to import it withing the `conversation-reward` that will post the comment itself as well, otherwise the architecture will be quite convoluted doing ping pong with everything.", "author": "gentlementlegen"}, {"id": 2036370459, "comment": "> I think each plugin should output JSON not html as it is not reliable to parse nor manipulate\r\n\r\nI know JSON makes things more complicated than it needs to be with serialization/transport. Then we have problems like this https://github.com/ubiquibot/conversation-rewards/issues/18. If the comment property is ONLY meant to be used for GitHub comments, then transporting HTML only is the most sensible. Besides, there's no other metadata thats necessary within the `comment` property, which is the main point of sending a JSON object (it has multiple properties.) \r\n\r\nAs a somewhat side note: there should also not be any variables inside of the HTML. We could look it as like server-side-rendering (or in this case, plugin-side-rendering) handle producing the final HTML output and then output it as a single string of html entities. \r\n\r\n> requires window instance to be instantiated which is annoying on node based projects.\r\n\r\nWe are currently using `mdast` in `@ubiquibot/comment-incentives` to generate virtual DOMs. No `window` needed.\r\n\r\n> My view on this, is to finalize https://github.com/ubiquibot/permit-generation/issues/5 to import it withing the conversation-reward that will post the comment itself as well, otherwise the architecture will be quite convoluted doing ping pong with everything.\r\n\r\nIf we agree that this is considered as technical debt, and that we are accruing this so that we can get back on schedule, ok.", "author": "0x4007"}, {"id": 2036385985, "comment": "If you want to manipulate and convey data, HTML really is not made for this. If you want something formatted similarly but made for data we can use XML format.\r\nThe new comment reward actually does instantiate a DOM through [JSDOM](https://github.com/ubiquibot/conversation-rewards/blob/ba434761281446a23566cd02c68bd3b0e79d4eb1/src/parser/formatting-evaluator-module.ts#L80) to make things way simpler instead of using Regex everywhere which is highly unreliable. But there it makes sense because we are parsing comments from an HTML page content.\r\n\r\nBiggest advantage from this is to have the comment reward fully standalone, while easy to integrate with the kernel.\r\n\r\nIf we do something that handles the comment it means each and every module has to send it there and that module should understand every different content / format we send which would be way easier if the module itself handled its own comments, formatting wise.", "author": "gentlementlegen"}, {"id": 2036393020, "comment": "> If you want to manipulate and convey data, HTML really is not made for this.\r\n\r\nGoing back to my \"plugin-side-rendering\" mention, the data manipulation happens inside of the logic of the plugin. Then when the plugin is finally done with all of its compute, it emits a single string in the `comment` output property. This string is the final, rendered HTML. \r\n\r\nThe `comment` output is not intended to be consumed by other plugins. In most cases, the kernel will concatenate all the `comment` outputs from every plugin and post a single comment at the end. Plugins will consume the `metadata` property output which will include raw values to work with. I forget where I originally proposed this, but imagine something like:\r\n```typescript\r\n\r\ntype HtmlString = string;\r\ntype RequestedPermits = { username: string; amount: string; token: string; }[]\r\n \r\ninterface PluginOutput {\r\n   metadata: Record<string, any>;\r\n   comment: HtmlString;\r\n   rewards: RequestedPermits;\r\n}\r\n```", "author": "0x4007"}, {"id": 2036411811, "comment": "But then how do we consider the formatting of that output?\r\n\r\nPractical case: we want to post a comment when a user queries a wallet. That comment is 'user name': 'wallet 0x0'\r\nKernel calls the comment plugin, saying that it wants a comment to be posted. Should the Kernel send the rendering it wants, should the comment plugin transform the data to HTML?\r\n\r\nThen, comment-reward wants to post the results. Should ask the Kernel to call the comment plugin, but then formatting is different. Should the Kernel notify the comment plugin that it wants a different output formatting? Should the Kernel compute beforehand the HTML and send it to the comment plugin?\r\n\r\nMight be something I don't grasp there. Because I do understand your use case but it seems to be very deterministic on what is the purpose of the plugins which kinda defeats the purpose of having plugins, looks more like a monolithic approach to me", "author": "gentlementlegen"}, {"id": 2036433646, "comment": "> But then how do we consider the formatting of that output?\r\n\r\nThe proposed `comment` output is intended for ease of composability for MOST situations, basically that the output from one plugin would just be appended to the final rendered comment. For simple plugin configurations (plugins not modifying the output results of others) this seems like straightforward architecture. \r\n\r\nHowever, as we know, there will be situations where a subsequent plugin will consider the results of a previous plugin, which means it would need to change the comment that is rendered. \r\n\r\nIn this situation the subsequent plugin should clobber the output of the previous plugin. It is now clear to be that this will be a new challenge to express to the kernel that it should ignore the comment output of a previous plugin. I suppose it would be straightforward in the metadata using the plugin ID i.e. `{ \"silences\": [\"ubiquibot/conversation-rewards\"] }`.\r\n\r\n> Kernel calls the comment plugin,\r\n\r\nI think that comments should be handled from within the kernel. There should not be a separate comment plugin. Read my explanation [here](https://github.com/ubiquibot/conversation-rewards/issues/5#issuecomment-2036355445). \r\n\r\n\r\n### Inputs\r\n\r\nFor completeness of my [previous comment](https://github.com/ubiquibot/conversation-rewards/issues/5#issuecomment-2036393020):\r\n\r\n```typescript\r\n\r\ntype PluginId = string; // i.e. `ubiquibot/conversation-rewards` `ubiquibot/assistive-pricing`\r\n \r\ninterface PluginInput {\r\n   metadata: Record<PluginId, any>;\r\n   context: GitHubEventContext;\r\n}\r\n```\r\n\r\nNotice that we should not pass in the HTML of other plugins. Instead, just the metadata (\"computed\") values from the previous plugins.", "author": "0x4007"}, {"id": 2036458775, "comment": "This can work, but we skyrocket coupling and to me defeat purpose of plugins that should be unaware of each other. If any plugin has to understand the result of a previous plugin, it means these plugins have necessarily to co-exist so basically they become a single plugin with no purpose to split them.", "author": "gentlementlegen"}]
    
    IMPORTANT CONTEXT:
    Consider all comments when evaluating connections. Relevant comments may appear before or after the comment being evaluated, so examine all of them closely.
    
    START EVALUATING:
    [{"id": 2036516869, "comment": "> so basically they become a single plugin with no purpose to split them.\r\n\r\nI understand your concern and I would need to put more thought into composability. Maybe the `comment` proposal is bad after all due to your point. It's only useful with simple plugins, but for more complex ones your point is valid. \r\n\r\nI need help to think through how any partner in the future can create new plugins that modify \r\n1. permit reward amounts \r\n2. XP reward amounts\r\n\r\nI don't like the idea of having a single monolithic rewards generation plugin that wraps all the ways possible to compute rewards (i.e. time estimation, comment rewards, review rewards etc)"}, {"id": 2036535332, "comment": "@pavlovcik To mitigate that that's why inside the comment reward itself I also integrated that `Module` principle so code is not coupled tightly and easy to build on. There is as usual pros and cons to both approaches (splitting or not) but biggest pro is that comments get evaluated once in the same spot, so we save calls to OpenAPI and speed up the process. Also makes it only one configuration file in one location. We need to think about our best options there."}, {"id": 2051094255, "comment": "I realized that to carry this task properly we need to handle flags for comment more delicately as they only indicate if the comment is `ISSUE | REVIEW` with the level `MEMBER | CONTRIBUTOR` etc. but doesn't specify if it is from a task, a specification and so on. Tags should be added to the config properly as well."}, {"id": 2053332029, "comment": "> I realized that to carry this task properly we need to handle flags for comment more delicately as they only indicate if the comment is `ISSUE | REVIEW` with the level `MEMBER | CONTRIBUTOR` etc. but doesn't specify if it is from a task, a specification and so on. Tags should be added to the config properly as well.\n\nI see, so you're suggesting that we must annotate each comment as well in order to properly handle the scoring at the end?\n\nOff hand I think there's:\n\n1. Specification \n2. Issue comment\n3. Pull request comment (a normal comment on a pull request, not related to posting a \"review\"\n4. Pull request review comment (associated with a \"review state\" I.e. approved, left comments, requested changes)\n\nI suppose we need a better naming convention for the pull related ones. They are considered as separate entities according to the GitHub api. They require different methods to obtain both types. "}, {"id": 2054424028, "comment": "Agreed, I think currently there are 3 possible things to annotate on the comments:\r\n- `SPECIFICATION` for the issue itself, `TASK` for the related PR fixing it, or simply `COMMENT`\r\n- `MEMBER` or `CONTRIBUTOR` for the status of the member\r\n- `REVIEW` or `ISSUE` for where the comment was added\r\n\r\nI think this shall cover all cases."}, {"id": 2055783331, "comment": "Consider calling it \"contributor\" and \"collaborator\" as that is how it is presented on the GitHub APIs as I recall. \n\nAlso I think you forgot about the \"review comments\""}]
    POST EVALUATION:
    Provide only the connection scores as floating-point values indicating the relevance of each comment based on its connection to the overall context.
    SCORING CRITERIA:
    Assign scores from 0 to 1, 0: Not related (e.g., spam), 1: Highly relevant (e.g., solutions, bugs)
    Consider the context of all comments; even minor details may be significant for resolving the issue. If a comment is unrelated to all comments, title, or issue specification, assign a score of 0.
    OUTPUT:
    Return a JSON object containing the ID and the connection score for each evaluated comment. The number of entries in the JSON response must match exactly 6

New Prompt Response

{
    "2036516869": 0.7,
    "2036535332": 0.8,
    "2051094255": 0.9,
    "2053332029": 0.85,
    "2054424028": 0.9,
    "2055783331": 0.75
}

[0x4007]

I reviewed the prompt and it doesn't make sense to me why there are redundant comments being passed in on ALL COMMENTS and START EVALUATING

What's going on here?

Also START EVALUATING starts from about halfway through the conversation.

#5

I really think JSON notation is not optimal for a prompt.

[jaykayudo]

I reviewed the prompt and it doesn't make sense to me why there are redundant comments being passed in on ALL COMMENTS and START EVALUATING

I pulled this comments from a closed issue of these repo which was used as a fallback issue url during testing. i did it to create a more complex example. I failed to indent it so that is why it looks that way which i will fix. these are all the comment from this issue.

I really think JSON notation is not optimal for a prompt.

Do you have anything in mind? because it being sent as a json notation enables the AI to grab the id and tell which comment belongs to which id and attach a relevance score based on that .

Here is a more clearer view of the prompt

    Instruction: 
    Start by thoroughly reading all comments and retaining their content for the evaluation.
    OUTPUT FORMAT:
    Provide a JSON object with the format: {ID: CONNECTION_SCORE} for each record in the evaluation section. 
    The CONNECTION_SCORE should reflect the average relevance based on all comments, title, and body.
    GLOBAL CONTEXT:
    Specification
    "In the v1 of the Ubiquibot, when a result gets evaluated, a recap is posted to the related issue with a summary and details of the rewards as well as the link to the rewards themselves Example: https://github.com/ubiquity/cloudflare-deploy-action/issues/9#issuecomment-2028623754

The same logic should be applied in the v2 version by creating a new Module responsible to post that comment. This module will receive a similar input than the one mentioned [here](https://github.com/ubiquity/cloudflare-deploy-action/issues/9#issuecomment-2028623754)

The module should be:
- possible to enable / disable
- eventually configurable (what data to show / hide)
- coming with tests"
    
    ALL COMMENTS:
    [
  {
    "id": 2030164289,
    "comment": "@whilefoo rfc on how we can deal with comment outputs. Perhaps we can have a standard recognized property on the output interface? Then the kernel can decide whether to pass it around or something. \r\n\r\n```ts\r\ninterface PluginOutput {\r\n  comment: string; // html comment\r\n  rewards: Rewards; // { \"whilefoo\": \"500\", \"token\": \"0x0\" } etc\r\n}\r\n```",
    "author": "0x4007"
  },
  {
    "id": 2033404518,
    "comment": "This needs https://github.com/ubiquibot/conversation-rewards/pull/7 to be merged first. Also probably needs https://github.com/ubiquibot/permit-generation/pull/2 to be able to generate the permits properly.",
    "author": "gentlementlegen"
  },
  {
    "id": 2033488255,
    "comment": "> This needs https://github.com/ubiquibot/conversation-rewards/pull/7 to be merged first. Also probably needs https://github.com/ubiquibot/permit-generation/pull/2 to be able to generate the permits properly.\n\nI think you should fork from and overtake that second pull due to us being behind schedule ",
    "author": "0x4007"
  },
  {
    "id": 2035427134,
    "comment": "> @whilefoo rfc on how we can deal with comment outputs. Perhaps we can have a standard recognized property on the output interface? Then the kernel can decide whether to pass it around or something.\r\n> \r\n> ```ts\r\n> interface PluginOutput {\r\n>   comment: string; // html comment\r\n>   rewards: Rewards; // { \"whilefoo\": \"500\", \"token\": \"0x0\" } etc\r\n> }\r\n> ```\r\n\r\nthere are a couple of options:\r\n1. we let the conversation-rewards plugin generate and post the comment\r\n2. we put comment as output and then another module is responsible for posting it or let conversation-rewards generate rewards and permit-generation generate permits and a third module that uses output from previous plugins to make a comment and post it\r\n3. we let the conversation-rewards plugin generate the comment and pass it as a standard property like you suggested.\r\n\r\nIn theory 2. option sounds good to separate concerns but it's another plugin which means another call to github actions thus more latency, so for the sake of speed it'd go with option 1 or 3, but going with these 2 options would mean there will 1 comment for rewards summary and 1 comment for permits.\r\nI'm not sure if option 3 is any better than option 1 because the plugin already has a token that has permissions to post comments so passing it to the kernel doesn't make much difference.\r\n",
    "author": "whilefoo"
  },
  {
    "id": 2036174312,
    "comment": "To me 1 is the most straightforward to do for few reasons:\r\n- the comment reward plugin has all the needed data already\r\n- it can import the https://github.com/ubiquibot/permit-generation to generate permits itself\r\n- if it is done this way it can be used as a complete standalone without the kernel\r\n\r\n3 might make more sense in terms of architecture however. In such case the kernel should pass down results. It is more of an architecture question. Although, if we ever have other plugins in the flow that have influence on the total incentives, it would make sense to go through the kernel to aggregate the total result.",
    "author": "gentlementlegen"
  },
  {
    "id": 2036355445,
    "comment": "> \r\n> there are a couple of options:\r\n> \r\n> 1. we let the conversation-rewards plugin generate and post the comment\r\n\r\nI think the most pure architecture would be that plugins can NOT inherit (write) authentication (only read if possible) of the kernel. As a consequence, no plugin should be able to post directly any issue. Ideally it should only be the kernel with a direct interface to issues. Plugins should just output comment HTML and the kernel can post it all in a single comment at the end of the webhook event invocation chain.\r\n\r\n> 2. we put comment as output and then another module is responsible for posting it or let conversation-rewards generate rewards and permit-generation generate permits and a third module that uses output from previous plugins to make a comment and post it\r\n\r\nCould be interesting to have a dedicated plugin to handle commenting, but because this seems like such an essential capability, I would more carefully consider the pros/cons of including this within the kernel. \r\n\r\n- On one hand, I really like the idea of making the kernel as pure and lean as possible\r\n- On the other hand there are practical considerations that if every plugin standard output interface includes `comment:` and then we don't have the `comment` capability available (the comment plugin is not included in the config) then why is this a standard output property required on every plugin?\r\n\r\n> 3. we let the conversation-rewards plugin generate the comment and pass it as a standard property like you suggested.\r\n> \r\n> In theory 2. option sounds good to separate concerns but it's another plugin which means another call to github actions thus more latency, so for the sake of speed it'd go with option 1 or 3, but going with these 2 options would mean there will 1 comment for rewards summary and 1 comment for permits. I'm not sure if option 3 is any better than option 1 because the plugin already has a token that has permissions to post comments so passing it to the kernel doesn't make much difference.\r\n\r\nI'm not concerned about latency now. Besides we can port any plugin to Cloudflare Workers (or combine several inside a single Worker) when we are ready to fix latency issues. To be honest though, except for setting labels, I have no problem with latency (like with permit generation) so there's no need to overengineer yet. I also love the GitHub Actions logs being available for easy debugging, and the fact that they are super generous with the compute. \r\n\r\nGiven the entire conversation and all the considerations, I definitely think we should do option 3. \r\n\r\nThis separates concerns the best, which will allow our plugin ecosystem to grow the fastest. ",
    "author": "0x4007"
  },
  {
    "id": 2036367126,
    "comment": "I think each plugin should output JSON not html as it is not reliable to parse nor manipulate and requires `window` instance to be instantiated which is annoying on `node` based projects. \r\n\r\nHaving a plugin handling commenting seems quite weird as commenting is done by calling Octokit and the REST api which is already a library by itself, so no need to encapsulate it within another one to do the same thing.\r\n\r\nMy view on this, is to finalize https://github.com/ubiquibot/permit-generation/issues/5 to import it withing the `conversation-reward` that will post the comment itself as well, otherwise the architecture will be quite convoluted doing ping pong with everything.",
    "author": "gentlementlegen"
  },
  {
    "id": 2036370459,
    "comment": "> I think each plugin should output JSON not html as it is not reliable to parse nor manipulate\r\n\r\nI know JSON makes things more complicated than it needs to be with serialization/transport. Then we have problems like this https://github.com/ubiquibot/conversation-rewards/issues/18. If the comment property is ONLY meant to be used for GitHub comments, then transporting HTML only is the most sensible. Besides, there's no other metadata thats necessary within the `comment` property, which is the main point of sending a JSON object (it has multiple properties.) \r\n\r\nAs a somewhat side note: there should also not be any variables inside of the HTML. We could look it as like server-side-rendering (or in this case, plugin-side-rendering) handle producing the final HTML output and then output it as a single string of html entities. \r\n\r\n> requires window instance to be instantiated which is annoying on node based projects.\r\n\r\nWe are currently using `mdast` in `@ubiquibot/comment-incentives` to generate virtual DOMs. No `window` needed.\r\n\r\n> My view on this, is to finalize https://github.com/ubiquibot/permit-generation/issues/5 to import it withing the conversation-reward that will post the comment itself as well, otherwise the architecture will be quite convoluted doing ping pong with everything.\r\n\r\nIf we agree that this is considered as technical debt, and that we are accruing this so that we can get back on schedule, ok.",
    "author": "0x4007"
  },
  {
    "id": 2036385985,
    "comment": "If you want to manipulate and convey data, HTML really is not made for this. If you want something formatted similarly but made for data we can use XML format.\r\nThe new comment reward actually does instantiate a DOM through [JSDOM](https://github.com/ubiquibot/conversation-rewards/blob/ba434761281446a23566cd02c68bd3b0e79d4eb1/src/parser/formatting-evaluator-module.ts#L80) to make things way simpler instead of using Regex everywhere which is highly unreliable. But there it makes sense because we are parsing comments from an HTML page content.\r\n\r\nBiggest advantage from this is to have the comment reward fully standalone, while easy to integrate with the kernel.\r\n\r\nIf we do something that handles the comment it means each and every module has to send it there and that module should understand every different content / format we send which would be way easier if the module itself handled its own comments, formatting wise.",
    "author": "gentlementlegen"
  },
  {
    "id": 2036393020,
    "comment": "> If you want to manipulate and convey data, HTML really is not made for this.\r\n\r\nGoing back to my \"plugin-side-rendering\" mention, the data manipulation happens inside of the logic of the plugin. Then when the plugin is finally done with all of its compute, it emits a single string in the `comment` output property. This string is the final, rendered HTML. \r\n\r\nThe `comment` output is not intended to be consumed by other plugins. In most cases, the kernel will concatenate all the `comment` outputs from every plugin and post a single comment at the end. Plugins will consume the `metadata` property output which will include raw values to work with. I forget where I originally proposed this, but imagine something like:\r\n```typescript\r\n\r\ntype HtmlString = string;\r\ntype RequestedPermits = { username: string; amount: string; token: string; }[]\r\n \r\ninterface PluginOutput {\r\n   metadata: Record<string, any>;\r\n   comment: HtmlString;\r\n   rewards: RequestedPermits;\r\n}\r\n```",
    "author": "0x4007"
  },
  {
    "id": 2036411811,
    "comment": "But then how do we consider the formatting of that output?\r\n\r\nPractical case: we want to post a comment when a user queries a wallet. That comment is 'user name': 'wallet 0x0'\r\nKernel calls the comment plugin, saying that it wants a comment to be posted. Should the Kernel send the rendering it wants, should the comment plugin transform the data to HTML?\r\n\r\nThen, comment-reward wants to post the results. Should ask the Kernel to call the comment plugin, but then formatting is different. Should the Kernel notify the comment plugin that it wants a different output formatting? Should the Kernel compute beforehand the HTML and send it to the comment plugin?\r\n\r\nMight be something I don't grasp there. Because I do understand your use case but it seems to be very deterministic on what is the purpose of the plugins which kinda defeats the purpose of having plugins, looks more like a monolithic approach to me",
    "author": "gentlementlegen"
  },
  {
    "id": 2036433646,
    "comment": "> But then how do we consider the formatting of that output?\r\n\r\nThe proposed `comment` output is intended for ease of composability for MOST situations, basically that the output from one plugin would just be appended to the final rendered comment. For simple plugin configurations (plugins not modifying the output results of others) this seems like straightforward architecture. \r\n\r\nHowever, as we know, there will be situations where a subsequent plugin will consider the results of a previous plugin, which means it would need to change the comment that is rendered. \r\n\r\nIn this situation the subsequent plugin should clobber the output of the previous plugin. It is now clear to be that this will be a new challenge to express to the kernel that it should ignore the comment output of a previous plugin. I suppose it would be straightforward in the metadata using the plugin ID i.e. `{ \"silences\": [\"ubiquibot/conversation-rewards\"] }`.\r\n\r\n> Kernel calls the comment plugin,\r\n\r\nI think that comments should be handled from within the kernel. There should not be a separate comment plugin. Read my explanation [here](https://github.com/ubiquibot/conversation-rewards/issues/5#issuecomment-2036355445). \r\n\r\n\r\n### Inputs\r\n\r\nFor completeness of my [previous comment](https://github.com/ubiquibot/conversation-rewards/issues/5#issuecomment-2036393020):\r\n\r\n```typescript\r\n\r\ntype PluginId = string; // i.e. `ubiquibot/conversation-rewards` `ubiquibot/assistive-pricing`\r\n \r\ninterface PluginInput {\r\n   metadata: Record<PluginId, any>;\r\n   context: GitHubEventContext;\r\n}\r\n```\r\n\r\nNotice that we should not pass in the HTML of other plugins. Instead, just the metadata (\"computed\") values from the previous plugins.",
    "author": "0x4007"
  },
  {
    "id": 2036458775,
    "comment": "This can work, but we skyrocket coupling and to me defeat purpose of plugins that should be unaware of each other. If any plugin has to understand the result of a previous plugin, it means these plugins have necessarily to co-exist so basically they become a single plugin with no purpose to split them.",
    "author": "gentlementlegen"
  }
]
    
    IMPORTANT CONTEXT:
    Consider all comments when evaluating connections. Relevant comments may appear before or after the comment being evaluated, so examine all of them closely.
    
    START EVALUATING:
    [
  {
    "id": 2036516869,
    "comment": "> so basically they become a single plugin with no purpose to split them.\r\n\r\nI understand your concern and I would need to put more thought into composability. Maybe the `comment` proposal is bad after all due to your point. It's only useful with simple plugins, but for more complex ones your point is valid. \r\n\r\nI need help to think through how any partner in the future can create new plugins that modify \r\n1. permit reward amounts \r\n2. XP reward amounts\r\n\r\nI don't like the idea of having a single monolithic rewards generation plugin that wraps all the ways possible to compute rewards (i.e. time estimation, comment rewards, review rewards etc)"
  },
  {
    "id": 2036535332,
    "comment": "@pavlovcik To mitigate that that's why inside the comment reward itself I also integrated that `Module` principle so code is not coupled tightly and easy to build on. There is as usual pros and cons to both approaches (splitting or not) but biggest pro is that comments get evaluated once in the same spot, so we save calls to OpenAPI and speed up the process. Also makes it only one configuration file in one location. We need to think about our best options there."
  },
  {
    "id": 2051094255,
    "comment": "I realized that to carry this task properly we need to handle flags for comment more delicately as they only indicate if the comment is `ISSUE | REVIEW` with the level `MEMBER | CONTRIBUTOR` etc. but doesn't specify if it is from a task, a specification and so on. Tags should be added to the config properly as well."
  },
  {
    "id": 2053332029,
    "comment": "> I realized that to carry this task properly we need to handle flags for comment more delicately as they only indicate if the comment is `ISSUE | REVIEW` with the level `MEMBER | CONTRIBUTOR` etc. but doesn't specify if it is from a task, a specification and so on. Tags should be added to the config properly as well.\n\nI see, so you're suggesting that we must annotate each comment as well in order to properly handle the scoring at the end?\n\nOff hand I think there's:\n\n1. Specification \n2. Issue comment\n3. Pull request comment (a normal comment on a pull request, not related to posting a \"review\"\n4. Pull request review comment (associated with a \"review state\" I.e. approved, left comments, requested changes)\n\nI suppose we need a better naming convention for the pull related ones. They are considered as separate entities according to the GitHub api. They require different methods to obtain both types. "
  },
  {
    "id": 2054424028,
    "comment": "Agreed, I think currently there are 3 possible things to annotate on the comments:\r\n- `SPECIFICATION` for the issue itself, `TASK` for the related PR fixing it, or simply `COMMENT`\r\n- `MEMBER` or `CONTRIBUTOR` for the status of the member\r\n- `REVIEW` or `ISSUE` for where the comment was added\r\n\r\nI think this shall cover all cases."
  },
  {
    "id": 2055783331,
    "comment": "Consider calling it \"contributor\" and \"collaborator\" as that is how it is presented on the GitHub APIs as I recall. \n\nAlso I think you forgot about the \"review comments\""
  }
]
    POST EVALUATION:
    Provide only the connection scores as floating-point values indicating the relevance of each comment based on its connection to the overall context.
    SCORING CRITERIA:
    Assign scores from 0 to 1, 0: Not related (e.g., spam), 1: Highly relevant (e.g., solutions, bugs)
    Consider the context of all comments; even minor details may be significant for resolving the issue. If a comment is unrelated to all comments, title, or issue specification, assign a score of 0.
    OUTPUT:
    Return a JSON object containing the ID and the connection score for each evaluated comment. The number of entries in the JSON response must match exactly 6

I will more into the prompt and what it needs to provide and improve it.

[0x4007]

Do you have anything in mind?

2036458775 - gentlementlegen: "This can work, but we skyrocket coupling and to me defeat purpose of plugins that should be unaware of each other. If any plugin has to understand the result of a previous plugin, it means these plugins have necessarily to co-exist so basically they become a single plugin with no purpose to split them."

[jaykayudo]

@0x4007 Check out this prompt format

    Instruction: 
    Start by thoroughly reading all comments and retaining their content for the evaluation.
    The comments in the ALL COMMENTS section will be in the format of: => {id} - {author}: "{comment}"

    OUTPUT FORMAT:
    For every record in the START EVALUATING section, provide a JSON object in the following format: {ID: CONNECTION_SCORE}. 
    The average relevance based on the comments in the ALL COMMENTS SECTION should be reflected in the CONNECTION_SCORE.

    GLOBAL CONTEXT:
    Specification
    "In the v1 of the Ubiquibot, when a result gets evaluated, a recap is posted to the related issue with a summary and details of the rewards as well as the link to the rewards themselves Example: https://github.com/ubiquity/cloudflare-deploy-action/issues/9#issuecomment-2028623754

The same logic should be applied in the v2 version by creating a new Module responsible to post that comment. This module will receive a similar input than the one mentioned [here](https://github.com/ubiquity/cloudflare-deploy-action/issues/9#issuecomment-2028623754)

The module should be:
- possible to enable / disable
- eventually configurable (what data to show / hide)
- coming with tests"
    
    ALL COMMENTS:
    [
  "=> 2030164289 - 0x4007: \"@whilefoo rfc on how we can deal with comment outputs. Perhaps we can have a standard recognized property on the output interface? Then the kernel can decide whether to pass it around or something. \r\n\r\n```ts\r\ninterface PluginOutput {\r\n  comment: string; // html comment\r\n  rewards: Rewards; // { \"whilefoo\": \"500\", \"token\": \"0x0\" } etc\r\n}\r\n```\"",
  "=> 2033404518 - gentlementlegen: \"This needs https://github.com/ubiquibot/conversation-rewards/pull/7 to be merged first. Also probably needs https://github.com/ubiquibot/permit-generation/pull/2 to be able to generate the permits properly.\"",
  "=> 2033488255 - 0x4007: \"> This needs https://github.com/ubiquibot/conversation-rewards/pull/7 to be merged first. Also probably needs https://github.com/ubiquibot/permit-generation/pull/2 to be able to generate the permits properly.\n\nI think you should fork from and overtake that second pull due to us being behind schedule \"",
  "=> 2035427134 - whilefoo: \"> @whilefoo rfc on how we can deal with comment outputs. Perhaps we can have a standard recognized property on the output interface? Then the kernel can decide whether to pass it around or something.\r\n> \r\n> ```ts\r\n> interface PluginOutput {\r\n>   comment: string; // html comment\r\n>   rewards: Rewards; // { \"whilefoo\": \"500\", \"token\": \"0x0\" } etc\r\n> }\r\n> ```\r\n\r\nthere are a couple of options:\r\n1. we let the conversation-rewards plugin generate and post the comment\r\n2. we put comment as output and then another module is responsible for posting it or let conversation-rewards generate rewards and permit-generation generate permits and a third module that uses output from previous plugins to make a comment and post it\r\n3. we let the conversation-rewards plugin generate the comment and pass it as a standard property like you suggested.\r\n\r\nIn theory 2. option sounds good to separate concerns but it's another plugin which means another call to github actions thus more latency, so for the sake of speed it'd go with option 1 or 3, but going with these 2 options would mean there will 1 comment for rewards summary and 1 comment for permits.\r\nI'm not sure if option 3 is any better than option 1 because the plugin already has a token that has permissions to post comments so passing it to the kernel doesn't make much difference.\r\n\"",
  "=> 2036174312 - gentlementlegen: \"To me 1 is the most straightforward to do for few reasons:\r\n- the comment reward plugin has all the needed data already\r\n- it can import the https://github.com/ubiquibot/permit-generation to generate permits itself\r\n- if it is done this way it can be used as a complete standalone without the kernel\r\n\r\n3 might make more sense in terms of architecture however. In such case the kernel should pass down results. It is more of an architecture question. Although, if we ever have other plugins in the flow that have influence on the total incentives, it would make sense to go through the kernel to aggregate the total result.\"",
  "=> 2036355445 - 0x4007: \"> \r\n> there are a couple of options:\r\n> \r\n> 1. we let the conversation-rewards plugin generate and post the comment\r\n\r\nI think the most pure architecture would be that plugins can NOT inherit (write) authentication (only read if possible) of the kernel. As a consequence, no plugin should be able to post directly any issue. Ideally it should only be the kernel with a direct interface to issues. Plugins should just output comment HTML and the kernel can post it all in a single comment at the end of the webhook event invocation chain.\r\n\r\n> 2. we put comment as output and then another module is responsible for posting it or let conversation-rewards generate rewards and permit-generation generate permits and a third module that uses output from previous plugins to make a comment and post it\r\n\r\nCould be interesting to have a dedicated plugin to handle commenting, but because this seems like such an essential capability, I would more carefully consider the pros/cons of including this within the kernel. \r\n\r\n- On one hand, I really like the idea of making the kernel as pure and lean as possible\r\n- On the other hand there are practical considerations that if every plugin standard output interface includes `comment:` and then we don't have the `comment` capability available (the comment plugin is not included in the config) then why is this a standard output property required on every plugin?\r\n\r\n> 3. we let the conversation-rewards plugin generate the comment and pass it as a standard property like you suggested.\r\n> \r\n> In theory 2. option sounds good to separate concerns but it's another plugin which means another call to github actions thus more latency, so for the sake of speed it'd go with option 1 or 3, but going with these 2 options would mean there will 1 comment for rewards summary and 1 comment for permits. I'm not sure if option 3 is any better than option 1 because the plugin already has a token that has permissions to post comments so passing it to the kernel doesn't make much difference.\r\n\r\nI'm not concerned about latency now. Besides we can port any plugin to Cloudflare Workers (or combine several inside a single Worker) when we are ready to fix latency issues. To be honest though, except for setting labels, I have no problem with latency (like with permit generation) so there's no need to overengineer yet. I also love the GitHub Actions logs being available for easy debugging, and the fact that they are super generous with the compute. \r\n\r\nGiven the entire conversation and all the considerations, I definitely think we should do option 3. \r\n\r\nThis separates concerns the best, which will allow our plugin ecosystem to grow the fastest. \"",
  "=> 2036367126 - gentlementlegen: \"I think each plugin should output JSON not html as it is not reliable to parse nor manipulate and requires `window` instance to be instantiated which is annoying on `node` based projects. \r\n\r\nHaving a plugin handling commenting seems quite weird as commenting is done by calling Octokit and the REST api which is already a library by itself, so no need to encapsulate it within another one to do the same thing.\r\n\r\nMy view on this, is to finalize https://github.com/ubiquibot/permit-generation/issues/5 to import it withing the `conversation-reward` that will post the comment itself as well, otherwise the architecture will be quite convoluted doing ping pong with everything.\"",
  "=> 2036370459 - 0x4007: \"> I think each plugin should output JSON not html as it is not reliable to parse nor manipulate\r\n\r\nI know JSON makes things more complicated than it needs to be with serialization/transport. Then we have problems like this https://github.com/ubiquibot/conversation-rewards/issues/18. If the comment property is ONLY meant to be used for GitHub comments, then transporting HTML only is the most sensible. Besides, there's no other metadata thats necessary within the `comment` property, which is the main point of sending a JSON object (it has multiple properties.) \r\n\r\nAs a somewhat side note: there should also not be any variables inside of the HTML. We could look it as like server-side-rendering (or in this case, plugin-side-rendering) handle producing the final HTML output and then output it as a single string of html entities. \r\n\r\n> requires window instance to be instantiated which is annoying on node based projects.\r\n\r\nWe are currently using `mdast` in `@ubiquibot/comment-incentives` to generate virtual DOMs. No `window` needed.\r\n\r\n> My view on this, is to finalize https://github.com/ubiquibot/permit-generation/issues/5 to import it withing the conversation-reward that will post the comment itself as well, otherwise the architecture will be quite convoluted doing ping pong with everything.\r\n\r\nIf we agree that this is considered as technical debt, and that we are accruing this so that we can get back on schedule, ok.\"",
  "=> 2036385985 - gentlementlegen: \"If you want to manipulate and convey data, HTML really is not made for this. If you want something formatted similarly but made for data we can use XML format.\r\nThe new comment reward actually does instantiate a DOM through [JSDOM](https://github.com/ubiquibot/conversation-rewards/blob/ba434761281446a23566cd02c68bd3b0e79d4eb1/src/parser/formatting-evaluator-module.ts#L80) to make things way simpler instead of using Regex everywhere which is highly unreliable. But there it makes sense because we are parsing comments from an HTML page content.\r\n\r\nBiggest advantage from this is to have the comment reward fully standalone, while easy to integrate with the kernel.\r\n\r\nIf we do something that handles the comment it means each and every module has to send it there and that module should understand every different content / format we send which would be way easier if the module itself handled its own comments, formatting wise.\"",
  "=> 2036393020 - 0x4007: \"> If you want to manipulate and convey data, HTML really is not made for this.\r\n\r\nGoing back to my \"plugin-side-rendering\" mention, the data manipulation happens inside of the logic of the plugin. Then when the plugin is finally done with all of its compute, it emits a single string in the `comment` output property. This string is the final, rendered HTML. \r\n\r\nThe `comment` output is not intended to be consumed by other plugins. In most cases, the kernel will concatenate all the `comment` outputs from every plugin and post a single comment at the end. Plugins will consume the `metadata` property output which will include raw values to work with. I forget where I originally proposed this, but imagine something like:\r\n```typescript\r\n\r\ntype HtmlString = string;\r\ntype RequestedPermits = { username: string; amount: string; token: string; }[]\r\n \r\ninterface PluginOutput {\r\n   metadata: Record<string, any>;\r\n   comment: HtmlString;\r\n   rewards: RequestedPermits;\r\n}\r\n```\"",
  "=> 2036411811 - gentlementlegen: \"But then how do we consider the formatting of that output?\r\n\r\nPractical case: we want to post a comment when a user queries a wallet. That comment is 'user name': 'wallet 0x0'\r\nKernel calls the comment plugin, saying that it wants a comment to be posted. Should the Kernel send the rendering it wants, should the comment plugin transform the data to HTML?\r\n\r\nThen, comment-reward wants to post the results. Should ask the Kernel to call the comment plugin, but then formatting is different. Should the Kernel notify the comment plugin that it wants a different output formatting? Should the Kernel compute beforehand the HTML and send it to the comment plugin?\r\n\r\nMight be something I don't grasp there. Because I do understand your use case but it seems to be very deterministic on what is the purpose of the plugins which kinda defeats the purpose of having plugins, looks more like a monolithic approach to me\"",
  "=> 2036433646 - 0x4007: \"> But then how do we consider the formatting of that output?\r\n\r\nThe proposed `comment` output is intended for ease of composability for MOST situations, basically that the output from one plugin would just be appended to the final rendered comment. For simple plugin configurations (plugins not modifying the output results of others) this seems like straightforward architecture. \r\n\r\nHowever, as we know, there will be situations where a subsequent plugin will consider the results of a previous plugin, which means it would need to change the comment that is rendered. \r\n\r\nIn this situation the subsequent plugin should clobber the output of the previous plugin. It is now clear to be that this will be a new challenge to express to the kernel that it should ignore the comment output of a previous plugin. I suppose it would be straightforward in the metadata using the plugin ID i.e. `{ \"silences\": [\"ubiquibot/conversation-rewards\"] }`.\r\n\r\n> Kernel calls the comment plugin,\r\n\r\nI think that comments should be handled from within the kernel. There should not be a separate comment plugin. Read my explanation [here](https://github.com/ubiquibot/conversation-rewards/issues/5#issuecomment-2036355445). \r\n\r\n\r\n### Inputs\r\n\r\nFor completeness of my [previous comment](https://github.com/ubiquibot/conversation-rewards/issues/5#issuecomment-2036393020):\r\n\r\n```typescript\r\n\r\ntype PluginId = string; // i.e. `ubiquibot/conversation-rewards` `ubiquibot/assistive-pricing`\r\n \r\ninterface PluginInput {\r\n   metadata: Record<PluginId, any>;\r\n   context: GitHubEventContext;\r\n}\r\n```\r\n\r\nNotice that we should not pass in the HTML of other plugins. Instead, just the metadata (\"computed\") values from the previous plugins.\"",
  "=> 2036458775 - gentlementlegen: \"This can work, but we skyrocket coupling and to me defeat purpose of plugins that should be unaware of each other. If any plugin has to understand the result of a previous plugin, it means these plugins have necessarily to co-exist so basically they become a single plugin with no purpose to split them.\""
]
    
    IMPORTANT CONTEXT:
    Consider all comments when evaluating connections. Relevant comments may appear before or after the comment being evaluated, so examine all of them closely.
    The comments in the start evaluating section will be in the format of: => {id}: "{comment}"
    
    START EVALUATING:
    [
  "=> 2036516869: \"> so basically they become a single plugin with no purpose to split them.\r\n\r\nI understand your concern and I would need to put more thought into composability. Maybe the `comment` proposal is bad after all due to your point. It's only useful with simple plugins, but for more complex ones your point is valid. \r\n\r\nI need help to think through how any partner in the future can create new plugins that modify \r\n1. permit reward amounts \r\n2. XP reward amounts\r\n\r\nI don't like the idea of having a single monolithic rewards generation plugin that wraps all the ways possible to compute rewards (i.e. time estimation, comment rewards, review rewards etc)\" - ",
  "=> 2036535332: \"@pavlovcik To mitigate that that's why inside the comment reward itself I also integrated that `Module` principle so code is not coupled tightly and easy to build on. There is as usual pros and cons to both approaches (splitting or not) but biggest pro is that comments get evaluated once in the same spot, so we save calls to OpenAPI and speed up the process. Also makes it only one configuration file in one location. We need to think about our best options there.\" - ",
  "=> 2051094255: \"I realized that to carry this task properly we need to handle flags for comment more delicately as they only indicate if the comment is `ISSUE | REVIEW` with the level `MEMBER | CONTRIBUTOR` etc. but doesn't specify if it is from a task, a specification and so on. Tags should be added to the config properly as well.\" - ",
  "=> 2053332029: \"> I realized that to carry this task properly we need to handle flags for comment more delicately as they only indicate if the comment is `ISSUE | REVIEW` with the level `MEMBER | CONTRIBUTOR` etc. but doesn't specify if it is from a task, a specification and so on. Tags should be added to the config properly as well.\n\nI see, so you're suggesting that we must annotate each comment as well in order to properly handle the scoring at the end?\n\nOff hand I think there's:\n\n1. Specification \n2. Issue comment\n3. Pull request comment (a normal comment on a pull request, not related to posting a \"review\"\n4. Pull request review comment (associated with a \"review state\" I.e. approved, left comments, requested changes)\n\nI suppose we need a better naming convention for the pull related ones. They are considered as separate entities according to the GitHub api. They require different methods to obtain both types. \" - ",
  "=> 2054424028: \"Agreed, I think currently there are 3 possible things to annotate on the comments:\r\n- `SPECIFICATION` for the issue itself, `TASK` for the related PR fixing it, or simply `COMMENT`\r\n- `MEMBER` or `CONTRIBUTOR` for the status of the member\r\n- `REVIEW` or `ISSUE` for where the comment was added\r\n\r\nI think this shall cover all cases.\" - ",
  "=> 2055783331: \"Consider calling it \"contributor\" and \"collaborator\" as that is how it is presented on the GitHub APIs as I recall. \n\nAlso I think you forgot about the \"review comments\"\" - "
]

    POST EVALUATION:
    Provide only the connection scores as floating-point values indicating the relevance of each comment based on its connection to the overall context.

    SCORING CRITERIA:
    Assign scores from 0 to 1, 0: Not related (e.g., spam), 1: Highly relevant (e.g., solutions, bugs)
    Consider the context of all comments; even minor details may be significant for resolving the issue. If a comment is unrelated to all comments, title, or issue specification, assign a score of 0.

    OUTPUT:
    Return a JSON object containing the ID and the connection score for each evaluated comment. The number of entries in the JSON response must match exactly 6.

Response

{
  "2036516869": 0.9,
  "2036535332": 0.8,
  "2051094255": 0.7,
  "2053332029": 0.6,
  "2054424028": 0.7,
  "2055783331": 0.5
}

I had to still put it in json array because the code blocks was confusing the LLM and this comment section as well

[0x4007]

Whats with the right arrows in the beginning of each comment?

[jaykayudo]

Whats with the right arrows in the beginning of each comment?

That was before i had to put it in json array. i used it to specify the start of a new comment for clearer understanding.

[jaykayudo]

Right arrow removed. same response

    Instruction: 
    Start by thoroughly reading all comments and retaining their content for the evaluation.
    The comments in the ALL COMMENTS section will be in the format of: => {id} - {author}: "{comment}"

    OUTPUT FORMAT:
    For every record in the START EVALUATING section, provide a JSON object in the following format: {ID: CONNECTION_SCORE}. 
    The average relevance based on the comments in the ALL COMMENTS SECTION should be reflected in the CONNECTION_SCORE.

    GLOBAL CONTEXT:
    Specification
    "In the v1 of the Ubiquibot, when a result gets evaluated, a recap is posted to the related issue with a summary and details of the rewards as well as the link to the rewards themselves Example: https://github.com/ubiquity/cloudflare-deploy-action/issues/9#issuecomment-2028623754

The same logic should be applied in the v2 version by creating a new Module responsible to post that comment. This module will receive a similar input than the one mentioned [here](https://github.com/ubiquity/cloudflare-deploy-action/issues/9#issuecomment-2028623754)

The module should be:
- possible to enable / disable
- eventually configurable (what data to show / hide)
- coming with tests"
    
    ALL COMMENTS:
    [
  "2030164289 - 0x4007: \"@whilefoo rfc on how we can deal with comment outputs. Perhaps we can have a standard recognized property on the output interface? Then the kernel can decide whether to pass it around or something. \r\n\r\n```ts\r\ninterface PluginOutput {\r\n  comment: string; // html comment\r\n  rewards: Rewards; // { \"whilefoo\": \"500\", \"token\": \"0x0\" } etc\r\n}\r\n```\"",
  "2033404518 - gentlementlegen: \"This needs https://github.com/ubiquibot/conversation-rewards/pull/7 to be merged first. Also probably needs https://github.com/ubiquibot/permit-generation/pull/2 to be able to generate the permits properly.\"",
  "2033488255 - 0x4007: \"> This needs https://github.com/ubiquibot/conversation-rewards/pull/7 to be merged first. Also probably needs https://github.com/ubiquibot/permit-generation/pull/2 to be able to generate the permits properly.\n\nI think you should fork from and overtake that second pull due to us being behind schedule \"",
  "2035427134 - whilefoo: \"> @whilefoo rfc on how we can deal with comment outputs. Perhaps we can have a standard recognized property on the output interface? Then the kernel can decide whether to pass it around or something.\r\n> \r\n> ```ts\r\n> interface PluginOutput {\r\n>   comment: string; // html comment\r\n>   rewards: Rewards; // { \"whilefoo\": \"500\", \"token\": \"0x0\" } etc\r\n> }\r\n> ```\r\n\r\nthere are a couple of options:\r\n1. we let the conversation-rewards plugin generate and post the comment\r\n2. we put comment as output and then another module is responsible for posting it or let conversation-rewards generate rewards and permit-generation generate permits and a third module that uses output from previous plugins to make a comment and post it\r\n3. we let the conversation-rewards plugin generate the comment and pass it as a standard property like you suggested.\r\n\r\nIn theory 2. option sounds good to separate concerns but it's another plugin which means another call to github actions thus more latency, so for the sake of speed it'd go with option 1 or 3, but going with these 2 options would mean there will 1 comment for rewards summary and 1 comment for permits.\r\nI'm not sure if option 3 is any better than option 1 because the plugin already has a token that has permissions to post comments so passing it to the kernel doesn't make much difference.\r\n\"",
  "2036174312 - gentlementlegen: \"To me 1 is the most straightforward to do for few reasons:\r\n- the comment reward plugin has all the needed data already\r\n- it can import the https://github.com/ubiquibot/permit-generation to generate permits itself\r\n- if it is done this way it can be used as a complete standalone without the kernel\r\n\r\n3 might make more sense in terms of architecture however. In such case the kernel should pass down results. It is more of an architecture question. Although, if we ever have other plugins in the flow that have influence on the total incentives, it would make sense to go through the kernel to aggregate the total result.\"",
  "2036355445 - 0x4007: \"> \r\n> there are a couple of options:\r\n> \r\n> 1. we let the conversation-rewards plugin generate and post the comment\r\n\r\nI think the most pure architecture would be that plugins can NOT inherit (write) authentication (only read if possible) of the kernel. As a consequence, no plugin should be able to post directly any issue. Ideally it should only be the kernel with a direct interface to issues. Plugins should just output comment HTML and the kernel can post it all in a single comment at the end of the webhook event invocation chain.\r\n\r\n> 2. we put comment as output and then another module is responsible for posting it or let conversation-rewards generate rewards and permit-generation generate permits and a third module that uses output from previous plugins to make a comment and post it\r\n\r\nCould be interesting to have a dedicated plugin to handle commenting, but because this seems like such an essential capability, I would more carefully consider the pros/cons of including this within the kernel. \r\n\r\n- On one hand, I really like the idea of making the kernel as pure and lean as possible\r\n- On the other hand there are practical considerations that if every plugin standard output interface includes `comment:` and then we don't have the `comment` capability available (the comment plugin is not included in the config) then why is this a standard output property required on every plugin?\r\n\r\n> 3. we let the conversation-rewards plugin generate the comment and pass it as a standard property like you suggested.\r\n> \r\n> In theory 2. option sounds good to separate concerns but it's another plugin which means another call to github actions thus more latency, so for the sake of speed it'd go with option 1 or 3, but going with these 2 options would mean there will 1 comment for rewards summary and 1 comment for permits. I'm not sure if option 3 is any better than option 1 because the plugin already has a token that has permissions to post comments so passing it to the kernel doesn't make much difference.\r\n\r\nI'm not concerned about latency now. Besides we can port any plugin to Cloudflare Workers (or combine several inside a single Worker) when we are ready to fix latency issues. To be honest though, except for setting labels, I have no problem with latency (like with permit generation) so there's no need to overengineer yet. I also love the GitHub Actions logs being available for easy debugging, and the fact that they are super generous with the compute. \r\n\r\nGiven the entire conversation and all the considerations, I definitely think we should do option 3. \r\n\r\nThis separates concerns the best, which will allow our plugin ecosystem to grow the fastest. \"",
  "2036367126 - gentlementlegen: \"I think each plugin should output JSON not html as it is not reliable to parse nor manipulate and requires `window` instance to be instantiated which is annoying on `node` based projects. \r\n\r\nHaving a plugin handling commenting seems quite weird as commenting is done by calling Octokit and the REST api which is already a library by itself, so no need to encapsulate it within another one to do the same thing.\r\n\r\nMy view on this, is to finalize https://github.com/ubiquibot/permit-generation/issues/5 to import it withing the `conversation-reward` that will post the comment itself as well, otherwise the architecture will be quite convoluted doing ping pong with everything.\"",
  "2036370459 - 0x4007: \"> I think each plugin should output JSON not html as it is not reliable to parse nor manipulate\r\n\r\nI know JSON makes things more complicated than it needs to be with serialization/transport. Then we have problems like this https://github.com/ubiquibot/conversation-rewards/issues/18. If the comment property is ONLY meant to be used for GitHub comments, then transporting HTML only is the most sensible. Besides, there's no other metadata thats necessary within the `comment` property, which is the main point of sending a JSON object (it has multiple properties.) \r\n\r\nAs a somewhat side note: there should also not be any variables inside of the HTML. We could look it as like server-side-rendering (or in this case, plugin-side-rendering) handle producing the final HTML output and then output it as a single string of html entities. \r\n\r\n> requires window instance to be instantiated which is annoying on node based projects.\r\n\r\nWe are currently using `mdast` in `@ubiquibot/comment-incentives` to generate virtual DOMs. No `window` needed.\r\n\r\n> My view on this, is to finalize https://github.com/ubiquibot/permit-generation/issues/5 to import it withing the conversation-reward that will post the comment itself as well, otherwise the architecture will be quite convoluted doing ping pong with everything.\r\n\r\nIf we agree that this is considered as technical debt, and that we are accruing this so that we can get back on schedule, ok.\"",
  "2036385985 - gentlementlegen: \"If you want to manipulate and convey data, HTML really is not made for this. If you want something formatted similarly but made for data we can use XML format.\r\nThe new comment reward actually does instantiate a DOM through [JSDOM](https://github.com/ubiquibot/conversation-rewards/blob/ba434761281446a23566cd02c68bd3b0e79d4eb1/src/parser/formatting-evaluator-module.ts#L80) to make things way simpler instead of using Regex everywhere which is highly unreliable. But there it makes sense because we are parsing comments from an HTML page content.\r\n\r\nBiggest advantage from this is to have the comment reward fully standalone, while easy to integrate with the kernel.\r\n\r\nIf we do something that handles the comment it means each and every module has to send it there and that module should understand every different content / format we send which would be way easier if the module itself handled its own comments, formatting wise.\"",
  "2036393020 - 0x4007: \"> If you want to manipulate and convey data, HTML really is not made for this.\r\n\r\nGoing back to my \"plugin-side-rendering\" mention, the data manipulation happens inside of the logic of the plugin. Then when the plugin is finally done with all of its compute, it emits a single string in the `comment` output property. This string is the final, rendered HTML. \r\n\r\nThe `comment` output is not intended to be consumed by other plugins. In most cases, the kernel will concatenate all the `comment` outputs from every plugin and post a single comment at the end. Plugins will consume the `metadata` property output which will include raw values to work with. I forget where I originally proposed this, but imagine something like:\r\n```typescript\r\n\r\ntype HtmlString = string;\r\ntype RequestedPermits = { username: string; amount: string; token: string; }[]\r\n \r\ninterface PluginOutput {\r\n   metadata: Record<string, any>;\r\n   comment: HtmlString;\r\n   rewards: RequestedPermits;\r\n}\r\n```\"",
  "2036411811 - gentlementlegen: \"But then how do we consider the formatting of that output?\r\n\r\nPractical case: we want to post a comment when a user queries a wallet. That comment is 'user name': 'wallet 0x0'\r\nKernel calls the comment plugin, saying that it wants a comment to be posted. Should the Kernel send the rendering it wants, should the comment plugin transform the data to HTML?\r\n\r\nThen, comment-reward wants to post the results. Should ask the Kernel to call the comment plugin, but then formatting is different. Should the Kernel notify the comment plugin that it wants a different output formatting? Should the Kernel compute beforehand the HTML and send it to the comment plugin?\r\n\r\nMight be something I don't grasp there. Because I do understand your use case but it seems to be very deterministic on what is the purpose of the plugins which kinda defeats the purpose of having plugins, looks more like a monolithic approach to me\"",
  "2036433646 - 0x4007: \"> But then how do we consider the formatting of that output?\r\n\r\nThe proposed `comment` output is intended for ease of composability for MOST situations, basically that the output from one plugin would just be appended to the final rendered comment. For simple plugin configurations (plugins not modifying the output results of others) this seems like straightforward architecture. \r\n\r\nHowever, as we know, there will be situations where a subsequent plugin will consider the results of a previous plugin, which means it would need to change the comment that is rendered. \r\n\r\nIn this situation the subsequent plugin should clobber the output of the previous plugin. It is now clear to be that this will be a new challenge to express to the kernel that it should ignore the comment output of a previous plugin. I suppose it would be straightforward in the metadata using the plugin ID i.e. `{ \"silences\": [\"ubiquibot/conversation-rewards\"] }`.\r\n\r\n> Kernel calls the comment plugin,\r\n\r\nI think that comments should be handled from within the kernel. There should not be a separate comment plugin. Read my explanation [here](https://github.com/ubiquibot/conversation-rewards/issues/5#issuecomment-2036355445). \r\n\r\n\r\n### Inputs\r\n\r\nFor completeness of my [previous comment](https://github.com/ubiquibot/conversation-rewards/issues/5#issuecomment-2036393020):\r\n\r\n```typescript\r\n\r\ntype PluginId = string; // i.e. `ubiquibot/conversation-rewards` `ubiquibot/assistive-pricing`\r\n \r\ninterface PluginInput {\r\n   metadata: Record<PluginId, any>;\r\n   context: GitHubEventContext;\r\n}\r\n```\r\n\r\nNotice that we should not pass in the HTML of other plugins. Instead, just the metadata (\"computed\") values from the previous plugins.\"",
  "2036458775 - gentlementlegen: \"This can work, but we skyrocket coupling and to me defeat purpose of plugins that should be unaware of each other. If any plugin has to understand the result of a previous plugin, it means these plugins have necessarily to co-exist so basically they become a single plugin with no purpose to split them.\""
]
    
    IMPORTANT CONTEXT:
    Consider all comments when evaluating connections. Relevant comments may appear before or after the comment being evaluated, so examine all of them closely.
    The comments in the start evaluating section will be in the format of: => {id}: "{comment}"
    
    START EVALUATING:
    [
  "2036516869: \"> so basically they become a single plugin with no purpose to split them.\r\n\r\nI understand your concern and I would need to put more thought into composability. Maybe the `comment` proposal is bad after all due to your point. It's only useful with simple plugins, but for more complex ones your point is valid. \r\n\r\nI need help to think through how any partner in the future can create new plugins that modify \r\n1. permit reward amounts \r\n2. XP reward amounts\r\n\r\nI don't like the idea of having a single monolithic rewards generation plugin that wraps all the ways possible to compute rewards (i.e. time estimation, comment rewards, review rewards etc)\" - ",
  "2036535332: \"@pavlovcik To mitigate that that's why inside the comment reward itself I also integrated that `Module` principle so code is not coupled tightly and easy to build on. There is as usual pros and cons to both approaches (splitting or not) but biggest pro is that comments get evaluated once in the same spot, so we save calls to OpenAPI and speed up the process. Also makes it only one configuration file in one location. We need to think about our best options there.\" - ",
  "2051094255: \"I realized that to carry this task properly we need to handle flags for comment more delicately as they only indicate if the comment is `ISSUE | REVIEW` with the level `MEMBER | CONTRIBUTOR` etc. but doesn't specify if it is from a task, a specification and so on. Tags should be added to the config properly as well.\" - ",
  "2053332029: \"> I realized that to carry this task properly we need to handle flags for comment more delicately as they only indicate if the comment is `ISSUE | REVIEW` with the level `MEMBER | CONTRIBUTOR` etc. but doesn't specify if it is from a task, a specification and so on. Tags should be added to the config properly as well.\n\nI see, so you're suggesting that we must annotate each comment as well in order to properly handle the scoring at the end?\n\nOff hand I think there's:\n\n1. Specification \n2. Issue comment\n3. Pull request comment (a normal comment on a pull request, not related to posting a \"review\"\n4. Pull request review comment (associated with a \"review state\" I.e. approved, left comments, requested changes)\n\nI suppose we need a better naming convention for the pull related ones. They are considered as separate entities according to the GitHub api. They require different methods to obtain both types. \" - ",
  "2054424028: \"Agreed, I think currently there are 3 possible things to annotate on the comments:\r\n- `SPECIFICATION` for the issue itself, `TASK` for the related PR fixing it, or simply `COMMENT`\r\n- `MEMBER` or `CONTRIBUTOR` for the status of the member\r\n- `REVIEW` or `ISSUE` for where the comment was added\r\n\r\nI think this shall cover all cases.\" - ",
  "2055783331: \"Consider calling it \"contributor\" and \"collaborator\" as that is how it is presented on the GitHub APIs as I recall. \n\nAlso I think you forgot about the \"review comments\"\" - "
]

    POST EVALUATION:
    Provide only the connection scores as floating-point values indicating the relevance of each comment based on its connection to the overall context.

    SCORING CRITERIA:
    Assign scores from 0 to 1, 0: Not related (e.g., spam), 1: Highly relevant (e.g., solutions, bugs)
    Consider the context of all comments; even minor details may be significant for resolving the issue. If a comment is unrelated to all comments, title, or issue specification, assign a score of 0.

    OUTPUT:
    Return a JSON object containing the ID and the connection score for each evaluated comment. The number of entries in the JSON response must match exactly 6.

[0x4007]

I would remove from array to conform to how the rest of the prompt is. Also clearer divisions of sections consider using headers ###

Lastly i would probably add a note about how we are using github flavored markdown and > indicates a block quote from another comment, and that those should not be considered as part of the scoring of that comment which is quoting another.

[jaykayudo]

Okay

[jaykayudo]

INSTRUCTION

Start by thoroughly reading all comments and retaining their content for the evaluation.
The comments in the ALL COMMENTS section will be in the format of: {id} - {author}: "{comment}"

OUTPUT FORMAT:

For every record in the START EVALUATING section, provide a JSON object in the following format: {ID: CONNECTION_SCORE}.
The average relevance based on the comments in the ALL COMMENTS SECTION should be reflected in the CONNECTION_SCORE.
Comments use GitHub-flavored Markdown, so words within symbols like ">" should not factor into scoring since they reference other comments.
Carefully distinguish between referenced words and regular words in the comments.

GLOBAL CONTEXT:

Specification
"In the v1 of the Ubiquibot, when a result gets evaluated, a recap is posted to the related issue with a summary and
details of the rewards as well as the link to the rewards themselves Example: https://github.com/ubiquity/cloudflare
deploy-action/issues/9#issuecomment-2028623754
The same logic should be applied in the v2 version by creating a new Module responsible to post that comment. This module will receive a similar input than the one mentioned here
The module should be:

• possible to enable / disable
• eventually configurable (what data to show / hide)
• coming with tests"

ALL COMMENTS:

2030164289 - 0x4007: "@whilefoo rfc on how we can deal with comment outputs. Perhaps we can have a standard recognized property on the output interface? Then the kernel can decide whether to pass it around or something.

interface PluginOutput {
  comment: string; // html comment
  rewards: Rewards; // { "whilefoo": "500", "token": "0x0" } etc
}
```"
2033404518 - gentlementlegen: "This needs https://github.com/ubiquibot/conversation-rewards/pull/7 to be merged first. Also probably needs https://github.com/ubiquibot/permit-generation/pull/2 to be able to generate the permits properly."
2033488255 - 0x4007: "> This needs https://github.com/ubiquibot/conversation-rewards/pull/7 to be merged first. Also probably needs https://github.com/ubiquibot/permit-generation/pull/2 to be able to generate the permits properly.

I think you should fork from and overtake that second pull due to us being behind schedule "
2035427134 - whilefoo: "> @whilefoo rfc on how we can deal with comment outputs. Perhaps we can have a standard recognized property on the output interface? Then the kernel can decide whether to pass it around or something.
> 
> ```ts
> interface PluginOutput {
>   comment: string; // html comment
>   rewards: Rewards; // { "whilefoo": "500", "token": "0x0" } etc
> }
> ```

there are a couple of options:
1. we let the conversation-rewards plugin generate and post the comment
2. we put comment as output and then another module is responsible for posting it or let conversation-rewards generate rewards and permit-generation generate permits and a third module that uses output from previous plugins to make a comment and post it
3. we let the conversation-rewards plugin generate the comment and pass it as a standard property like you suggested.

In theory 2. option sounds good to separate concerns but it's another plugin which means another call to github actions thus more latency, so for the sake of speed it'd go with option 1 or 3, but going with these 2 options would mean there will 1 comment for rewards summary and 1 comment for permits.
I'm not sure if option 3 is any better than option 1 because the plugin already has a token that has permissions to post comments so passing it to the kernel doesn't make much difference.
"
2036174312 - gentlementlegen: "To me 1 is the most straightforward to do for few reasons:
- the comment reward plugin has all the needed data already
- it can import the https://github.com/ubiquibot/permit-generation to generate permits itself
- if it is done this way it can be used as a complete standalone without the kernel

3 might make more sense in terms of architecture however. In such case the kernel should pass down results. It is more of an architecture question. Although, if we ever have other plugins in the flow that have influence on the total incentives, it would make sense to go through the kernel to aggregate the total result."
2036355445 - 0x4007: "> 
> there are a couple of options:
> 
> 1. we let the conversation-rewards plugin generate and post the comment

I think the most pure architecture would be that plugins can NOT inherit (write) authentication (only read if possible) of the kernel. As a consequence, no plugin should be able to post directly any issue. Ideally it should only be the kernel with a direct interface to issues. Plugins should just output comment HTML and the kernel can post it all in a single comment at the end of the webhook event invocation chain.

> 2. we put comment as output and then another module is responsible for posting it or let conversation-rewards generate rewards and permit-generation generate permits and a third module that uses output from previous plugins to make a comment and post it

Could be interesting to have a dedicated plugin to handle commenting, but because this seems like such an essential capability, I would more carefully consider the pros/cons of including this within the kernel. 

- On one hand, I really like the idea of making the kernel as pure and lean as possible
- On the other hand there are practical considerations that if every plugin standard output interface includes `comment:` and then we don't have the `comment` capability available (the comment plugin is not included in the config) then why is this a standard output property required on every plugin?

> 3. we let the conversation-rewards plugin generate the comment and pass it as a standard property like you suggested.
> 
> In theory 2. option sounds good to separate concerns but it's another plugin which means another call to github actions thus more latency, so for the sake of speed it'd go with option 1 or 3, but going with these 2 options would mean there will 1 comment for rewards summary and 1 comment for permits. I'm not sure if option 3 is any better than option 1 because the plugin already has a token that has permissions to post comments so passing it to the kernel doesn't make much difference.

I'm not concerned about latency now. Besides we can port any plugin to Cloudflare Workers (or combine several inside a single Worker) when we are ready to fix latency issues. To be honest though, except for setting labels, I have no problem with latency (like with permit generation) so there's no need to overengineer yet. I also love the GitHub Actions logs being available for easy debugging, and the fact that they are super generous with the compute. 

Given the entire conversation and all the considerations, I definitely think we should do option 3. 

This separates concerns the best, which will allow our plugin ecosystem to grow the fastest. "
2036367126 - gentlementlegen: "I think each plugin should output JSON not html as it is not reliable to parse nor manipulate and requires `window` instance to be instantiated which is annoying on `node` based projects. 

Having a plugin handling commenting seems quite weird as commenting is done by calling Octokit and the REST api which is already a library by itself, so no need to encapsulate it within another one to do the same thing.

My view on this, is to finalize https://github.com/ubiquibot/permit-generation/issues/5 to import it withing the `conversation-reward` that will post the comment itself as well, otherwise the architecture will be quite convoluted doing ping pong with everything."
2036370459 - 0x4007: "> I think each plugin should output JSON not html as it is not reliable to parse nor manipulate

I know JSON makes things more complicated than it needs to be with serialization/transport. Then we have problems like this https://github.com/ubiquibot/conversation-rewards/issues/18. If the comment property is ONLY meant to be used for GitHub comments, then transporting HTML only is the most sensible. Besides, there's no other metadata thats necessary within the `comment` property, which is the main point of sending a JSON object (it has multiple properties.) 

As a somewhat side note: there should also not be any variables inside of the HTML. We could look it as like server-side-rendering (or in this case, plugin-side-rendering) handle producing the final HTML output and then output it as a single string of html entities. 

> requires window instance to be instantiated which is annoying on node based projects.

We are currently using `mdast` in `@ubiquibot/comment-incentives` to generate virtual DOMs. No `window` needed.

> My view on this, is to finalize https://github.com/ubiquibot/permit-generation/issues/5 to import it withing the conversation-reward that will post the comment itself as well, otherwise the architecture will be quite convoluted doing ping pong with everything.

If we agree that this is considered as technical debt, and that we are accruing this so that we can get back on schedule, ok."
2036385985 - gentlementlegen: "If you want to manipulate and convey data, HTML really is not made for this. If you want something formatted similarly but made for data we can use XML format.
The new comment reward actually does instantiate a DOM through [JSDOM](https://github.com/ubiquibot/conversation-rewards/blob/ba434761281446a23566cd02c68bd3b0e79d4eb1/src/parser/formatting-evaluator-module.ts#L80) to make things way simpler instead of using Regex everywhere which is highly unreliable. But there it makes sense because we are parsing comments from an HTML page content.

Biggest advantage from this is to have the comment reward fully standalone, while easy to integrate with the kernel.

If we do something that handles the comment it means each and every module has to send it there and that module should understand every different content / format we send which would be way easier if the module itself handled its own comments, formatting wise."
2036393020 - 0x4007: "> If you want to manipulate and convey data, HTML really is not made for this.

Going back to my "plugin-side-rendering" mention, the data manipulation happens inside of the logic of the plugin. Then when the plugin is finally done with all of its compute, it emits a single string in the `comment` output property. This string is the final, rendered HTML. 

The `comment` output is not intended to be consumed by other plugins. In most cases, the kernel will concatenate all the `comment` outputs from every plugin and post a single comment at the end. Plugins will consume the `metadata` property output which will include raw values to work with. I forget where I originally proposed this, but imagine something like:
```typescript

type HtmlString = string;
type RequestedPermits = { username: string; amount: string; token: string; }[]
 
interface PluginOutput {
   metadata: Record<string, any>;
   comment: HtmlString;
   rewards: RequestedPermits;
}
```"
2036411811 - gentlementlegen: "But then how do we consider the formatting of that output?

Practical case: we want to post a comment when a user queries a wallet. That comment is 'user name': 'wallet 0x0'
Kernel calls the comment plugin, saying that it wants a comment to be posted. Should the Kernel send the rendering it wants, should the comment plugin transform the data to HTML?

Then, comment-reward wants to post the results. Should ask the Kernel to call the comment plugin, but then formatting is different. Should the Kernel notify the comment plugin that it wants a different output formatting? Should the Kernel compute beforehand the HTML and send it to the comment plugin?

Might be something I don't grasp there. Because I do understand your use case but it seems to be very deterministic on what is the purpose of the plugins which kinda defeats the purpose of having plugins, looks more like a monolithic approach to me"
2036433646 - 0x4007: "> But then how do we consider the formatting of that output?

The proposed `comment` output is intended for ease of composability for MOST situations, basically that the output from one plugin would just be appended to the final rendered comment. For simple plugin configurations (plugins not modifying the output results of others) this seems like straightforward architecture. 

However, as we know, there will be situations where a subsequent plugin will consider the results of a previous plugin, which means it would need to change the comment that is rendered. 

In this situation the subsequent plugin should clobber the output of the previous plugin. It is now clear to be that this will be a new challenge to express to the kernel that it should ignore the comment output of a previous plugin. I suppose it would be straightforward in the metadata using the plugin ID i.e. `{ "silences": ["ubiquibot/conversation-rewards"] }`.

> Kernel calls the comment plugin,

I think that comments should be handled from within the kernel. There should not be a separate comment plugin. Read my explanation [here](https://github.com/ubiquibot/conversation-rewards/issues/5#issuecomment-2036355445). 


### Inputs

For completeness of my [previous comment](https://github.com/ubiquibot/conversation-rewards/issues/5#issuecomment-2036393020):

```typescript

type PluginId = string; // i.e. `ubiquibot/conversation-rewards` `ubiquibot/assistive-pricing`
 
interface PluginInput {
   metadata: Record<PluginId, any>;
   context: GitHubEventContext;
}

Notice that we should not pass in the HTML of other plugins. Instead, just the metadata ("computed") values from the previous plugins."
2036458775 - gentlementlegen: "This can work, but we skyrocket coupling and to me defeat purpose of plugins that should be unaware of each other. If any plugin has to understand the result of a previous plugin, it means these plugins have necessarily to co-exist so basically they become a single plugin with no purpose to split them."

### IMPORTANT CONTEXT:
Consider all comments when evaluating connections. Relevant comments may appear before or after the comment being evaluated, so examine all of them closely.
The comments in the start evaluating section will be in the format of: {id}: "{comment}"

### START EVALUATING:
2036516869: "> so basically they become a single plugin with no purpose to split them.

I understand your concern and I would need to put more thought into composability. Maybe the comment proposal is bad after all due to your point. It's only useful with simple plugins, but for more complex ones your point is valid.

I need help to think through how any partner in the future can create new plugins that modify

1. permit reward amounts
2. XP reward amounts

I don't like the idea of having a single monolithic rewards generation plugin that wraps all the ways possible to compute rewards (i.e. time estimation, comment rewards, review rewards etc)" -
2036535332: "@pavlovcik To mitigate that that's why inside the comment reward itself I also integrated that Module principle so code is not coupled tightly and easy to build on. There is as usual pros and cons to both approaches (splitting or not) but biggest pro is that comments get evaluated once in the same spot, so we save calls to OpenAPI and speed up the process. Also makes it only one configuration file in one location. We need to think about our best options there." -
2051094255: "I realized that to carry this task properly we need to handle flags for comment more delicately as they only indicate if the comment is ISSUE | REVIEW with the level MEMBER | CONTRIBUTOR etc. but doesn't specify if it is from a task, a specification and so on. Tags should be added to the config properly as well." -
2053332029: "> I realized that to carry this task properly we need to handle flags for comment more delicately as they only indicate if the comment is ISSUE | REVIEW with the level MEMBER | CONTRIBUTOR etc. but doesn't specify if it is from a task, a specification and so on. Tags should be added to the config properly as well.

I see, so you're suggesting that we must annotate each comment as well in order to properly handle the scoring at the end?

Off hand I think there's:

1. Specification
2. Issue comment
3. Pull request comment (a normal comment on a pull request, not related to posting a "review"
4. Pull request review comment (associated with a "review state" I.e. approved, left comments, requested changes)

I suppose we need a better naming convention for the pull related ones. They are considered as separate entities according to the GitHub api. They require different methods to obtain both types. " -
2054424028: "Agreed, I think currently there are 3 possible things to annotate on the comments:

• SPECIFICATION for the issue itself, TASK for the related PR fixing it, or simply COMMENT
• MEMBER or CONTRIBUTOR for the status of the member
• REVIEW or ISSUE for where the comment was added

I think this shall cover all cases." -
2055783331: "Consider calling it "contributor" and "collaborator" as that is how it is presented on the GitHub APIs as I recall.

Also I think you forgot about the "review comments"" -

POST EVALUATION:

Provide only the connection scores as floating-point values indicating the relevance of each comment based on its connection to the overall context.

SCORING CRITERIA:

Assign scores from 0 to 1, 0: Not related (e.g., spam), 1: Highly relevant (e.g., solutions, bugs)
Consider the context of all comments; even minor details may be significant for resolving the issue. If a comment is unrelated to all comments, title, or issue specification, assign a score of 0.

OUTPUT:

Return a JSON object containing the ID and the connection score for each evaluated comment. The number of entries in the JSON response must match exactly 6.

Response

{
  "2036516869": 0.8,
  "2036535332": 0.7,
  "2051094255": 0.6,
  "2053332029": 0.8,
  "2054424028": 0.7,
  "2055783331": 0.5
}

[jaykayudo]

@0x4007 New prompt. any inconsistency will be because of the markdown in the comment samples.

[0x4007]

I meant some type of moderation capability that will hide the off topic (based on the specification) comments so that they don't get any credit.

[jaykayudo]

@0x4007 I decided to take a different approach. here is the new approach and the response

  Evaluate the relevance of GitHub comments to an issue. Provide a JSON object with comment IDs and their relevance scores.

  ### Issue: 
  In the v1 of the Ubiquibot, when a result gets evaluated, a recap is posted to the related issue with a summary and details of the rewards as well as the link to the rewards themselves Example: https://github.com/ubiquity/cloudflare-deploy-action/issues/9#issuecomment-2028623754

  The same logic should be applied in the v2 version by creating a new Module responsible to post that comment. This module will receive a similar input than the one mentioned [here](https://github.com/ubiquity/cloudflare-deploy-action/issues/9#issuecomment-2028623754)

  The module should be:
  - possible to enable / disable
  - eventually configurable (what data to show / hide)
  - coming with tests

  ### All comments:
  2030164289 - 0x4007: "@whilefoo rfc on how we can deal with comment outputs. Perhaps we can have a standard recognized property on the output interface? Then the kernel can decide whether to pass it around or something. 

  ```ts
  interface PluginOutput {
  comment: string; // html comment
  rewards: Rewards; // { "whilefoo": "500", "token": "0x0" } etc
  }
  ```"
  2033404518 - gentlementlegen: "This needs https://github.com/ubiquibot/conversation-rewards/pull/7 to be merged first. Also probably needs https://github.com/ubiquibot/permit-generation/pull/2 to be able to generate the permits properly."
  2033488255 - 0x4007: "> This needs https://github.com/ubiquibot/conversation-rewards/pull/7 to be merged first. Also probably needs https://github.com/ubiquibot/permit-generation/pull/2 to be able to generate the permits properly.

  I think you should fork from and overtake that second pull due to us being behind schedule "
  2035427134 - whilefoo: "> @whilefoo rfc on how we can deal with comment outputs. Perhaps we can have a standard recognized property on the output interface? Then the kernel can decide whether to pass it around or something.
  > 
  > ```ts
  > interface PluginOutput {
  >   comment: string; // html comment
  >   rewards: Rewards; // { "whilefoo": "500", "token": "0x0" } etc
  > }
  > ```

  there are a couple of options:
  1. we let the conversation-rewards plugin generate and post the comment
  2. we put comment as output and then another module is responsible for posting it or let conversation-rewards generate rewards and permit-generation generate permits and a third module that uses output from previous plugins to make a comment and post it
  3. we let the conversation-rewards plugin generate the comment and pass it as a standard property like you suggested.

  In theory 2. option sounds good to separate concerns but it's another plugin which means another call to github actions thus more latency, so for the sake of speed it'd go with option 1 or 3, but going with these 2 options would mean there will 1 comment for rewards summary and 1 comment for permits.
  I'm not sure if option 3 is any better than option 1 because the plugin already has a token that has permissions to post comments so passing it to the kernel doesn't make much difference.
  "
  2036174312 - gentlementlegen: "To me 1 is the most straightforward to do for few reasons:
  - the comment reward plugin has all the needed data already
  - it can import the https://github.com/ubiquibot/permit-generation to generate permits itself
  - if it is done this way it can be used as a complete standalone without the kernel

  3 might make more sense in terms of architecture however. In such case the kernel should pass down results. It is more of an architecture question. Although, if we ever have other plugins in the flow that have influence on the total incentives, it would make sense to go through the kernel to aggregate the total result."
  2036355445 - 0x4007: "> 
  > there are a couple of options:
  > 
  > 1. we let the conversation-rewards plugin generate and post the comment

  I think the most pure architecture would be that plugins can NOT inherit (write) authentication (only read if possible) of the kernel. As a consequence, no plugin should be able to post directly any issue. Ideally it should only be the kernel with a direct interface to issues. Plugins should just output comment HTML and the kernel can post it all in a single comment at the end of the webhook event invocation chain.

  > 2. we put comment as output and then another module is responsible for posting it or let conversation-rewards generate rewards and permit-generation generate permits and a third module that uses output from previous plugins to make a comment and post it

  Could be interesting to have a dedicated plugin to handle commenting, but because this seems like such an essential capability, I would more carefully consider the pros/cons of including this within the kernel. 

  - On one hand, I really like the idea of making the kernel as pure and lean as possible
  - On the other hand there are practical considerations that if every plugin standard output interface includes `comment:` and then we don't have the `comment` capability available (the comment plugin is not included in the config) then why is this a standard output property required on every plugin?

  > 3. we let the conversation-rewards plugin generate the comment and pass it as a standard property like you suggested.
  > 
  > In theory 2. option sounds good to separate concerns but it's another plugin which means another call to github actions thus more latency, so for the sake of speed it'd go with option 1 or 3, but going with these 2 options would mean there will 1 comment for rewards summary and 1 comment for permits. I'm not sure if option 3 is any better than option 1 because the plugin already has a token that has permissions to post comments so passing it to the kernel doesn't make much difference.

  I'm not concerned about latency now. Besides we can port any plugin to Cloudflare Workers (or combine several inside a single Worker) when we are ready to fix latency issues. To be honest though, except for setting labels, I have no problem with latency (like with permit generation) so there's no need to overengineer yet. I also love the GitHub Actions logs being available for easy debugging, and the fact that they are super generous with the compute. 

  Given the entire conversation and all the considerations, I definitely think we should do option 3. 

  This separates concerns the best, which will allow our plugin ecosystem to grow the fastest. "
  2036367126 - gentlementlegen: "I think each plugin should output JSON not html as it is not reliable to parse nor manipulate and requires `window` instance to be instantiated which is annoying on `node` based projects. 

  Having a plugin handling commenting seems quite weird as commenting is done by calling Octokit and the REST api which is already a library by itself, so no need to encapsulate it within another one to do the same thing.

  My view on this, is to finalize https://github.com/ubiquibot/permit-generation/issues/5 to import it withing the `conversation-reward` that will post the comment itself as well, otherwise the architecture will be quite convoluted doing ping pong with everything."
  2036370459 - 0x4007: "> I think each plugin should output JSON not html as it is not reliable to parse nor manipulate

  I know JSON makes things more complicated than it needs to be with serialization/transport. Then we have problems like this https://github.com/ubiquibot/conversation-rewards/issues/18. If the comment property is ONLY meant to be used for GitHub comments, then transporting HTML only is the most sensible. Besides, there's no other metadata thats necessary within the `comment` property, which is the main point of sending a JSON object (it has multiple properties.) 

  As a somewhat side note: there should also not be any variables inside of the HTML. We could look it as like server-side-rendering (or in this case, plugin-side-rendering) handle producing the final HTML output and then output it as a single string of html entities. 

  > requires window instance to be instantiated which is annoying on node based projects.

  We are currently using `mdast` in `@ubiquibot/comment-incentives` to generate virtual DOMs. No `window` needed.

  > My view on this, is to finalize https://github.com/ubiquibot/permit-generation/issues/5 to import it withing the conversation-reward that will post the comment itself as well, otherwise the architecture will be quite convoluted doing ping pong with everything.

  If we agree that this is considered as technical debt, and that we are accruing this so that we can get back on schedule, ok."
  2036385985 - gentlementlegen: "If you want to manipulate and convey data, HTML really is not made for this. If you want something formatted similarly but made for data we can use XML format.
  The new comment reward actually does instantiate a DOM through [JSDOM](https://github.com/ubiquibot/conversation-rewards/blob/ba434761281446a23566cd02c68bd3b0e79d4eb1/src/parser/formatting-evaluator-module.ts#L80) to make things way simpler instead of using Regex everywhere which is highly unreliable. But there it makes sense because we are parsing comments from an HTML page content.

  Biggest advantage from this is to have the comment reward fully standalone, while easy to integrate with the kernel.

  If we do something that handles the comment it means each and every module has to send it there and that module should understand every different content / format we send which would be way easier if the module itself handled its own comments, formatting wise."
  2036393020 - 0x4007: "> If you want to manipulate and convey data, HTML really is not made for this.

  Going back to my "plugin-side-rendering" mention, the data manipulation happens inside of the logic of the plugin. Then when the plugin is finally done with all of its compute, it emits a single string in the `comment` output property. This string is the final, rendered HTML. 

  The `comment` output is not intended to be consumed by other plugins. In most cases, the kernel will concatenate all the `comment` outputs from every plugin and post a single comment at the end. Plugins will consume the `metadata` property output which will include raw values to work with. I forget where I originally proposed this, but imagine something like:
  ```typescript

  type HtmlString = string;
  type RequestedPermits = { username: string; amount: string; token: string; }[]
  
  interface PluginOutput {
     metadata: Record<string, any>;
     comment: HtmlString;
     rewards: RequestedPermits;
  }
  ```"
  2036411811 - gentlementlegen: "But then how do we consider the formatting of that output?

  Practical case: we want to post a comment when a user queries a wallet. That comment is 'user name': 'wallet 0x0'
  Kernel calls the comment plugin, saying that it wants a comment to be posted. Should the Kernel send the rendering it wants, should the comment plugin transform the data to HTML?

  Then, comment-reward wants to post the results. Should ask the Kernel to call the comment plugin, but then formatting is different. Should the Kernel notify the comment plugin that it wants a different output formatting? Should the Kernel compute beforehand the HTML and send it to the comment plugin?

  Might be something I don't grasp there. Because I do understand your use case but it seems to be very deterministic on what is the purpose of the plugins which kinda defeats the purpose of having plugins, looks more like a monolithic approach to me"
  2036433646 - 0x4007: "> But then how do we consider the formatting of that output?

  The proposed `comment` output is intended for ease of composability for MOST situations, basically that the output from one plugin would just be appended to the final rendered comment. For simple plugin configurations (plugins not modifying the output results of others) this seems like straightforward architecture. 

  However, as we know, there will be situations where a subsequent plugin will consider the results of a previous plugin, which means it would need to change the comment that is rendered. 

  In this situation the subsequent plugin should clobber the output of the previous plugin. It is now clear to be that this will be a new challenge to express to the kernel that it should ignore the comment output of a previous plugin. I suppose it would be straightforward in the metadata using the plugin ID i.e. `{ "silences": ["ubiquibot/conversation-rewards"] }`.

  > Kernel calls the comment plugin,

  I think that comments should be handled from within the kernel. There should not be a separate comment plugin. Read my explanation [here](https://github.com/ubiquibot/conversation-rewards/issues/5#issuecomment-2036355445). 


  ### Inputs

  For completeness of my [previous comment](https://github.com/ubiquibot/conversation-rewards/issues/5#issuecomment-2036393020):

  ```typescript

  type PluginId = string; // i.e. `ubiquibot/conversation-rewards` `ubiquibot/assistive-pricing`
  
  interface PluginInput {
     metadata: Record<PluginId, any>;
     context: GitHubEventContext;
  }
  ```

  Notice that we should not pass in the HTML of other plugins. Instead, just the metadata ("computed") values from the previous plugins."
  2036458775 - gentlementlegen: "This can work, but we skyrocket coupling and to me defeat purpose of plugins that should be unaware of each other. If any plugin has to understand the result of a previous plugin, it means these plugins have necessarily to co-exist so basically they become a single plugin with no purpose to split them."

  Comments to evaluate:
  2036516869: "> so basically they become a single plugin with no purpose to split them.

  I understand your concern and I would need to put more thought into composability. Maybe the `comment` proposal is bad after all due to your point. It's only useful with simple plugins, but for more complex ones your point is valid. 

  I need help to think through how any partner in the future can create new plugins that modify 
  1. permit reward amounts 
  2. XP reward amounts

  I don't like the idea of having a single monolithic rewards generation plugin that wraps all the ways possible to compute rewards (i.e. time estimation, comment rewards, review rewards etc)" - 
  2036535332: "@pavlovcik To mitigate that that's why inside the comment reward itself I also integrated that `Module` principle so code is not coupled tightly and easy to build on. There is as usual pros and cons to both approaches (splitting or not) but biggest pro is that comments get evaluated once in the same spot, so we save calls to OpenAPI and speed up the process. Also makes it only one configuration file in one location. We need to think about our best options there." - 
  2051094255: "I realized that to carry this task properly we need to handle flags for comment more delicately as they only indicate if the comment is `ISSUE | REVIEW` with the level `MEMBER | CONTRIBUTOR` etc. but doesn't specify if it is from a task, a specification and so on. Tags should be added to the config properly as well." - 
  2053332029: "> I realized that to carry this task properly we need to handle flags for comment more delicately as they only indicate if the comment is `ISSUE | REVIEW` with the level `MEMBER | CONTRIBUTOR` etc. but doesn't specify if it is from a task, a specification and so on. Tags should be added to the config properly as well.

  I see, so you're suggesting that we must annotate each comment as well in order to properly handle the scoring at the end?

  Off hand I think there's:

  1. Specification 
  2. Issue comment
  3. Pull request comment (a normal comment on a pull request, not related to posting a "review"
  4. Pull request review comment (associated with a "review state" I.e. approved, left comments, requested changes)

  I suppose we need a better naming convention for the pull related ones. They are considered as separate entities according to the GitHub api. They require different methods to obtain both types. " - 
  2054424028: "Agreed, I think currently there are 3 possible things to annotate on the comments:
  - `SPECIFICATION` for the issue itself, `TASK` for the related PR fixing it, or simply `COMMENT`
  - `MEMBER` or `CONTRIBUTOR` for the status of the member
  - `REVIEW` or `ISSUE` for where the comment was added

  I think this shall cover all cases." - 
  2055783331: "Consider calling it "contributor" and "collaborator" as that is how it is presented on the GitHub APIs as I recall. 

  Also I think you forgot about the "review comments"" - 

  ### Instructions:
  1. Read all comments carefully, considering their context and content.
  2. Evaluate each comment in the "Comments to evaluate" section.
  3. Assign a relevance score from 0 to 1 for each comment:
     - 0: Not related (e.g., spam)
     - 1: Highly relevant (e.g., solutions, bug reports)
  4. Consider:
     - Relation to the issue description
     - Connection to other comments
     - Contribution to issue resolution
  5. Handle GitHub-flavored markdown:
     - Ignore text beginning with '>' as it references another comment
     - Distinguish between referenced text and the commenter's own words
     - Only evaluate the relevance of the commenter's original content
  6. Return only a JSON object: {ID: score}

  ### Notes:
  - Even minor details may be significant.
  - Comments may reference earlier or later comments.
  - The number of entries in the JSON response must equal 6.

[jaykayudo]

Response

{
  "2036516869": 0.8,
  "2036535332": 0.7,
  "2051094255": 0.6,
  "2053332029": 0.7,
  "2054424028": 0.5,
  "2055783331": 0.4
}

[0x4007]

Prompt looks decent

-  - Comments may reference earlier or later comments.
+  - Comments may reference earlier comments.

Your pull isn't updated. You should run this and link your GitHub Actions logs for QA.

[0x4007]

I already reviewed the latest state of this pull.

[jaykayudo]

Your pull isn't updated. You should run this and link your GitHub Actions logs for QA.

On it

[jaykayudo]

Pull Updated

[0x4007]

I think we need to ignore changelog from prettier. Also you should clobber dist/index.js and resolve so that this can be merged. @jaykayudo

[jaykayudo]

@0x4007 Conflicts resolved. Ready for merge.

[ubiquity-os-beta]

@jaykayudo, this task has been idle for a while. Please provide an update.

[gentlementlegen]

@jaykayudo After merge, the prompt build was always failing which is a problem. I reverted this pull-request. Here are the detailed logs

	⚠ Failed to run comment evaluation. SyntaxError: Unterminated string in JSON at position 27
	⚠ {
	    "caller": "/home/runner/work/text-conversation-rewards/text-conversation-rewards/dist/index.js:291:6135886"
	  }
	⚠   ↳  JSON.parse (<anonymous>)
	    ↳  ContentEvaluatorModule._submitPrompt (/home/runner/work/text-conversation-rewards/text-conversation-rewards/dist/index.js:291:5799849)
	    ↳  process.processTicksAndRejections (node:internal/process/task_queues:95:5)
	    ↳  async ContentEvaluatorModule._evaluateComments (/home/runner/work/text-conversation-rewards/text-conversation-rewards/dist/index.js:291:5799178)
	    ↳  async ContentEvaluatorModule._processComment (/home/runner/work/text-conversation-rewards/text-conversation-rewards/dist/index.js:291:5797980)
	    ↳  async Promise.all (index 1)
	    ↳  async ContentEvaluatorModule.transform (/home/runner/work/text-conversation-rewards/text-conversation-rewards/dist/index.js:291:5797545)
	    ↳  async Processor.run (/home/runner/work/text-conversation-rewards/text-conversation-rewards/dist/index.js:291:6133577)
	    ↳  async run (/home/runner/work/text-conversation-rewards/text-conversation-rewards/dist/index.js:291:6134971)

[jaykayudo]

@gentlementlegen I have looked through it thoroughly via the LLM and i found no issue in the json format. i think the issue came from when i fixed the conflicts in the dist/index.js file, i accepted the incoming changes instead of the current changes. Could that be the issue?

If so, i will make a new PR and have the bot recompile it, will that work?

[jaykayudo]

Also, will it the possible to get the logs of the raw prompt from this line. so as to know more of what i am dealing with.

[jaykayudo]

@gentlementlegen i seem to have discovered the issue, the LLM was returning an explanation for the result as well. The prompt did not specify to return only a json object. I will make a new PR with the new prompt.

[ubiquity-os-beta]

@jaykayudo, this task has been idle for a while. Please provide an update.

[0x4007]

This is unexpected

[gentlementlegen]

@0x4007 I think it does because ubiquity-os-marketplace/daemon-disqualifier#39 was merged and this task is priority 2 medium.

[ubiquity-os-beta]

@jaykayudo, this task has been idle for a while. Please provide an update.