Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

repo-gardening: Update AI prompt for labelling #39558

Open
wants to merge 1 commit into
base: trunk
Choose a base branch
from

Conversation

inaikem
Copy link

@inaikem inaikem commented Sep 27, 2024

Following further tests, I updated the OpenAI prompt to ensure the 'Product,' 'Platform', and 'Type' labels are added.

Fixes #

Proposed changes:

Other information:

  • Have you written new tests for your changes, if applicable?
  • Have you checked the E2E test CI results, and verified that your changes do not break them?
  • Have you tested your changes on WordPress.com, if applicable (if so, you'll see a generated comment below with a script to run)?

Jetpack product discussion

Does this pull request change what data or activity we track or use?

Testing instructions:

  • Go to '..'

Following further tests, I updated the OpenAI prompt to ensure the 'Product,' 'Platform', and 'Type' labels are added.
@github-actions github-actions bot added [Action] Repo Gardening Github Action: manage PR and issues in your Open Source project Actions GitHub actions used to automate some of the work around releases and repository management labels Sep 27, 2024
Copy link
Contributor

Thank you for your PR!

When contributing to Jetpack, we have a few suggestions that can help us test and review your patch:

  • ✅ Include a description of your PR changes.
  • 🔴 Add a "[Status]" label (In Progress, Needs Team Review, ...).
  • ✅ Add testing instructions.
  • ✅ Specify whether this PR includes any changes to data or privacy.
  • 🔴 Add changelog entries to affected projects

This comment will be updated as you work on your PR and make changes. If you think that some of those checks are not needed for your PR, please explain why you think so. Thanks for cooperation 🤖


The e2e test report can be found here. Please note that it can take a few minutes after the e2e tests checks are complete for the report to be available.


🔴 Action required: Please add missing changelog entries for the following projects: projects/github-actions/repo-gardening

Use the Jetpack CLI tool to generate changelog entries by running the following command: jetpack changelog add.
Guidelines: /docs/writing-a-good-changelog-entry.md


Follow this PR Review Process:

  1. Ensure all required checks appearing at the bottom of this PR are passing.
  2. Choose a review path based on your changes:
    • A. Team Review: add the "[Status] Needs Team Review" label
      • For most changes, including minor cross-team impacts.
      • Example: Updating a team-specific component or a small change to a shared library.
    • B. Crew Review: add the "[Status] Needs Review" label
      • For significant changes to core functionality.
      • Example: Major updates to a shared library or complex features.
    • C. Both: Start with Team, then request Crew
      • For complex changes or when you need extra confidence.
      • Example: Refactor affecting multiple systems.
  3. Get at least one approval before merging.

Still unsure? Reach out in #jetpack-developers for guidance!

@github-actions github-actions bot added the [Status] Needs Author Reply We would need you to make some changes or provide some more details about your PR. Thank you! label Sep 27, 2024
@inaikem inaikem requested review from anomiex and jeherve and removed request for anomiex September 27, 2024 05:06
@inaikem
Copy link
Author

inaikem commented Sep 27, 2024

@anomiex 👋 This is my first time submitting a PR, and any help/guidance is appreciated. This change updates only the prompt used to generate issue labels.

@anomiex anomiex changed the title Update index.js repo-gardening: Update AI prompt for labelling Sep 27, 2024
Copy link
Contributor

@anomiex anomiex left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since you asked for feedback 🙂 regarding the formatting of the PR itself,

  • I adjusted the title for you to be more descriptive.
  • You should fill in the template in the description. What you added at the top is decent for the "Proposed changes" section, but "Testing instructions" would be helpful.

You'll need to add a change entry. If you haven't done a pnpm install in your local copy of the monorepo yet, do that. Then you should be able to run pnpm jetpack changelog add to generate one.

As for the PR itself, I'm not terribly familiar with LLM prompt engineering but I left a bunch of inline comments about stuff that would confuse me if trying to follow the instructions. 😅

@@ -66,11 +66,21 @@ ${ repoLabels
.join( '\n' ) }

Analyze the issue and suggest relevant labels. Rules:
# Updated prompt: Ensures 'Product', 'Platform' and 'Type' issues are always added.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the AI supposed to ignore this line because it looks like a comment?

- Use only existing labels provided.
- Include 1 '[Product]' label.
Copy link
Contributor

@anomiex anomiex Sep 27, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems like it will confuse the AI, since no "Product" labels are included in the list above and the AI was instructed to only use labels from the list above.

Comment on lines +74 to +75
- Include the "[Platform] Simple" AND/OR "[Platform] Atomic" labels as appropriate.
- Only if NOT present: Add 1 of the following labels based on your assessment of the issue type: '[Type] Bug', '[Type] Enhancement', or '[Type] Feature'.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here, neither "Platform" nor "Type" are included in the list above.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, I wonder whether the AI will have any clue about what either of the platforms are and when they might apply.

- Briefly explain each label choice in 1 sentence.
- Format your response as a JSON object, with each suggested label as a key, and your explanation of the label choice as the value.
- Format your response as a JSON object, with 'labels' and 'explanations' keys.- Use only existing labels provided.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This changed instruction does not match the example response format below.

Comment on lines +79 to +83
# Original prompt
#- Include 1 '[Feature Group]' label.
#- Include 1 to 3 '[Feature]' labels.
#- Briefly explain each label choice in 1 sentence.
#- Format your response as a JSON object, with each suggested label as a key, and your explanation of the label choice as the value.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

More comments the AI is supposed to figure out to ignore?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
[Action] Repo Gardening Github Action: manage PR and issues in your Open Source project Actions GitHub actions used to automate some of the work around releases and repository management [Status] Needs Author Reply We would need you to make some changes or provide some more details about your PR. Thank you!
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants