-
Notifications
You must be signed in to change notification settings - Fork 658
Add a MCP tool that helps review alt text in context of surrounding text #6820
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
5 commits
Select commit
Hold shift + click to select a range
bd54558
Add a MCP tool that helps review alt text in context of surrounding text
kendallgassner 4ecc67f
add changeset
kendallgassner 101b43c
add details on how the review_alt_text tool should be used
kendallgassner 42fe3f0
clean up spelling
kendallgassner f98c701
Update next steps to avoid looping and allow file paths to be passed …
kendallgassner File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,5 @@ | ||
| --- | ||
| '@primer/mcp': minor | ||
| --- | ||
|
|
||
| Adds review_alt_text tool to the Primer MCP |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -629,4 +629,61 @@ The following list of coding guidelines must be followed: | |
| }, | ||
| ) | ||
|
|
||
| // ----------------------------------------------------------------------------- | ||
| // Accessibility | ||
| // ----------------------------------------------------------------------------- | ||
|
|
||
| /** | ||
| * The `review_alt_text` tool is experimental and may be removed in future versions. | ||
| * | ||
| * The intent of this tool is to assist products like Copilot Code Review and Copilot Coding Agent | ||
| * in reviewing both user- and AI-generated alt text for images, ensuring compliance with accessibility guidelines. | ||
| * This tool is not intended to replace human-generated alt text; rather, it supports the review process | ||
| * by providing suggestions for improvement. It should be used alongside human review, not as a substitute. | ||
| * | ||
| * | ||
| **/ | ||
| server.tool( | ||
| 'review_alt_text', | ||
| 'Evaluates image alt text against accessibility best practices and context relevance.', | ||
| { | ||
| surroundingText: z.string().describe('Text surrounding the image, relevant to the image.'), | ||
| alt: z.string().describe('The alt text of the image being evaluated'), | ||
| image: z | ||
| .union([ | ||
| z.instanceof(File).describe('The image src file being evaluated'), | ||
| z.string().url().describe('The URL of the image src being evaluated'), | ||
| z.string().describe('The file path of the image src being evaluated'), | ||
| ]) | ||
| .describe('The image file, file path, or URL being evaluated'), | ||
| }, | ||
| async ({surroundingText, alt, image}) => { | ||
| // Call the LLM through MCP sampling | ||
| const response = await server.server.createMessage({ | ||
| messages: [ | ||
| { | ||
| role: 'user', | ||
| content: { | ||
| type: 'text', | ||
| text: `Does this alt text: '${alt}' meet accessibility guidelines and describe the image: ${image} accurately in context of this surrounding text: '${surroundingText}'?\n\n`, | ||
| }, | ||
| }, | ||
| ], | ||
| sampling: {temperature: 0.4}, | ||
| maxTokens: 500, | ||
| }) | ||
|
|
||
| return { | ||
| content: [ | ||
| { | ||
| type: 'text', | ||
| text: response.content.type === 'text' ? response.content.text : 'Unable to generate summary', | ||
| }, | ||
| ], | ||
| altTextEvaluation: response.content.type === 'text' ? response.content.text : 'Unable to generate summary', | ||
|
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @joshblack I updated the |
||
| nextSteps: `If the evaluation indicates issues with the alt text, provide more meaningful alt text based on the feedback. DO NOT run this tool repeatedly on the same image - evaluations may vary slightly with each run.`, | ||
| } | ||
| }, | ||
| ) | ||
|
|
||
| export {server} | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
temperature-> Controls how “random” or “creative” the model’s output is. Low temperature is more predictable/conservative while a high temp is more creative/varied. I chose .4 (almost mid range) to allow some flexibility in how the LLM reviews alt text.