Skip to content

Improve openai_ask tool-use behavior#719

Merged
mokagio merged 12 commits into
trunkfrom
iangmaia/openai-ask-tool-safety-fixes
May 14, 2026
Merged

Improve openai_ask tool-use behavior#719
mokagio merged 12 commits into
trunkfrom
iangmaia/openai-ask-tool-safety-fixes

Conversation

@iangmaia
Copy link
Copy Markdown
Contributor

@iangmaia iangmaia commented May 12, 2026

What does it do?

Improves openai_ask follow-up behavior after the initial tool-use support:

  • Uses max_completion_tokens for Chat Completions requests.
  • Adds store: false to opt out of OpenAI request storage.
  • Validates function tool definitions before sending them to OpenAI.
  • Keeps malformed or unsupported tool calls from invoking local handlers.
  • Caps local tool execution without dropping the final allowed model response.
  • Avoids logging raw tool arguments, handler exception messages, or backtraces.

Checklist before requesting a review

  • Run bundle exec rubocop to test for code style violations and recommendations.
  • Add Unit Tests (aka specs/*_spec.rb) if applicable.
  • Run bundle exec rspec to run the whole test suite and ensure all your tests pass.
  • Make sure you added an entry in the CHANGELOG.md file to describe your changes under the appropriate existing ### subsection of the existing ## Trunk section.
  • If applicable, add an entry in the MIGRATION.md file to describe how the changes will affect the migration from the previous major version and what the clients will need to change and consider.

Copilot AI review requested due to automatic review settings May 12, 2026 12:54
@iangmaia iangmaia self-assigned this May 12, 2026
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR tightens and hardens openai_ask’s Chat Completions + tool-use loop behavior, with a focus on safer logging, stricter tool validation, and clearer iteration limiting semantics.

Changes:

  • Switch Chat Completions requests to max_completion_tokens and opt out of request storage via store: false.
  • Add validation for tool definitions and returned tool-call payloads, preventing malformed/unsupported tool calls from invoking local handlers.
  • Adjust the tool-use loop to cap local tool execution rounds while still allowing one final model turn to produce an answer; reduce sensitive data exposure in logs.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File Description
lib/fastlane/plugin/wpmreleasetoolkit/actions/common/openai_ask_action.rb Updates request parameters, adds tool/max-iteration validation, hardens tool-call execution and logging, and refines loop termination semantics.
spec/openai_ask_action_spec.rb Adds/updates specs for new request fields, iteration cap behavior, tool validation, and sensitive logging guarantees.
CHANGELOG.md Documents the openai_ask tool-use behavior improvements under Trunk bug fixes.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@iangmaia iangmaia requested a review from Copilot May 12, 2026 13:25
@iangmaia iangmaia requested review from jaclync, mokagio and twstokes May 12, 2026 13:28
@iangmaia iangmaia added bug Something isn't working enhancement New feature or request labels May 12, 2026
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

Comment thread lib/fastlane/plugin/wpmreleasetoolkit/actions/common/openai_ask_action.rb Outdated
module Actions
class OpenaiAskAction < Action
OPENAI_API_ENDPOINT = URI('https://api.openai.com/v1/chat/completions').freeze
DEFAULT_MAX_COMPLETION_TOKENS = 2048
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be good to document where this number comes from. What's its rationale?

I feel the 5 in the max tool iterations is much easier to accept at face value than 2048 tokens.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in 1e9c94c by documenting that DEFAULT_MAX_COMPLETION_TOKENS preserves the previous max_tokens ceiling while moving to the current API field.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! Also, only later in the review I realized that this was a previously introduced magic number, I should have acknowledge that in the comment.


validate_tools_array!(tools)
validate_max_tool_iterations!(max_tool_iterations)
validate_tools!(tools)
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nitpick: What do you think of an empty line to visually separate the validation from the execution?

Suggested change
validate_tools!(tools)
validate_tools!(tools)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Applied in 1e9c94c.

end

def self.tool_type(tool)
return '' unless tool.is_a?(Hash)
Copy link
Copy Markdown
Contributor

@mokagio mokagio May 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if returning nil would be clearer here.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. Changed tool_type to return nil for missing/non-hash definitions in 1e9c94c.

Comment thread CHANGELOG.md Outdated
Comment on lines +15 to +17
### Bug Fixes

_None_
- `openai_ask`: improve tool-use handling by requiring named function tools, using `max_completion_tokens`, opting out of OpenAI request storage, omitting sensitive tool diagnostics from logs, and refusing to execute additional tool calls after `max_tool_iterations`. [#719]
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this a bug fix then? All of the entries seem improvements to me. Should this be an internal changes or new feature entry?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's both 😅 omitting sensitive tool diagnostics from logs and not running additional tool calls after max_tool_iterations are actually bug fixes. But indeed other points are improvements.

Copy link
Copy Markdown
Contributor Author

@iangmaia iangmaia May 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated to split in 1e9c94c.

@mokagio mokagio merged commit 823c370 into trunk May 14, 2026
6 checks passed
@mokagio mokagio deleted the iangmaia/openai-ask-tool-safety-fixes branch May 14, 2026 05:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants