Migrate from using readline to prompt-toolkit

[tleonhardt]

This PR migrates from using GNU Readline for user input, tab-completion, and history to using prompt-toolkit for the same purpose.

prompt-toolkit is a pure-Python readline replacement that is fully cross-platform compatible and doesn't rely on the presence of underlying C dynamic libraries. It also opens the door to some advanced features in the future.

All use of readline is removed and the cmd2.rl_utils module has been deleted. There is a new cmd2.pt_utils module in its place. Currently all linting is passing and the documentation builds. This is essentially final draft that needs more testing. There are likely a some things that need to be considered related to asynicio support, signal handlers, and blocking popen calls.

All tests are passing on all platforms. I have done some manual testing on both macOS and Linux. But I don't have a Windows VM to do any manual testing on Windows however.

NOTE FOR REVIEWERS: The substantive code changes are in the 3 following files (all in the cmd2 directory):

• argparse_completer.py (modified)
• cmd2.py (modified)
• pt_utils.py (new)

TODO:

• Fix so tab-completion hints shown up above or below the prompt but not in the the bottom bar
• Fix tab completion for for command/sub-command names and arguments so it uses prompt-toolkit completion
• Fix subcommand suggestion for alias create <TAB> - currently showing nothing
• Increaes code coverage for new code
• Modify getting_started.py example to show how to use the optional bottom toolbar feature
• Update documentation
• Self review and code / comment cleeanup
• Check type-hints for new/modified methods and functions
• Verify that all arguments to cmd2.Cmd.read_input work like they did previously - test using examples/read_input.py
• Do a bunch of manual testing related to edge cases on all platforms
  • Windows
  • macOS
  • Linux
• Think about interactions with signal handlers
  • Ensure <Ctrl>+c works for SIGINT when using shell command - need to fix how prompt is redrawn after
  • Ensure terminated processes still properly handle SIGHUP/SIGTERM so that cmd2 exit handler runs to save persistent history and such
  • Should we worry about SIGKILL? - used to work fine and works normally now, but not anymore if inside ppaged (get traceback with termios.error)
• Thank about blocking calls to things like popen
  • Ensure output redirection works, to both file and clipboard
  • Ensure piping command output to shell command works
  • Ensure ppaged works properly like it did before - works, but doesn't handle SIGKILL correctly - get traceback
• Provide a flag to use readline-style completions instead of the prompt-toolkit style
• Dynamically switch from CompleteStyle.COLUMN to CompleteStyle.MULTI_COLUMN when there are a lot of completions - based on a settable which defaults to 7; switching disabled when user specifies READLINE_LIKE style
• Test in embedded py and ipy shells
  • Make sure locals self still works correctly in ipy
  • Make sure locals self still work correctly in py
  • See if we can get tab-completion working in embedded Python py shell for systems with LibEdit
• Think about whether there might be a race condition risk with self._in_prompt - I added a threadling.lock object to protect access to self._in_prompt to be safe
• Make sure we are happy with test coverage for new code
• Think about asyncio cases - prompt-toolkit natively uses asycnio and starts an event loop
  • Fix examples/async_call.py example to account for prompt-toolkit starting an asyncio event loop
  • Investigate whether cmd2 can use asyncio to replace things like self.terminal_lock, async_alert, and async_update_prompt - removed self.terminal_lock and updated sync_alert, and other stuff as well as async_printing.py example; also removed unused async_refresh_prompt and need_prompt_refresh methods
  • Can we remove the requirement for cmd2 apps to run in the main thread? - I don't think so. From what I can see, prompt-toolkit needs to run in the main thread for various reasons, though it does have good support for handling backkground tasks in other threads
  • Review example for how include async do_* commands is in examples/async_commands.py - if we are happy with it, we can move the decorator there into cmd2/decorators.py and the utility function into cmd2.utils.pyand add some unit tests
• Make sure changes are well documented in both CHANGELOG.md and in the Zensical documentation

[github-actions]

🤖 Hi @tleonhardt, I've received your request, and I'm working on it now! You can track my progress in the logs for more details.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.17%. Comparing base (cb0c75e) to head (73abb7d).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1553      +/-   ##
==========================================
+ Coverage   98.94%   99.17%   +0.22%     
==========================================
  Files          21       21              
  Lines        4942     4940       -2     
==========================================
+ Hits         4890     4899       +9     
+ Misses         52       41      -11     

Flag	Coverage Δ
unittests	99.17% <100.00%> (+0.22%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[github-actions]

🤖 I'm sorry @tleonhardt, but I was unable to process your request. Please see the logs for more details.

[notnotbrandon]

These are some observations / thoughts I had. I hope these are helpful!

1 Uncancelled Tasks

I changed examples/async_commands.py to sleep 10s, and ctrl+ced the task before completion. I was able to run other commands (help for example in the screenshot) and the command still completed in the background (see the "async work complete" print in the screenshot). This seems inconsistent with cancellation of synchronous commands, and could lead to unintuitive behavior. If you explore async cancellation, see also item 3 below.

2 Thread Unclosed

Updating examples/async_commands.py to look like the above, results in this:

Minor thing... might be worth keeping a refcount to the global thread var, and closing the thread when the (or all) cmd2 objects quit?

3 AnyIO

When writing async python in the past, I have preferred the use of anyio. Not sure if it is helpful, but this page is at least worth a read / being aware of. Its a small lib. They solve some edge cases that can arise when using asyncio, especially around cancellation.

[tleonhardt]

@notnotbrandon Thanks for the feedback on the new async_commands.py example.

I fixed the problem with uncancelled async tasks when an async do_* command is cancelled with a keyboard interrupt.

There is only ever 1 thread created for the async commands and it is a daemon thread. The main thread will automatically clean it up when it is ready to exit. So I'm not worried about that.

I don't think we want to add any extra dependencies on cmd2 itself at this time, but I'll keep AnyIO in mind for the future.