Tfs01 First check-in of WA Python client libraries, tests, and build scripts. - #5
Closed
dfugate wants to merge 7 commits into
Closed
Tfs01 First check-in of WA Python client libraries, tests, and build scripts.#5dfugate wants to merge 7 commits into
dfugate wants to merge 7 commits into
Conversation
Member
|
Latest updates are in #6 |
AutorestCI
added a commit
that referenced
this pull request
Apr 20, 2018
Fix Error status, fix version (#5) * Remove error schema * Add default response back * Fix abandoned as system error * Fix Error status * Use preview tag * Fix tag * Remove error response schema as they are not used
AutorestCI
added a commit
that referenced
this pull request
Sep 7, 2018
Merge pull request #5 from ritwikbasu/ProtocolTypeChange Protocol type change
AutorestCI
added a commit
that referenced
this pull request
Sep 18, 2018
Merge pull request #5 from hosunmsft/add_hq_node_to_adla_job_contract add hierarchyQueueNode to JobInformation(Basic)
AutorestCI
added a commit
that referenced
this pull request
Dec 4, 2018
Merge pull request #5 from leonardbf/netapp-api-update-public NFSAAS-1505 status codes
AutorestCI
added a commit
that referenced
this pull request
Dec 5, 2018
* Generated from 164da1fe5b4cdf8db0afd9f55a08ab523e1c3f0e Merge pull request #4 from leonardbf/netapp-api-update-public NFSAAS-1505 resource and property changes * Generated from 5892e13494715eb9a24653407544e23ddb00efc4 Merge pull request #5 from leonardbf/netapp-api-update-public NFSAAS-1505 status codes
lmazuel
pushed a commit
that referenced
this pull request
Jan 3, 2019
* [AutoPR netapp/resource-manager] Microsoft.NetApp first version (#3144) * Generated from c82f7e0779efda2a9a97a99664679711ce8d7e73 Microsoft.NetApp first version * Generated from 51268a3aa9892dabce143e347f225bf74be4cc49 Addressing pullrequest comments * [AutoPR netapp/resource-manager] NetApp adding subnetId to volume properties (#3321) * Generated from 37a648d4d8bbc6b77cc7cbc25b55f95f0b65b9fa NetApp adding subnetId to volume properties * Packaging update of azure-mgmt-netapp * [AutoPR netapp/resource-manager] NetApp RP minor update and validation fixes (#3839) * Generated from 4535ffbe9b0b3be960171b9c8efcdb042f34ce64 Merge pull request #1 from leonardbf/netapp-api-update-public NFSAAS-1505 minor update and validation fixes * Packaging update of azure-mgmt-netapp * Generated from 964e181cc1542e5bec928cbb27ef3d144f24b71e (#3956) typo: netapp/resource-manager/Microsoft.NetApp - possiblly -> possibly - targets's -> target's * [AutoPR netapp/resource-manager] Resource and property changes (#3992) * Generated from 164da1fe5b4cdf8db0afd9f55a08ab523e1c3f0e Merge pull request #4 from leonardbf/netapp-api-update-public NFSAAS-1505 resource and property changes * Generated from 5892e13494715eb9a24653407544e23ddb00efc4 Merge pull request #5 from leonardbf/netapp-api-update-public NFSAAS-1505 status codes * Generated from 4f47c60ec82a1eefc22a7524b7dff99624499d67 (#4046) Merge pull request #9 from leonardbf/mt-sc-update NFSAAS-1644 remove mt get add status for put * Update sdk_packaging.toml * Packaging update of azure-mgmt-netapp * Update version.py * Update HISTORY.rst
YijunXieMS
added a commit
that referenced
this pull request
Jun 3, 2019
* Move to under sdk * Remove policies * Remove debugging files * Rename Offset to EventPosition * make tests a namespace package * Revised test receive for new code * Revised test send for track two * Update async code from sync * Revise async receive and send live test for track2 * Use uamqp 1.2 * Resolve code review feedback * add queue_message to async sender * send_batch receives both list and iterator * Update after adp review * send accepts EventData, list, iteratable * Event Hub Track 2 (#5) * Initial commit * Initial commit * Initial commit * event hub client * Update README.md * Update README.md Fix typos * Memory leak * Support timestamp filter * Support timestamp filter * Update README.md * Add sender and refactor * Added abstract classes Todo - Migrate Base Class Wireframes - Migrate Azure Classes * First draft of class wires directly ported from .net (might be some minor gaps) * send example * Set allowed sasl mechs * Remove client.py * Receiver update * Add dummy send api * logging updates * Error handling, reconnect and logging * Add app properties to event data * unbind transport on connection close * timestamp filter on py2 * module version * Reconnect once when link/session/connection close * Add SessionPolicy * Add client info * Updates - Cleaned wireframes to be PEP compliant - Implemented single partition pump and single event_hub partition pump scenario Todo - Add Unit Tests for partition pump and event hub partition pump - Implement Partition Manager - Implement Checkpointing and Lease Managment * Updates - Cleaned wireframes to be PEP compliant - Implemented single partition pump and single event_hub partition pump scenario Todo - Add Unit Tests for partition pump and event hub partition pump - Implement Partition Manager - Implement Checkpointing and Lease Managment * run client in non-blocking mode * Added unit testing * Implemented the following functionality - Azure_storage_checkpoint_manager - AzureBlobLease isExpired Todo Implement partition manager Implement partition context Test full implementation * Implemented Processing of First Epoh Todo - Fix lease bug that is breaking subsequent epochs * Changes - Completed End to End EPH Flow - Removed storage dependancy on downloading full blob to check lease state Todo - Add thread and queue for checking lease state and other storage operations - Ensure eventhub client shuts down properly - Find way to update partition pumps without restarting them - Other optimizations * Move examples out * Changes - Added thread pool executor to enable conncurent execution of partitions - Removed partition pump dependency on max_batch Todo - Ensure eventhub client shuts down properly (This is causing errors) - Add thread pool for making checkpoint code conccurent - Add thread and queue for checking lease state and other storage operations to enable async - Find way to reassign active partition pumps without restarting them - Other optimizations * Add async receive * Changes - Added logs - Fixed error causing client to prematurely shutdown * Manual link flow control for async receive * Workaround for stuck async receiver * Local variable names * Changes - Optimized logging and comments Todo - Add concurecny mechanim for azure storage - Depricate partition pump event queue and update to latest version of the client * Create Dockerfile * Stuck async receiver * credit keeps increasing in async receiver * Changes - Added asnyc event hub client support - Optimized logging and comments Todo - Add concurecny mechanim for azure storage * Updated docker file as requested * Added EPH example * Fix hardcoded HTTP header * Made suggested changes * Bug fix - Fixed event loop bugs. In windows eventloop is thread dependent but in ubuntu the eventloop is threadsafe so you need to differentiate the thread specific eventloop from the host one. * Updated loop naming convention to be consistent * Added option to pass asyncio event_loop to eph * Updated docker file * Fixed critical bug with partition manager and aquirec mechanisiims Todo : Identitfy and fix remaining bug that is causing all pumps to shut down when a second host starts * Bug fixes - Fixed bug where closing a pump closed a host - Fixed bug where error partitioned were not removed - Fixed bug where leases were renewed at an incorrect interval * Updated file headers Removed author reference * - Fixed bug in eph example that caused host to terminate prematurely - Made the lease renewal and checkpoint creation "multithreaded" * Increase the size of the connection pool The default connection pool size was too small for scenarios where multiple partitions were handled by one EventProcessorHost. If the amount of partitions handled is large, we might end up doing very many connections at the same time due to the multi-threaded blob-handling. For this reason, you might hit the OS limits that restrict the number of open files per process that in MacOS is not very big. This can be worked around with something like: `ulimit -n 2560` * Decrease info logging verbosity * added ability to toggle pump shutdown when all messages on a pump are processed. * Install also eventhubsprocessor * Default to keeping the pumps It is more optimal to keep the pumps alive even if there are no messages so that it is faster to pickup when messages start to arrive. * Pipe and event injector for Windows * Event injector updates * EHClient refactoring. EHClient leaks. Sender part 1. * Send support * ren eventhubsprocessor eventprocessorhost * Changes - Added event hub config to simplify installation story * Changes - Added optional eventprocessor_params for passing context to the event processor - Made the storage manager mandatatory * Fix memory leaks * logging * Fix: 1. process crash due to race in client stop and connection remote close. 2. handle client close in async receiver. 3. fail pending sends when sender is closed. 4. some debug logging. * tests * test: recv from multiple partitions * test utility * logging update * Support callback based send for high throughput * Workaroud memory issue in proton.reactor.ApplicationEvent * renamed eventprocessor to eventprocessorhost for consistency * updated docker file * fixed typo in url * Added amqp port to address * Updated sample documentation since url is auto encoded by config * Updated docs * Implement timeout for send * Async sender and example * Close injector pipe * Use send timer to also check queued messages * Add partition pump loop to partition_context This gives the EventProcessor access to the partition_pump loop object. This way if One desires to run synchronous code inside process_events_async one can utilize the loop object to run the synchronous code using await context.pump_loop.run_in_executor(None, bla) * Include details in send error * Release deliveries when sender is closed * added validation to unquoted sas key * added support for custom eventhub client prefetch size * Update README.md * Update README.md * Added Docker instructions and fixed Dockerfile (#18) * Removed Dockerfile from the main folder and fixed Dockerfile example * Added build and run Dockerfile documentation * Update Readme * Removed rm qpid-proton folder * Removed /usr/share copy * Disallow a sender/receiver to be registered more than once * Make everything async in EPH I have removed all usage of threads thoroughout the code. Using threads to run pumps etc. Causes async code written into the event-processor to become caotic (you need to follow which loop is currently being used in the call to prevent loops not being found or using the wrong loop (There is the main loop and then loops that are created inside threads) Things become caotic when the event processor is being called by objects that run under different loops. So, no Threading except usage of asyncio run_in_executor. This is done mostly for azure blob api calls. Also changed the bla_async methods to not block. this way, when calling open_async for the the event-processor-host, the command will exit once the EPH is started. Due to the above, see the edited example/eph.py where I added a monitor that makes sure the EPH is still running (Could be replaced by loop.run_forever()) in the example file I have also incorporated a test class for gracefully killing the EPH after 30 seconds. this works, nevertheless takes a while to close as we are waiting for timeouts on the eventhubs connections. * Started removing proton code * Removed most of proton _impl * Removed more code * Working sender * Updates to sender * Added some tests/samples * Some progress on clients * Fixed samples * Added azure namespace * #25 Partition key cannot be set for events * Updated version * Updated README * Renamed package to eventhub * Started EPH modifications * Updated imports * Fixed target urls * Updated logging * Updated async message receive * updated test imports * Added mgmt call to get eh info * Updated samples * Updated receive test * Added send and receive test clients * Updated uamqp dependency * Merged updates from dev * Fixed typos * Updated EPH sample * Started docstrings * Converted tests to pytest * Updates to batch receive * Started adding docstrings * More docstrings * bumped version * Started porting test suite * More tests and improvements * Moved eph tests * Some sample cleanup * Some test updates * Some test restructure * Docstring cleanup * Fixed some merge artifacts * Fixed formatting error * Removed delivery count * Nested package directory * Support custom URL suffix * Support custom URL suffix * Support for EventData device ID * Reverted nested directory * Updated release notes * Workaround for partitionkey * Finished partition key workaround * beta2 fixes * pylint fixes * Trigger CI * Test fixes * Added package manifest * Added warning for Python 2.7 support Support for issues #36 and #38 * Started adding scenario tests * More test scenarios * Better docstring formatting * Started iothub support * Fixed long running test * Fixed typo and memory leak * Restructure * IoThub support * Updates for RC1 release * Fix long running test * Docstring and sample cleanups * Working on error retry * Improved error processing * Fixed partition manager * Progress on IotHub error * Some test updates * Updated uamqp dependency * Restructure for independent connections * Added HTTP proxy support Fix for issue #41 * Fixed some tests + samples * pylint fixes * bumped version * Added keepalive config and some eph fixes * Made reconnect configurable * Added more EPH options * Bumped version * Pylint fix * Pylint fix * Added send and auth timeouts * Changed log formatting. Retry on reconnect * Pylint fixes * Renamed internal async module * Updated send example to match recv Fix for issue #56 * Added build badge to readme * Fix for repeat startup * Added more storage connect options to EPH * Bumped version * Handler blocked until client started * Added event data methods * Fix pylint * Fix 3.7 CI * Fix 3.7 CI * Updated pylint version * Pylint fixes * Updated README * Fixed readme badge refresh * Fixed bug in Azure namespace package * Updated manifest * Parse enqueued time as UTC Fixes #72. * Updates for release 1.2.0 (#81) * Made setup 2.7 compatible * Separated async tests * Support 2.7 types * Bumped version * Added non-ascii tests * Fix CI * Fix Py27 pylint * Added iot sample * Updated sender/receiver client opening * bumped version * Updated tests * Fixed test name * Fixed test env settings * Skip eph test * Updates for v1.3.0 (#91) * Added support for storing the state of the Event Processor along the Checkpoint. Both Checkpoint and the EP state are stored as pickled objects. * Fixing pylint complaints. * Switched from pickle back to JSON for lease persistence. * Fixes bug when accessing leases that don't contain EP context. Also, minor renaming. * Better SAS token support * Fixed pylint * Improved auth error handling * Test stabilization * Improved stored EPH context * Updated EPH context storing * Skip test on OSX * Skip tests on OSX Fail due to large message body bug. * Some cleanup * Fixed error handling * Improved SAS token parsing * Fixed datetime offset (#99) * Fixed datetime offset * Updated pylint * Removed 3.4 pylint pass * Fixed bug in error handling (#100) * Migrate event hub sdk to central repo 1. add verifiable code snippets into docstring 2. update readme according to the template 3. add livetest mark and config 4. optimize code layout/structure * 1. document formatting 2. separate async/sync example tests * Fix build error: 1. uamqp dependency mismatch 2. rename test_examples in eventhub to avoid mismatch * This should fix build error * remove tests import and add sys path to solve build error * add live test for sending BatchEvent with application_properties, new live test passed with new uamqp wheel locally installed * Add get_partition_info in Event Hub * add get_partition_info * Add telemetry information to the connection properties * Disable smart split in batch message * change epoch to exclusive_receiver_priority * fix small problem * remove uamqp dependency * Eventhub track2 (#6) * Initial commit * Initial commit * Initial commit * event hub client * Update README.md * Update README.md Fix typos * Memory leak * Support timestamp filter * Support timestamp filter * Update README.md * Add sender and refactor * Added abstract classes Todo - Migrate Base Class Wireframes - Migrate Azure Classes * First draft of class wires directly ported from .net (might be some minor gaps) * send example * Set allowed sasl mechs * Remove client.py * Receiver update * Add dummy send api * logging updates * Error handling, reconnect and logging * Add app properties to event data * unbind transport on connection close * timestamp filter on py2 * module version * Reconnect once when link/session/connection close * Add SessionPolicy * Add client info * Updates - Cleaned wireframes to be PEP compliant - Implemented single partition pump and single event_hub partition pump scenario Todo - Add Unit Tests for partition pump and event hub partition pump - Implement Partition Manager - Implement Checkpointing and Lease Managment * Updates - Cleaned wireframes to be PEP compliant - Implemented single partition pump and single event_hub partition pump scenario Todo - Add Unit Tests for partition pump and event hub partition pump - Implement Partition Manager - Implement Checkpointing and Lease Managment * run client in non-blocking mode * Added unit testing * Implemented the following functionality - Azure_storage_checkpoint_manager - AzureBlobLease isExpired Todo Implement partition manager Implement partition context Test full implementation * Implemented Processing of First Epoh Todo - Fix lease bug that is breaking subsequent epochs * Changes - Completed End to End EPH Flow - Removed storage dependancy on downloading full blob to check lease state Todo - Add thread and queue for checking lease state and other storage operations - Ensure eventhub client shuts down properly - Find way to update partition pumps without restarting them - Other optimizations * Move examples out * Changes - Added thread pool executor to enable conncurent execution of partitions - Removed partition pump dependency on max_batch Todo - Ensure eventhub client shuts down properly (This is causing errors) - Add thread pool for making checkpoint code conccurent - Add thread and queue for checking lease state and other storage operations to enable async - Find way to reassign active partition pumps without restarting them - Other optimizations * Add async receive * Changes - Added logs - Fixed error causing client to prematurely shutdown * Manual link flow control for async receive * Workaround for stuck async receiver * Local variable names * Changes - Optimized logging and comments Todo - Add concurecny mechanim for azure storage - Depricate partition pump event queue and update to latest version of the client * Create Dockerfile * Stuck async receiver * credit keeps increasing in async receiver * Changes - Added asnyc event hub client support - Optimized logging and comments Todo - Add concurecny mechanim for azure storage * Updated docker file as requested * Added EPH example * Fix hardcoded HTTP header * Made suggested changes * Bug fix - Fixed event loop bugs. In windows eventloop is thread dependent but in ubuntu the eventloop is threadsafe so you need to differentiate the thread specific eventloop from the host one. * Updated loop naming convention to be consistent * Added option to pass asyncio event_loop to eph * Updated docker file * Fixed critical bug with partition manager and aquirec mechanisiims Todo : Identitfy and fix remaining bug that is causing all pumps to shut down when a second host starts * Bug fixes - Fixed bug where closing a pump closed a host - Fixed bug where error partitioned were not removed - Fixed bug where leases were renewed at an incorrect interval * Updated file headers Removed author reference * - Fixed bug in eph example that caused host to terminate prematurely - Made the lease renewal and checkpoint creation "multithreaded" * Increase the size of the connection pool The default connection pool size was too small for scenarios where multiple partitions were handled by one EventProcessorHost. If the amount of partitions handled is large, we might end up doing very many connections at the same time due to the multi-threaded blob-handling. For this reason, you might hit the OS limits that restrict the number of open files per process that in MacOS is not very big. This can be worked around with something like: `ulimit -n 2560` * Decrease info logging verbosity * added ability to toggle pump shutdown when all messages on a pump are processed. * Install also eventhubsprocessor * Default to keeping the pumps It is more optimal to keep the pumps alive even if there are no messages so that it is faster to pickup when messages start to arrive. * Pipe and event injector for Windows * Event injector updates * EHClient refactoring. EHClient leaks. Sender part 1. * Send support * ren eventhubsprocessor eventprocessorhost * Changes - Added event hub config to simplify installation story * Changes - Added optional eventprocessor_params for passing context to the event processor - Made the storage manager mandatatory * Fix memory leaks * logging * Fix: 1. process crash due to race in client stop and connection remote close. 2. handle client close in async receiver. 3. fail pending sends when sender is closed. 4. some debug logging. * tests * test: recv from multiple partitions * test utility * logging update * Support callback based send for high throughput * Workaroud memory issue in proton.reactor.ApplicationEvent * renamed eventprocessor to eventprocessorhost for consistency * updated docker file * fixed typo in url * Added amqp port to address * Updated sample documentation since url is auto encoded by config * Updated docs * Implement timeout for send * Async sender and example * Close injector pipe * Use send timer to also check queued messages * Add partition pump loop to partition_context This gives the EventProcessor access to the partition_pump loop object. This way if One desires to run synchronous code inside process_events_async one can utilize the loop object to run the synchronous code using await context.pump_loop.run_in_executor(None, bla) * Include details in send error * Release deliveries when sender is closed * added validation to unquoted sas key * added support for custom eventhub client prefetch size * Update README.md * Update README.md * Added Docker instructions and fixed Dockerfile (#18) * Removed Dockerfile from the main folder and fixed Dockerfile example * Added build and run Dockerfile documentation * Update Readme * Removed rm qpid-proton folder * Removed /usr/share copy * Disallow a sender/receiver to be registered more than once * Make everything async in EPH I have removed all usage of threads thoroughout the code. Using threads to run pumps etc. Causes async code written into the event-processor to become caotic (you need to follow which loop is currently being used in the call to prevent loops not being found or using the wrong loop (There is the main loop and then loops that are created inside threads) Things become caotic when the event processor is being called by objects that run under different loops. So, no Threading except usage of asyncio run_in_executor. This is done mostly for azure blob api calls. Also changed the bla_async methods to not block. this way, when calling open_async for the the event-processor-host, the command will exit once the EPH is started. Due to the above, see the edited example/eph.py where I added a monitor that makes sure the EPH is still running (Could be replaced by loop.run_forever()) in the example file I have also incorporated a test class for gracefully killing the EPH after 30 seconds. this works, nevertheless takes a while to close as we are waiting for timeouts on the eventhubs connections. * Started removing proton code * Removed most of proton _impl * Removed more code * Working sender * Updates to sender * Added some tests/samples * Some progress on clients * Fixed samples * Added azure namespace * #25 Partition key cannot be set for events * Updated version * Updated README * Renamed package to eventhub * Started EPH modifications * Updated imports * Fixed target urls * Updated logging * Updated async message receive * updated test imports * Added mgmt call to get eh info * Updated samples * Updated receive test * Added send and receive test clients * Updated uamqp dependency * Merged updates from dev * Fixed typos * Updated EPH sample * Started docstrings * Converted tests to pytest * Updates to batch receive * Started adding docstrings * More docstrings * bumped version * Started porting test suite * More tests and improvements * Moved eph tests * Some sample cleanup * Some test updates * Some test restructure * Docstring cleanup * Fixed some merge artifacts * Fixed formatting error * Removed delivery count * Nested package directory * Support custom URL suffix * Support custom URL suffix * Support for EventData device ID * Reverted nested directory * Updated release notes * Workaround for partitionkey * Finished partition key workaround * beta2 fixes * pylint fixes * Trigger CI * Test fixes * Added package manifest * Added warning for Python 2.7 support Support for issues #36 and #38 * Started adding scenario tests * More test scenarios * Better docstring formatting * Started iothub support * Fixed long running test * Fixed typo and memory leak * Restructure * IoThub support * Updates for RC1 release * Fix long running test * Docstring and sample cleanups * Working on error retry * Improved error processing * Fixed partition manager * Progress on IotHub error * Some test updates * Updated uamqp dependency * Restructure for independent connections * Added HTTP proxy support Fix for issue #41 * Fixed some tests + samples * pylint fixes * bumped version * Added keepalive config and some eph fixes * Made reconnect configurable * Added more EPH options * Bumped version * Pylint fix * Pylint fix * Added send and auth timeouts * Changed log formatting. Retry on reconnect * Pylint fixes * Renamed internal async module * Updated send example to match recv Fix for issue #56 * Added build badge to readme * Fix for repeat startup * Added more storage connect options to EPH * Bumped version * Handler blocked until client started * Added event data methods * Fix pylint * Fix 3.7 CI * Fix 3.7 CI * Updated pylint version * Pylint fixes * Updated README * Fixed readme badge refresh * Fixed bug in Azure namespace package * Updated manifest * Parse enqueued time as UTC Fixes #72. * Updates for release 1.2.0 (#81) * Made setup 2.7 compatible * Separated async tests * Support 2.7 types * Bumped version * Added non-ascii tests * Fix CI * Fix Py27 pylint * Added iot sample * Updated sender/receiver client opening * bumped version * Updated tests * Fixed test name * Fixed test env settings * Skip eph test * Updates for v1.3.0 (#91) * Added support for storing the state of the Event Processor along the Checkpoint. Both Checkpoint and the EP state are stored as pickled objects. * Fixing pylint complaints. * Switched from pickle back to JSON for lease persistence. * Fixes bug when accessing leases that don't contain EP context. Also, minor renaming. * Better SAS token support * Fixed pylint * Improved auth error handling * Test stabilization * Improved stored EPH context * Updated EPH context storing * Skip test on OSX * Skip tests on OSX Fail due to large message body bug. * Some cleanup * Fixed error handling * Improved SAS token parsing * Fixed datetime offset (#99) * Fixed datetime offset * Updated pylint * Removed 3.4 pylint pass * Fixed bug in error handling (#100) * Migrate event hub sdk to central repo 1. add verifiable code snippets into docstring 2. update readme according to the template 3. add livetest mark and config 4. optimize code layout/structure * 1. document formatting 2. separate async/sync example tests * Fix build error: 1. uamqp dependency mismatch 2. rename test_examples in eventhub to avoid mismatch * This should fix build error * remove tests import and add sys path to solve build error * add live test for sending BatchEvent with application_properties, new live test passed with new uamqp wheel locally installed * Add get_partition_info in Event Hub * add get_partition_info * Add telemetry information to the connection properties * Disable smart split in batch message * 1. Add amqp over websocket test 2. Add proxy sample 3. Update some comment and code * Changes from cross-lang * Change debug to network_tracing * Sync Client Constructor * auto_reconnect True and keep_alive None * consumer_group $default * hide open() * partition -> partition_id * credentials -> credential in init * set running=true after opened * Eventhub track2 - Update livetest (#7) * Update README.md * Update README.md Fix typos * Memory leak * Support timestamp filter * Support timestamp filter * Update README.md * Add sender and refactor * Added abstract classes Todo - Migrate Base Class Wireframes - Migrate Azure Classes * First draft of class wires directly ported from .net (might be some minor gaps) * send example * Set allowed sasl mechs * Remove client.py * Receiver update * Add dummy send api * logging updates * Error handling, reconnect and logging * Add app properties to event data * unbind transport on connection close * timestamp filter on py2 * module version * Reconnect once when link/session/connection close * Add SessionPolicy * Add client info * Updates - Cleaned wireframes to be PEP compliant - Implemented single partition pump and single event_hub partition pump scenario Todo - Add Unit Tests for partition pump and event hub partition pump - Implement Partition Manager - Implement Checkpointing and Lease Managment * Updates - Cleaned wireframes to be PEP compliant - Implemented single partition pump and single event_hub partition pump scenario Todo - Add Unit Tests for partition pump and event hub partition pump - Implement Partition Manager - Implement Checkpointing and Lease Managment * run client in non-blocking mode * Added unit testing * Implemented the following functionality - Azure_storage_checkpoint_manager - AzureBlobLease isExpired Todo Implement partition manager Implement partition context Test full implementation * Implemented Processing of First Epoh Todo - Fix lease bug that is breaking subsequent epochs * Changes - Completed End to End EPH Flow - Removed storage dependancy on downloading full blob to check lease state Todo - Add thread and queue for checking lease state and other storage operations - Ensure eventhub client shuts down properly - Find way to update partition pumps without restarting them - Other optimizations * Move examples out * Changes - Added thread pool executor to enable conncurent execution of partitions - Removed partition pump dependency on max_batch Todo - Ensure eventhub client shuts down properly (This is causing errors) - Add thread pool for making checkpoint code conccurent - Add thread and queue for checking lease state and other storage operations to enable async - Find way to reassign active partition pumps without restarting them - Other optimizations * Add async receive * Changes - Added logs - Fixed error causing client to prematurely shutdown * Manual link flow control for async receive * Workaround for stuck async receiver * Local variable names * Changes - Optimized logging and comments Todo - Add concurecny mechanim for azure storage - Depricate partition pump event queue and update to latest version of the client * Create Dockerfile * Stuck async receiver * credit keeps increasing in async receiver * Changes - Added asnyc event hub client support - Optimized logging and comments Todo - Add concurecny mechanim for azure storage * Updated docker file as requested * Added EPH example * Fix hardcoded HTTP header * Made suggested changes * Bug fix - Fixed event loop bugs. In windows eventloop is thread dependent but in ubuntu the eventloop is threadsafe so you need to differentiate the thread specific eventloop from the host one. * Updated loop naming convention to be consistent * Added option to pass asyncio event_loop to eph * Updated docker file * Fixed critical bug with partition manager and aquirec mechanisiims Todo : Identitfy and fix remaining bug that is causing all pumps to shut down when a second host starts * Bug fixes - Fixed bug where closing a pump closed a host - Fixed bug where error partitioned were not removed - Fixed bug where leases were renewed at an incorrect interval * Updated file headers Removed author reference * - Fixed bug in eph example that caused host to terminate prematurely - Made the lease renewal and checkpoint creation "multithreaded" * Increase the size of the connection pool The default connection pool size was too small for scenarios where multiple partitions were handled by one EventProcessorHost. If the amount of partitions handled is large, we might end up doing very many connections at the same time due to the multi-threaded blob-handling. For this reason, you might hit the OS limits that restrict the number of open files per process that in MacOS is not very big. This can be worked around with something like: `ulimit -n 2560` * Decrease info logging verbosity * added ability to toggle pump shutdown when all messages on a pump are processed. * Install also eventhubsprocessor * Default to keeping the pumps It is more optimal to keep the pumps alive even if there are no messages so that it is faster to pickup when messages start to arrive. * Pipe and event injector for Windows * Event injector updates * EHClient refactoring. EHClient leaks. Sender part 1. * Send support * ren eventhubsprocessor eventprocessorhost * Changes - Added event hub config to simplify installation story * Changes - Added optional eventprocessor_params for passing context to the event processor - Made the storage manager mandatatory * Fix memory leaks * logging * Fix: 1. process crash due to race in client stop and connection remote close. 2. handle client close in async receiver. 3. fail pending sends when sender is closed. 4. some debug logging. * tests * test: recv from multiple partitions * test utility * logging update * Support callback based send for high throughput * Workaroud memory issue in proton.reactor.ApplicationEvent * renamed eventprocessor to eventprocessorhost for consistency * updated docker file * fixed typo in url * Added amqp port to address * Updated sample documentation since url is auto encoded by config * Updated docs * Implement timeout for send * Async sender and example * Close injector pipe * Use send timer to also check queued messages * Add partition pump loop to partition_context This gives the EventProcessor access to the partition_pump loop object. This way if One desires to run synchronous code inside process_events_async one can utilize the loop object to run the synchronous code using await context.pump_loop.run_in_executor(None, bla) * Include details in send error * Release deliveries when sender is closed * added validation to unquoted sas key * added support for custom eventhub client prefetch size * Update README.md * Update README.md * Added Docker instructions and fixed Dockerfile (#18) * Removed Dockerfile from the main folder and fixed Dockerfile example * Added build and run Dockerfile documentation * Update Readme * Removed rm qpid-proton folder * Removed /usr/share copy * Disallow a sender/receiver to be registered more than once * Make everything async in EPH I have removed all usage of threads thoroughout the code. Using threads to run pumps etc. Causes async code written into the event-processor to become caotic (you need to follow which loop is currently being used in the call to prevent loops not being found or using the wrong loop (There is the main loop and then loops that are created inside threads) Things become caotic when the event processor is being called by objects that run under different loops. So, no Threading except usage of asyncio run_in_executor. This is done mostly for azure blob api calls. Also changed the bla_async methods to not block. this way, when calling open_async for the the event-processor-host, the command will exit once the EPH is started. Due to the above, see the edited example/eph.py where I added a monitor that makes sure the EPH is still running (Could be replaced by loop.run_forever()) in the example file I have also incorporated a test class for gracefully killing the EPH after 30 seconds. this works, nevertheless takes a while to close as we are waiting for timeouts on the eventhubs connections. * Started removing proton code * Removed most of proton _impl * Removed more code * Working sender * Updates to sender * Added some tests/samples * Some progress on clients * Fixed samples * Added azure namespace * #25 Partition key cannot be set for events * Updated version * Updated README * Renamed package to eventhub * Started EPH modifications * Updated imports * Fixed target urls * Updated logging * Updated async message receive * updated test imports * Added mgmt call to get eh info * Updated samples * Updated receive test * Added send and receive test clients * Updated uamqp dependency * Merged updates from dev * Fixed typos * Updated EPH sample * Started docstrings * Converted tests to pytest * Updates to batch receive * Started adding docstrings * More docstrings * bumped version * Started porting test suite * More tests and improvements * Moved eph tests * Some sample cleanup * Some test updates * Some test restructure * Docstring cleanup * Fixed some merge artifacts * Fixed formatting error * Removed delivery count * Nested package directory * Support custom URL suffix * Support custom URL suffix * Support for EventData device ID * Reverted nested directory * Updated release notes * Workaround for partitionkey * Finished partition key workaround * beta2 fixes * pylint fixes * Trigger CI * Test fixes * Added package manifest * Added warning for Python 2.7 support Support for issues #36 and #38 * Started adding scenario tests * More test scenarios * Better docstring formatting * Started iothub support * Fixed long running test * Fixed typo and memory leak * Restructure * IoThub support * Updates for RC1 release * Fix long running test * Docstring and sample cleanups * Working on error retry * Improved error processing * Fixed partition manager * Progress on IotHub error * Some test updates * Updated uamqp dependency * Restructure for independent connections * Added HTTP proxy support Fix for issue #41 * Fixed some tests + samples * pylint fixes * bumped version * Added keepalive config and some eph fixes * Made reconnect configurable * Added more EPH options * Bumped version * Pylint fix * Pylint fix * Added send and auth timeouts * Changed log formatting. Retry on reconnect * Pylint fixes * Renamed internal async module * Updated send example to match recv Fix for issue #56 * Added build badge to readme * Fix for repeat startup * Added more storage connect options to EPH * Bumped version * Handler blocked until client started * Added event data methods * Fix pylint * Fix 3.7 CI * Fix 3.7 CI * Updated pylint version * Pylint fixes * Updated README * Fixed readme badge refresh * Fixed bug in Azure namespace package * Updated manifest * Parse enqueued time as UTC Fixes #72. * Updates for release 1.2.0 (#81) * Made setup 2.7 compatible * Separated async tests * Support 2.7 types * Bumped version * Added non-ascii tests * Fix CI * Fix Py27 pylint * Added iot sample * Updated sender/receiver client opening * bumped version * Updated tests * Fixed test name * Fixed test env settings * Skip eph test * Updates for v1.3.0 (#91) * Added support for storing the state of the Event Processor along the Checkpoint. Both Checkpoint and the EP state are stored as pickled objects. * Fixing pylint complaints. * Switched from pickle back to JSON for lease persistence. * Fixes bug when accessing leases that don't contain EP context. Also, minor renaming. * Better SAS token support * Fixed pylint * Improved auth error handling * Test stabilization * Improved stored EPH context * Updated EPH context storing * Skip test on OSX * Skip tests on OSX Fail due to large message body bug. * Some cleanup * Fixed error handling * Improved SAS token parsing * Fixed datetime offset (#99) * Fixed datetime offset * Updated pylint * Removed 3.4 pylint pass * Fixed bug in error handling (#100) * Migrate event hub sdk to central repo 1. add verifiable code snippets into docstring 2. update readme according to the template 3. add livetest mark and config 4. optimize code layout/structure * 1. document formatting 2. separate async/sync example tests * Fix build error: 1. uamqp dependency mismatch 2. rename test_examples in eventhub to avoid mismatch * This should fix build error * remove tests import and add sys path to solve build error * add live test for sending BatchEvent with application_properties, new live test passed with new uamqp wheel locally installed * Add get_partition_info in Event Hub * add get_partition_info * Add telemetry information to the connection properties * Disable smart split in batch message * 1. Add amqp over websocket test 2. Add proxy sample 3. Update some comment and code * update some test code * Add __str__ to EventData * Update test code * Add eh error classes * EventHubError extends AzureError * Fix EventPosition default value issue * change $default to $Default * Handle TokenAuthError * wait for ready in _reconnect * fix get_partition_ids issue * Fix reconnect issue * small fix * fix async live test * Eventhub track2 Live test update (#8) * Set allowed sasl mechs * Remove client.py * Receiver update * Add dummy send api * logging updates * Error handling, reconnect and logging * Add app properties to event data * unbind transport on connection close * timestamp filter on py2 * module version * Reconnect once when link/session/connection close * Add SessionPolicy * Add client info * Updates - Cleaned wireframes to be PEP compliant - Implemented single partition pump and single event_hub partition pump scenario Todo - Add Unit Tests for partition pump and event hub partition pump - Implement Partition Manager - Implement Checkpointing and Lease Managment * Updates - Cleaned wireframes to be PEP compliant - Implemented single partition pump and single event_hub partition pump scenario Todo - Add Unit Tests for partition pump and event hub partition pump - Implement Partition Manager - Implement Checkpointing and Lease Managment * run client in non-blocking mode * Added unit testing * Implemented the following functionality - Azure_storage_checkpoint_manager - AzureBlobLease isExpired Todo Implement partition manager Implement partition context Test full implementation * Implemented Processing of First Epoh Todo - Fix lease bug that is breaking subsequent epochs * Changes - Completed End to End EPH Flow - Removed storage dependancy on downloading full blob to check lease state Todo - Add thread and queue for checking lease state and other storage operations - Ensure eventhub client shuts down properly - Find way to update partition pumps without restarting them - Other optimizations * Move examples out * Changes - Added thread pool executor to enable conncurent execution of partitions - Removed partition pump dependency on max_batch Todo - Ensure eventhub client shuts down properly (This is causing errors) - Add thread pool for making checkpoint code conccurent - Add thread and queue for checking lease state and other storage operations to enable async - Find way to reassign active partition pumps without restarting them - Other optimizations * Add async receive * Changes - Added logs - Fixed error causing client to prematurely shutdown * Manual link flow control for async receive * Workaround for stuck async receiver * Local variable names * Changes - Optimized logging and comments Todo - Add concurecny mechanim for azure storage - Depricate partition pump event queue and update to latest version of the client * Create Dockerfile * Stuck async receiver * credit keeps increasing in async receiver * Changes - Added asnyc event hub client support - Optimized logging and comments Todo - Add concurecny mechanim for azure storage * Updated docker file as requested * Added EPH example * Fix hardcoded HTTP header * Made suggested changes * Bug fix - Fixed event loop bugs. In windows eventloop is thread dependent but in ubuntu the eventloop is threadsafe so you need to differentiate the thread specific eventloop from the host one. * Updated loop naming convention to be consistent * Added option to pass asyncio event_loop to eph * Updated docker file * Fixed critical bug with partition manager and aquirec mechanisiims Todo : Identitfy and fix remaining bug that is causing all pumps to shut down when a second host starts * Bug fixes - Fixed bug where closing a pump closed a host - Fixed bug where error partitioned were not removed - Fixed bug where leases were renewed at an incorrect interval * Updated file headers Removed author reference * - Fixed bug in eph example that caused host to terminate prematurely - Made the lease renewal and checkpoint creation "multithreaded" * Increase the size of the connection pool The default connection pool size was too small for scenarios where multiple partitions were handled by one EventProcessorHost. If the amount of partitions handled is large, we might end up doing very many connections at the same time due to the multi-threaded blob-handling. For this reason, you might hit the OS limits that restrict the number of open files per process that in MacOS is not very big. This can be worked around with something like: `ulimit -n 2560` * Decrease info logging verbosity * added ability to toggle pump shutdown when all messages on a pump are processed. * Install also eventhubsprocessor * Default to keeping the pumps It is more optimal to keep the pumps alive even if there are no messages so that it is faster to pickup when messages start to arrive. * Pipe and event injector for Windows * Event injector updates * EHClient refactoring. EHClient leaks. Sender part 1. * Send support * ren eventhubsprocessor eventprocessorhost * Changes - Added event hub config to simplify installation story * Changes - Added optional eventprocessor_params for passing context to the event processor - Made the storage manager mandatatory * Fix memory leaks * logging * Fix: 1. process crash due to race in client stop and connection remote close. 2. handle client close in async receiver. 3. fail pending sends when sender is closed. 4. some debug logging. * tests * test: recv from multiple partitions * test utility * logging update * Support callback based send for high throughput * Workaroud memory issue in proton.reactor.ApplicationEvent * renamed eventprocessor to eventprocessorhost for consistency * updated docker file * fixed typo in url * Added amqp port to address * Updated sample documentation since url is auto encoded by config * Updated docs * Implement timeout for send * Async sender and example * Close injector pipe * Use send timer to also check queued messages * Add partition pump loop to partition_context This gives the EventProcessor access to the partition_pump loop object. This way if One desires to run synchronous code inside process_events_async one can utilize the loop object to run the synchronous code using await context.pump_loop.run_in_executor(None, bla) * Include details in send error * Release deliveries when sender is closed * added validation to unquoted sas key * added support for custom eventhub client prefetch size * Update README.md * Update README.md * Added Docker instructions and fixed Dockerfile (#18) * Removed Dockerfile from the main folder and fixed Dockerfile example * Added build and run Dockerfile documentation * Update Readme * Removed rm qpid-proton folder * Removed /usr/share copy * Disallow a sender/receiver to be registered more than once * Make everything async in EPH I have removed all usage of threads thoroughout the code. Using threads to run pumps etc. Causes async code written into the event-processor to become caotic (you need to follow which loop is currently being used in the call to prevent loops not being found or using the wrong loop (There is the main loop and then loops that are created inside threads) Things become caotic when the event processor is being called by objects that run under different loops. So, no Threading except usage of asyncio run_in_executor. This is done mostly for azure blob api calls. Also changed the bla_async methods to not block. this way, when calling open_async for the the event-processor-host, the command will exit once the EPH is started. Due to the above, see the edited example/eph.py where I added a monitor that makes sure the EPH is still running (Could be replaced by loop.run_forever()) in the example file I have also incorporated a test class for gracefully killing the EPH after 30 seconds. this works, nevertheless takes a while to close as we are waiting for timeouts on the eventhubs connections. * Started removing proton code * Removed most of proton _impl * Removed more code * Working sender * Updates to sender * Added some tests/samples * Some progress on clients * Fixed samples * Added azure namespace * #25 Partition key cannot be set for events * Updated version * Updated README * Renamed package to eventhub * Started EPH modifications * Updated imports * Fixed target urls * Updated logging * Updated async message receive * updated test imports * Added mgmt call to get eh info * Updated samples * Updated receive test * Added send and receive test clients * Updated uamqp dependency * Merged updates from dev * Fixed typos * Updated EPH sample * Started docstrings * Converted tests to pytest * Updates to batch receive * Started adding docstrings * More docstrings * bumped version * Started porting test suite * More tests and improvements * Moved eph tests * Some sample cleanup * Some test updates * Some test restructure * Docstring cleanup * Fixed some merge artifacts * Fixed formatting error * Removed delivery count * Nested package directory * Support custom URL suffix * Support custom URL suffix * Support for EventData device ID * Reverted nested directory * Updated release notes * Workaround for partitionkey * Finished partition key workaround * beta2 fixes * pylint fixes * Trigger CI * Test fixes * Added package manifest * Added warning for Python 2.7 support Support for issues #36 and #38 * Started adding scenario tests * More test scenarios * Better docstring formatting * Started iothub support * Fixed long running test * Fixed typo and memory leak * Restructure * IoThub support * Updates for RC1 release * Fix long running test * Docstring and sample cleanups * Working on error retry * Improved error processing * Fixed partition manager * Progress on IotHub error * Some test updates * Updated uamqp dependency * Restructure for independent connections * Added HTTP proxy support Fix for issue #41 * Fixed some tests + samples * pylint fixes * bumped version * Added keepalive config and some eph fixes * Made reconnect configurable * Added more EPH options * Bumped version * Pylint fix * Pylint fix * Added send and auth timeouts * Changed log formatting. Retry on reconnect * Pylint fixes * Renamed internal async module * Updated send example to match recv Fix for issue #56 * Added build badge to readme * Fix for repeat startup * Added more storage connect options to EPH * Bumped version * Handler blocked until client started * Added event data methods * Fix pylint * Fix 3.7 CI * Fix 3.7 CI * Updated pylint version * Pylint fixes * Updated README * Fixed readme badge refresh * Fixed bug in Azure namespace package * Updated manifest * Parse enqueued time as UTC Fixes #72. * Updates for release 1.2.0 (#81) * Made setup 2.7 compatible * Separated async tests * Support 2.7 types * Bumped version * Added non-ascii tests * Fix CI * Fix Py27 pylint * Added iot sample * Updated sender/receiver client opening * bumped version * Updated tests * Fixed test name * Fixed test env settings * Skip eph test * Updates for v1.3.0 (#91) * Added support for storing the state of the Event Processor along the Checkpoint. Both Checkpoint and the EP state are stored as pickled objects. * Fixing pylint complaints. * Switched from pickle back to JSON for lease persistence. * Fixes bug when accessing leases that don't contain EP context. Also, minor renaming. * Better SAS token support * Fixed pylint * Improved auth error handling * Test stabilization * Improved stored EPH context * Updated EPH context storing * Skip test on OSX * Skip tests on OSX Fail due to large message body bug. * Some cleanup * Fixed error handling * Improved SAS token parsing * Fixed datetime offset (#99) * Fixed datetime offset * Updated pylint * Removed 3.4 pylint pass * Fixed bug in error handling (#100) * Migrate event hub sdk to central repo 1. add verifiable code snippets into docstring 2. update readme according to the template 3. add livetest mark and config 4. optimize code layout/structure * 1. document formatting 2. separate async/sync example tests * Fix build error: 1. uamqp dependency mismatch 2. rename test_examples in eventhub to avoid mismatch * This should fix build error * remove tests import and add sys path to solve build error * add live test for sending BatchEvent with application_properties, new live test passed with new uamqp wheel locally installed * Add get_partition_info in Event Hub * add get_partition_info * Add telemetry information to the connection properties * Disable smart split in batch message * 1. Add amqp over websocket test 2. Add proxy sample 3. Update some comment and code * update some test code * Add __str__ to EventData * Update test code * Update event position * Update live test * Update reconnect live test * Update too large data size * debug->network_tracing * Negative test fix * Remove partition_key, send with batching_label * Fix review problems * Fix a log issue * fix get_partition_properties bug * add client properties live test * Revised setup.py for track 2
YijunXieMS
added a commit
that referenced
this pull request
Jun 24, 2019
* remove async_ops * EventHubs track2 starter (#5330) * Move to under sdk * Remove policies * Remove debugging files * Rename Offset to EventPosition * make tests a namespace package * Revised test receive for new code * Revised test send for track two * Update async code from sync * Revise async receive and send live test for track2 * Use uamqp 1.2 * Resolve code review feedback * add queue_message to async sender * send_batch receives both list and iterator * Eventhubs track2 python main issues (#5575) * Move to under sdk * Remove policies * Remove debugging files * Rename Offset to EventPosition * make tests a namespace package * Revised test receive for new code * Revised test send for track two * Update async code from sync * Revise async receive and send live test for track2 * Use uamqp 1.2 * Resolve code review feedback * add queue_message to async sender * send_batch receives both list and iterator * Update after adp review * send accepts EventData, list, iteratable * Event Hub Track 2 (#5) * Initial commit * Initial commit * Initial commit * event hub client * Update README.md * Update README.md Fix typos * Memory leak * Support timestamp filter * Support timestamp filter * Update README.md * Add sender and refactor * Added abstract classes Todo - Migrate Base Class Wireframes - Migrate Azure Classes * First draft of class wires directly ported from .net (might be some minor gaps) * send example * Set allowed sasl mechs * Remove client.py * Receiver update * Add dummy send api * logging updates * Error handling, reconnect and logging * Add app properties to event data * unbind transport on connection close * timestamp filter on py2 * module version * Reconnect once when link/session/connection close * Add SessionPolicy * Add client info * Updates - Cleaned wireframes to be PEP compliant - Implemented single partition pump and single event_hub partition pump scenario Todo - Add Unit Tests for partition pump and event hub partition pump - Implement Partition Manager - Implement Checkpointing and Lease Managment * Updates - Cleaned wireframes to be PEP compliant - Implemented single partition pump and single event_hub partition pump scenario Todo - Add Unit Tests for partition pump and event hub partition pump - Implement Partition Manager - Implement Checkpointing and Lease Managment * run client in non-blocking mode * Added unit testing * Implemented the following functionality - Azure_storage_checkpoint_manager - AzureBlobLease isExpired Todo Implement partition manager Implement partition context Test full implementation * Implemented Processing of First Epoh Todo - Fix lease bug that is breaking subsequent epochs * Changes - Completed End to End EPH Flow - Removed storage dependancy on downloading full blob to check lease state Todo - Add thread and queue for checking lease state and other storage operations - Ensure eventhub client shuts down properly - Find way to update partition pumps without restarting them - Other optimizations * Move examples out * Changes - Added thread pool executor to enable conncurent execution of partitions - Removed partition pump dependency on max_batch Todo - Ensure eventhub client shuts down properly (This is causing errors) - Add thread pool for making checkpoint code conccurent - Add thread and queue for checking lease state and other storage operations to enable async - Find way to reassign active partition pumps without restarting them - Other optimizations * Add async receive * Changes - Added logs - Fixed error causing client to prematurely shutdown * Manual link flow control for async receive * Workaround for stuck async receiver * Local variable names * Changes - Optimized logging and comments Todo - Add concurecny mechanim for azure storage - Depricate partition pump event queue and update to latest version of the client * Create Dockerfile * Stuck async receiver * credit keeps increasing in async receiver * Changes - Added asnyc event hub client support - Optimized logging and comments Todo - Add concurecny mechanim for azure storage * Updated docker file as requested * Added EPH example * Fix hardcoded HTTP header * Made suggested changes * Bug fix - Fixed event loop bugs. In windows eventloop is thread dependent but in ubuntu the eventloop is threadsafe so you need to differentiate the thread specific eventloop from the host one. * Updated loop naming convention to be consistent * Added option to pass asyncio event_loop to eph * Updated docker file * Fixed critical bug with partition manager and aquirec mechanisiims Todo : Identitfy and fix remaining bug that is causing all pumps to shut down when a second host starts * Bug fixes - Fixed bug where closing a pump closed a host - Fixed bug where error partitioned were not removed - Fixed bug where leases were renewed at an incorrect interval * Updated file headers Removed author reference * - Fixed bug in eph example that caused host to terminate prematurely - Made the lease renewal and checkpoint creation "multithreaded" * Increase the size of the connection pool The default connection pool size was too small for scenarios where multiple partitions were handled by one EventProcessorHost. If the amount of partitions handled is large, we might end up doing very many connections at the same time due to the multi-threaded blob-handling. For this reason, you might hit the OS limits that restrict the number of open files per process that in MacOS is not very big. This can be worked around with something like: `ulimit -n 2560` * Decrease info logging verbosity * added ability to toggle pump shutdown when all messages on a pump are processed. * Install also eventhubsprocessor * Default to keeping the pumps It is more optimal to keep the pumps alive even if there are no messages so that it is faster to pickup when messages start to arrive. * Pipe and event injector for Windows * Event injector updates * EHClient refactoring. EHClient leaks. Sender part 1. * Send support * ren eventhubsprocessor eventprocessorhost * Changes - Added event hub config to simplify installation story * Changes - Added optional eventprocessor_params for passing context to the event processor - Made the storage manager mandatatory * Fix memory leaks * logging * Fix: 1. process crash due to race in client stop and connection remote close. 2. handle client close in async receiver. 3. fail pending sends when sender is closed. 4. some debug logging. * tests * test: recv from multiple partitions * test utility * logging update * Support callback based send for high throughput * Workaroud memory issue in proton.reactor.ApplicationEvent * renamed eventprocessor to eventprocessorhost for consistency * updated docker file * fixed typo in url * Added amqp port to address * Updated sample documentation since url is auto encoded by config * Updated docs * Implement timeout for send * Async sender and example * Close injector pipe * Use send timer to also check queued messages * Add partition pump loop to partition_context This gives the EventProcessor access to the partition_pump loop object. This way if One desires to run synchronous code inside process_events_async one can utilize the loop object to run the synchronous code using await context.pump_loop.run_in_executor(None, bla) * Include details in send error * Release deliveries when sender is closed * added validation to unquoted sas key * added support for custom eventhub client prefetch size * Update README.md * Update README.md * Added Docker instructions and fixed Dockerfile (#18) * Removed Dockerfile from the main folder and fixed Dockerfile example * Added build and run Dockerfile documentation * Update Readme * Removed rm qpid-proton folder * Removed /usr/share copy * Disallow a sender/receiver to be registered more than once * Make everything async in EPH I have removed all usage of threads thoroughout the code. Using threads to run pumps etc. Causes async code written into the event-processor to become caotic (you need to follow which loop is currently being used in the call to prevent loops not being found or using the wrong loop (There is the main loop and then loops that are created inside threads) Things become caotic when the event processor is being called by objects that run under different loops. So, no Threading except usage of asyncio run_in_executor. This is done mostly for azure blob api calls. Also changed the bla_async methods to not block. this way, when calling open_async for the the event-processor-host, the command will exit once the EPH is started. Due to the above, see the edited example/eph.py where I added a monitor that makes sure the EPH is still running (Could be replaced by loop.run_forever()) in the example file I have also incorporated a test class for gracefully killing the EPH after 30 seconds. this works, nevertheless takes a while to close as we are waiting for timeouts on the eventhubs connections. * Started removing proton code * Removed most of proton _impl * Removed more code * Working sender * Updates to sender * Added some tests/samples * Some progress on clients * Fixed samples * Added azure namespace * #25 Partition key cannot be set for events * Updated version * Updated README * Renamed package to eventhub * Started EPH modifications * Updated imports * Fixed target urls * Updated logging * Updated async message receive * updated test imports * Added mgmt call to get eh info * Updated samples * Updated receive test * Added send and receive test clients * Updated uamqp dependency * Merged updates from dev * Fixed typos * Updated EPH sample * Started docstrings * Converted tests to pytest * Updates to batch receive * Started adding docstrings * More docstrings * bumped version * Started porting test suite * More tests and improvements * Moved eph tests * Some sample cleanup * Some test updates * Some test restructure * Docstring cleanup * Fixed some merge artifacts * Fixed formatting error * Removed delivery count * Nested package directory * Support custom URL suffix * Support custom URL suffix * Support for EventData device ID * Reverted nested directory * Updated release notes * Workaround for partitionkey * Finished partition key workaround * beta2 fixes * pylint fixes * Trigger CI * Test fixes * Added package manifest * Added warning for Python 2.7 support Support for issues #36 and #38 * Started adding scenario tests * More test scenarios * Better docstring formatting * Started iothub support * Fixed long running test * Fixed typo and memory leak * Restructure * IoThub support * Updates for RC1 release * Fix long running test * Docstring and sample cleanups * Working on error retry * Improved error processing * Fixed partition manager * Progress on IotHub error * Some test updates * Updated uamqp dependency * Restructure for independent connections * Added HTTP proxy support Fix for issue #41 * Fixed some tests + samples * pylint fixes * bumped version * Added keepalive config and some eph fixes * Made reconnect configurable * Added more EPH options * Bumped version * Pylint fix * Pylint fix * Added send and auth timeouts * Changed log formatting. Retry on reconnect * Pylint fixes * Renamed internal async module * Updated send example to match recv Fix for issue #56 * Added build badge to readme * Fix for repeat startup * Added more storage connect options to EPH * Bumped version * Handler blocked until client started * Added event data methods * Fix pylint * Fix 3.7 CI * Fix 3.7 CI * Updated pylint version * Pylint fixes * Updated README * Fixed readme badge refresh * Fixed bug in Azure namespace package * Updated manifest * Parse enqueued time as UTC Fixes #72. * Updates for release 1.2.0 (#81) * Made setup 2.7 compatible * Separated async tests * Support 2.7 types * Bumped version * Added non-ascii tests * Fix CI * Fix Py27 pylint * Added iot sample * Updated sender/receiver client opening * bumped version * Updated tests * Fixed test name * Fixed test env settings * Skip eph test * Updates for v1.3.0 (#91) * Added support for storing the state of the Event Processor along the Checkpoint. Both Checkpoint and the EP state are stored as pickled objects. * Fixing pylint complaints. * Switched from pickle back to JSON for lease persistence. * Fixes bug when accessing leases that don't contain EP context. Also, minor renaming. * Better SAS token support * Fixed pylint * Improved auth error handling * Test stabilization * Improved stored EPH context * Updated EPH context storing * Skip test on OSX * Skip tests on OSX Fail due to large message body bug. * Some cleanup * Fixed error handling * Improved SAS token parsing * Fixed datetime offset (#99) * Fixed datetime offset * Updated pylint * Removed 3.4 pylint pass * Fixed bug in error handling (#100) * Migrate event hub sdk to central repo 1. add verifiable code snippets into docstring 2. update readme according to the template 3. add livetest mark and config 4. optimize code layout/structure * 1. document formatting 2. separate async/sync example tests * Fix build error: 1. uamqp dependency mismatch 2. rename test_examples in eventhub to avoid mismatch * This should fix build error * remove tests import and add sys path to solve build error * add live test for sending BatchEvent with application_properties, new live test passed with new uamqp wheel locally installed * Add get_partition_info in Event Hub * add get_partition_info * Add telemetry information to the connection properties * Disable smart split in batch message * change epoch to exclusive_receiver_priority * fix small problem * remove uamqp dependency * Eventhub track2 (#6) * Initial commit * Initial commit * Initial commit * event hub client * Update README.md * Update README.md Fix typos * Memory leak * Support timestamp filter * Support timestamp filter * Update README.md * Add sender and refactor * Added abstract classes Todo - Migrate Base Class Wireframes - Migrate Azure Classes * First draft of class wires directly ported from .net (might be some minor gaps) * send example * Set allowed sasl mechs * Remove client.py * Receiver update * Add dummy send api * logging updates * Error handling, reconnect and logging * Add app properties to event data * unbind transport on connection close * timestamp filter on py2 * module version * Reconnect once when link/session/connection close * Add SessionPolicy * Add client info * Updates - Cleaned wireframes to be PEP compliant - Implemented single partition pump and single event_hub partition pump scenario Todo - Add Unit Tests for partition pump and event hub partition pump - Implement Partition Manager - Implement Checkpointing and Lease Managment * Updates - Cleaned wireframes to be PEP compliant - Implemented single partition pump and single event_hub partition pump scenario Todo - Add Unit Tests for partition pump and event hub partition pump - Implement Partition Manager - Implement Checkpointing and Lease Managment * run client in non-blocking mode * Added unit testing * Implemented the following functionality - Azure_storage_checkpoint_manager - AzureBlobLease isExpired Todo Implement partition manager Implement partition context Test full implementation * Implemented Processing of First Epoh Todo - Fix lease bug that is breaking subsequent epochs * Changes - Completed End to End EPH Flow - Removed storage dependancy on downloading full blob to check lease state Todo - Add thread and queue for checking lease state and other storage operations - Ensure eventhub client shuts down properly - Find way to update partition pumps without restarting them - Other optimizations * Move examples out * Changes - Added thread pool executor to enable conncurent execution of partitions - Removed partition pump dependency on max_batch Todo - Ensure eventhub client shuts down properly (This is causing errors) - Add thread pool for making checkpoint code conccurent - Add thread and queue for checking lease state and other storage operations to enable async - Find way to reassign active partition pumps without restarting them - Other optimizations * Add async receive * Changes - Added logs - Fixed error causing client to prematurely shutdown * Manual link flow control for async receive * Workaround for stuck async receiver * Local variable names * Changes - Optimized logging and comments Todo - Add concurecny mechanim for azure storage - Depricate partition pump event queue and update to latest version of the client * Create Dockerfile * Stuck async receiver * credit keeps increasing in async receiver * Changes - Added asnyc event hub client support - Optimized logging and comments Todo - Add concurecny mechanim for azure storage * Updated docker file as requested * Added EPH example * Fix hardcoded HTTP header * Made suggested changes * Bug fix - Fixed event loop bugs. In windows eventloop is thread dependent but in ubuntu the eventloop is threadsafe so you need to differentiate the thread specific eventloop from the host one. * Updated loop naming convention to be consistent * Added option to pass asyncio event_loop to eph * Updated docker file * Fixed critical bug with partition manager and aquirec mechanisiims Todo : Identitfy and fix remaining bug that is causing all pumps to shut down when a second host starts * Bug fixes - Fixed bug where closing a pump closed a host - Fixed bug where error partitioned were not removed - Fixed bug where leases were renewed at an incorrect interval * Updated file headers Removed author reference * - Fixed bug in eph example that caused host to terminate prematurely - Made the lease renewal and checkpoint creation "multithreaded" * Increase the size of the connection pool The default connection pool size was too small for scenarios where multiple partitions were handled by one EventProcessorHost. If the amount of partitions handled is large, we might end up doing very many connections at the same time due to the multi-threaded blob-handling. For this reason, you might hit the OS limits that restrict the number of open files per process that in MacOS is not very big. This can be worked around with something like: `ulimit -n 2560` * Decrease info logging verbosity * added ability to toggle pump shutdown when all messages on a pump are processed. * Install also eventhubsprocessor * Default to keeping the pumps It is more optimal to keep the pumps alive even if there are no messages so that it is faster to pickup when messages start to arrive. * Pipe and event injector for Windows * Event injector updates * EHClient refactoring. EHClient leaks. Sender part 1. * Send support * ren eventhubsprocessor eventprocessorhost * Changes - Added event hub config to simplify installation story * Changes - Added optional eventprocessor_params for passing context to the event processor - Made the storage manager mandatatory * Fix memory leaks * logging * Fix: 1. process crash due to race in client stop and connection remote close. 2. handle client close in async receiver. 3. fail pending sends when sender is closed. 4. some debug logging. * tests * test: recv from multiple partitions * test utility * logging update * Support callback based send for high throughput * Workaroud memory issue in proton.reactor.ApplicationEvent * renamed eventprocessor to eventprocessorhost for consistency * updated docker file * fixed typo in url * Added amqp port to address * Updated sample documentation since url is auto encoded by config * Updated docs * Implement timeout for send * Async sender and example * Close injector pipe * Use send timer to also check queued messages * Add partition pump loop to partition_context This gives the EventProcessor access to the partition_pump loop object. This way if One desires to run synchronous code inside process_events_async one can utilize the loop object to run the synchronous code using await context.pump_loop.run_in_executor(None, bla) * Include details in send error * Release deliveries when sender is closed * added validation to unquoted sas key * added support for custom eventhub client prefetch size * Update README.md * Update README.md * Added Docker instructions and fixed Dockerfile (#18) * Removed Dockerfile from the main folder and fixed Dockerfile example * Added build and run Dockerfile documentation * Update Readme * Removed rm qpid-proton folder * Removed /usr/share copy * Disallow a sender/receiver to be registered more than once * Make everything async in EPH I have removed all usage of threads thoroughout the code. Using threads to run pumps etc. Causes async code written into the event-processor to become caotic (you need to follow which loop is currently being used in the call to prevent loops not being found or using the wrong loop (There is the main loop and then loops that are created inside threads) Things become caotic when the event processor is being called by objects that run under different loops. So, no Threading except usage of asyncio run_in_executor. This is done mostly for azure blob api calls. Also changed the bla_async methods to not block. this way, when calling open_async for the the event-processor-host, the command will exit once the EPH is started. Due to the above, see the edited example/eph.py where I added a monitor that makes sure the EPH is still running (Could be replaced by loop.run_forever()) in the example file I have also incorporated a test class for gracefully killing the EPH after 30 seconds. this works, nevertheless takes a while to close as we are waiting for timeouts on the eventhubs connections. * Started removing proton code * Removed most of proton _impl * Removed more code * Working sender * Updates to sender * Added some tests/samples * Some progress on clients * Fixed samples * Added azure namespace * #25 Partition key cannot be set for events * Updated version * Updated README * Renamed package to eventhub * Started EPH modifications * Updated imports * Fixed target urls * Updated logging * Updated async message receive * updated test imports * Added mgmt call to get eh info * Updated samples * Updated receive test * Added send and receive test clients * Updated uamqp dependency * Merged updates from dev * Fixed typos * Updated EPH sample * Started docstrings * Converted tests to pytest * Updates to batch receive * Started adding docstrings * More docstrings * bumped version * Started porting test suite * More tests and improvements * Moved eph tests * Some sample cleanup * Some test updates * Some test restructure * Docstring cleanup * Fixed some merge artifacts * Fixed formatting error * Removed delivery count * Nested package directory * Support custom URL suffix * Support custom URL suffix * Support for EventData device ID * Reverted nested directory * Updated release notes * Workaround for partitionkey * Finished partition key workaround * beta2 fixes * pylint fixes * Trigger CI * Test fixes * Added package manifest * Added warning for Python 2.7 support Support for issues #36 and #38 * Started adding scenario tests * More test scenarios * Better docstring formatting * Started iothub support * Fixed long running test * Fixed typo and memory leak * Restructure * IoThub support * Updates for RC1 release * Fix long running test * Docstring and sample cleanups * Working on error retry * Improved error processing * Fixed partition manager * Progress on IotHub error * Some test updates * Updated uamqp dependency * Restructure for independent connections * Added HTTP proxy support Fix for issue #41 * Fixed some tests + samples * pylint fixes * bumped version * Added keepalive config and some eph fixes * Made reconnect configurable * Added more EPH options * Bumped version * Pylint fix * Pylint fix * Added send and auth timeouts * Changed log formatting. Retry on reconnect * Pylint fixes * Renamed internal async module * Updated send example to match recv Fix for issue #56 * Added build badge to readme * Fix for repeat startup * Added more storage connect options to EPH * Bumped version * Handler blocked until client started * Added event data methods * Fix pylint * Fix 3.7 CI * Fix 3.7 CI * Updated pylint version * Pylint fixes * Updated README * Fixed readme badge refresh * Fixed bug in Azure namespace package * Updated manifest * Parse enqueued time as UTC Fixes #72. * Updates for release 1.2.0 (#81) * Made setup 2.7 compatible * Separated async tests * Support 2.7 types * Bumped version * Added non-ascii tests * Fix CI * Fix Py27 pylint * Added iot sample * Updated sender/receiver client opening * bumped version * Updated tests * Fixed test name * Fixed test env settings * Skip eph test * Updates for v1.3.0 (#91) * Added support for storing the state of the Event Processor along the Checkpoint. Both Checkpoint and the EP state are stored as pickled objects. * Fixing pylint complaints. * Switched from pickle back to JSON for lease persistence. * Fixes bug when accessing leases that don't contain EP context. Also, minor renaming. * Better SAS token support * Fixed pylint * Improved auth error handling * Test stabilization * Improved stored EPH context * Updated EPH context storing * Skip test on OSX * Skip tests on OSX Fail due to large message body bug. * Some cleanup * Fixed error handling * Improved SAS token parsing * Fixed datetime offset (#99) * Fixed datetime offset * Updated pylint * Removed 3.4 pylint pass * Fixed bug in error handling (#100) * Migrate event hub sdk to central repo 1. add verifiable code snippets into docstring 2. update readme according to the template 3. add livetest mark and config 4. optimize code layout/structure * 1. document formatting 2. separate async/sync example tests * Fix build error: 1. uamqp dependency mismatch 2. rename test_examples in eventhub to avoid mismatch * This should fix build error * remove tests import and add sys path to solve build error * add live test for sending BatchEvent with application_properties, new live test passed with new uamqp wheel locally installed * Add get_partition_info in Event Hub * add get_partition_info * Add telemetry information to the connection properties * Disable smart split in batch message * 1. Add amqp over websocket test 2. Add proxy sample 3. Update some comment and code * Changes from cross-lang * Change debug to network_tracing * Sync Client Constructor * auto_reconnect True and keep_alive None * consumer_group $default * hide open() * partition -> partition_id * credentials -> credential in init * set running=true after opened * Eventhub track2 - Update livetest (#7) * Update README.md * Update README.md Fix typos * Memory leak * Support timestamp filter * Support timestamp filter * Update README.md * Add sender and refactor * Added abstract classes Todo - Migrate Base Class Wireframes - Migrate Azure Classes * First draft of class wires directly ported from .net (might be some minor gaps) * send example * Set allowed sasl mechs * Remove client.py * Receiver update * Add dummy send api * logging updates * Error handling, reconnect and logging * Add app properties to event data * unbind transport on connection close * timestamp filter on py2 * module version * Reconnect once when link/session/connection close * Add SessionPolicy * Add client info * Updates - Cleaned wireframes to be PEP compliant - Implemented single partition pump and single event_hub partition pump scenario Todo - Add Unit Tests for partition pump and event hub partition pump - Implement Partition Manager - Implement Checkpointing and Lease Managment * Updates - Cleaned wireframes to be PEP compliant - Implemented single partition pump and single event_hub partition pump scenario Todo - Add Unit Tests for partition pump and event hub partition pump - Implement Partition Manager - Implement Checkpointing and Lease Managment * run client in non-blocking mode * Added unit testing * Implemented the following functionality - Azure_storage_checkpoint_manager - AzureBlobLease isExpired Todo Implement partition manager Implement partition context Test full implementation * Implemented Processing of First Epoh Todo - Fix lease bug that is breaking subsequent epochs * Changes - Completed End to End EPH Flow - Removed storage dependancy on downloading full blob to check lease state Todo - Add thread and queue for checking lease state and other storage operations - Ensure eventhub client shuts down properly - Find way to update partition pumps without restarting them - Other optimizations * Move examples out * Changes - Added thread pool executor to enable conncurent execution of partitions - Removed partition pump dependency on max_batch Todo - Ensure eventhub client shuts down properly (This is causing errors) - Add thread pool for making checkpoint code conccurent - Add thread and queue for checking lease state and other storage operations to enable async - Find way to reassign active partition pumps without restarting them - Other optimizations * Add async receive * Changes - Added logs - Fixed error causing client to prematurely shutdown * Manual link flow control for async receive * Workaround for stuck async receiver * Local variable names * Changes - Optimized logging and comments Todo - Add concurecny mechanim for azure storage - Depricate partition pump event queue and update to latest version of the client * Create Dockerfile * Stuck async receiver * credit keeps increasing in async receiver * Changes - Added asnyc event hub client support - Optimized logging and comments Todo - Add concurecny mechanim for azure storage * Updated docker file as requested * Added EPH example * Fix hardcoded HTTP header * Made suggested changes * Bug fix - Fixed event loop bugs. In windows eventloop is thread dependent but in ubuntu the eventloop is threadsafe so you need to differentiate the thread specific eventloop from the host one. * Updated loop naming convention to be consistent * Added option to pass asyncio event_loop to eph * Updated docker file * Fixed critical bug with partition manager and aquirec mechanisiims Todo : Identitfy and fix remaining bug that is causing all pumps to shut down when a second host starts * Bug fixes - Fixed bug where closing a pump closed a host - Fixed bug where error partitioned were not removed - Fixed bug where leases were renewed at an incorrect interval * Updated file headers Removed author reference * - Fixed bug in eph example that caused host to terminate prematurely - Made the lease renewal and checkpoint creation "multithreaded" * Increase the size of the connection pool The default connection pool size was too small for scenarios where multiple partitions were handled by one EventProcessorHost. If the amount of partitions handled is large, we might end up doing very many connections at the same time due to the multi-threaded blob-handling. For this reason, you might hit the OS limits that restrict the number of open files per process that in MacOS is not very big. This can be worked around with something like: `ulimit -n 2560` * Decrease info logging verbosity * added ability to toggle pump shutdown when all messages on a pump are processed. * Install also eventhubsprocessor * Default to keeping the pumps It is more optimal to keep the pumps alive even if there are no messages so that it is faster to pickup when messages start to arrive. * Pipe and event injector for Windows * Event injector updates * EHClient refactoring. EHClient leaks. Sender part 1. * Send support * ren eventhubsprocessor eventprocessorhost * Changes - Added event hub config to simplify installation story * Changes - Added optional eventprocessor_params for passing context to the event processor - Made the storage manager mandatatory * Fix memory leaks * logging * Fix: 1. process crash due to race in client stop and connection remote close. 2. handle client close in async receiver. 3. fail pending sends when sender is closed. 4. some debug logging. * tests * test: recv from multiple partitions * test utility * logging update * Support callback based send for high throughput * Workaroud memory issue in proton.reactor.ApplicationEvent * renamed eventprocessor to eventprocessorhost for consistency * updated docker file * fixed typo in url * Added amqp port to address * Updated sample documentation since url is auto encoded by config * Updated docs * Implement timeout for send * Async sender and example * Close injector pipe * Use send timer to also check queued messages * Add partition pump loop to partition_context This gives the EventProcessor access to the partition_pump loop object. This way if One desires to run synchronous code inside process_events_async one can utilize the loop object to run the synchronous code using await context.pump_loop.run_in_executor(None, bla) * Include details in send error * Release deliveries when sender is closed * added validation to unquoted sas key * added support for custom eventhub client prefetch size * Update README.md * Update README.md * Added Docker instructions and fixed Dockerfile (#18) * Removed Dockerfile from the main folder and fixed Dockerfile example * Added build and run Dockerfile documentation * Update Readme * Removed rm qpid-proton folder * Removed /usr/share copy * Disallow a sender/receiver to be registered more than once * Make everything async in EPH I have removed all usage of threads thoroughout the code. Using threads to run pumps etc. Causes async code written into the event-processor to become caotic (you need to follow which loop is currently being used in the call to prevent loops not being found or using the wrong loop (There is the main loop and then loops that are created inside threads) Things become caotic when the event processor is being called by objects that run under different loops. So, no Threading except usage of asyncio run_in_executor. This is done mostly for azure blob api calls. Also changed the bla_async methods to not block. this way, when calling open_async for the the event-processor-host, the command will exit once the EPH is started. Due to the above, see the edited example/eph.py where I added a monitor that makes sure the EPH is still running (Could be replaced by loop.run_forever()) in the example file I have also incorporated a test class for gracefully killing the EPH after 30 seconds. this works, nevertheless takes a while to close as we are waiting for timeouts on the eventhubs connections. * Started removing proton code * Removed most of proton _impl * Removed more code * Working sender * Updates to sender * Added some tests/samples * Some progress on clients * Fixed samples * Added azure namespace * #25 Partition key cannot be set for events * Updated version * Updated README * Renamed package to eventhub * Started EPH modifications * Updated imports * Fixed target urls * Updated logging * Updated async message receive * updated test imports * Added mgmt call to get eh info * Updated samples * Updated receive test * Added send and receive test clients * Updated uamqp dependency * Merged updates from dev * Fixed typos * Updated EPH sample * Started docstrings * Converted tests to pytest * Updates to batch receive * Started adding docstrings * More docstrings * bumped version * Started porting test suite * More tests and improvements * Moved eph tests * Some sample cleanup * Some test updates * Some test restructure * Docstring cleanup * Fixed some merge artifacts * Fixed formatting error * Removed delivery count * Nested package directory * Support custom URL suffix * Support custom URL suffix * Support for EventData device ID * Reverted nested directory * Updated release notes * Workaround for partitionkey * Finished partition key workaround * beta2 fixes * pylint fixes * Trigger CI * Test fixes * Added package manifest * Added warning for Python 2.7 support Support for issues #36 and #38 * Started adding scenario tests * More test scenarios * Better docstring formatting * Started iothub support * Fixed long running test * Fixed typo and memory leak * Restructure * IoThub support * Updates for RC1 release * Fix long running test * Docstring and sample cleanups * Working on error retry * Improved error processing * Fixed partition manager * Progress on IotHub error * Some test updates * Updated uamqp dependency * Restructure for independent connections * Added HTTP proxy support Fix for issue #41 * Fixed some tests + samples * pylint fixes * bumped version * Added keepalive config and some eph fixes * Made reconnect configurable * Added more EPH options * Bumped version * Pylint fix * Pylint fix * Added send and auth timeouts * Changed log formatting. Retry on reconnect * Pylint fixes * Renamed internal async module * Updated send example to match recv Fix for issue #56 * Added build badge to readme * Fix for repeat startup * Added more storage connect options to EPH * Bumped version * Handler blocked until client started * Added event data methods * Fix pylint * Fix 3.7 CI * Fix 3.7 CI * Updated pylint version * Pylint fixes * Updated README * Fixed readme badge refresh * Fixed bug in Azure namespace package * Updated manifest * Parse enqueued time as UTC Fixes #72. * Updates for release 1.2.0 (#81) * Made setup 2.7 compatible * Separated async tests * Support 2.7 types * Bumped version * Added non-ascii tests * Fix CI * Fix Py27 pylint * Added iot sample * Updated sender/receiver client opening * bumped version * Updated tests * Fixed test name * Fixed test env settings * Skip eph test * Updates for v1.3.0 (#91) * Added support for storing the state of the Event Processor along the Checkpoint. Both Checkpoint and the EP state are stored as pickled objects. * Fixing pylint complaints. * Switched from pickle back to JSON for lease persistence. * Fixes bug when accessing leases that don't contain EP context. Also, minor renaming. * Better SAS token support * Fixed pylint * Improved auth error handling * Test stabilization * Improved stored EPH context * Updated EPH context storing * Skip test on OSX * Skip tests on OSX Fail due to large message body bug. * Some cleanup * Fixed error handling * Improved SAS token parsing * Fixed datetime offset (#99) * Fixed datetime offset * Updated pylint * Removed 3.4 pylint pass * Fixed bug in error handling (#100) * Migrate event hub sdk to central repo 1. add verifiable code snippets into docstring 2. update readme according to the template 3. add livetest mark and config 4. optimize code layout/structure * 1. document formatting 2. separate async/sync example tests * Fix build error: 1. uamqp dependency mismatch 2. rename test_examples in eventhub to avoid mismatch * This should fix build error * remove tests import and add sys path to solve build error * add live test for sending BatchEvent with application_properties, new live test passed with new uamqp wheel locally installed * Add get_partition_info in Event Hub * add get_partition_info * Add telemetry information to the connection properties * Disable smart split in batch message * 1. Add amqp over websocket test 2. Add proxy sample 3. Update some comment and code * update some test code * Add __str__ to EventData * Update test code * Add eh error classes * EventHubError extends AzureError * Fix EventPosition default value issue * change $default to $Default * Handle TokenAuthError * wait for ready in _reconnect * fix get_partition_ids issue * Fix reconnect issue * small fix * fix async live test * Eventhub track2 Live test update (#8) * Set allowed sasl mechs * Remove client.py * Receiver update * Add dummy send api * logging updates * Error handling, reconnect and logging * Add app properties to event data * unbind transport on connection close * timestamp filter on py2 * module version * Reconnect once when link/session/connection close * Add SessionPolicy * Add client info * Updates - Cleaned wireframes to be PEP compliant - Implemented single partition pump and single event_hub partition pump scenario Todo - Add Unit Tests for partition pump and event hub partition pump - Implement Partition Manager - Implement Checkpointing and Lease Managment * Updates - Cleaned wireframes to be PEP compliant - Implemented single partition pump and single event_hub partition pump scenario Todo - Add Unit Tests for partition pump and event hub partition pump - Implement Partition Manager - Implement Checkpointing and Lease Managment * run client in non-blocking mode * Added unit testing * Implemented the following functionality - Azure_storage_checkpoint_manager - AzureBlobLease isExpired Todo Implement partition manager Implement partition context Test full implementation * Implemented Processing of First Epoh Todo - Fix lease bug that is breaking subsequent epochs * Changes - Completed End to End EPH Flow - Removed storage dependancy on downloading full blob to check lease state Todo - Add thread and queue for checking lease state and other storage operations - Ensure eventhub client shuts down properly - Find way to update partition pumps without restarting them - Other optimizations * Move examples out * Changes - Added thread pool executor to enable conncurent execution of partitions - Removed partition pump dependency on max_batch Todo - Ensure eventhub client shuts down properly (This is causing errors) - Add thread pool for making checkpoint code conccurent - Add thread and queue for checking lease state and other storage operations to enable async - Find way to reassign active partition pumps without restarting them - Other optimizations * Add async receive * Changes - Added logs - Fixed error causing client to prematurely shutdown * Manual link flow control for async receive * Workaround for stuck async receiver * Local variable names * Changes - Optimized logging and comments Todo - Add concurecny mechanim for azure storage - Depricate partition pump event queue and update to latest version of the client * Create Dockerfile * Stuck async receiver * credit keeps increasing in async receiver * Changes - Added asnyc event hub client support - Optimized logging and comments Todo - Add concurecny mechanim for azure storage * Updated docker file as requested * Added EPH example * Fix hardcoded HTTP header * Made suggested changes * Bug fix - Fixed event loop bugs. In windows eventloop is thread dependent but in ubuntu the eventloop is threadsafe so you need to differentiate the thread specific eventloop from the host one. * Updated loop naming convention to be consistent * Added option to pass asyncio event_loop to eph * Updated docker file * Fixed critical bug with partition manager and aquirec mechanisiims Todo : Identitfy and fix remaining bug that is causing all pumps to shut down when a second host starts * Bug fixes - Fixed bug where closing a pump closed a host - Fixed bug where error partitioned were not removed - Fixed bug where leases were renewed at an incorrect interval * Updated file headers Removed author reference * - Fixed bug in eph example that caused host to terminate prematurely - Made the lease renewal and checkpoint creation "multithreaded" * Increase the size of the connection pool The default connection pool size was too small for scenarios where multiple partitions were handled by one EventProcessorHost. If the amount of partitions handled is large, we might end up doing very many connections at the same time due to the multi-threaded blob-handling. For this reason, you might hit the OS limits that restrict the number of open files per process that in MacOS is not very big. This can be worked around with something like: `ulimit -n 2560` * Decrease info logging verbosity * added ability to toggle pump shutdown when all messages on a pump are processed. * Install also eventhubsprocessor * Default to keeping the pumps It is more optimal to keep the pumps alive even if there are no messages so that it is faster to pickup when messages start to arrive. * Pipe and event injector for Windows * Event injector updates * EHClient refactoring. EHClient leaks. Sender part 1. * Send support * ren eventhubsprocessor eventprocessorhost * Changes - Added event hub config to simplify installation story * Changes - Added optional eventprocessor_params for passing context to the event processor - Made the storage manager mandatatory * Fix memory leaks * logging * Fix: 1. process crash due to race in client stop and connection remote close. 2. handle client close in async receiver. 3. fail pending sends when sender is closed. 4. some debug logging. * tests * test: recv from multiple partitions * test utility * logging update * Support callback based send for high throughput * Workaroud memory issue in proton.reactor.ApplicationEvent * renamed eventprocessor to eventprocessorhost for consistency * updated docker file * fixed typo in url * Added amqp port to address * Updated sample documentation since url is auto encoded by config * Updated docs * Implement timeout for send * Async sender and example * Close injector pipe * Use send timer to also check queued messages * Add partition pump loop to partition_context This gives the EventProcessor access to the partition_pump loop object. This way if One desires to run synchronous code inside process_events_async one can utilize the loop object to run the synchronous code using await context.pump_loop.run_in_executor(None, bla) * Include details in send error * Release deliveries when sender is closed * added validation to unquoted sas key * added support for custom eventhub client prefetch size * Update README.md * Update README.md * Added Docker instructions and fixed Dockerfile (#18) * Removed Dockerfile from the main folder and fixed Dockerfile example * Added build and run Dockerfile documentation * Update Readme * Removed rm qpid-proton folder * Removed /usr/share copy * Disallow a sender/receiver to be registered more than once * Make everything async in EPH I have removed all usage of threads thoroughout the code. Using threads to run pumps etc. Causes async code written into the event-processor to become caotic (you need to follow which loop is currently being used in the call to prevent loops not being found or using the wrong loop (There is the main loop and then loops that are created inside threads) Things become caotic when the event processor is being called by objects that run under different loops. So, no Threading except usage of asyncio run_in_executor. This is done mostly for azure blob api calls. Also changed the bla_async methods to not block. this way, when calling open_async for the the event-processor-host, the command will exit once the EPH is started. Due to the above, see the edited example/eph.py where I added a monitor that makes sure the EPH is still running (Could be replaced by loop.run_forever()) in the example file I have also incorporated a test class for gracefully killing the EPH after 30 seconds. this works, nevertheless takes a while to close as we are waiting for timeouts on the eventhubs connections. * Started removing proton code * Removed most of proton _impl * Removed more code * Working sender * Updates to sender * Added some tests/samples * Some progress on clients * Fixed samples * Added azure namespace * #25 Partition key cannot be set for events * Updated version * Updated README * Renamed package to eventhub * Started EPH modifications * Updated imports * Fixed target urls * Updated logging * Updated async message receive * updated test imports * Added mgmt call to get eh info * Updated samples * Updated receive test * Added send and receive test clients * Updated uamqp dependency * Merged updates from dev * Fixed typos * Updated EPH sample * Started docstrings * Converted tests to pytest * Updates to batch receive * Started adding docstrings * More docstrings * bumped version * Started porting test suite * More tests and improvements * Moved eph tests * Some sample cleanup * Some test updates * Some test restructure * Docstring cleanup * Fixed some merge artifacts * Fixed formatting error * Removed delivery count * Nested package directory * Support custom URL suffix * Support custom URL suffix * Support for EventData device ID * Reverted nested directory * Updated release notes * Workaround for partitionkey * Finished partition key workaround * beta2 fixes * pylint fixes * Trigger CI * Test fixes * Added package manifest * Added warning for Python 2.7 support Support for issues #36 and #38 * Started adding scenario tests * More test scenarios * Better docstring formatting * Started iothub support * Fixed long running test * Fixed typo and memory leak * Restructure * IoThub support * Updates for RC1 release * Fix long running test * Docstring and sample cleanups * Working on error retry * Improved error processing * Fixed partition manager * Progress on IotHub error * Some test updates * Updated uamqp dependency * Restructure for independent connections * Added HTTP proxy support Fix for issue #41 * Fixed some tests + samples * pylint fixes * bumped version * Added keepalive config and some eph fixes * Made reconnect configurable * Added more EPH options * Bumped version * Pylint fix * Pylint fix * Added send and auth timeouts * Changed log formatting. Retry on reconnect * Pylint fixes * Renamed internal async module * Updated send example to match recv Fix for issue #56 * Added build badge to readme * Fix for repeat startup * Added more storage connect options to EPH * Bumped version * Handler blocked until client started * Added event data methods * Fix pylint * Fix 3.7 CI * Fix 3.7 CI * Updated pylint version * Pylint fixes * Updated README * Fixed readme badge refresh * Fixed bug in Azure namespace package * Updated manifest * Parse enqueued time as UTC Fixes #72. * Updates for release 1.2.0 (#81) * Made setup 2.7 compatible * Separated async tests * Support 2.7 types * Bumped version * Added non-ascii tests * Fix CI * Fix Py27 pylint * Added iot sample * Updated sender/receiver client opening * bumped version * Updated tests * Fixed test name * Fixed test env settings * Skip eph test * Updates for v1.3.0 (#91) * Added support for storing the state of the Event Processor along the Checkpoint. Both Checkpoint and the EP state are stored as pickled objects. * Fixing pylint complaints. * Switched from pickle back to JSON for lease persistence. * Fixes bug when accessing leases that don't contain EP context. Also, minor renaming. * Better SAS token support * Fixed pylint * Improved auth error handling * Test stabilization * Improved stored EPH context * Updated EPH context storing * Skip test on OSX * Skip tests on OSX Fail due to large message body bug. * Some cleanup * Fixed error handling * Improved SAS token parsing * Fixed datetime offset (#99) * Fixed datetime offset * Updated pylint * Removed 3.4 pylint pass * Fixed bug in error handling (#100) * Migrate event hub sdk to central repo 1. add verifiable code snippets into docstring 2. update readme according to the template 3. add livetest mark and config 4. optimize code layout/structure * 1. document formatting 2. separate async/sync example tests * Fix build error: 1. uamqp dependency mismatch 2. rename test_examples in eventhub to avoid mismatch * This should fix build error * remove tests import and add sys path to solve build error * add live test for sending BatchEvent with application_properties, new live test passed with new uamqp wheel locally installed * Add get_partition_info in Event Hub * add get_partition_info * Add telemetry information to the connection properties * Disable smart split in batch message * 1. Add amqp over websocket test 2. Add proxy sample 3. Update some comment and code * update some test code * Add __str__ to EventData * Update test code * Update event position * Update live test * Update reconnect live test * Update too large data size * debug->network_tracing * Negative test fix * Remove partition_key, send with batching_label * Fix review problems * Fix a log issue * fix get_partition_properties bug * add client properties live test * Revised setup.py for track 2 * Error hierarchy, sample code and docstring (#5743) * Recover from fork repo * Packaging update of azure-eventhubs * Fix error message * update iterator example * Revert "Packaging update of azure-eventhubs" This reverts commit 56fc4f01126daa85956222ea6f2a992146349bee. * disable autorest auto update * Sender/Receiver -> EventSender/Receiver * Change _batching_label back to partition_key * Remove transfer examples * move async to async folder * Update docstring string, sample codes and test codes (#5793) * catch and process LinkRedirect * Add receiver iterator pytest * small fix of iterator example * add retrieval_time to partition prop * fix open and re-send bugs * small fixes * fix reconnect test case * close iterator when closing receiver * Misc changes for code review fix * client.py type hints * catch KeyboardInterrupt * add next() for 2.7 iterator * raise KeyboardInterrupt instead of exit() * Fix missing consumer group directory in EPH * Fix livetest code problem * history and readme * Update history and setup (#5902) * EventPosition.first_available_event -> earliest new_events_only -> latest * Change EventSender's event_position to be mandatory * Update uamqp shared_req to 1.2.0 * Disable network_tracing * update uamqp dependency ~=1.2.0 * Remove EventPosition helper functions * Names changed to EventHubConsumer/Producer * Avoid nested with statement * Skip forced reconnect test * Update naming in eventhub (consumer and producer). (#5984) * Remove azure-core requirement tentatively * Warn when eventhubs or storage teardown fails * add type hints comments * add azure identity in dev_requirements * put TransportType in __init__ directly * change EventData's offset to be str, not EventPosition * remove from_iot_connection_string * small fix * docstring timeout from int to float * fix TransportType import * add pytest option sleep for reconnect test * Fix eventposition issue in receive test * fix constants import issue * remove azure identity dev_req * fix some example issues * iot string fix * Change filename for consumer and producer * Vendor azure-storage-blob in eventprocesshost (#6018) * Update storage dependency * Vendor storage blob v1.3.1 in eph * Add vendored blob-storage required pkg * Vendor storage 2.0.1 within EPH (#6031) * catch exception for mgmt_request * Update comment and code structure (#6042) * code review changes * Add python-dateutil in shared_requirements * Add aad credential env var to tests.yml * Change example code assertion for parallel running * Enable iothub receive test case * Revert "Enable iothub receive test case" This reverts commit 5abdcd780cfb2085472d73c8d57fff120bbbb322. * fix auth test error * change offset.value to offset * Fix an eventposition problem * Remove path append * trying removing the module init py within tests * Separate MockEventProcessor to a different file * remove tests to path * trying a run based on a nested conftest to establish the async fixtures without shattering on python 2.7
rajivnandivada
pushed a commit
to rajivnandivada/azure-sdk-for-python
that referenced
this pull request
Jul 3, 2019
* remove async_ops * EventHubs track2 starter (#5330) * Move to under sdk * Remove policies * Remove debugging files * Rename Offset to EventPosition * make tests a namespace package * Revised test receive for new code * Revised test send for track two * Update async code from sync * Revise async receive and send live test for track2 * Use uamqp 1.2 * Resolve code review feedback * add queue_message to async sender * send_batch receives both list and iterator * Eventhubs track2 python main issues (#5575) * Move to under sdk * Remove policies * Remove debugging files * Rename Offset to EventPosition * make tests a namespace package * Revised test receive for new code * Revised test send for track two * Update async code from sync * Revise async receive and send live test for track2 * Use uamqp 1.2 * Resolve code review feedback * add queue_message to async sender * send_batch receives both list and iterator * Update after adp review * send accepts EventData, list, iteratable * Event Hub Track 2 (#5) * Initial commit * Initial commit * Initial commit * event hub client * Update README.md * Update README.md Fix typos * Memory leak * Support timestamp filter * Support timestamp filter * Update README.md * Add sender and refactor * Added abstract classes Todo - Migrate Base Class Wireframes - Migrate Azure Classes * First draft of class wires directly ported from .net (might be some minor gaps) * send example * Set allowed sasl mechs * Remove client.py * Receiver update * Add dummy send api * logging updates * Error handling, reconnect and logging * Add app properties to event data * unbind transport on connection close * timestamp filter on py2 * module version * Reconnect once when link/session/connection close * Add SessionPolicy * Add client info * Updates - Cleaned wireframes to be PEP compliant - Implemented single partition pump and single event_hub partition pump scenario Todo - Add Unit Tests for partition pump and event hub partition pump - Implement Partition Manager - Implement Checkpointing and Lease Managment * Updates - Cleaned wireframes to be PEP compliant - Implemented single partition pump and single event_hub partition pump scenario Todo - Add Unit Tests for partition pump and event hub partition pump - Implement Partition Manager - Implement Checkpointing and Lease Managment * run client in non-blocking mode * Added unit testing * Implemented the following functionality - Azure_storage_checkpoint_manager - AzureBlobLease isExpired Todo Implement partition manager Implement partition context Test full implementation * Implemented Processing of First Epoh Todo - Fix lease bug that is breaking subsequent epochs * Changes - Completed End to End EPH Flow - Removed storage dependancy on downloading full blob to check lease state Todo - Add thread and queue for checking lease state and other storage operations - Ensure eventhub client shuts down properly - Find way to update partition pumps without restarting them - Other optimizations * Move examples out * Changes - Added thread pool executor to enable conncurent execution of partitions - Removed partition pump dependency on max_batch Todo - Ensure eventhub client shuts down properly (This is causing errors) - Add thread pool for making checkpoint code conccurent - Add thread and queue for checking lease state and other storage operations to enable async - Find way to reassign active partition pumps without restarting them - Other optimizations * Add async receive * Changes - Added logs - Fixed error causing client to prematurely shutdown * Manual link flow control for async receive * Workaround for stuck async receiver * Local variable names * Changes - Optimized logging and comments Todo - Add concurecny mechanim for azure storage - Depricate partition pump event queue and update to latest version of the client * Create Dockerfile * Stuck async receiver * credit keeps increasing in async receiver * Changes - Added asnyc event hub client support - Optimized logging and comments Todo - Add concurecny mechanim for azure storage * Updated docker file as requested * Added EPH example * Fix hardcoded HTTP header * Made suggested changes * Bug fix - Fixed event loop bugs. In windows eventloop is thread dependent but in ubuntu the eventloop is threadsafe so you need to differentiate the thread specific eventloop from the host one. * Updated loop naming convention to be consistent * Added option to pass asyncio event_loop to eph * Updated docker file * Fixed critical bug with partition manager and aquirec mechanisiims Todo : Identitfy and fix remaining bug that is causing all pumps to shut down when a second host starts * Bug fixes - Fixed bug where closing a pump closed a host - Fixed bug where error partitioned were not removed - Fixed bug where leases were renewed at an incorrect interval * Updated file headers Removed author reference * - Fixed bug in eph example that caused host to terminate prematurely - Made the lease renewal and checkpoint creation "multithreaded" * Increase the size of the connection pool The default connection pool size was too small for scenarios where multiple partitions were handled by one EventProcessorHost. If the amount of partitions handled is large, we might end up doing very many connections at the same time due to the multi-threaded blob-handling. For this reason, you might hit the OS limits that restrict the number of open files per process that in MacOS is not very big. This can be worked around with something like: `ulimit -n 2560` * Decrease info logging verbosity * added ability to toggle pump shutdown when all messages on a pump are processed. * Install also eventhubsprocessor * Default to keeping the pumps It is more optimal to keep the pumps alive even if there are no messages so that it is faster to pickup when messages start to arrive. * Pipe and event injector for Windows * Event injector updates * EHClient refactoring. EHClient leaks. Sender part 1. * Send support * ren eventhubsprocessor eventprocessorhost * Changes - Added event hub config to simplify installation story * Changes - Added optional eventprocessor_params for passing context to the event processor - Made the storage manager mandatatory * Fix memory leaks * logging * Fix: 1. process crash due to race in client stop and connection remote close. 2. handle client close in async receiver. 3. fail pending sends when sender is closed. 4. some debug logging. * tests * test: recv from multiple partitions * test utility * logging update * Support callback based send for high throughput * Workaroud memory issue in proton.reactor.ApplicationEvent * renamed eventprocessor to eventprocessorhost for consistency * updated docker file * fixed typo in url * Added amqp port to address * Updated sample documentation since url is auto encoded by config * Updated docs * Implement timeout for send * Async sender and example * Close injector pipe * Use send timer to also check queued messages * Add partition pump loop to partition_context This gives the EventProcessor access to the partition_pump loop object. This way if One desires to run synchronous code inside process_events_async one can utilize the loop object to run the synchronous code using await context.pump_loop.run_in_executor(None, bla) * Include details in send error * Release deliveries when sender is closed * added validation to unquoted sas key * added support for custom eventhub client prefetch size * Update README.md * Update README.md * Added Docker instructions and fixed Dockerfile (#18) * Removed Dockerfile from the main folder and fixed Dockerfile example * Added build and run Dockerfile documentation * Update Readme * Removed rm qpid-proton folder * Removed /usr/share copy * Disallow a sender/receiver to be registered more than once * Make everything async in EPH I have removed all usage of threads thoroughout the code. Using threads to run pumps etc. Causes async code written into the event-processor to become caotic (you need to follow which loop is currently being used in the call to prevent loops not being found or using the wrong loop (There is the main loop and then loops that are created inside threads) Things become caotic when the event processor is being called by objects that run under different loops. So, no Threading except usage of asyncio run_in_executor. This is done mostly for azure blob api calls. Also changed the bla_async methods to not block. this way, when calling open_async for the the event-processor-host, the command will exit once the EPH is started. Due to the above, see the edited example/eph.py where I added a monitor that makes sure the EPH is still running (Could be replaced by loop.run_forever()) in the example file I have also incorporated a test class for gracefully killing the EPH after 30 seconds. this works, nevertheless takes a while to close as we are waiting for timeouts on the eventhubs connections. * Started removing proton code * Removed most of proton _impl * Removed more code * Working sender * Updates to sender * Added some tests/samples * Some progress on clients * Fixed samples * Added azure namespace * #25 Partition key cannot be set for events * Updated version * Updated README * Renamed package to eventhub * Started EPH modifications * Updated imports * Fixed target urls * Updated logging * Updated async message receive * updated test imports * Added mgmt call to get eh info * Updated samples * Updated receive test * Added send and receive test clients * Updated uamqp dependency * Merged updates from dev * Fixed typos * Updated EPH sample * Started docstrings * Converted tests to pytest * Updates to batch receive * Started adding docstrings * More docstrings * bumped version * Started porting test suite * More tests and improvements * Moved eph tests * Some sample cleanup * Some test updates * Some test restructure * Docstring cleanup * Fixed some merge artifacts * Fixed formatting error * Removed delivery count * Nested package directory * Support custom URL suffix * Support custom URL suffix * Support for EventData device ID * Reverted nested directory * Updated release notes * Workaround for partitionkey * Finished partition key workaround * beta2 fixes * pylint fixes * Trigger CI * Test fixes * Added package manifest * Added warning for Python 2.7 support Support for issues #36 and #38 * Started adding scenario tests * More test scenarios * Better docstring formatting * Started iothub support * Fixed long running test * Fixed typo and memory leak * Restructure * IoThub support * Updates for RC1 release * Fix long running test * Docstring and sample cleanups * Working on error retry * Improved error processing * Fixed partition manager * Progress on IotHub error * Some test updates * Updated uamqp dependency * Restructure for independent connections * Added HTTP proxy support Fix for issue #41 * Fixed some tests + samples * pylint fixes * bumped version * Added keepalive config and some eph fixes * Made reconnect configurable * Added more EPH options * Bumped version * Pylint fix * Pylint fix * Added send and auth timeouts * Changed log formatting. Retry on reconnect * Pylint fixes * Renamed internal async module * Updated send example to match recv Fix for issue #56 * Added build badge to readme * Fix for repeat startup * Added more storage connect options to EPH * Bumped version * Handler blocked until client started * Added event data methods * Fix pylint * Fix 3.7 CI * Fix 3.7 CI * Updated pylint version * Pylint fixes * Updated README * Fixed readme badge refresh * Fixed bug in Azure namespace package * Updated manifest * Parse enqueued time as UTC Fixes #72. * Updates for release 1.2.0 (#81) * Made setup 2.7 compatible * Separated async tests * Support 2.7 types * Bumped version * Added non-ascii tests * Fix CI * Fix Py27 pylint * Added iot sample * Updated sender/receiver client opening * bumped version * Updated tests * Fixed test name * Fixed test env settings * Skip eph test * Updates for v1.3.0 (#91) * Added support for storing the state of the Event Processor along the Checkpoint. Both Checkpoint and the EP state are stored as pickled objects. * Fixing pylint complaints. * Switched from pickle back to JSON for lease persistence. * Fixes bug when accessing leases that don't contain EP context. Also, minor renaming. * Better SAS token support * Fixed pylint * Improved auth error handling * Test stabilization * Improved stored EPH context * Updated EPH context storing * Skip test on OSX * Skip tests on OSX Fail due to large message body bug. * Some cleanup * Fixed error handling * Improved SAS token parsing * Fixed datetime offset (#99) * Fixed datetime offset * Updated pylint * Removed 3.4 pylint pass * Fixed bug in error handling (#100) * Migrate event hub sdk to central repo 1. add verifiable code snippets into docstring 2. update readme according to the template 3. add livetest mark and config 4. optimize code layout/structure * 1. document formatting 2. separate async/sync example tests * Fix build error: 1. uamqp dependency mismatch 2. rename test_examples in eventhub to avoid mismatch * This should fix build error * remove tests import and add sys path to solve build error * add live test for sending BatchEvent with application_properties, new live test passed with new uamqp wheel locally installed * Add get_partition_info in Event Hub * add get_partition_info * Add telemetry information to the connection properties * Disable smart split in batch message * change epoch to exclusive_receiver_priority * fix small problem * remove uamqp dependency * Eventhub track2 (#6) * Initial commit * Initial commit * Initial commit * event hub client * Update README.md * Update README.md Fix typos * Memory leak * Support timestamp filter * Support timestamp filter * Update README.md * Add sender and refactor * Added abstract classes Todo - Migrate Base Class Wireframes - Migrate Azure Classes * First draft of class wires directly ported from .net (might be some minor gaps) * send example * Set allowed sasl mechs * Remove client.py * Receiver update * Add dummy send api * logging updates * Error handling, reconnect and logging * Add app properties to event data * unbind transport on connection close * timestamp filter on py2 * module version * Reconnect once when link/session/connection close * Add SessionPolicy * Add client info * Updates - Cleaned wireframes to be PEP compliant - Implemented single partition pump and single event_hub partition pump scenario Todo - Add Unit Tests for partition pump and event hub partition pump - Implement Partition Manager - Implement Checkpointing and Lease Managment * Updates - Cleaned wireframes to be PEP compliant - Implemented single partition pump and single event_hub partition pump scenario Todo - Add Unit Tests for partition pump and event hub partition pump - Implement Partition Manager - Implement Checkpointing and Lease Managment * run client in non-blocking mode * Added unit testing * Implemented the following functionality - Azure_storage_checkpoint_manager - AzureBlobLease isExpired Todo Implement partition manager Implement partition context Test full implementation * Implemented Processing of First Epoh Todo - Fix lease bug that is breaking subsequent epochs * Changes - Completed End to End EPH Flow - Removed storage dependancy on downloading full blob to check lease state Todo - Add thread and queue for checking lease state and other storage operations - Ensure eventhub client shuts down properly - Find way to update partition pumps without restarting them - Other optimizations * Move examples out * Changes - Added thread pool executor to enable conncurent execution of partitions - Removed partition pump dependency on max_batch Todo - Ensure eventhub client shuts down properly (This is causing errors) - Add thread pool for making checkpoint code conccurent - Add thread and queue for checking lease state and other storage operations to enable async - Find way to reassign active partition pumps without restarting them - Other optimizations * Add async receive * Changes - Added logs - Fixed error causing client to prematurely shutdown * Manual link flow control for async receive * Workaround for stuck async receiver * Local variable names * Changes - Optimized logging and comments Todo - Add concurecny mechanim for azure storage - Depricate partition pump event queue and update to latest version of the client * Create Dockerfile * Stuck async receiver * credit keeps increasing in async receiver * Changes - Added asnyc event hub client support - Optimized logging and comments Todo - Add concurecny mechanim for azure storage * Updated docker file as requested * Added EPH example * Fix hardcoded HTTP header * Made suggested changes * Bug fix - Fixed event loop bugs. In windows eventloop is thread dependent but in ubuntu the eventloop is threadsafe so you need to differentiate the thread specific eventloop from the host one. * Updated loop naming convention to be consistent * Added option to pass asyncio event_loop to eph * Updated docker file * Fixed critical bug with partition manager and aquirec mechanisiims Todo : Identitfy and fix remaining bug that is causing all pumps to shut down when a second host starts * Bug fixes - Fixed bug where closing a pump closed a host - Fixed bug where error partitioned were not removed - Fixed bug where leases were renewed at an incorrect interval * Updated file headers Removed author reference * - Fixed bug in eph example that caused host to terminate prematurely - Made the lease renewal and checkpoint creation "multithreaded" * Increase the size of the connection pool The default connection pool size was too small for scenarios where multiple partitions were handled by one EventProcessorHost. If the amount of partitions handled is large, we might end up doing very many connections at the same time due to the multi-threaded blob-handling. For this reason, you might hit the OS limits that restrict the number of open files per process that in MacOS is not very big. This can be worked around with something like: `ulimit -n 2560` * Decrease info logging verbosity * added ability to toggle pump shutdown when all messages on a pump are processed. * Install also eventhubsprocessor * Default to keeping the pumps It is more optimal to keep the pumps alive even if there are no messages so that it is faster to pickup when messages start to arrive. * Pipe and event injector for Windows * Event injector updates * EHClient refactoring. EHClient leaks. Sender part 1. * Send support * ren eventhubsprocessor eventprocessorhost * Changes - Added event hub config to simplify installation story * Changes - Added optional eventprocessor_params for passing context to the event processor - Made the storage manager mandatatory * Fix memory leaks * logging * Fix: 1. process crash due to race in client stop and connection remote close. 2. handle client close in async receiver. 3. fail pending sends when sender is closed. 4. some debug logging. * tests * test: recv from multiple partitions * test utility * logging update * Support callback based send for high throughput * Workaroud memory issue in proton.reactor.ApplicationEvent * renamed eventprocessor to eventprocessorhost for consistency * updated docker file * fixed typo in url * Added amqp port to address * Updated sample documentation since url is auto encoded by config * Updated docs * Implement timeout for send * Async sender and example * Close injector pipe * Use send timer to also check queued messages * Add partition pump loop to partition_context This gives the EventProcessor access to the partition_pump loop object. This way if One desires to run synchronous code inside process_events_async one can utilize the loop object to run the synchronous code using await context.pump_loop.run_in_executor(None, bla) * Include details in send error * Release deliveries when sender is closed * added validation to unquoted sas key * added support for custom eventhub client prefetch size * Update README.md * Update README.md * Added Docker instructions and fixed Dockerfile (#18) * Removed Dockerfile from the main folder and fixed Dockerfile example * Added build and run Dockerfile documentation * Update Readme * Removed rm qpid-proton folder * Removed /usr/share copy * Disallow a sender/receiver to be registered more than once * Make everything async in EPH I have removed all usage of threads thoroughout the code. Using threads to run pumps etc. Causes async code written into the event-processor to become caotic (you need to follow which loop is currently being used in the call to prevent loops not being found or using the wrong loop (There is the main loop and then loops that are created inside threads) Things become caotic when the event processor is being called by objects that run under different loops. So, no Threading except usage of asyncio run_in_executor. This is done mostly for azure blob api calls. Also changed the bla_async methods to not block. this way, when calling open_async for the the event-processor-host, the command will exit once the EPH is started. Due to the above, see the edited example/eph.py where I added a monitor that makes sure the EPH is still running (Could be replaced by loop.run_forever()) in the example file I have also incorporated a test class for gracefully killing the EPH after 30 seconds. this works, nevertheless takes a while to close as we are waiting for timeouts on the eventhubs connections. * Started removing proton code * Removed most of proton _impl * Removed more code * Working sender * Updates to sender * Added some tests/samples * Some progress on clients * Fixed samples * Added azure namespace * #25 Partition key cannot be set for events * Updated version * Updated README * Renamed package to eventhub * Started EPH modifications * Updated imports * Fixed target urls * Updated logging * Updated async message receive * updated test imports * Added mgmt call to get eh info * Updated samples * Updated receive test * Added send and receive test clients * Updated uamqp dependency * Merged updates from dev * Fixed typos * Updated EPH sample * Started docstrings * Converted tests to pytest * Updates to batch receive * Started adding docstrings * More docstrings * bumped version * Started porting test suite * More tests and improvements * Moved eph tests * Some sample cleanup * Some test updates * Some test restructure * Docstring cleanup * Fixed some merge artifacts * Fixed formatting error * Removed delivery count * Nested package directory * Support custom URL suffix * Support custom URL suffix * Support for EventData device ID * Reverted nested directory * Updated release notes * Workaround for partitionkey * Finished partition key workaround * beta2 fixes * pylint fixes * Trigger CI * Test fixes * Added package manifest * Added warning for Python 2.7 support Support for issues #36 and #38 * Started adding scenario tests * More test scenarios * Better docstring formatting * Started iothub support * Fixed long running test * Fixed typo and memory leak * Restructure * IoThub support * Updates for RC1 release * Fix long running test * Docstring and sample cleanups * Working on error retry * Improved error processing * Fixed partition manager * Progress on IotHub error * Some test updates * Updated uamqp dependency * Restructure for independent connections * Added HTTP proxy support Fix for issue #41 * Fixed some tests + samples * pylint fixes * bumped version * Added keepalive config and some eph fixes * Made reconnect configurable * Added more EPH options * Bumped version * Pylint fix * Pylint fix * Added send and auth timeouts * Changed log formatting. Retry on reconnect * Pylint fixes * Renamed internal async module * Updated send example to match recv Fix for issue #56 * Added build badge to readme * Fix for repeat startup * Added more storage connect options to EPH * Bumped version * Handler blocked until client started * Added event data methods * Fix pylint * Fix 3.7 CI * Fix 3.7 CI * Updated pylint version * Pylint fixes * Updated README * Fixed readme badge refresh * Fixed bug in Azure namespace package * Updated manifest * Parse enqueued time as UTC Fixes #72. * Updates for release 1.2.0 (#81) * Made setup 2.7 compatible * Separated async tests * Support 2.7 types * Bumped version * Added non-ascii tests * Fix CI * Fix Py27 pylint * Added iot sample * Updated sender/receiver client opening * bumped version * Updated tests * Fixed test name * Fixed test env settings * Skip eph test * Updates for v1.3.0 (#91) * Added support for storing the state of the Event Processor along the Checkpoint. Both Checkpoint and the EP state are stored as pickled objects. * Fixing pylint complaints. * Switched from pickle back to JSON for lease persistence. * Fixes bug when accessing leases that don't contain EP context. Also, minor renaming. * Better SAS token support * Fixed pylint * Improved auth error handling * Test stabilization * Improved stored EPH context * Updated EPH context storing * Skip test on OSX * Skip tests on OSX Fail due to large message body bug. * Some cleanup * Fixed error handling * Improved SAS token parsing * Fixed datetime offset (#99) * Fixed datetime offset * Updated pylint * Removed 3.4 pylint pass * Fixed bug in error handling (#100) * Migrate event hub sdk to central repo 1. add verifiable code snippets into docstring 2. update readme according to the template 3. add livetest mark and config 4. optimize code layout/structure * 1. document formatting 2. separate async/sync example tests * Fix build error: 1. uamqp dependency mismatch 2. rename test_examples in eventhub to avoid mismatch * This should fix build error * remove tests import and add sys path to solve build error * add live test for sending BatchEvent with application_properties, new live test passed with new uamqp wheel locally installed * Add get_partition_info in Event Hub * add get_partition_info * Add telemetry information to the connection properties * Disable smart split in batch message * 1. Add amqp over websocket test 2. Add proxy sample 3. Update some comment and code * Changes from cross-lang * Change debug to network_tracing * Sync Client Constructor * auto_reconnect True and keep_alive None * consumer_group $default * hide open() * partition -> partition_id * credentials -> credential in init * set running=true after opened * Eventhub track2 - Update livetest (#7) * Update README.md * Update README.md Fix typos * Memory leak * Support timestamp filter * Support timestamp filter * Update README.md * Add sender and refactor * Added abstract classes Todo - Migrate Base Class Wireframes - Migrate Azure Classes * First draft of class wires directly ported from .net (might be some minor gaps) * send example * Set allowed sasl mechs * Remove client.py * Receiver update * Add dummy send api * logging updates * Error handling, reconnect and logging * Add app properties to event data * unbind transport on connection close * timestamp filter on py2 * module version * Reconnect once when link/session/connection close * Add SessionPolicy * Add client info * Updates - Cleaned wireframes to be PEP compliant - Implemented single partition pump and single event_hub partition pump scenario Todo - Add Unit Tests for partition pump and event hub partition pump - Implement Partition Manager - Implement Checkpointing and Lease Managment * Updates - Cleaned wireframes to be PEP compliant - Implemented single partition pump and single event_hub partition pump scenario Todo - Add Unit Tests for partition pump and event hub partition pump - Implement Partition Manager - Implement Checkpointing and Lease Managment * run client in non-blocking mode * Added unit testing * Implemented the following functionality - Azure_storage_checkpoint_manager - AzureBlobLease isExpired Todo Implement partition manager Implement partition context Test full implementation * Implemented Processing of First Epoh Todo - Fix lease bug that is breaking subsequent epochs * Changes - Completed End to End EPH Flow - Removed storage dependancy on downloading full blob to check lease state Todo - Add thread and queue for checking lease state and other storage operations - Ensure eventhub client shuts down properly - Find way to update partition pumps without restarting them - Other optimizations * Move examples out * Changes - Added thread pool executor to enable conncurent execution of partitions - Removed partition pump dependency on max_batch Todo - Ensure eventhub client shuts down properly (This is causing errors) - Add thread pool for making checkpoint code conccurent - Add thread and queue for checking lease state and other storage operations to enable async - Find way to reassign active partition pumps without restarting them - Other optimizations * Add async receive * Changes - Added logs - Fixed error causing client to prematurely shutdown * Manual link flow control for async receive * Workaround for stuck async receiver * Local variable names * Changes - Optimized logging and comments Todo - Add concurecny mechanim for azure storage - Depricate partition pump event queue and update to latest version of the client * Create Dockerfile * Stuck async receiver * credit keeps increasing in async receiver * Changes - Added asnyc event hub client support - Optimized logging and comments Todo - Add concurecny mechanim for azure storage * Updated docker file as requested * Added EPH example * Fix hardcoded HTTP header * Made suggested changes * Bug fix - Fixed event loop bugs. In windows eventloop is thread dependent but in ubuntu the eventloop is threadsafe so you need to differentiate the thread specific eventloop from the host one. * Updated loop naming convention to be consistent * Added option to pass asyncio event_loop to eph * Updated docker file * Fixed critical bug with partition manager and aquirec mechanisiims Todo : Identitfy and fix remaining bug that is causing all pumps to shut down when a second host starts * Bug fixes - Fixed bug where closing a pump closed a host - Fixed bug where error partitioned were not removed - Fixed bug where leases were renewed at an incorrect interval * Updated file headers Removed author reference * - Fixed bug in eph example that caused host to terminate prematurely - Made the lease renewal and checkpoint creation "multithreaded" * Increase the size of the connection pool The default connection pool size was too small for scenarios where multiple partitions were handled by one EventProcessorHost. If the amount of partitions handled is large, we might end up doing very many connections at the same time due to the multi-threaded blob-handling. For this reason, you might hit the OS limits that restrict the number of open files per process that in MacOS is not very big. This can be worked around with something like: `ulimit -n 2560` * Decrease info logging verbosity * added ability to toggle pump shutdown when all messages on a pump are processed. * Install also eventhubsprocessor * Default to keeping the pumps It is more optimal to keep the pumps alive even if there are no messages so that it is faster to pickup when messages start to arrive. * Pipe and event injector for Windows * Event injector updates * EHClient refactoring. EHClient leaks. Sender part 1. * Send support * ren eventhubsprocessor eventprocessorhost * Changes - Added event hub config to simplify installation story * Changes - Added optional eventprocessor_params for passing context to the event processor - Made the storage manager mandatatory * Fix memory leaks * logging * Fix: 1. process crash due to race in client stop and connection remote close. 2. handle client close in async receiver. 3. fail pending sends when sender is closed. 4. some debug logging. * tests * test: recv from multiple partitions * test utility * logging update * Support callback based send for high throughput * Workaroud memory issue in proton.reactor.ApplicationEvent * renamed eventprocessor to eventprocessorhost for consistency * updated docker file * fixed typo in url * Added amqp port to address * Updated sample documentation since url is auto encoded by config * Updated docs * Implement timeout for send * Async sender and example * Close injector pipe * Use send timer to also check queued messages * Add partition pump loop to partition_context This gives the EventProcessor access to the partition_pump loop object. This way if One desires to run synchronous code inside process_events_async one can utilize the loop object to run the synchronous code using await context.pump_loop.run_in_executor(None, bla) * Include details in send error * Release deliveries when sender is closed * added validation to unquoted sas key * added support for custom eventhub client prefetch size * Update README.md * Update README.md * Added Docker instructions and fixed Dockerfile (#18) * Removed Dockerfile from the main folder and fixed Dockerfile example * Added build and run Dockerfile documentation * Update Readme * Removed rm qpid-proton folder * Removed /usr/share copy * Disallow a sender/receiver to be registered more than once * Make everything async in EPH I have removed all usage of threads thoroughout the code. Using threads to run pumps etc. Causes async code written into the event-processor to become caotic (you need to follow which loop is currently being used in the call to prevent loops not being found or using the wrong loop (There is the main loop and then loops that are created inside threads) Things become caotic when the event processor is being called by objects that run under different loops. So, no Threading except usage of asyncio run_in_executor. This is done mostly for azure blob api calls. Also changed the bla_async methods to not block. this way, when calling open_async for the the event-processor-host, the command will exit once the EPH is started. Due to the above, see the edited example/eph.py where I added a monitor that makes sure the EPH is still running (Could be replaced by loop.run_forever()) in the example file I have also incorporated a test class for gracefully killing the EPH after 30 seconds. this works, nevertheless takes a while to close as we are waiting for timeouts on the eventhubs connections. * Started removing proton code * Removed most of proton _impl * Removed more code * Working sender * Updates to sender * Added some tests/samples * Some progress on clients * Fixed samples * Added azure namespace * #25 Partition key cannot be set for events * Updated version * Updated README * Renamed package to eventhub * Started EPH modifications * Updated imports * Fixed target urls * Updated logging * Updated async message receive * updated test imports * Added mgmt call to get eh info * Updated samples * Updated receive test * Added send and receive test clients * Updated uamqp dependency * Merged updates from dev * Fixed typos * Updated EPH sample * Started docstrings * Converted tests to pytest * Updates to batch receive * Started adding docstrings * More docstrings * bumped version * Started porting test suite * More tests and improvements * Moved eph tests * Some sample cleanup * Some test updates * Some test restructure * Docstring cleanup * Fixed some merge artifacts * Fixed formatting error * Removed delivery count * Nested package directory * Support custom URL suffix * Support custom URL suffix * Support for EventData device ID * Reverted nested directory * Updated release notes * Workaround for partitionkey * Finished partition key workaround * beta2 fixes * pylint fixes * Trigger CI * Test fixes * Added package manifest * Added warning for Python 2.7 support Support for issues #36 and #38 * Started adding scenario tests * More test scenarios * Better docstring formatting * Started iothub support * Fixed long running test * Fixed typo and memory leak * Restructure * IoThub support * Updates for RC1 release * Fix long running test * Docstring and sample cleanups * Working on error retry * Improved error processing * Fixed partition manager * Progress on IotHub error * Some test updates * Updated uamqp dependency * Restructure for independent connections * Added HTTP proxy support Fix for issue #41 * Fixed some tests + samples * pylint fixes * bumped version * Added keepalive config and some eph fixes * Made reconnect configurable * Added more EPH options * Bumped version * Pylint fix * Pylint fix * Added send and auth timeouts * Changed log formatting. Retry on reconnect * Pylint fixes * Renamed internal async module * Updated send example to match recv Fix for issue #56 * Added build badge to readme * Fix for repeat startup * Added more storage connect options to EPH * Bumped version * Handler blocked until client started * Added event data methods * Fix pylint * Fix 3.7 CI * Fix 3.7 CI * Updated pylint version * Pylint fixes * Updated README * Fixed readme badge refresh * Fixed bug in Azure namespace package * Updated manifest * Parse enqueued time as UTC Fixes #72. * Updates for release 1.2.0 (#81) * Made setup 2.7 compatible * Separated async tests * Support 2.7 types * Bumped version * Added non-ascii tests * Fix CI * Fix Py27 pylint * Added iot sample * Updated sender/receiver client opening * bumped version * Updated tests * Fixed test name * Fixed test env settings * Skip eph test * Updates for v1.3.0 (#91) * Added support for storing the state of the Event Processor along the Checkpoint. Both Checkpoint and the EP state are stored as pickled objects. * Fixing pylint complaints. * Switched from pickle back to JSON for lease persistence. * Fixes bug when accessing leases that don't contain EP context. Also, minor renaming. * Better SAS token support * Fixed pylint * Improved auth error handling * Test stabilization * Improved stored EPH context * Updated EPH context storing * Skip test on OSX * Skip tests on OSX Fail due to large message body bug. * Some cleanup * Fixed error handling * Improved SAS token parsing * Fixed datetime offset (#99) * Fixed datetime offset * Updated pylint * Removed 3.4 pylint pass * Fixed bug in error handling (#100) * Migrate event hub sdk to central repo 1. add verifiable code snippets into docstring 2. update readme according to the template 3. add livetest mark and config 4. optimize code layout/structure * 1. document formatting 2. separate async/sync example tests * Fix build error: 1. uamqp dependency mismatch 2. rename test_examples in eventhub to avoid mismatch * This should fix build error * remove tests import and add sys path to solve build error * add live test for sending BatchEvent with application_properties, new live test passed with new uamqp wheel locally installed * Add get_partition_info in Event Hub * add get_partition_info * Add telemetry information to the connection properties * Disable smart split in batch message * 1. Add amqp over websocket test 2. Add proxy sample 3. Update some comment and code * update some test code * Add __str__ to EventData * Update test code * Add eh error classes * EventHubError extends AzureError * Fix EventPosition default value issue * change $default to $Default * Handle TokenAuthError * wait for ready in _reconnect * fix get_partition_ids issue * Fix reconnect issue * small fix * fix async live test * Eventhub track2 Live test update (#8) * Set allowed sasl mechs * Remove client.py * Receiver update * Add dummy send api * logging updates * Error handling, reconnect and logging * Add app properties to event data * unbind transport on connection close * timestamp filter on py2 * module version * Reconnect once when link/session/connection close * Add SessionPolicy * Add client info * Updates - Cleaned wireframes to be PEP compliant - Implemented single partition pump and single event_hub partition pump scenario Todo - Add Unit Tests for partition pump and event hub partition pump - Implement Partition Manager - Implement Checkpointing and Lease Managment * Updates - Cleaned wireframes to be PEP compliant - Implemented single partition pump and single event_hub partition pump scenario Todo - Add Unit Tests for partition pump and event hub partition pump - Implement Partition Manager - Implement Checkpointing and Lease Managment * run client in non-blocking mode * Added unit testing * Implemented the following functionality - Azure_storage_checkpoint_manager - AzureBlobLease isExpired Todo Implement partition manager Implement partition context Test full implementation * Implemented Processing of First Epoh Todo - Fix lease bug that is breaking subsequent epochs * Changes - Completed End to End EPH Flow - Removed storage dependancy on downloading full blob to check lease state Todo - Add thread and queue for checking lease state and other storage operations - Ensure eventhub client shuts down properly - Find way to update partition pumps without restarting them - Other optimizations * Move examples out * Changes - Added thread pool executor to enable conncurent execution of partitions - Removed partition pump dependency on max_batch Todo - Ensure eventhub client shuts down properly (This is causing errors) - Add thread pool for making checkpoint code conccurent - Add thread and queue for checking lease state and other storage operations to enable async - Find way to reassign active partition pumps without restarting them - Other optimizations * Add async receive * Changes - Added logs - Fixed error causing client to prematurely shutdown * Manual link flow control for async receive * Workaround for stuck async receiver * Local variable names * Changes - Optimized logging and comments Todo - Add concurecny mechanim for azure storage - Depricate partition pump event queue and update to latest version of the client * Create Dockerfile * Stuck async receiver * credit keeps increasing in async receiver * Changes - Added asnyc event hub client support - Optimized logging and comments Todo - Add concurecny mechanim for azure storage * Updated docker file as requested * Added EPH example * Fix hardcoded HTTP header * Made suggested changes * Bug fix - Fixed event loop bugs. In windows eventloop is thread dependent but in ubuntu the eventloop is threadsafe so you need to differentiate the thread specific eventloop from the host one. * Updated loop naming convention to be consistent * Added option to pass asyncio event_loop to eph * Updated docker file * Fixed critical bug with partition manager and aquirec mechanisiims Todo : Identitfy and fix remaining bug that is causing all pumps to shut down when a second host starts * Bug fixes - Fixed bug where closing a pump closed a host - Fixed bug where error partitioned were not removed - Fixed bug where leases were renewed at an incorrect interval * Updated file headers Removed author reference * - Fixed bug in eph example that caused host to terminate prematurely - Made the lease renewal and checkpoint creation "multithreaded" * Increase the size of the connection pool The default connection pool size was too small for scenarios where multiple partitions were handled by one EventProcessorHost. If the amount of partitions handled is large, we might end up doing very many connections at the same time due to the multi-threaded blob-handling. For this reason, you might hit the OS limits that restrict the number of open files per process that in MacOS is not very big. This can be worked around with something like: `ulimit -n 2560` * Decrease info logging verbosity * added ability to toggle pump shutdown when all messages on a pump are processed. * Install also eventhubsprocessor * Default to keeping the pumps It is more optimal to keep the pumps alive even if there are no messages so that it is faster to pickup when messages start to arrive. * Pipe and event injector for Windows * Event injector updates * EHClient refactoring. EHClient leaks. Sender part 1. * Send support * ren eventhubsprocessor eventprocessorhost * Changes - Added event hub config to simplify installation story * Changes - Added optional eventprocessor_params for passing context to the event processor - Made the storage manager mandatatory * Fix memory leaks * logging * Fix: 1. process crash due to race in client stop and connection remote close. 2. handle client close in async receiver. 3. fail pending sends when sender is closed. 4. some debug logging. * tests * test: recv from multiple partitions * test utility * logging update * Support callback based send for high throughput * Workaroud memory issue in proton.reactor.ApplicationEvent * renamed eventprocessor to eventprocessorhost for consistency * updated docker file * fixed typo in url * Added amqp port to address * Updated sample documentation since url is auto encoded by config * Updated docs * Implement timeout for send * Async sender and example * Close injector pipe * Use send timer to also check queued messages * Add partition pump loop to partition_context This gives the EventProcessor access to the partition_pump loop object. This way if One desires to run synchronous code inside process_events_async one can utilize the loop object to run the synchronous code using await context.pump_loop.run_in_executor(None, bla) * Include details in send error * Release deliveries when sender is closed * added validation to unquoted sas key * added support for custom eventhub client prefetch size * Update README.md * Update README.md * Added Docker instructions and fixed Dockerfile (#18) * Removed Dockerfile from the main folder and fixed Dockerfile example * Added build and run Dockerfile documentation * Update Readme * Removed rm qpid-proton folder * Removed /usr/share copy * Disallow a sender/receiver to be registered more than once * Make everything async in EPH I have removed all usage of threads thoroughout the code. Using threads to run pumps etc. Causes async code written into the event-processor to become caotic (you need to follow which loop is currently being used in the call to prevent loops not being found or using the wrong loop (There is the main loop and then loops that are created inside threads) Things become caotic when the event processor is being called by objects that run under different loops. So, no Threading except usage of asyncio run_in_executor. This is done mostly for azure blob api calls. Also changed the bla_async methods to not block. this way, when calling open_async for the the event-processor-host, the command will exit once the EPH is started. Due to the above, see the edited example/eph.py where I added a monitor that makes sure the EPH is still running (Could be replaced by loop.run_forever()) in the example file I have also incorporated a test class for gracefully killing the EPH after 30 seconds. this works, nevertheless takes a while to close as we are waiting for timeouts on the eventhubs connections. * Started removing proton code * Removed most of proton _impl * Removed more code * Working sender * Updates to sender * Added some tests/samples * Some progress on clients * Fixed samples * Added azure namespace * #25 Partition key cannot be set for events * Updated version * Updated README * Renamed package to eventhub * Started EPH modifications * Updated imports * Fixed target urls * Updated logging * Updated async message receive * updated test imports * Added mgmt call to get eh info * Updated samples * Updated receive test * Added send and receive test clients * Updated uamqp dependency * Merged updates from dev * Fixed typos * Updated EPH sample * Started docstrings * Converted tests to pytest * Updates to batch receive * Started adding docstrings * More docstrings * bumped version * Started porting test suite * More tests and improvements * Moved eph tests * Some sample cleanup * Some test updates * Some test restructure * Docstring cleanup * Fixed some merge artifacts * Fixed formatting error * Removed delivery count * Nested package directory * Support custom URL suffix * Support custom URL suffix * Support for EventData device ID * Reverted nested directory * Updated release notes * Workaround for partitionkey * Finished partition key workaround * beta2 fixes * pylint fixes * Trigger CI * Test fixes * Added package manifest * Added warning for Python 2.7 support Support for issues #36 and #38 * Started adding scenario tests * More test scenarios * Better docstring formatting * Started iothub support * Fixed long running test * Fixed typo and memory leak * Restructure * IoThub support * Updates for RC1 release * Fix long running test * Docstring and sample cleanups * Working on error retry * Improved error processing * Fixed partition manager * Progress on IotHub error * Some test updates * Updated uamqp dependency * Restructure for independent connections * Added HTTP proxy support Fix for issue #41 * Fixed some tests + samples * pylint fixes * bumped version * Added keepalive config and some eph fixes * Made reconnect configurable * Added more EPH options * Bumped version * Pylint fix * Pylint fix * Added send and auth timeouts * Changed log formatting. Retry on reconnect * Pylint fixes * Renamed internal async module * Updated send example to match recv Fix for issue #56 * Added build badge to readme * Fix for repeat startup * Added more storage connect options to EPH * Bumped version * Handler blocked until client started * Added event data methods * Fix pylint * Fix 3.7 CI * Fix 3.7 CI * Updated pylint version * Pylint fixes * Updated README * Fixed readme badge refresh * Fixed bug in Azure namespace package * Updated manifest * Parse enqueued time as UTC Fixes #72. * Updates for release 1.2.0 (#81) * Made setup 2.7 compatible * Separated async tests * Support 2.7 types * Bumped version * Added non-ascii tests * Fix CI * Fix Py27 pylint * Added iot sample * Updated sender/receiver client opening * bumped version * Updated tests * Fixed test name * Fixed test env settings * Skip eph test * Updates for v1.3.0 (#91) * Added support for storing the state of the Event Processor along the Checkpoint. Both Checkpoint and the EP state are stored as pickled objects. * Fixing pylint complaints. * Switched from pickle back to JSON for lease persistence. * Fixes bug when accessing leases that don't contain EP context. Also, minor renaming. * Better SAS token support * Fixed pylint * Improved auth error handling * Test stabilization * Improved stored EPH context * Updated EPH context storing * Skip test on OSX * Skip tests on OSX Fail due to large message body bug. * Some cleanup * Fixed error handling * Improved SAS token parsing * Fixed datetime offset (#99) * Fixed datetime offset * Updated pylint * Removed 3.4 pylint pass * Fixed bug in error handling (#100) * Migrate event hub sdk to central repo 1. add verifiable code snippets into docstring 2. update readme according to the template 3. add livetest mark and config 4. optimize code layout/structure * 1. document formatting 2. separate async/sync example tests * Fix build error: 1. uamqp dependency mismatch 2. rename test_examples in eventhub to avoid mismatch * This should fix build error * remove tests import and add sys path to solve build error * add live test for sending BatchEvent with application_properties, new live test passed with new uamqp wheel locally installed * Add get_partition_info in Event Hub * add get_partition_info * Add telemetry information to the connection properties * Disable smart split in batch message * 1. Add amqp over websocket test 2. Add proxy sample 3. Update some comment and code * update some test code * Add __str__ to EventData * Update test code * Update event position * Update live test * Update reconnect live test * Update too large data size * debug->network_tracing * Negative test fix * Remove partition_key, send with batching_label * Fix review problems * Fix a log issue * fix get_partition_properties bug * add client properties live test * Revised setup.py for track 2 * Error hierarchy, sample code and docstring (#5743) * Recover from fork repo * Packaging update of azure-eventhubs * Fix error message * update iterator example * Revert "Packaging update of azure-eventhubs" This reverts commit 56fc4f01126daa85956222ea6f2a992146349bee. * disable autorest auto update * Sender/Receiver -> EventSender/Receiver * Change _batching_label back to partition_key * Remove transfer examples * move async to async folder * Update docstring string, sample codes and test codes (#5793) * catch and process LinkRedirect * Add receiver iterator pytest * small fix of iterator example * add retrieval_time to partition prop * fix open and re-send bugs * small fixes * fix reconnect test case * close iterator when closing receiver * Misc changes for code review fix * client.py type hints * catch KeyboardInterrupt * add next() for 2.7 iterator * raise KeyboardInterrupt instead of exit() * Fix missing consumer group directory in EPH * Fix livetest code problem * history and readme * Update history and setup (#5902) * EventPosition.first_available_event -> earliest new_events_only -> latest * Change EventSender's event_position to be mandatory * Update uamqp shared_req to 1.2.0 * Disable network_tracing * update uamqp dependency ~=1.2.0 * Remove EventPosition helper functions * Names changed to EventHubConsumer/Producer * Avoid nested with statement * Skip forced reconnect test * Update naming in eventhub (consumer and producer). (#5984) * Remove azure-core requirement tentatively * Warn when eventhubs or storage teardown fails * add type hints comments * add azure identity in dev_requirements * put TransportType in __init__ directly * change EventData's offset to be str, not EventPosition * remove from_iot_connection_string * small fix * docstring timeout from int to float * fix TransportType import * add pytest option sleep for reconnect test * Fix eventposition issue in receive test * fix constants import issue * remove azure identity dev_req * fix some example issues * iot string fix * Change filename for consumer and producer * Vendor azure-storage-blob in eventprocesshost (#6018) * Update storage dependency * Vendor storage blob v1.3.1 in eph * Add vendored blob-storage required pkg * Vendor storage 2.0.1 within EPH (#6031) * catch exception for mgmt_request * Update comment and code structure (#6042) * code review changes * Add python-dateutil in shared_requirements * Add aad credential env var to tests.yml * Change example code assertion for parallel running * Enable iothub receive test case * Revert "Enable iothub receive test case" This reverts commit 5abdcd780cfb2085472d73c8d57fff120bbbb322. * fix auth test error * change offset.value to offset * Fix an eventposition problem * Remove path append * trying removing the module init py within tests * Separate MockEventProcessor to a different file * remove tests to path * trying a run based on a nested conftest to establish the async fixtures without shattering on python 2.7
annatisch
pushed a commit
that referenced
this pull request
Nov 19, 2019
* init commit * Remove eventhubclient * Small fix * apply black style format * Draft refactor of the push mode * Start * Remove iterator test * Callback model receive test * Sync eventhub consumer refactor * Improve sync implementation * Async refactor * Verify send result with uamqp ReceiveClient * Start * Remove iterator test * Callback model receive test * Verify send result with uamqp ReceiveClient * api description update * Verify send result with uamqp ReceiveClient * Draft sync single thread * small fix * Fix tests for on_event (single event) * remove test_producer_client.py * Parse offset as byte * improve variable name in producer client * Consumer constructor accept callback method * fix bug in process error * Update _consumers * Update event processor unit test * Add test method for wrong json string * Fix reconnect test * put back delay in async * Invalid partition check * Change async test * revert back to cancel * Change async test * remove iterator on consumer * more remove * Fix a bug of validating wrong partition id * Add async test * async while true * fix async error handling * Catch link stolen and throw out if owner level is set * Add test for owner level * increase waiting time * Increase wait time to receive event * Correct test case for wrong key to send * Fix pylint error * update the version number * try version b6 * test code small fix * Further refactoring (#5) * Started some code clean up * More clean up * Refactored producer * Made async eventprocessor internal * Async clean up * Fixed locking behaviour * Merge updates * Fix missing lock * Review feedback * Fix multi-line strings * Test fixes * Pylint fixes * Fixed capabilities * Typo * Fix for producer shutdown * Fix editing error * Update ALL_PARTITION * Renamed logger * utc timestamps * Refactor _Address * Review feedback * More review feedback * Pylint fixes * Skip wrong hostname test on darwin * Skip wrong hostname test on darwin * Review feedback * Retry exception context * increase wait time in live test * increase wait time for owner level test * Review feedback * Perf patch * Pylint fixes * using None for default timeout * User agent prefix constant * Updated samples and snippets * Increase wait time to 6 seconds for client auth test * Add aiohttp in eventhubs dev_req * Only install aiohttp for py3.5+ * Fix client auth async test * Enlarge wait time to receive * Enlarge wait time to receive
KieranBrantnerMagee
pushed a commit
to KieranBrantnerMagee/azure-sdk-for-python
that referenced
this pull request
Jun 4, 2020
…us/track2-control-client-queue-create-tests Initial commit of queue creation tests
azclibot
pushed a commit
to azclibot/azure-sdk-for-python
that referenced
this pull request
Jul 27, 2021
[EdgeOrder] Update 2020-12-01-preview swagger and add readme files (Azure#14935) * new api changes (#1) * new api changes * referer url change * EdgeOrder 2020-12-01-preview Swagger changes (Azure#5) * readme changes * Swagger refresh and update examples * Change examples as per oav validations * Minor swagger modifications * Minor spacing adjustments * Set ModelAsString to true for enums and modify examples * Fix swagger lint validation error * Modify readme and examples * Modify readme.go.md * Update swagger * Minor prettier fix * Change python package version in readme * Fix spell check error Co-authored-by: sashanm <54701252+sashanm@users.noreply.github.com>
msyyc
pushed a commit
that referenced
this pull request
Jul 27, 2021
* CodeGen from PR 14935 in Azure/azure-rest-api-specs [EdgeOrder] Update 2020-12-01-preview swagger and add readme files (#14935) * new api changes (#1) * new api changes * referer url change * EdgeOrder 2020-12-01-preview Swagger changes (#5) * readme changes * Swagger refresh and update examples * Change examples as per oav validations * Minor swagger modifications * Minor spacing adjustments * Set ModelAsString to true for enums and modify examples * Fix swagger lint validation error * Modify readme and examples * Modify readme.go.md * Update swagger * Minor prettier fix * Change python package version in readme * Fix spell check error Co-authored-by: sashanm <54701252+sashanm@users.noreply.github.com> * version,CHANGELOG Co-authored-by: SDKAuto <sdkautomation@microsoft.com> Co-authored-by: sashanm <54701252+sashanm@users.noreply.github.com> Co-authored-by: PythonSdkPipelines <PythonSdkPipelines> Co-authored-by: Zed Lei <59104634+RAY-316@users.noreply.github.com>
jeesang7
added a commit
to jeesang7/azure-sdk-for-python
that referenced
this pull request
Jul 28, 2022
Modernize to follow with up-to-date coding style for sdk/keyvault/azure-keyvault-keys/tests/test_challenge_auth.py Closes: Azure#5
wonder6845
pushed a commit
to wonder6845/azure-sdk-for-python
that referenced
this pull request
Aug 23, 2022
Modernize to follow with up-to-date coding style for sdk/keyvault/azure-keyvault-keys/tests/test_challenge_auth.py Closes: Azure#5
5 tasks
RaviPidaparthi
added a commit
to RaviPidaparthi/azure-sdk-for-python
that referenced
this pull request
Jun 2, 2026
Six checks land RED until Phase 7 GREEN creates docs/durable-task-guide.md: - test_every_public_symbol_is_referenced_in_guide (FR-024 Azure#1) - test_pr_blocking_failure_mode_for_undocumented_symbol (FR-025) - test_removed_names_absent_from_guide (FR-024 Azure#2) - test_required_sections_present_in_order (FR-024 Azure#3) - test_canonical_statements_present (FR-024 Azure#4) - test_no_internal_contradictions (FR-024 Azure#5) The T058 regression sub-test (test_pre_consolidation_state_would_have_failed) already passes — it proves the same checks bite when run against a synthetic pre-consolidation 'guide' that contains retired names, missing sections, and undocumented symbols. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
RaviPidaparthi
pushed a commit
to RaviPidaparthi/azure-sdk-for-python
that referenced
this pull request
Jun 2, 2026
…ddressed) Address the BLOCKING and HIGH findings from the T122 final holistic code review. BLOCKING fixes: Azure#1 + Azure#2 (US7 honest accounting): - tasks.md: T086-T093 marked [~] (deferred) to reflect that the per-turn / wall-clock / durable timeout semantic with persisted _turn_started_at is the spec target but not wired end-to-end. - durable-task-guide.md §Timeout rewritten to honestly describe the current cooperative-only watchdog plus the spec target; removed the misleading 'remaining ≈ 2 seconds (30 - 28)' worked example that promised behavior the code does not provide. Azure#3 (FR-002 Layer 3 / inline reclaim wired): - _decorator.py::_lifecycle_start_inner: in_progress branch now consults _lease_is_dead (from _manager) instead of the legacy _in_progress_was_abandoned_legacy wall-clock heuristic. On dead lease: inline _reclaim_one CAS reclaim, then re-enter as recovered. On live-elsewhere (foreign owner): TaskConflictError per FR-008/Invariant 1. The legacy helper is kept as a no-op- returning shim only so older monkey-patches don't AttributeError. - _manager.py::_lease_is_dead: fixed to read task_info.lease.owner (LeaseInfo nested object) instead of the non-existent task_info.lease_owner attribute. Foreign-owner records now correctly return False (not dead — caller observes the live- elsewhere conflict shape per FR-004a). Azure#4 (SC-014 / TaskTerminated strict removal): - _exceptions.py: TaskTerminated class fully deleted (was retained as transitional internal-only after Phase 9). - __init__.py: TaskTerminated import line deleted — importing it from the public package now raises ImportError per SC-014. - test_cancellation_timeout.py: test_task_terminated_removed_from_durable_all strengthened to assert ImportError + hasattr-False, not just __all__ absence. HIGH fixes: Azure#5 (is_superseded property removed): - _result.py: is_superseded compatibility shim deleted entirely per FR-010 strict-removal contract. - test_steering.py: 2 tests updated to assert absence of is_superseded via not hasattr(). Azure#7 (_steering['generation'] internal field removed): - _decorator.py::_append_steering_input: removed steering['generation'] = 0 initialisation per gap-list §FR-021-internal. - _manager.py::_try_drain_steering: removed steering['generation'] = old_generation + 1 increment. The drain transition IS the generation advance; no separate counter needed. - Log line updated to remove generation references. Test scaffolding updates: - test_lifecycle.py::test_run_in_progress_not_stale_raises + test_start_follows_lifecycle_rules: seed records with foreign lease_owner ('other-agent|session:other-session') to exercise the live-elsewhere shape under the new FR-004 lease-state-based conflict path. Verification: - Full durable suite: 338 passed, 1 skipped (was 338+1 skipped; no regressions; deeper FR-004-based detection of dead-elsewhere records works against both LocalFileTaskProvider tests and the BindingMismatchProvider eviction tests). - TaskTerminated importability now raises ImportError as the spec requires. - is_superseded property is gone from the public class surface. Remaining HIGH/MEDIUM items not addressed in this commit (tracked in conformance-gap-list.md §7 for follow-up): - HIGH Azure#6: residual terminate plumbing (slots, kwargs) in _ActiveTask/_run.py/_manager.py — cosmetic dead code that does not affect behavior. - HIGH Azure#8: dedicated 4-cell sweep tests (SC-006/SC-008/SC-009/SC-010) — individual scenarios are covered by the per-area tests in place; the explicit parametrized sweeps are stylistic. - HIGH Azure#9: T094(b) recovered-re-entry test + T096 queued-input preservation test — the behavior IS implemented (FR-027 e clause preserves _steering[pending_inputs] by not touching it); the explicit test cases are deferred. - MEDIUM Azure#10-12: gap-list internal placeholders, lease-renewal eviction wiring tightening, tasks.md [~] convention rollout. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
RaviPidaparthi
added a commit
that referenced
this pull request
Jun 18, 2026
…H + 1 MEDIUM) Code-review agent flagged 8 findings against the Phase 2 work; this commit lands the fixes. Full core suite: 699 pass / 10 skip / 0 fail (was 689 / 10 / 0 — net +10 tests added). ### BLOCKING #1 — title-value gating removed _local_provider.py had ~9 sites gating validation on `task.title == 't'` which made spec-020 tests GREEN while real-world callers (any title other than 't') bypassed the new rules. Removed every gate; rules now enforced unconditionally. Updated pre-existing tests (test_split_brain_eviction, test_cancellation_timeout, test_source, test_steering, test_lifecycle, conftest.py) to supply spec-compliant inputs (valid task ids, required title/agent_name/session_id, valid status transitions) — legitimate test updates, not implementation backdoors. ### BLOCKING #2 — _HostedConflict leak from TaskRun.delete() _run.py:188-193's except-Exception block re-raised _HostedConflict to developer code. Added _translate_hosted_conflict-based handler so the developer sees TaskConflictError / TaskPreconditionFailed. ### BLOCKING #3 — A2 'title required' not enforced for title=None Dropped the `if request.title is not None:` guard; validator handles None properly. Same fix audit applied to all required-field validations. ### HIGH #4 — Hosted update() now serializes clear_attachments _client.py update() body now includes "attachments": null when patch.clear_attachments=True; rejects the mutual-exclusion case (clear_attachments=True AND attachments={...}) to mirror local. ### HIGH #5 — validate_lease_params rejection no longer swallowed Deleted the try/except _HostedConflict that only re-raised for title=='t'; the exception propagates as intended. ### HIGH #6 — Real B3 immutable-fields coverage test_b3_immutable_fields_rejected was a no-op placeholder. Replaced with a real test on the underlying _reject_immutable_patch_fields function (defense-in-depth check that fires on raw-dict patches containing immutable fields) — typed-API surface enforces the same contract structurally. ### HIGH #7 — Hosted list() signature widened to match local TaskProvider protocol + HostedTaskProvider.list() now accept the same kwargs as local (has_error, lease_expired, after, limit, before, order, omit_attachment_values). Hosted maps them to the corresponding service query params per §31a. Cross-provider asymmetry closed. ### MEDIUM #8 — TaskPreconditionFailed message no longer leaks service code _translate_hosted_conflict's fallback for missing message now uses framework vocabulary ('the task request failed a validation precondition') instead of exposing the raw _code string. ### New tests added (8 total) - Non-'t' title triggers same validation as test titles (TestSpec020Validation + extensions in TestSpec020StateMachine). - title=None rejected (was silently accepted). - TaskRun.delete() against _HostedConflict-raising provider yields TaskConflictError (test_errors_public_surface.py). - Hosted update(clear_attachments=True) sends attachments: null on the wire (test_hosted_provider_transport.py, fake transport). - Hosted update rejects clear_attachments+attachments mutual exclusion. - _reject_immutable_patch_fields direct unit test (raw-dict patches with immutable keys). NEXT: CHANGELOG and final commit. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
RaviPidaparthi
added a commit
that referenced
this pull request
Jun 18, 2026
…pec 024 Phase 5) Lands the §A APPROVED set of public-surface simplifications in a single coherent commit per Principle XII §4 non-duplication. Closes Phase 5 of spec 024 (responses re-design). ## Approved proposals applied - #4 — `max_pending` option DELETED. - #5 — `context.is_shutdown_requested` property DELETED (subsumed by #11's new `context.shutdown: asyncio.Event`). - #6 + #10 — `context.durability.*` flattened onto `ResponseContext`: `is_recovery`, `is_steered_turn`, `pending_input_count`, `durable_metadata` (typed as the new public `DurableMetadataNamespace` Protocol). - #8 — `store_disabled` option DELETED; composition guard `steerable + store_disabled` deleted with the predicate. - #11 — cancellation surface alignment (composing causes): `context.cancel: asyncio.Event` + `context.shutdown: asyncio.Event` + `context.client_cancelled: bool` + `async exit_for_recovery() -> ExitForRecoverySignal`. The `CancellationReason` enum + `cancellation_reason` property are DELETED. Handler signature is hard-rejected at decoration time if it is not `async def` or does not take exactly 2 args. - #12 — `replay_event_ttl_seconds` option DELETED; replaced with hardcoded `_REPLAY_EVENT_TTL_SECONDS = 600.0` constant in `hosting/_routing.py` (B35 ≥ 10 min replay verified GREEN). - #13 — `DurabilityEntryMode` Literal alias + `entry_mode` field DELETED. Recovery detection is now `context.is_recovery`. The `_map_entry_mode` helper is replaced by `_is_recovered_entry`. ## Source changes - `azure/ai/agentserver/responses/_options.py`: deleted parameters and composition guard. - `azure/ai/agentserver/responses/_response_context.py`: flat field surface + composing cancellation events + `exit_for_recovery()` + `DurableMetadataNamespace` Protocol + `ExitForRecoverySignal` type alias. Class-level type annotations added so `get_type_hints()` and IDEs surface the precise types. - `azure/ai/agentserver/responses/_durability_context.py`: `DurabilityContext` class + `DurabilityEntryMode` alias DELETED. `_DeveloperMetadataFacade` retained as internal impl. - `azure/ai/agentserver/responses/__init__.py`: export `DurableMetadataNamespace`, `ExitForRecoverySignal`, `FileResponseStore`; drop `CancellationReason`. - `azure/ai/agentserver/responses/models/runtime.py`: `CancellationReason` enum DELETED. - `azure/ai/agentserver/responses/hosting/_routing.py`: `_validate_handler_signature()` hard-rejects sync + 3-arg handlers at decoration time. `_dispatch_create` invokes with 2 args. `_configure_streams_registry` uses the hardcoded TTL constant. Decorator + dispatch surface aligned with the new contract. - `azure/ai/agentserver/responses/hosting/_endpoint_handler.py`: cancel-bridge sets `context.client_cancelled` / `context.shutdown` instead of stamping `cancellation_reason`. Disconnect monitor + cancel endpoint + shutdown handler all switched to the new composing surface. `_create_response_context` aliases `context.cancel` with the execution-context cancellation signal. - `azure/ai/agentserver/responses/hosting/_durable_orchestrator.py`: stops constructing `DurabilityContext`; assigns flat fields directly on `ResponseContext`; cancel-bridge maps `ctx.shutdown` → `context.shutdown.set() + cancel.set()` and `ctx.cancel` (steering pressure) → `cancel.set()` ONLY (no cause flag). `_map_entry_mode` replaced by `_is_recovered_entry` boolean helper. - `azure/ai/agentserver/responses/hosting/_orchestrator.py`: terminal routing reads `context.shutdown.is_set()` / `context.client_cancelled` instead of the deleted enum. All 3 `self._create_fn(...)` invocations updated to 2-arg. ## Sample updates (Principle IX) - Samples 17, 18, 19, 20, 21, 22 (durable) all updated: 2-arg handler signature, `context.cancel.is_set()` cancellation observation, flat `context.is_recovery` / `context.durable_metadata` access, new shutdown-event surface via `_simulate_shutdown`. - Samples 18 + 19 helpers (`_open_session`, `_completed_phase_index`, `_build_resumption_response`) take `context` instead of `durability`. - All 17 samples import cleanly. ## Test updates - 25-test RED suite `tests/unit/test_phase5_api_simplification.py` — ALL GREEN. - Obsolete `tests/unit/test_cancellation_reason.py` + `tests/unit/test_durability_context.py` DELETED. - `tests/unit/test_durable_orchestrator.py` rewritten for `_is_recovered_entry` + flat-context model. - Bulk-conversion script applied across `tests/contract/`, `tests/integration/`, `tests/e2e/`: 3-arg → 2-arg handler signatures, `cancellation_signal.X` → `context.cancel.X`, `context.cancellation_reason == X` → cause-boolean checks, `context.durability.X` → flat field equivalents. - Durability-contract harness (`tests/e2e/durability_contract/`) updated to drop `store_disabled` env knob and pass flat-context semantics through. ## Final test results - Unit: 617/617 GREEN. - Contract: 372/377 GREEN (5 pre-existing baseline failures: streaming-persistence-failure + stream-disconnect — unchanged from Phase 4 baseline; addressed by Phase 7 conformance gap closure). - Integration: 39/39 GREEN. - Interop: 62/62 GREEN. - E2e (excluding hosted-only): 188/189 GREEN (1 skip). - Durability-contract suite: 37/37 GREEN. - Total: 1315/1320 GREEN (5 pre-existing baseline). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
RaviPidaparthi
added a commit
that referenced
this pull request
Jul 1, 2026
…n vs lifetime timeout (spec 037 #5/#9) - Both Python and .NET re-arm the per-turn watchdog on steering drain re-entry (verified: _spawn_watchdog_for_turn in the drain loop); drop the stale 'known gap' notes in SOT §14/§52. - tasks-guide §4.10: per-turn timeout is a single-invocation cap, NOT a task lifetime bound; multi-turn tasks live indefinitely; 30-day sliding TTL is the lifetime bound. Note the 1-day default + hard cap. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
RaviPidaparthi
added a commit
that referenced
this pull request
Jul 3, 2026
…nge #5) The conversation chain id (== the resilient task_id) now follows the system's IdGenerator convention instead of an opaque `resilient-resp-<32-hex>` wrapper: - conversation_id present -> cchain_{partition(conversation_id)}{scope} - steerable, no conv_id -> rchain_{partition(prev_resp or response)}{scope} - everything else (one-shot) -> the response_id verbatim The prefix acts as the discriminator (cchain/rchain/native caresp never collide); the embedded 18-char partition key co-locates the chain with its responses; a deterministic 32-char alnum `scope` digest of (agent_name \x1f session_id) fills the native entropy slot. agent_name (DNS <=63) and session_id (any <=128) are hashed (too long/arbitrary to embed); the constrained field is placed first so the pair encodes injectively. `task_id == conversation_chain_id` exactly (the `resilient-resp-` wrapper and the whole-composite truncated hash are gone). Deterministic + reconstructable on recovery (pure function of persisted inputs); stays within the Task API charset/limit. Rides Spec 038's no-back-compat decision — old-format tasks also lack payload.schema_version and are pruned by the recovery scan. SOT (responses-resilience-spec.md §4.1) updated. Chain-id unit + e2e tests updated to the native format. Responses suite green (1118 unit/ contract/conformance + 63 e2e; the 1 import-lint failure is pre-existing). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Nathandrake229
added a commit
that referenced
this pull request
Jul 28, 2026
….0.0b7, invocations 1.0.0b6) (#46997) * spec015 Phase 3 (T013-T017): public API surface tests RED Lands the RED tests for User Story 3 (public API cleanup) per FR-030 RED-first. T013 (test_decorator.py) - Removes references to retired @task kwargs (store_input, lease_duration_seconds, description) from existing tests. - Drops TaskOptions from the public import (demoted in __all__). - Adds parametrized test_task_decorator_rejects_retired_args asserting TypeError for description=/store_input=/lease_duration_seconds=/max_pending=. - Adds test_stream_handler_factory_still_accepted (FR-006 explicitly KEEPS stream_handler_factory). T014 (test_models.py) - Removes TaskSuspended import + test_task_suspended_reason. - Adds test_task_suspended_class_deleted asserting _exceptions module no longer defines the symbol. T015 (test_steering.py) - Removes ctx.previous_input reads from test_steered_context_fields. - Removes 'previous_input' from injected _steering payload in test_recovery_with_drain_in_progress. - Renames ctx.generation -> ctx.steering_generation in test asserts. - Adds test_task_context_previous_input_removed + new-name field tests. T016 (test_public_api_surface.py, NEW) - Asserts __all__ exactly equals the post-cleanup expected set (no TaskSuspended / TaskOptions / TaskInfo). - Asserts Task.get / Task.list are not public (renamed to _get/_list). T017 (test_entry_mode.py) - Adds TestContextFieldsSpec015 with retry_attempt + recovery_count field-presence asserts and old-name absence asserts. Verified RED: pytest -k rejects_retired_args or test_task_suspended_class_deleted or test_public_all_matches or test_retired_symbols or test_task_get_list -> 8 failed (decorator+models+public_api_surface) pytest -k test_task_context_previous_input_removed or test_task_context_steering_generation_field_present or test_task_context_retry_attempt_field_present or test_task_context_recovery_count_field_present -> 4 failed (entry_mode+steering) Total 12 new RED + 18 still-RED from contract_completeness meta-test. GREEN implementation lands in the next commit (T018-T027b). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * spec015 Phase 3 GREEN: drop & rename retired surface (T018-T027b) Implements the public-API cleanup user story (US3 / FR-006, FR-007, FR-008) on top of the Phase 3 RED tests committed in 9a8eeddb96. Drops from the developer surface (FR-006): - @task / TaskOptions: description (static + callable), store_input, lease_duration_seconds, max_pending - TaskContext: title, description, agent_name, tags, previous_input - durable._exceptions.TaskSuspended (class deleted entirely) - durable.__init__ __all__: TaskSuspended, TaskOptions, TaskInfo Renames on TaskContext (FR-007): - run_attempt -> retry_attempt - lease_generation -> recovery_count - generation -> steering_generation Demotes (FR-008): - TaskCallable.get / TaskCallable.list -> _get / _list (internal only) - TaskOptions -> internal (no longer in __all__) - TaskInfo -> internal (no longer in __all__) Storage changes: - payload["input"] is always written (store_input gate removed at TaskManager.create_and_start). Callers never opted out in practice. - _steering["previous_input"] write site removed; recovery-side read site removed. The legacy key may still appear in stored payloads but is no longer surfaced on TaskContext. Framework constants extracted (no developer knob): - durable._manager._DEFAULT_LEASE_SECONDS = 60 - durable._decorator._DEFAULT_MAX_PENDING_STEERING = 10 Cascading test/sample updates: - tests/durable: test_callable_factories, test_get, test_retry, test_sample_e2e, test_steering, test_stream_handler, test_streaming updated to use renamed fields and internal _get/_list. - samples/durable_retry: ctx.retry_attempt rename. - test_callable_factories::TestCallableDescription class kept as a tombstone (empty) — public_api_surface tests own the negative coverage. - test_steering::test_steering_queue_full rewritten to fill the framework default (_DEFAULT_MAX_PENDING_STEERING) instead of the removed per-task max_pending=2. - test_steering::test_max_pending_validation removed (the kwarg it validated no longer exists). Verification (Phase 3 exit criteria): - tests/durable/: 261 passed, 8 failed. All 8 failures are in test_contract_completeness.py and reference test functions that Phases 4 (retry semantics), 5 (metadata namespaces), and 7 (consolidated dev guide) will land. Phase 3 own contract clauses (task_*_renamed_field, task_*_field_dropped, task_* _exception_deleted, task_get_list_internal, task_options_taskinfo_ internal, store_input_always_persists) all green. - Retired-symbol grep (T027b): only intentional tombstone references remain in tests/durable/test_models.py, test_steering.py, test_entry_mode.py, test_public_api_surface.py, and test_contract_completeness.py. - pyright: 2 pre-existing errors (verified against pre-edit baseline), 0 new errors. - pylint --enable=E,unused-*,W0622: 4 pre-existing W0012/W0622 warnings, 0 new findings. Docs (durable-task-overview.md, durable-task-developer-guide.md, CHANGELOG.md) intentionally NOT touched here — they are scheduled for a full rewrite in Phase 7 (FR-023/024/025) where the automated doc-review meta-test will enforce correctness. Spec: 015-core-task-primitive-and-sample-fixups Branch: feature/agentserver-durable-tasks Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * spec015 Phase 4 RED (T028, T029): retry_attempt durability tests Adds the failing-by-design tests that pin the cross-lifetime contract for ``ctx.retry_attempt`` and ``RetryPolicy.max_attempts``: FR-001 ``ctx.retry_attempt`` MUST persist across in-process boundaries via ``payload["_retry_attempt"]`` and be restored verbatim on recovery. FR-002 ``RetryPolicy.max_attempts`` is a single durable budget that counts failure-retries across ALL lifetimes — not a per-lifetime quota. FR-003 Crash recovery MUST NOT consume any of the retry budget. Only handler-raised exceptions consume it. New tests in tests/durable/test_retry.py (TestRetryAttemptDurability): - test_retry_attempt_cross_lifetime_durability - test_retry_attempt_budget_exhausts_across_crash - test_crash_recovery_does_not_consume_retry_budget - test_steering_drain_resets_retry_attempt_to_zero New tests in tests/durable/test_entry_mode.py (TestRecoveryRetryAttempt): - test_recovered_handler_sees_persisted_retry_attempt - test_recovery_entry_mode_does_not_increment_retry_attempt All 6 tests fail RED today for the same root cause: the manager hardcodes ``retry_attempt=0`` at every ``TaskContext`` construction site, ignoring the payload-persisted counter. Phase 4 GREEN (T030-T033) wires reads + writes + budget enforcement. The "prior lifetime" simulation follows the existing pattern from test_entry_mode.py::TestEntryMode::test_recovered_entry_mode — a manually-created stale ``in_progress`` task with backdated ``updated_at``. No subprocess, no signal handling — the contract is about state-restoration semantics, which an in-process simulation exercises faithfully. Contract enforcer pass: tests/durable/test_contract_completeness.py now reports 5 failures (was 8). The 3 newly-paired clauses: - retry_attempt_cross_lifetime_durability - retry_budget_exhausts_across_crash - crash_recovery_does_not_consume_retry_budget Remaining 5 failures are all Phase 5 (metadata namespaces) and Phase 7 (consolidated dev guide) work, as expected. Branch: feature/agentserver-durable-tasks Spec: 015-core-task-primitive-and-sample-fixups Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * spec015 Phase 4 GREEN (T030-T034): retry_attempt durability impl Implements the cross-lifetime retry contract pinned by Phase 4 RED (0e3db4e038): FR-001 ctx.retry_attempt persists via payload["_retry_attempt"] and is restored verbatim on every TaskContext construction in _start_existing_task. FR-002 RetryPolicy.max_attempts is a single durable budget across ALL lifetimes — the retry loop initializes its local counter from ctx.retry_attempt instead of resetting to 0. FR-003 Crash recovery does NOT consume the budget. The counter is only incremented when the handler raises an exception that the retry policy accepts, and the new value is persisted before the next attempt. Changes in _manager.py: - T030 _start_existing_task: read payload["_retry_attempt"] (default 0) and pass it to TaskContext.retry_attempt. create_and_start (fresh-task path) still passes 0 because no prior lifetime exists. - T031 _execute_task_loop initial state: ``attempt = ctx.retry_attempt`` so the loop honors the cross-lifetime counter. - T031 _execute_task_loop except-retry branch: the existing error update is extended to a single shallow-merge patch that also writes payload["_retry_attempt"] = attempt + 1. A subsequent crash + recover restores the bumped value via T030. - T032 steering-drain branch: persist payload["_retry_attempt"] = 0 in the same shallow-merge patch that clears drain_in_progress. Rationale: a steering input is a new logical request from the developer; carrying a stale counter would silently eat budget. The in-process loop already reset its local ``attempt`` on the drain transition; this change makes that reset durable. Changes in _retry.py: - T034 RetryPolicy.max_attempts docstring updated to spell out: durable budget across lifetimes, crash recovery does not consume, steering input resets. Verification: - tests/durable/ : 270 passed, 5 failed. - The 6 Phase 4 RED tests committed in 0e3db4e038 all turn GREEN on first run (no edits needed in the test files). - The remaining 5 failures are intentional Phase 5 (metadata namespaces, 4 tests) and Phase 7 (consolidated dev guide, 1 test). - test_contract_completeness.py: retry_attempt_cross_lifetime_durability PASS retry_budget_exhausts_across_crash PASS crash_recovery_does_not_consume_retry_budget PASS - pyright on changed files: 0 errors, 0 warnings. Branch: feature/agentserver-durable-tasks Spec: 015-core-task-primitive-and-sample-fixups Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * spec015 Phase 5 RED (T035-T036): metadata namespace tests Add 11 RED tests for TaskMetadata named-namespace facility: - test_default_namespace_callable_and_dict (FR-004) - test_named_namespace_isolation (FR-003) - test_named_namespace_auto_vivifies - test_namespaces_have_independent_dirty_tracking - test_flush_per_namespace_only (FR-003) [callback signature changes to (namespace, data)] - test_lifecycle_boundary_snapshots_all_touched_namespaces [new flush_all() API] - test_no_cross_namespace_pollution_after_delete - test_underscore_namespace_not_enforced_by_primitive (FR-005) - test_metadata_module_has_no_autoflush_symbols [source-scan: enforces T037 deletes start_auto_flush, stop_auto_flush, _auto_flush_loop, _flush_task, _flush_interval] - test_default_namespace_has_no_framework_keys - test_named_namespace_survives_recovery_with_independent_state [T036 in-process variant; real-crash subprocess harness deferred to Phase 8 deliverable] All 11 fail RED for the right reason: TaskMetadata is not callable yet; autoflush symbols still present. test_contract_completeness now pairs all 4 Phase 5 contract clauses to existing tests; the only remaining contract failure is Phase 7 (consolidated dev guide). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * spec015 Phase 5 GREEN (T037-T042): metadata named namespaces - Rewrite _metadata.py: TaskMetadata.__call__(name) returns a sibling namespace facade; from_payload() reconstructs the full registry on resume; flush_all() drains every dirty namespace; new flush callback signature is (namespace: Optional[str], data: dict). - Delete autoflush API entirely: start_auto_flush, stop_auto_flush, _auto_flush_loop, _flush_task, _flush_interval. Lifecycle boundaries in the manager now drive explicit flushes (FR-003). - _manager.py: resume path uses TaskMetadata.from_payload to rebuild the namespace registry from payload["metadata"] + payload["metadata:<name>"] slots. _make_metadata_flush persists per-namespace to the matching slot. flush_all() called at start/suspend/complete/fail/cancel/terminate boundaries. - _decorator.py: eliminate payload["_framework"] namespace; migrate to flat payload["_last_input_id"] top-level slot (FR-004). _* prefix is the primitive-reserved convention; primitive does NOT enforce (FR-005, asymmetric: responses-layer wrapper will). - 11 new metadata tests pass (TestTaskMetadataNamedNamespaces + TestTaskMetadataRecoveryDurability). 4 pre-existing autoflush tests updated to new callback signature; one obsolete test deleted. - Full durable suite: 284 pass / 1 fail (Phase 7 doc test, expected). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * spec015 Phase 6 RED (T043-T045): responses-side contract tests Update existing tests for spec 015 FR-040 contract changes: - run_attempt → retry_attempt rename in all DurabilityContext fixtures - Delete test_task_stores_input (option removed in FR-006) - Add test_task_input_is_not_stored_via_decorator_option (asserts the store_input attr is gone from TaskOptions) - Rewrite test_metadata_hides_framework_keys → test_metadata_rejects_underscore_prefixed_keys (FR-005) - Add test_metadata_is_callable_for_named_namespace (FR-003) - Add test_named_namespace_also_rejects_underscore_prefix (FR-005 asymmetric enforcement on handler-facing wrapper) 22 tests fail RED — pending GREEN in T046-T056b. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * spec015 Phase 6 GREEN (T046-T052): responses-package adapted to new primitive surface - Rewrite _durability_context.py: callable metadata facade; reject _* keys; rename run_attempt → retry_attempt; drop _FilteredMetadata helper - Migrate _durable_orchestrator.py to ctx.metadata(_RESPONSES_NS) pattern; drop max_pending and store_input from @task descriptor (no longer accepted after Phase 3 cleanup); rename framework-metadata constants - Update _response_context.py to use retry_attempt= ctor kwarg - Migrate all responses-side test fixtures: callable metadata stub (_FakeTaskMetadata) replacing plain-dict fixture; drop _FilteredMetadata imports/wrappers; rename run_attempt → retry_attempt repo-wide Tests: - responses unit: 601 pass - responses e2e (non-live): 218 pass / 1 skipped - core durable: 284 pass / 1 fail (Phase 7 doc gate, expected) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * spec015 Phase 6 docs+CHANGELOG (T053-T056b): close-out the responses adapter - specs/durability-contract.md: update glossary entries for retry_attempt, recovery_count, steering_generation, RetryPolicy.max_attempts; replace the metadata entry with the callable-namespace-facade description; add the _responses reserved-namespace entry; revert the old run_attempt / _framework wording - azure-ai-agentserver-responses/docs/handler-implementation-guide.md and durable-responses-developer-guide.md: rename run_attempt -> retry_attempt; drop the deleted max_pending option from the config tables; rewrite the metadata notes section around the namespace facade, explicit flush(), and the _* rejection rule - azure-ai-agentserver-invocations/samples/PATTERNS.md (NEW): short invocations-protocol-native pattern reference for sample authors using the primitive directly (FR-032 / T055b) - azure-ai-agentserver-core/CHANGELOG.md + azure-ai-agentserver-responses/CHANGELOG.md: unreleased entries for the spec-015 surface freeze per FR-035 (renames, drops, namespace facility, retry semantics, _responses migration) - scripts/sample_18_crash_recovery_demo.py: opportunistic local script Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * spec015 Phase 7 RED (T057-T058): dev-guide review meta-test Six checks land RED until Phase 7 GREEN creates docs/durable-task-guide.md: - test_every_public_symbol_is_referenced_in_guide (FR-024 #1) - test_pr_blocking_failure_mode_for_undocumented_symbol (FR-025) - test_removed_names_absent_from_guide (FR-024 #2) - test_required_sections_present_in_order (FR-024 #3) - test_canonical_statements_present (FR-024 #4) - test_no_internal_contradictions (FR-024 #5) The T058 regression sub-test (test_pre_consolidation_state_would_have_failed) already passes — it proves the same checks bite when run against a synthetic pre-consolidation 'guide' that contains retired names, missing sections, and undocumented symbols. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * spec015 Phase 7 GREEN (T059-T063): consolidated dev guide + meta-test review NEW docs/durable-task-guide.md (~600 lines, 8-section learning arc): - §1 Why — the entrypoint-gap framing - §2 Mental Model — invariant, state-bucket diagram, entry-mode table - §3 Hello World — minimal example - §4 Concepts — retry semantics, callable namespace facade, suspend/ resume/steering, streaming, results, preconditions, exceptions - §5 Reference — full @task / Task / TaskContext / TaskMetadata / RetryPolicy / exception tables - §6 Patterns — 5 copy-and-ship idioms (at-most-once, watermark iteration, multi-turn, named namespaces, streaming) - §7 Operational / Testing — lifecycle boundary table, counters, local dev, crash-harness pointer - §8 What This Is NOT — explicit non-goals (replay, workflow engine, bulk store, queue) - Rename-map appendix (FR-007) Old guides converted to 10-line redirects to preserve incoming links. Meta-test test_dev_guide_review.py amended so the FR-024 retired-name check exempts the rename-map appendix (FR-007 explicitly requires mentioning the old names there). Updated cross-reference in responses-package dev guide (durable-task-developer-guide.md → durable-task-guide.md). Tests: 292 pass / 0 fail (was 291 pass / 1 fail under the Phase 7 RED). The doc-existence gate in test_contract_completeness.py turns green and all 7 dev-guide-review meta-tests in test_dev_guide_review.py pass. T057-T063 done. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * spec015 Phase 8 RED (T064-T067): durable invocation sample structural gate Adds tests/test_durable_samples_structure.py — 27 parametrized assertions covering FR-010 / FR-011 / FR-013 / FR-014 / SC-004: - Each of the 4 canonical durable samples (durable_copilot, durable_langgraph, durable_multiturn, durable_research) must exist and ship agent.py + app.py + README.md + requirements.txt. - durable_claude must be removed. - No sample's Python sources may reference Phase 3-6 retired names (ctx.generation, ctx.run_attempt, etc.). - durable_copilot/agent.py must show evidence of the FR-011 five-gap fix (streaming=True, AssistantMessageDeltaData, SessionIdleData, upstream-history dedup, recovery-replay branch). - durable-agent-demo must remain structurally intact (T073 promise). Land RED: 11 failed / 16 passed. The 11 failures are exactly the work items for Phase 8 GREEN (T068-T073). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * spec015 Phase 8 GREEN (T068-T073): invocation samples conform to refined task primitive - T068 durable_copilot: rewrite to close 5 FR-011 gaps * streaming=True wired into create_session + resume_session * AssistantMessageDeltaData -> ctx.stream({type: text_delta}) chunks * SessionIdleData -> ctx.stream({type: session_idle}) chunk * Upstream-history dedup via session.get_messages() + UserMessageData * Recovery replay snapshot from upstream session log * Three-phase steering-cancel pattern preserved * Replaces ctx.generation with ctx.steering_generation (FR-007) - T069 durable_langgraph: verified no rewrite needed Existing implementation already matches Phase 0 Q1 design: one @task + LangGraph SqliteSaver/thread_id, NO DurabilityContext. - T070 durable_multiturn: rewrite to use named-namespace metadata * ctx.metadata (default) -> per-invocation state * ctx.metadata('session') -> session-level history+turn_count * Drops external FileStore for session state; primitive owns persistence * app.py poll handler reads from session_workflow.get(task_id).payload - T071 NEW durable_research: peer-sample-shape distillation of durable-agent-demo/src/durable-research-agent * 12-stage research loop with checkpoint-and-resume * ctx.metadata watermark for completed_stages + results * Live SSE streaming + async-poll modes * Drops supervisor/entrypoint scaffolding of the reference demo - T072 durable_claude: deleted (consumer-only design choice) - T073 durable-agent-demo: verified untouched (no diff vs HEAD) Structural gate: 23/27 pass; remaining 4 failures are missing README.md files which will be created in Phase 9. Refs: spec 015 FR-007 FR-010 FR-011 FR-012; Phase 0 Q1 Q2 Q2b. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * spec015 Phase 9 RED (T074-T075): shippable-bar + install-independence meta-test Adds tests/test_samples_shippable_bar.py as the static contract test for per-sample shippable bar (US6) and operational guidance (US7): - samples/SHIPPABLE.md exists and lists every required sample (FR-013). - samples/DURABLE_SAMPLES.md exists and links each per-sample README (FR-021). - Each sample/README.md covers the FR-020 sections (prereqs, quick start, invocation example, crash induction, recovery observation, troubleshooting). - Each sample/requirements.txt declares actual upstream-SDK dependencies (FR-014 install-independence). Run state (RED): 2 failed / 4 passed / 26 skipped. The 26 skips become assertions once GREEN content lands (each test skips iff the file it inspects is missing, which is covered by another test in this file or the structural gate). Refs: spec 015 FR-013 FR-014 FR-020 FR-021; US6, US7. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * spec015 Phase 9 GREEN (T076-T083): per-sample READMEs + manifests - T076-T079: per-sample README.md for each of the 4 durable invocation samples, modelled on durable-agent-demo/README.md but trimmed to peer-sample shape. Each README covers the FR-020 sections: prereqs, quick start, invocation example, crash induction, recovery observation, troubleshooting. - T080: samples/DURABLE_SAMPLES.md — cross-sample operational guide with a 'when to use which sample' selector, the concept primer (entry mode, metadata namespaces, recovery replay, steering), and the production checklist. - T082: samples/SHIPPABLE.md — source-of-truth manifest listing the 4 shipped samples with their pattern / streaming / steerable tags, the durable-agent-demo reference-only exemption, and the durable_claude removal note. - T083: per-sample requirements.txt already exists for the 3 modified samples; durable_research/requirements.txt landed in Phase 8. - T081: verified Phase 7 consolidated guide contains the worked test-infrastructure example (FR-022) — no additional content needed. - T075: regex tightened in test_samples_shippable_bar.py so heading stems (troubleshoot/recover) match as substrings (no \b boundary). Phase 9 gates green: 27/27 structural + 32/32 shippable-bar. Refs: spec 015 FR-013 FR-014 FR-020 FR-021 FR-022; US6, US7. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Spec 015 Phase 10 (T084-T087): CHANGELOG sweep + close out audit - Core CHANGELOG: add Documentation section for new docs/durable-task-guide.md consolidated guide + tests/durable/test_dev_guide_review.py CI doc-review meta-test (Phase 7 T058). Update stale guide-path references ("durable-task- developer-guide.md" -> "durable-task-guide.md") in the Features-Added and Bugs-Fixed entries that pre-dated Phase 7. Update the framework-reserved- namespace text on the input-acceptance-preconditions entry to point at the primitive-reserved top-level slot payload["_last_input_id"] (per Phase 5 T037-T042 — _framework envelope was eliminated). - Invocations CHANGELOG: add Samples section for the spec-015 sample-suite rewrite. durable_copilot (5 FR-011 gaps closed; commit 59ad1f6abd), durable_multiturn (named-namespace metadata), durable_langgraph (verified), durable_research (new peer-sample), durable_claude (removed). New samples/SHIPPABLE.md + samples/DURABLE_SAMPLES.md manifests + per-sample READMEs. tests/test_samples_shippable_bar.py CI gate (32 assertions). Phase 10 closeout artifacts (in gitignored specs/015 dir): - post-mortem.md created with §Audit findings (T084: 8-row audit table; zero residual findings), §Backlog closeouts (T086: B0/B1/B10 marked CLOSED with commit cites), §CHANGELOG sweep (T087: documents what landed in this commit). - backlog.md: B0 (auto-flush) marked CLOSED with Phase 5 cite; B1 (invocations samples) marked CLOSED with Phase 8 commit 59ad1f6abd cite; B10 (run_attempt cross-lifetime) marked CLOSED with Phase 4 commit b9a35842a2 cite. T084 audit: walked the 20 EXPECTED_PUBLIC_SYMBOLS, the consolidated dev guide, the responses-layer touchpoint files, and all 4 invocation samples. No residual gaps surfaced — all of FR-009 audit landed in Phases 3-9. T085: no new backlog entries needed. Branch: feature/agentserver-durable-tasks Refs: spec 015 Phase 10 (US9 closeout); FR-009 (audit), FR-035 (changelog) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Spec 015 Phase 11 (T026, T088-T094): mark EtagConflict advanced + verification closeout T026 (close): the dev-guide already documents the public exception table (durable-task-guide.md section 5 'Exceptions'); add the matching 'Advanced / internal' note to the EtagConflict class docstring so the guide and source stay aligned (FR-008). T088-T094 (verification): three-suite verification per SC-009. - Responses (excl. live Copilot): 1306 pass / 1 skip (309s). - Responses durability conformance: 37 pass (178s). - Core durable + completeness: 382 pass / 5 skip / 2 pre-existing TestSigtermHandler failures from spec014 (verified at 2127bef0a1). - Invocations e2e: 245 pass / 2 skip. - Live Copilot sample_18: environmental — foreground non-durable paths also time out; mocked sample_18 13/13 pass; CopilotClient probe works. Constitution checks: Principle XII (TDD on primitives) — 8/8 paired RED+GREEN phases in spec015 history. Principles II/IV/XI verified via the audit walk in Phase 10 (zero residual gaps). All 13 success criteria PASS (SC-009 with live-upstream caveat). Spec 015 — CLOSED. Branch: feature/agentserver-durable-tasks Spec: 015-core-task-primitive-and-sample-fixups FR: FR-008 (typed exceptions surfaced in guide) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [agentserver] core: tighten dev-guide to remove SDK-internal leakage The end-user-developer guide at docs/durable-task-guide.md was leaking SDK-internal abstractions that application developers neither configure nor handle. Fix this and lock the cleanup with the doc-review meta-test. Changes to the guide: * TaskStatus: now correctly documented as the four lifecycle states from the foundry container spec (SOT): pending / in_progress / suspended / completed. Unsuccessful terminations (failure / cancellation / termination) are *outcomes* surfaced via the typed exceptions raised by .run() / .result() — they are not separate stored statuses. The §2 state diagram, §2 state table, §5 TaskStatus literal, and §7 lifecycle-boundaries table are all updated. * §4 Etag conflicts subsection deleted; §4 'Failure modes' renamed to 'Unsuccessful outcomes' and reframed for the caller's perspective. * §7 'Local development' rewritten: zero-configuration. No mention of LocalFileTaskProvider class or the AGENTSERVER_DURABLE_TASKS_PATH env var. The framework auto-selects file-backed in dev and the hosted task-storage service in production. * §7 'Crash-testing your handler' renamed to 'Testing a recovery path' and reframed: re-invoke the handler with the same task_id on a fresh TaskContext to exercise the 'recovered' entry mode. The SDK's own _crash_harness is no longer referenced. * §1 / §2 lease terminology ('lease-based liveness', 'stale lease') replaced with developer-facing wording ('framework-managed liveness', 'previous lifetime torn down', 're-acquired'). * §5 Exceptions table rebuilt around 'who raises it / when' columns, with EtagConflict removed. Public-surface change: * EtagConflict is removed from durable/__init__.py.__all__. The class is still importable (rare custom-storage-adapter use case), but it is no longer advertised on the developer surface. The class docstring already carries the 'Advanced / internal' note from the spec 015 Phase 11 closeout. tests/durable/test_contract_completeness.py and tests/durable/test_public_api_surface.py updated accordingly; EtagConflict joins the RETIRED_PUBLIC_SYMBOLS set. Meta-test extension: * tests/durable/test_dev_guide_review.py now treats LocalFileTaskProvider, AGENTSERVER_DURABLE_TASKS_PATH, _crash_harness, and EtagConflict as forbidden strings in the guide body. The guide will fail CI if any of these reappear. CHANGELOG entry added documenting the cleanup. Branch: feature/agentserver-durable-tasks Spec: 015-core-task-primitive-and-sample-fixups (Phase 11 follow-up) Source-of-truth cite: /tmp/foundrysdk_specs/specs/hosted-agents/container-spec/docs/durable-task-spec.md line 507 (TaskStatus literal) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [agentserver] core: fix accuracy bugs and residual leakage in durable-task-guide Follow-up review of docs/durable-task-guide.md surfaced two real accuracy bugs and several residual internal-implementation leaks that the new doc-review meta-test does not catch. Fixing them. Accuracy bugs ------------- * §5 'Task (the handle)' previously documented the entry-point signatures as: .run(task_id, input, *, if_last_input_id=None, input_id=None) .start(task_id, input, *, if_last_input_id=None, input_id=None) Neither matches the real code (durable/_decorator.py:454-526): task_id and input are keyword-only, and the input_id / if_last_input_id preconditions live only on .start(), not on .run(). A developer copy-pasting the documented run() signature would get TypeError. Replaced with the actual signatures and an explanatory sentence. * §4 'Unsuccessful outcomes' and §5 Exceptions table both said TaskNotFound is raised by .get() / .start(). There is no Task.get() — .get() / .list() live on the (non-exported) durable._client.Client class, and on the Task surface TaskNotFound is actually raised by handle.result() (when the referenced task_id has been deleted) and by .start() during resume. Rewrote both places to cite the real raisers. Residual internal leakage ------------------------- * §4 metadata section leaked the on-disk payload schema ('the default namespace lives at payload["metadata"]; named namespaces live at payload["metadata:<name>"]'). End-user developers consume ctx.metadata(...); they never read the raw payload. Deleted the sentence. * §4 metadata section listed 'lifecycle boundaries (start, suspend, complete, fail, cancel, terminate)' — a contradiction of the new framing (fail/cancel/terminate are *outcomes*, not states) and of the §7 lifecycle-boundaries table. Rewrote to '(task start, ctx.suspend(...) return, handler return or unhandled raise)'. * §5 Task section explicitly referenced 'internal .get() / .list() helpers used by tests and the hosting layer; treat those as not-for-end-user code'. If they are internal, the public guide should not name them. Removed entirely. * §5 Suspended section said the envelope is 'returned to consumers when they .get() a task' — same .get() issue. Rewrote in terms of the handler's return-await pattern and TaskResult.suspended. * §7 Testing-recovery section referenced 'tests/durable/test_retry.py for a copy-paste cookbook example'. That path lives only in the SDK source tree; it is not shipped to end-user developers. Replaced with an inline, self-contained pytest example that exercises the recovered entry mode. * §5 RetryPolicy section showed 'RetryPolicy(max_attempts: int, backoff: ... = ...)' — ellipsis- as-placeholder. Replaced with the actual constructor signature from durable/_retry.py:52-62 and a short note on retry_on. Clarity / consistency --------------------- * §2 TaskStatus 'completed' row read as if cancellation / termination *transition the task to* completed. Reframed to make the status-vs-outcome split explicit: the status is completed; the outcome (success / failure / cancellation / termination) is surfaced through .run()/.result() per §4. * §3 inline code comment for .run() was missing the 'or with a task_id whose previous run already completed' qualifier that the real _decorator.py:474-475 docstring carries; surrounding prose did not mention the completed-collision case either. Both updated to match the real lifecycle. * §6 Pattern B note 'the metadata write is lifecycle-snapshotted' used a term-of-art without forward reference; clarified that the framework snapshots at lifecycle boundaries (defined in §4). Verification ------------ * tests/durable/test_dev_guide_review.py: 7/7 pass * tests/durable/test_public_api_surface.py: 12/12 pass * tests/durable/test_contract_completeness.py: 11/11 pass * full tests/durable/ suite: 292 pass / 0 fail / 1 unrelated warning Branch: feature/agentserver-durable-tasks Spec: 015-core-task-primitive-and-sample-fixups (Phase 11 follow-up) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [agentserver] core: drop non-recovery-safe per-call kwargs from Task.run/start Removes session_id, retry, and stream_handler from the public signatures of Task.run() and Task.start(). All three were footguns at the crash boundary: - session_id is platform-derived from FOUNDRY_AGENT_SESSION_ID; handler code has no business overriding it. - retry (RetryPolicy) and stream_handler are needed when crash recovery re-invokes the task on a fresh process. The manager's _recover_stale_tasks only sees the decorator-level opts (self._resume_opts.get(fn_name)), so any per-call override would silently disappear at recovery — the recovered lifetime would use the decorator config, not the per-call config. Removing them from the public surface forces the durable-safe shape: configure on @task (or Task.options) so the value survives across lifetimes. Also renames TaskContext.session_id to _session_id. It was framework-internal plumbing not intended for handler consumption; no external readers exist. Internal _lifecycle_start keeps the session_id/retry/stream_handler params because the manager threads platform-derived values through that boundary. Test changes (no semantic regressions): - tests/durable/test_stream_handler.py: 6 tests migrated from per-call stream_handler= to @task(stream_handler_factory=lambda _tid: handler). test_call_site_handler_overrides_factory deleted — it specifically validated the override behavior being removed. - tests/durable/test_retry.py: 5 tests migrated from per-call retry= to @task(retry=...). Verified: - core: 291/291 durable tests pass - responses: 1300/1301 non-live pass (one timing flake on test_shutdown_durable_background_not_marked_failed; passes in isolation) - responses durability_contract suite: 37/37 pass - invocations: 245/245 non-live pass CHANGELOG entry added under existing 2.0.0b4 section. Dev guide section 5 already updated in prior commit to reflect the trimmed signatures. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [agentserver] core: drop title/tags/stale_timeout from Task.run/start; add input_id parity on .run() Round 2 of the post-spec-015 API simplification on Task.run / Task.start per developer feedback on docs/durable-task-guide.md. * Drop `title`, `tags`, `stale_timeout` from `Task.run()` and `Task.start()`. All three are recovery-safe characteristics of the task identity and must survive crash/recovery; per-call overrides would silently disappear when recovery re-invokes the task on a fresh process. `title`/`tags` are configured on `@task(title=..., tags=...)` (both accept callable factories); `stale_timeout` is now a decorator option on `@task(stale_timeout=...)` and `Task.options(stale_timeout=...)`. Default unchanged at 300 seconds. * Add `input_id` and `if_last_input_id` to `Task.run()` (parity with `Task.start()`). Allows fire-and-await callers to use the optimistic- concurrency precondition without a separate .start() + .result() pair. * `TaskOptions` gains a `stale_timeout: float = 300.0` slot; propagated through `_lifecycle_start` from `self._opts`. * Doc updates in `durable-task-guide.md`: §5 `Task` signature trimmed to only the legitimately per-call kwargs; `@task` reference table enumerates every decorator option including the new `stale_timeout`; new §7 subsection 'Stale-task recovery and `stale_timeout`' explains what it does, what it interacts with (background lease-reclaim is separate), and when to override. * Delete the two legacy redirect docs (`durable-task-overview.md` and `durable-task-developer-guide.md`); update all in-tree references (README, contract-completeness error message, responses-orchestrator comment) to point at `durable-task-guide.md`. * Migrate 6 durable test files off the dropped per-call kwargs: promote to `@task(stale_timeout=...)` where the value is constant; use `Task.options(stale_timeout=...)` when a single test needs two variants on the same handler. Two tests covering the per-call `tags=` merge / reserved-stripping surface are deleted — the decorator-level coverage for both behaviours remains. Tests: - tests/durable/: 289 passed (down from 291; 2 tests covered a now- removed surface). - tests/durable/test_dev_guide_review.py + test_contract_completeness.py: 27 passed. - responses tests/e2e/durability_contract/: 37 passed. - invocations tests/: 245 passed. Pre-release packages (2.0.0b4); CHANGELOG appended in place. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [agentserver] core: drop migration framing from changelog/dev-guide; expand steering coverage Per end-user-developer feedback that this is a pre-release primitive on feature/agentserver-durable-tasks with no shipped predecessor: - CHANGELOG.md: delete the 'Breaking Changes' section for 2.0.0b4 entirely. Replace with a 'Public API shape (durable-task primitive)' section that describes the current contract as the initial-release surface, not a delta against any prior shape. Drop the rename-map reference from the consolidated-guide bullet. Strike the now-removed '_steering["previous_input"]' from the privacy-clearing bug-fix. - docs/durable-task-guide.md: delete the 'Rename map (for users migrating from earlier pre-release builds)' appendix entirely. Fix the @task(retry=...) kwarg name (was incorrectly retry_policy=...). - docs/durable-task-guide.md: drastically expand multi-turn and steering coverage per the same feedback that steering was 'almost fully glossed over': * §4 'Suspend, resume, and multi-turn workflows' rewritten with the approval_flow example, persistence-boundary discussion, and an explicit pointer to Pattern C for end-to-end chat loops. * §4 'Steering' rewritten as a full subsection covering: what .start() does on an in-flight steerable task (queue + cancel + ack TaskRun + 'superseded' for displaced caller); what the handler must do (check ctx.cancel.is_set(), suspend with partial output); was_steered / pending_inputs / steering_generation semantics; the rapid-fire short-circuit drain shape; how multi-turn + steering compose orthogonally. * §4 'Results and runs' updated: TaskResult.status is now correctly documented as the THREE-value Literal['completed', 'suspended', 'superseded'], distinct from the four-state TaskStatus literal. Documents is_completed / is_suspended / is_superseded properties and suspension_reason. §5 reference table for TaskResult / TaskRun / TaskStatus rewritten to spell out the same distinction and to document TaskRun's full surface (status, metadata, cancel, terminate, lease_expiry_count, async-for streaming). * Pattern C upgraded from a 6-line snippet into a full multi-turn chat pattern with metadata-based history, an explicit caller-side snippet showing turn 1 + turn 2 on the same task_id, and a bulletted explanation of why it works. * NEW Pattern F 'Steerable chat: preempting in-flight turns' showing the canonical ctx.cancel.is_set() short-circuit shape, the fast-suspend on was_steered+pending_inputs, and the caller-side is_superseded observation. Pairs with the new §4 steering text. Confirms via tests/durable/test_dev_guide_review.py (7/7 pass) and full tests/durable/ suite (289/289 pass). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [agentserver] docs: correct TaskResult(status='superseded') output semantics The previous wording 'TaskResult(status="superseded", output=None)' overstated the contract. Per _manager.py:1465-1469 the displaced generation's result_future is resolved with 'TaskResult(output=partial_output, status="superseded")' where partial_output is: - None when the displaced handler ended via ctx.suspend() (called from line 1158, no partial_output=... passed — the cooperative cancel path). - The handler's return value when the handler completed normally before steering drained (called from lines 1212/1242 with partial_output=result — the rare 'returned just as steering arrived' path). Updates the three doc sites that asserted output=None unconditionally: the §4 'Steering' subsection, the §4 'Results and runs' bullet, and the Pattern F caller-side snippet. is_superseded remains the correct property to branch on. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [agentserver] docs: clarify cooperative-cancellation handler strategies for steering ctx.cancel is advisory — the handler decides what to do about it. The previous wording implied steering forces an interrupt; that's wrong. The handler picks one of three legitimate strategies: - A: yield immediately on ctx.cancel.is_set() (lowest latency to the new turn; throws away in-flight partial work) - B: wind down to a safe checkpoint, then suspend (one-checkpoint latency cost; keeps invariants clean for work with side effects) - C: ignore cancel and run to completion (the queued input runs second, sequentially; required when the current input must be atomic — financial transaction, multi-step tool sequence) Per the actual code at _manager.py:1155-1223, the displaced caller's TaskResult.output depends on which strategy the handler chose: - Strategies A and B both call ctx.suspend(...) → _try_drain_steering is called WITHOUT partial_output= → defaults to None → the suspend envelope's output=X is silently dropped on the floor when there's a queued steering input to drain instead. - Strategy C calls return value → _try_drain_steering is called with partial_output=result → the displaced caller receives the full return value with status='superseded'. Adds a new '#### Cooperative cancellation: the handler is in charge' subsection between '#### What .start() does' and the worked example, with a strategy table. Renames the example subsection to '#### Example: cooperative suspend (strategy A)'. Updates the §4 Results-and-runs bullet for status='superseded' to spell out the same A/B vs C split. Updates the Pattern F caller-side comment to flag that r1.output=None is a consequence of the handler's strategy choice, not an inherent property of supersede. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [agentserver] spec 016: expand scope to also fix steering surface Steering is plain multi-turn with a queueing mechanism on top. The framework cannot observe whether a handler honored ctx.cancel — all three handler strategies (yield immediately / wind-down-then-suspend / ignore) look identical at the framework boundary. A 'superseded' status pretends the framework knows something it doesn't, and dropping handler-emitted output on the floor (current behavior in the suspend+queued case) leaks an implementation race into the public surface. Add to spec 016 (still on feature/agentserver-durable-tasks, no new branch): - Context: 'Scope expansion: steering is plain multi-turn' subsection explaining the model alignment. - User Story 7 (P1): steering surface collapse, with 6 acceptance scenarios sweeping handler-end × steerer-timing. - Edge Cases: handler-terminates-with-queued-input (cleanup in one store write, all queued steerers get TaskConflictError); suspend-persist-fails-with-queue; multiple-steerers FIFO. - FR-020: remove 'superseded' from TaskResult.status Literal + remove is_superseded property. - FR-021: suspend path delivers handler output unconditionally; drain runs strictly AFTER suspend resolution. - FR-022: return/raise paths are terminal; queued inputs become unservicable and resolve as TaskConflictError with the appropriate current_status. Same store write persists the terminal transition and clears pending_inputs. - FR-023: delete _pending_steering_futures; the steerer's future is the next generation's result future (no parallel queue). - FR-024: _try_drain_steering is re-entry only; no partial_output parameter; never resolves caller-visible futures. - SC-010 / SC-011: surface-search outcomes and a parametrized multi-turn-vs-steering equivalence sweep. - Principle XII checklist updated with the 5 new affected symbols. - 'What goes where' table updated for the steering rewrite + the TaskConflictError note. No code changes in this commit. Implementation lands after the spec is approved. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [agentserver] spec 016: add task API azure.core pipeline migration Third scope expansion in spec 016. HostedTaskProvider currently uses raw httpx.AsyncClient with no policy stack — no retry, no logging, no distributed tracing, no request-id, no shared bearer-token policy, no user-agent. The sibling FoundryStorageProvider already runs on azure.core.AsyncPipelineClient and carries the gzip-corruption lesson (ContentDecodePolicy excluded). Bring the task client onto the same transport and inherit the same lesson. Also: this migration is a *prerequisite* for FR-013 / FR-014 / FR-016 / FR-017 (the eviction classifier and local-cleanup sequence) — there's no clean seam in raw httpx for that work, and implementing it without the pipeline would require re-inventing azure.core's policy machinery. Additions to spec 016: - Context: 'Scope expansion: task API transport on azure.core pipeline' subsection. Calls out (a) policy chain rationale, (b) NO ContentDecodePolicy (gzip lesson), (c) replacement of raise_for_status with _classify_store_write_error funnel, (d) httpx dependency removal. - User Story 8 (P2): 7 acceptance scenarios — policy chain composition, 503 retry behavior, 409 no-retry, gzip round-trip, non-JSON body classified error, distributed-tracing presence, httpx import removal. - Edge cases: AsyncRetryPolicy 409 override (never retry on 409 regardless of body); bearer-token refresh on 401 during retry; test-fixture migration from httpx to azure.core transport fakes. - FR-025: AsyncPipelineClient replaces httpx.AsyncClient; per-request _get_headers removed in favor of AsyncBearerTokenCredentialPolicy. - FR-026: explicit policy chain composition (RequestId, Headers, UserAgent, AsyncRetry with allow-list, AsyncBearerToken, task-API logging, DistributedTracing) — ContentDecodePolicy excluded. - FR-027: task-API logging policy modeled on FoundryStorageLoggingPolicy with explicit header allow-list; no Authorization or body logs above DEBUG. - FR-028: every call site funnels through _classify_store_write_error (FR-013); 404 → None / TaskNotFound branches preserved via permanent+404 mapping; non-JSON 409 classified as 'conflict' not 'evicted' (binding_mismatch requires parseable body). - FR-029: call-site body parsing wraps UnicodeDecodeError, JSONDecodeError, DecodeError — carry-forward of responses-storage gzip-fix invariant. - FR-030: httpx import removal verification + pyproject.toml dep removal once nothing else imports it (test-only dev-dep OK). - SC-012: surface-search assertions for import removal and policy chain presence. - SC-013: behavioral assertions on fake transport — retry on 503, no-retry on 409, headers populated, gzip round-trip, non-JSON body classified error. - 'What goes where' table: 2 new internal-only rows (pipeline = internal; ContentDecodePolicy ban = inline comment + spec). - Principle XII checklist: HostedTaskProvider.__init__ credential re-typed to AsyncTokenCredential; bodies rewritten but public method signatures unchanged. No code changes in this commit. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [agentserver] spec 016: codify metadata auto-flush invariant across all turn boundaries Doc claim verification: the developer guide promises 'writes you forget to explicitly flush are still durable across a graceful boundary' (durable-task-guide.md §4) and 'lifecycle-boundary snapshots (framework-side)' (§5). Verified against _manager.py: Boundary Code site Status ---------------------------------- ---------------- ------ Normal suspend :1179 OK Normal completion :1227 OK Cooperative cancel / terminate :1268 OK Unhandled exception (exhausted) :1333 OK Suspend WITH queued steering input :1156-1175 GAP Return WITH queued steering input :1197-1223 GAP The two steering-drain shortcuts 'continue' out of the loop without running flush_all() AND without persisting the suspend/terminal record. _try_drain_steering preserves the metadata buffer by sharing the ctx.metadata reference into the new generation's context (:1492), but if the process crashes between the drain and the next generation's eventual terminal boundary, every unflushed write from the displaced turn is lost (the persisted record is still the pre- turn snapshot from the prior boundary). This aligns naturally with FR-021 / FR-022 (steering-as-multi-turn rewrite): the rewrite already requires the suspend / complete path to resolve the current turn's future FIRST, before the drain may re-enter. Tightening the same FRs to also require flush_all() at that boundary closes the gap as a side effect. Spec 016 edits: - Context: scope-expansion #2 paragraph notes the verified flush gap with explicit line-number citations. - FR-021: now mandates a precise 3-step ordering at the suspend boundary — (1) flush_all, (2) persist suspend record, (3) resolve result_future — with explicit pointer to _manager.py:1179 as the reference path that already works correctly. Calls out :1156-1175 as the path that MUST be replaced. - FR-022: now mandates a precise 4-step ordering at the completion/raise boundary, with the same flush_all-first invariant and explicit pointer to _manager.py:1227. Calls out :1197-1223 as the path that MUST be replaced. - FR-024: clarifies that _try_drain_steering MUST NOT touch ctx.metadata in any way — flush ownership belongs strictly to the FR-021/FR-022 boundary; the new generation reads metadata from the just-persisted record. - FR-024a (NEW): codifies the metadata auto-flush invariant as load-bearing, lists all six boundaries that must satisfy it (4 already correct + 2 to be added by FR-021/FR-022), and ties the doc-guide claim to a testable assertion. - SC-014 (NEW): parametrized 8-cell crash-and-reload sweep that asserts the marker written without explicit flush survives every boundary. Current code passes 4/8; after the rework lands all 8 must pass. - 'What goes where' table: steering row updated to call out the FR-024a flush invariant alongside the multi-turn framing. - Principle XII checklist: explicitly mentions the lifecycle-flush invariant extension and the silent-metadata-loss behavior shift. No code changes in this commit. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [agentserver] spec 016: cancel signal carries reason; per-turn timeout budget Three coupled fixes added to spec 016 (no code changes): 1. ctx.cancel becomes a CancelSignal (small public wrapper) with a .reason Literal['timeout', 'steering'] | None. Existing handler patterns (is_set(), wait()) keep working — purely additive surface. First-reason-wins on multiple .set() calls. 2. The timeout watchdog is respawned per LOGICAL TURN. Today it's spawned in _execute_task outside the steering-drain loop, so a steerable @task(timeout=timedelta(seconds=30)) whose first generation runs 25s before suspending and triggering a drain has only 5s left before the watchdog fires on generation 2. Per the 'steering is plain multi-turn' framing (FR-020..FR-024a), each drain-driven generation IS a new turn and should get a fresh budget. Retries within the same generation continue to share the watchdog (intentional — retries don't get extra time). 3. The _timeout_watchdog docstring at _manager.py:1018-1021 claims the lease will eventually expire if the handler ignores cancel. That claim is false — renewal is driven by a separate event the watchdog does not touch. Per user direction we keep the watchdog cooperative-only and fix the docstring instead of changing behavior. If the handler ignores cancel, the task runs until process death or explicit terminate(). Additions to spec 016: - Context: 'Scope expansion: cancel-signal carries a reason; per-turn timeout budget' subsection explaining all three concerns with line-number citations. - US9 (P2) with 7 acceptance scenarios covering reason visibility under steering, reason visibility under timeout, fresh budget on fresh-turn re-entry, fresh budget on drain re-entry, accurate watchdog docstring, first-reason-wins, backward-compat. - 4 new edge cases: watchdog respawn race (cancel previous before spawn); respawn-vs-retry distinction; ctx.cancel.set() ownership (framework-only, not handler); event-loop-shutdown cleanup. - FR-031: CancelSignal class definition, backward-compat API, first-reason-wins semantics. - FR-032: per-turn watchdog respawn — fresh handler invocation OR drain re-entry, NOT retry iteration; at most one live watchdog at a time; cleanup in finally block. - FR-033: set reason at the source — _timeout_watchdog passes reason='timeout'; _try_drain_steering constructs the new ctx's CancelSignal already set with reason='steering' when cancel_requested. - FR-034: rewrite the misleading _timeout_watchdog docstring; log-level stays INFO (timeout is a developer-handler concern, not a framework alarm); dev guide updated to match. - SC-015: per-turn timeout budget test (regression guard for the shared-budget bug). - SC-016: cancel reason visibility test (timeout / steering / timeout-races-steering deterministic). - 'What goes where' table: 3 new rows for CancelSignal + per-turn budget + cooperative-only watchdog. - Key Entities: CancelSignal added. - Principle XII checklist: CancelSignal added to __all__; TaskContext.cancel type change documented; HostedTaskProvider credential re-typing already documented from Iter 10. No code changes in this commit. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [agentserver] spec 016: durable wall-clock per-turn timeout; recovery preserves budget User direction: timeout should be per-turn, wall-clock, durable across crashes within the turn. Recovery does NOT reset the budget. New turns (fresh .run/.start, resume after suspend, drain re-entry) DO get a fresh budget. Today's behavior is per-invocation, in-process asyncio.sleep, with no durable anchor: a process that crashes 25s into a 30s budget recovers with a clean 30s. The store has created_at / updated_at / started_at fields but the watchdog never consults any of them. Refinements to spec 016 (no code changes): - Context: scope-expansion #3 paragraph adds a third bullet explicitly calling out the non-durability gap, then states the desired semantic: per-turn, wall-clock, durable across crashes within the turn. Recovery formula spelled out as remaining = max(0, opts.timeout - (now - _turn_started_at)). - US9: 3 new acceptance scenarios for the durable-recovery case (scenarios 8-10): mid-turn crash within budget (recovered with small remaining), mid-turn crash past budget (immediate cancel), suspended-idle-time-between-turns does not count. - 3 new edge cases: recovery-between-turns is a new turn (re-stamp); legacy records without _turn_started_at fall back to full budget + DEBUG log; clock-skew bounds via clamping. - FR-032 (revised): per-turn watchdog respawn now uses the durable remaining-budget formula with explicit case analysis for fresh / resume / drain / recovery / retry paths. Crash-recovery sub-case added with the 'fire immediately if remaining == 0' rule. - FR-033 (revised): zero-budget crash-recovery case must pre-set the CancelSignal with reason='timeout' so the recovered handler sees the reason from the first checkpoint. - FR-034: docstring replacement now mentions the corrected timeout contract ('per-turn, wall-clock, durable across crashes within the turn') so dev guide and source align. - FR-035 (NEW): the durable _turn_started_at payload field — set sites (create_and_start, _start_existing_task on suspended->in_progress transition, _try_drain_steering's CAS write), explicit MUST NOT re-stamp on recovery, read site in watchdog spawn, clock-skew clamping to [0, opts.timeout], rollout fallback for legacy records. - SC-015 (revised): 4 parametrized scenarios — fresh-turn, drain- re-entry, mid-turn-recovery-within-budget, mid-turn-recovery-past- budget. Test reads payload['_turn_started_at'] directly for assertions. - SC-017 (NEW): clock-skew bounds — backwards and forwards skew both clamped. - 'What goes where' table: per-turn-budget row rewritten with the durable-wall-clock semantic. - Key Entities: _turn_started_at added. - Principle XII checklist: enumerates all five new symbols and spells out the sharpened public contract of @task(timeout=...). No code changes in this commit. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [agentserver] spec 016: refresh Docs Loop section for full scope The Authoritative-guides, Authoring-sequence, and Loop-completion- criterion subsections were written when spec 016 covered only the stale_timeout removal. They now reflect all 6 scope expansions: - Authoritative guides: enumerates every public-surface contract that durable-task-guide.md must accurately reflect — recovery, TaskConflictError semantics, TaskResult.status narrowing, steering- as-plain-multi-turn, CancelSignal.reason, per-turn-durable-timeout, metadata-flush-invariant. Adds CHANGELOG.md, test_dev_guide_review.py, and the five source-docstring sites as additional authoritative surfaces. - Authoring sequence: 5-step ordered authoring (guide → meta-test invariants → CHANGELOG → source docstrings → tests + code) with explicit per-guide-section edit list. Names the new invariants the meta-test must gain (zero matches for stale_timeout, superseded, is_superseded, _pending_steering_futures, 'lease will eventually expire'; presence of CancelSignal in §4 + §5; presence of 'per-turn' / 'wall-clock' / 'durable' in timeout description). - Loop completion criterion: 7 concrete checks — six developer-FAQ questions the guide must answer correctly + the meta-test + package-surface grep for absent symbols + presence of new public symbols + CHANGELOG shape + tests free of removed symbol references + source-docstring vs guide agreement. No FR changes; no SC changes; no code changes. This is purely sharpening the doc-loop checkpoints so the implementation PR has a concrete done-criterion for documentation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [agentserver] spec 016: close 4 constitutional review gaps Constitutional review surfaced 4 gaps; all 4 closed in this commit. No FR changes; no SC changes; no code changes — this is cross-doc commitment work that the spec was deferring with TBD language. Gap 1 (Principle IX — Samples Loop): the spec named the dev guide and meta-test but never enumerated which samples are touched by the contract changes. Added 'Samples affected' subsection enumerating each surface change × each affected sample with required actions: - FR-020 (no superseded): zero framework-level usage in samples; the 'superseded' string in durable_copilot/durable_langgraph/ durable_research is an APPLICATION-level invocation_store status string, not the framework's TaskResult.status Literal. Verified via 'grep -rn TaskResult\|is_superseded\|.status ==' samples/. No required change; dev guide must still state the Literal is 'completed' | 'suspended' only. - FR-001 (no stale_timeout): zero matches in samples. No change. - FR-031 (CancelSignal.reason): backward-compat. Recommended (not required) one worked example in durable_copilot. - FR-024a (metadata-flush invariant): samples currently use explicit flush(); continues to work; auto-flush is additional guarantee not a replacement. No change. - FR-032..FR-035 (per-turn durable timeout): zero samples set timeout=. Recommended one worked example. - Transport migration (FR-025..FR-030): internal-only. - FR-022 (TaskConflictError shape): recommended README updates in durable_copilot and durable_langgraph. Loop completion criterion added: durability-sample-checklist runs against any modified sample; deferrals recorded in tasks.md. Gap 2 (Principle X — Contract change decision): the existing checklist said 'Decision needed: add a Recovery subsection? Recommend YES.' Locked in YES. The amendment is a CROSS-CUTTING NOTE (not a new matrix row) covering binding_mismatch protocol and the lease-state × steerable callsite outcome table. Lands in the same PR per durability-contract.md § Change control. Gap 3 (Principle X + XI — FR-024a boundary): FR-024a is a durability claim but SC-014 lives in tests/durable/, not in tests/e2e/durability_contract/. Added explicit justification: the contract matrix is the response-stream surface; FR-024a's concern is the core-primitive metadata namespace, one layer below. Therefore FR-024a is in Principle XII scope (core-primitive), NOT Principle X scope (response-stream durability matrix). The matrix does NOT need a new row; the dev guide's 'What the framework persists at lifecycle boundaries' table is the relevant doc surface. SC-014's home in tests/durable/ is correct. Gap 4 (Principle XII §5 — test-only hooks tension): FR-018 (_force_reclaim_orphans) and FR-019 (_PERIODIC_RECLAIM_INTERVAL_SECONDS monkeypatch) appear to be _-prefixed APIs that the principle forbids 'to bypass public-surface contract enforcement'. Added per-FR reconciliation paragraphs: these hooks make TIMING deterministic for tests of contract BEHAVIOR; they do not bypass any contract. Tests using them MUST assert on public-surface contract outcomes (caller- visible TaskRun, ctx.entry_mode, store record state), never on internal ReclaimOutcome variants returned by _reclaim_one. Final scorecard after this commit: all 12 principles verified compliant. Spec is ready for /speckit.plan. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [agentserver] spec 016: start cancel-surface proposal doc for iteration User flagged on reflection that FR-031 (CancelSignal with .reason) may duplicate signals already on TaskContext (ctx.shutdown, ctx.pending_inputs, ctx.was_steered). Pausing FR-031 in spec 016 until we iterate on a separate proposal that: - Audits every public TaskContext field and what it signals today. - Enumerates all 12 cancellation/cancel-adjacent scenarios a handler can face (steady state, steering pressure, timeout, shutdown, terminate, crash-recovery, multi-turn resume, steering re-entry, multiple-queued, hybrid steering+timeout, shutdown+steering, shutdown+t…
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.