From 13919fbbbb3f3e0be8316ffd49d1e48dcdf81942 Mon Sep 17 00:00:00 2001 From: Vinay Sajip Date: Mon, 31 Oct 2022 19:19:15 +0000 Subject: [PATCH 1/3] Update cookbook example for socket-based logging in a production setting. --- Doc/howto/logging-cookbook.rst | 72 ++++++++++++++++++++++++++++++---- 1 file changed, 65 insertions(+), 7 deletions(-) diff --git a/Doc/howto/logging-cookbook.rst b/Doc/howto/logging-cookbook.rst index ae101e3cdf2501c..7a868dd3baf7c10 100644 --- a/Doc/howto/logging-cookbook.rst +++ b/Doc/howto/logging-cookbook.rst @@ -765,13 +765,71 @@ serialization. Running a logging socket listener in production ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -To run a logging listener in production, you may need to use a process-management tool -such as `Supervisor `_. `Here -`_ is a Gist which -provides the bare-bones files to run the above functionality using Supervisor: you -will need to change the ``/path/to/`` parts in the Gist to reflect the actual paths you -want to use. - +To run a logging listener in production, you may need to use a +process-management tool such as `Supervisor `_. `Here +`_ is a Gist +which provides the bare-bones files to run the above functionality using +Supervisor. It consists of the following files: + ++-------------------------+----------------------------------------------------+ +| File | Purpose | ++=========================+====================================================+ +| :file:`prepare.sh` | A Bash script to prepare the environment for | +| | testing | ++-------------------------+----------------------------------------------------+ +| :file:`supervisor.conf` | The Supervisor configuration file, which has | +| | entries for the listener and a multi-process web | +| | application | ++-------------------------+----------------------------------------------------+ +| :file:`ensure_app.sh` | A Bash script to ensure that Supervisor is running | +| | with the above configuration | ++-------------------------+----------------------------------------------------+ +| :file:`log_listener.py` | The socket listener program which receives log | +| | events and logs them to a file | ++-------------------------+----------------------------------------------------+ +| :file:`main.py` | A simple web application which does logging via a | +| | socket connected to the listener | ++-------------------------+----------------------------------------------------+ +| :file:`webapp.json` | A JSON configuration file for the web application | ++-------------------------+----------------------------------------------------+ +| :file:`client.py` | A Python script to exercise the web application | ++-------------------------+----------------------------------------------------+ + +The web application uses `Gunicorn `_, which is a +popular web application server which starts multiple worker processes to handle +requests. The whole setup shows how the workers can write to the same log file +without treading on each other's toes --- they go through the socket listener. + +To test these files, do the following in a POSIX environment: + +1. Download `the Gist + `_, using + the "Download ZIP" button, as a :file:`zip` archive. + +2. Unzip the above files from the archive into a scratch directory. + +3. In the scratch directory, run :program:`bash prepare.sh` to get things ready. + This creates a :file:`run` subdirectory to contain Supervisor related and + log files, and a :file:`venv` subdirectory to contain a virtual environment + into which ``bottle``, ``gunicorn`` and ``supervisor`` are installed. + +4. Run :program:`bash ensure_app.bash` to ensure that Supervisor is running with + the above configuration. + +5. Run :program:`venv/bin/python client.py` to exercise the web application, + which will lead to records being written to the log. + +6. Inspect the log files in the :file:`run` subdirectory. You should see the + most recent log lines in files matching :file:`app.log*`. They won't be in + any particular order, since they have been handled concurrently by different + worker processes, and that does not happend in a deterministic way. + +7. You can shut down the listener and the web application by running + :program:`venv/bin/supervisorctl -c supervisor.conf shutdown`. + +You may need to tweak the configuration files in the unlikely event that the +ports configured in them cause a clash with something else in your test +environment. .. _context-info: From d06b59fe19dc0667ff0c3d4e83b96cad20a11f33 Mon Sep 17 00:00:00 2001 From: Vinay Sajip Date: Tue, 1 Nov 2022 08:40:21 +0000 Subject: [PATCH 2/3] Apply suggestions from code review Co-authored-by: C.A.M. Gerlach --- Doc/howto/logging-cookbook.rst | 45 +++++++++++++++++----------------- 1 file changed, 22 insertions(+), 23 deletions(-) diff --git a/Doc/howto/logging-cookbook.rst b/Doc/howto/logging-cookbook.rst index 7a868dd3baf7c10..c203f2ce6b6cb0e 100644 --- a/Doc/howto/logging-cookbook.rst +++ b/Doc/howto/logging-cookbook.rst @@ -766,8 +766,8 @@ Running a logging socket listener in production ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ To run a logging listener in production, you may need to use a -process-management tool such as `Supervisor `_. `Here -`_ is a Gist +process-management tool such as `Supervisor `_. +`Here is a Gist `__ which provides the bare-bones files to run the above functionality using Supervisor. It consists of the following files: @@ -785,10 +785,10 @@ Supervisor. It consists of the following files: | | with the above configuration | +-------------------------+----------------------------------------------------+ | :file:`log_listener.py` | The socket listener program which receives log | -| | events and logs them to a file | +| | events and records them to a file | +-------------------------+----------------------------------------------------+ -| :file:`main.py` | A simple web application which does logging via a | -| | socket connected to the listener | +| :file:`main.py` | A simple web application which performs logging | +| | via a socket connected to the listener | +-------------------------+----------------------------------------------------+ | :file:`webapp.json` | A JSON configuration file for the web application | +-------------------------+----------------------------------------------------+ @@ -796,40 +796,39 @@ Supervisor. It consists of the following files: +-------------------------+----------------------------------------------------+ The web application uses `Gunicorn `_, which is a -popular web application server which starts multiple worker processes to handle -requests. The whole setup shows how the workers can write to the same log file -without treading on each other's toes --- they go through the socket listener. +popular web application server that starts multiple worker processes to handle +requests. This example setup shows how the workers can write to the same log file +without conflicting with one another --- they all go through the socket listener. To test these files, do the following in a POSIX environment: -1. Download `the Gist - `_, using - the "Download ZIP" button, as a :file:`zip` archive. +#. Download `the Gist + `_, + as a ZIP archive using the :guilabel:`Download ZIP` button. -2. Unzip the above files from the archive into a scratch directory. +#. Unzip the above files from the archive into a scratch directory. -3. In the scratch directory, run :program:`bash prepare.sh` to get things ready. - This creates a :file:`run` subdirectory to contain Supervisor related and +#. In the scratch directory, run ``bash prepare.sh`` to get things ready. + This creates a :file:`run` subdirectory to contain Supervisor-related and log files, and a :file:`venv` subdirectory to contain a virtual environment into which ``bottle``, ``gunicorn`` and ``supervisor`` are installed. -4. Run :program:`bash ensure_app.bash` to ensure that Supervisor is running with +#. Run ``bash ensure_app.sh`` to ensure that Supervisor is running with the above configuration. -5. Run :program:`venv/bin/python client.py` to exercise the web application, +#. Run ``venv/bin/python client.py`` to exercise the web application, which will lead to records being written to the log. -6. Inspect the log files in the :file:`run` subdirectory. You should see the - most recent log lines in files matching :file:`app.log*`. They won't be in +#. Inspect the log files in the :file:`run` subdirectory. You should see the + most recent log lines in files matching the pattern :file:`app.log*`. They won't be in any particular order, since they have been handled concurrently by different - worker processes, and that does not happend in a deterministic way. + worker processes in a non-deterministic way. -7. You can shut down the listener and the web application by running - :program:`venv/bin/supervisorctl -c supervisor.conf shutdown`. +#. You can shut down the listener and the web application by running + ``venv/bin/supervisorctl -c supervisor.conf shutdown``. You may need to tweak the configuration files in the unlikely event that the -ports configured in them cause a clash with something else in your test -environment. +configured ports clash with something else in your test environment. .. _context-info: From 10e846dd43d13795a74a3417c71e11288569e4e4 Mon Sep 17 00:00:00 2001 From: Vinay Sajip Date: Tue, 1 Nov 2022 22:49:05 +0000 Subject: [PATCH 3/3] Apply more suggestions from code review Co-authored-by: C.A.M. Gerlach --- Doc/howto/logging-cookbook.rst | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/Doc/howto/logging-cookbook.rst b/Doc/howto/logging-cookbook.rst index c203f2ce6b6cb0e..bf6f54a841a7b9d 100644 --- a/Doc/howto/logging-cookbook.rst +++ b/Doc/howto/logging-cookbook.rst @@ -765,9 +765,11 @@ serialization. Running a logging socket listener in production ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +.. _socket-listener-gist: https://gist.github.com/vsajip/4b227eeec43817465ca835ca66f75e2b + To run a logging listener in production, you may need to use a process-management tool such as `Supervisor `_. -`Here is a Gist `__ +`Here is a Gist `__ which provides the bare-bones files to run the above functionality using Supervisor. It consists of the following files: @@ -802,8 +804,7 @@ without conflicting with one another --- they all go through the socket listener To test these files, do the following in a POSIX environment: -#. Download `the Gist - `_, +#. Download `the Gist `__ as a ZIP archive using the :guilabel:`Download ZIP` button. #. Unzip the above files from the archive into a scratch directory.