Update IPS-portal docs

doc/user_guides/portal_guides.rst

@@ -19,7 +19,44 @@ To use the portal include
 
 .. code-block:: text
 
+   USE_PORTAL = True
    PORTAL_URL = http://lb.ipsportal.production.svc.spin.nersc.org
 
+
+The source code for the portal can be found one `GitHub
+<https://github.com/HPC-SimTools/IPS-portal>`_ and issues can be
+reported using `GitHub issues
+<https://github.com/HPC-SimTools/IPS-portal/issues>`_.
+
 in either your :doc:`Platform Configuration File<platform>` or your
 :doc:`Simulation Configuration File<config_file>`.
+
+
+Tracing
+-------
+
+IPS (version >= 0.6.0) has the ability to capture a trace of the
+workflow to allow analysis and visualizations. The traces are captured
+in the `Zipkin Span format <https://zipkin.io/zipkin-api/>`_ and
+viewed within IPS portal using `Jaeger
+<https://www.jaegertracing.io/>`_.
+
+After selecting a run in the portal there will be a link to the trace:
+
+.. image:: run_list.png
+
+.. image:: trace_link.png
+
+The default view is the Trace Timeline but other useful views are
+Trace Graph and Trace Statistic which can be selected from the menu in
+the top-right:
+
+.. image:: jaeger_options.png
+
+The statistics can be further broken down by operation.
+
+.. image:: statistics_sub_group.png
+
+.. note::
+
+   Self time (ST) is the total time spent in a span when it was not waiting on children. For example, a 10ms span with two 4ms non-overlapping children would have self-time = 10ms - 2 * 4ms = 2ms.