Skip to content

Commit 7983938

Browse files
rimeydhermes
rimey
authored and
dhermes
committed
Monitoring API (#1691)
* Add gcloud.monitoring with fetching of metric descriptors. * Add fetching of resource descriptors. * Add querying of time series. * Add unit tests for everything but time series. * Added some links to the documentation, and a comment. * Dropped two nested generators. * Require start time and end time to be datetimes rather than strings. * Change how query intervals are specified. 1. Removed start_time from the query constructor/factory. 2. Added a select_interval() method for when you actually do have a start time in your hands, or for when you want to set the time interval on an existing query object. * Change how one filters on the resource type. Several related changes: - Removed resource_type from the constructor/factory parameters. - Added support for filtering on "resource_type" as a pseudo-label. - Renamed select_resource_labels() to select_resources() and select_metric_labels() to select_metrics(). * Add some pseudo-enums providing constants you can use if you wish. * More unit tests, and a few related changes. The signature of the reduce() method is now as follows: reduce(self, cross_series_reducer, *group_by_fields) * Added a metric_type property. I had dropped this, but I think we want it. * The descriptor classes are no longer named tuples. MetricDescriptor ResourceDescriptor LabelDescriptor * Move Query into a new query.py file. * Isolated the pandas-dependent code in _dataframe.py. * Add unit tests for _build_dataframe(). * Make the pseudo-enums importable from gcloud.monitoring. * Skip tests of _build_dataframe() if pandas is not available. * Fix lint errors. * Add "NO COVER" pragmas due to not having pandas in the testenv:cover environment. * Add a py27-pandas tox test environment. * as_matrix() -> values * Utilize the pseudo-enums in the docstrings and add a few more usage examples. * Add some basic usage documentation. * Fix some remaining references to the pseudo-enums in a docstring. * Add prompts (">>>") to examples showing interactive usage. * Add basic system tests. * Expect that the service will never return an empty name, type, or key. * Use print() in docstring examples. * Add pandas to the intersphinx configuration. * Alphabetical order. * Change TOP_RESOURCE_LABELS from list to tuple. * Rephrase a complex if-statement. * p -> point * Use DataFrame.from_records() to build from a list of rows. * Add a comment about the system test getting no time series data. * Use Query.DEFAULT_METRIC_TYPE cross-reference in doc strings. * Add some docstring text about NotFound exceptions. * Rename "filter" parameter to "filter_" in private methods. * Move status comments into module docstrings. * Use :data: for docstring cross references to pseudo-enums. * Change some empty lists to empty tuples. * _NOW -> _UTCNOW * Remove docstring text explaining about named tuples. * Reformat a docstring example. * _page_size -> page_size * Introduce an _iter_fragments() helper method. * Make _build_query_params() a generator. * Use six.iteritems(). * Reorganize test data around DIMENSIONS instead of N and M. * Restyle an import. * Unpack points more elegantly. * Remove pylint comment disabling redefined-builtin. (It's disabled project-wide.) * Cache the label map without having to disable pylint. * Excercise the test harness fully by reading past the available responses. * Change some more empty lists into empty tuples. * filter/filter_ -> filter_string * '%Y-%m-%dT%H:%M:%S.%fZ' -> _RFC3339_MICROS * Use parentheses instead of backslashes. * Reorganize a hard-to-understand bit of code for building multi-level column headers. * type -> type_
1 parent 18c92fe commit 7983938

31 files changed

+4307
-1
lines changed

‎docs/.DS_Store‎

6 KB
Binary file not shown.

‎docs/conf.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -292,5 +292,6 @@
292292
intersphinx_mapping = {
293293
'httplib2': ('http://bitworking.org/projects/httplib2/doc/html/', None),
294294
'oauth2client': ('http://oauth2client.readthedocs.org/en/latest/', None),
295+
'pandas': ('http://pandas.pydata.org/pandas-docs/stable/', None),
295296
'python': ('https://docs.python.org/', None),
296297
}

‎docs/index.rst‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -119,6 +119,19 @@
119119
logging-metric
120120
logging-sink
121121

122+
.. toctree::
123+
:maxdepth: 0
124+
:hidden:
125+
:caption: Cloud Monitoring
126+
127+
monitoring-usage
128+
Client <monitoring-client>
129+
monitoring-metric
130+
monitoring-resource
131+
monitoring-query
132+
monitoring-timeseries
133+
monitoring-label
134+
122135
.. toctree::
123136
:maxdepth: 0
124137
:hidden:

‎docs/monitoring-client.rst‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
Monitoring Client
2+
=================
3+
4+
.. automodule:: gcloud.monitoring.client
5+
:members:
6+
:undoc-members:
7+
:show-inheritance:
8+
9+
Connection
10+
~~~~~~~~~~
11+
12+
.. automodule:: gcloud.monitoring.connection
13+
:members:
14+
:undoc-members:
15+
:show-inheritance:
16+

‎docs/monitoring-label.rst‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
Label Descriptors
2+
=================
3+
4+
.. automodule:: gcloud.monitoring.label
5+
:members:
6+
:undoc-members:
7+
:show-inheritance:

‎docs/monitoring-metric.rst‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
Metric Descriptors
2+
==================
3+
4+
.. automodule:: gcloud.monitoring.metric
5+
:members:
6+
:undoc-members:
7+
:show-inheritance:

‎docs/monitoring-query.rst‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
Time Series Query
2+
=================
3+
4+
.. automodule:: gcloud.monitoring.query
5+
:members:
6+
:undoc-members:
7+
:show-inheritance:

‎docs/monitoring-resource.rst‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
Resource Descriptors
2+
====================
3+
4+
.. automodule:: gcloud.monitoring.resource
5+
:members:
6+
:undoc-members:
7+
:show-inheritance:

‎docs/monitoring-timeseries.rst‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
Time Series
2+
===========
3+
4+
.. automodule:: gcloud.monitoring.timeseries
5+
:members:
6+
:undoc-members:
7+
:show-inheritance:

‎docs/monitoring-usage.rst‎

Lines changed: 155 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,155 @@
1+
Using the API
2+
=============
3+
4+
5+
Introduction
6+
------------
7+
8+
With the Monitoring API, you can work with Stackdriver metric data
9+
pertaining to monitored resources in Google Cloud Platform (GCP)
10+
or elsewhere.
11+
12+
Essential concepts:
13+
14+
- Metric data is associated with a **monitored resource**. A monitored
15+
resource has a *resource type* and a set of *resource labels* —
16+
key-value pairs — that identify the particular resource.
17+
- A **metric** further identifies the particular kind of data that
18+
is being collected. It has a *metric type* and a set of *metric
19+
labels* that, when combined with the resource labels, identify
20+
a particular time series.
21+
- A **time series** is a collection of data points associated with
22+
points or intervals in time.
23+
24+
Please refer to the documentation for the `Monitoring API`_ for
25+
more information.
26+
27+
At present, this client library supports querying of time series,
28+
metric descriptors, and resource descriptors.
29+
30+
.. _Monitoring API: https://cloud.google.com/monitoring/api/
31+
32+
33+
The Monitoring Client Object
34+
----------------------------
35+
36+
The monitoring client library generally makes its
37+
functionality available as methods of the monitoring
38+
:class:`~gcloud.monitoring.client.Client` class.
39+
A :class:`~gcloud.monitoring.client.Client` instance holds
40+
authentication credentials and the ID of the target project with
41+
which the metric data of interest is associated. This project ID
42+
will often refer to a `Stackdriver account`_ binding multiple
43+
GCP projects and AWS accounts. It can also simply be the ID of
44+
a monitored project.
45+
46+
Most often the authentication credentials will be determined
47+
implicitly from your environment. See :doc:`gcloud-auth` for
48+
more information.
49+
50+
It is thus typical to create a client object as follows::
51+
52+
>>> from gcloud import monitoring
53+
>>> client = monitoring.Client(project='target-project')
54+
55+
If you are running in Google Compute Engine or Google App Engine,
56+
the current project is the default target project. This default
57+
can be further overridden with the :envvar:`GCLOUD_PROJECT`
58+
environment variable. Using the default target project is
59+
even easier::
60+
61+
>>> client = monitoring.Client()
62+
63+
If necessary, you can pass in ``credentials`` and ``project`` explicitly::
64+
65+
>>> client = monitoring.Client(project='target-project', credentials=...)
66+
67+
.. _Stackdriver account: https://cloud.google.com/monitoring/accounts/
68+
69+
70+
Monitored Resource Descriptors
71+
------------------------------
72+
73+
The available monitored resource types are defined by *monitored resource
74+
descriptors*. You can fetch a list of these with the
75+
:meth:`~gcloud.monitoring.client.Client.list_resource_descriptors` method::
76+
77+
>>> for descriptor in client.list_resource_descriptors():
78+
... print(descriptor.type)
79+
80+
Each :class:`~gcloud.monitoring.resource.ResourceDescriptor`
81+
has a type, a display name, a description, and a list of
82+
:class:`~gcloud.monitoring.label.LabelDescriptor` instances.
83+
See the documentation about `Monitored Resources`_
84+
for more information.
85+
86+
.. _Monitored Resources:
87+
https://cloud.google.com/monitoring/api/v3/monitored-resources
88+
89+
90+
Metric Descriptors
91+
------------------
92+
93+
The available metric types are defined by *metric descriptors*.
94+
They include `platform metrics`_, `agent metrics`_, and `custom metrics`_.
95+
You can list all of these with the
96+
:meth:`~gcloud.monitoring.client.Client.list_metric_descriptors` method::
97+
98+
>>> for descriptor in client.list_metric_descriptors():
99+
... print(descriptor.type)
100+
101+
See :class:`~gcloud.monitoring.metric.MetricDescriptor` and the
102+
`Metric Descriptors`_ API documentation for more information.
103+
104+
.. _platform metrics: https://cloud.google.com/monitoring/api/metrics
105+
.. _agent metrics: https://cloud.google.com/monitoring/agent/
106+
.. _custom metrics: https://cloud.google.com/monitoring/custom-metrics/
107+
.. _Metric Descriptors:
108+
https://cloud.google.com/monitoring/api/ref_v3/rest/v3/\
109+
projects.metricDescriptors
110+
111+
112+
Time Series Queries
113+
-------------------
114+
115+
A time series includes a collection of data points and a set of
116+
resource and metric label values.
117+
See :class:`~gcloud.monitoring.timeseries.TimeSeries` and the
118+
`Time Series`_ API documentation for more information.
119+
120+
While you can obtain time series objects by iterating over a
121+
:class:`~gcloud.monitoring.query.Query` object, usually it is
122+
more useful to retrieve time series data in the form of a
123+
:class:`pandas.DataFrame`, where each column corresponds to a
124+
single time series. For this, you must have :mod:`pandas` installed;
125+
it is not a required dependency of ``gcloud-python``.
126+
127+
You can display CPU utilization across your GCE instances during
128+
the last five minutes as follows::
129+
130+
>>> METRIC = 'compute.googleapis.com/instance/cpu/utilization'
131+
>>> query = client.query(METRIC, minutes=5)
132+
>>> print(query.as_dataframe())
133+
134+
:class:`~gcloud.monitoring.query.Query` objects provide a variety of
135+
methods for refining the query. You can request temporal alignment
136+
and cross-series reduction, and you can filter by label values.
137+
See the client :meth:`~gcloud.monitoring.client.Client.query` method
138+
and the :class:`~gcloud.monitoring.query.Query` class for more
139+
information.
140+
141+
For example, you can display CPU utilization during the last hour
142+
across GCE instances with names beginning with ``"mycluster-"``,
143+
averaged over five-minute intervals and aggregated per zone, as
144+
follows::
145+
146+
>>> from gcloud.monitoring import Aligner, Reducer
147+
>>> METRIC = 'compute.googleapis.com/instance/cpu/utilization'
148+
>>> query = (client.query(METRIC, hours=1)
149+
... .select_metrics(instance_name_prefix='mycluster-')
150+
... .align(Aligner.ALIGN_MEAN, minutes=5)
151+
... .reduce(Reducer.REDUCE_MEAN, 'resource.zone'))
152+
>>> print(query.as_dataframe())
153+
154+
.. _Time Series:
155+
https://cloud.google.com/monitoring/api/ref_v3/rest/v3/TimeSeries

0 commit comments

Comments
 (0)