Skip to content

Commit 4fbb41a

Browse files
ruoning-ngmichael-hossDominik Dienlin
authored
Build and test with Python 3.12 (#1166)
* WIP upgrade to pymotmetrics 1.4.0 * replace MOTAccumulatorCustom by 1.4.0 implementation * add pred_frequencies, as this is now required * fix printing of pred_frequencies * fix ana in signature * leave notes for weird pandas issue. to be debugged further * tidy up and fix bug of messing up an empty MultiIndex * fix requirements specification for compatibility with python3.12 * fix issue with setting title via matplotlib when using newer versions of matplotlib * Support CI in GitHub Actions * Modifications to support Python 3.12 * Make pipeline extensible to support multiple Python versions * Update Python versions supported * Remove Jenkinsfile --------- Co-authored-by: Michael Hoss <michael.hoss@rwth-aachen.de> Co-authored-by: Dominik Dienlin <dominik.dienlin@mobis.com>
1 parent a1e2178 commit 4fbb41a

File tree

24 files changed

+539
-240
lines changed

24 files changed

+539
-240
lines changed
Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,62 @@
1+
name: 'Run Python Tests'
2+
description: 'Run Python Tests'
3+
inputs:
4+
python-version:
5+
description: 'Python version'
6+
required: true
7+
runs:
8+
using: "composite"
9+
steps:
10+
- name: Display Python version
11+
shell: bash
12+
env:
13+
PYTHON_VERSION: ${{ inputs.python-version }}
14+
run: echo "Running Python tests with version ${PYTHON_VERSION}"
15+
- name: Set up Python
16+
uses: actions/setup-python@v3
17+
with:
18+
python-version: ${{ inputs.python-version }}
19+
- name: Install datasets
20+
shell: bash
21+
run: |
22+
mkdir -p ${NUSCENES} && mkdir -p ${NUIMAGES}
23+
24+
echo "Installing: v1.0-mini.tgz"
25+
curl -fsSL https://motional-nuscenes.s3-ap-northeast-1.amazonaws.com/public/v1.0/v1.0-mini.tgz | tar -xzf - -C ${NUSCENES} --exclude sweeps
26+
27+
echo "Installing: nuimages-v1.0-mini.tgz"
28+
curl -fsSL https://motional-nuscenes.s3-ap-northeast-1.amazonaws.com/public/nuimages-v1.0/nuimages-v1.0-mini.tgz | tar -xzf - -C ${NUIMAGES}
29+
30+
echo "Installing: nuScenes-lidarseg-mini-v1.0.tar.bz2"
31+
curl -fsSL https://motional-nuscenes.s3-ap-northeast-1.amazonaws.com/public/nuscenes-lidarseg-v1.0/nuScenes-lidarseg-mini-v1.0.tar.bz2 | tar -xjf - -C ${NUSCENES}
32+
33+
echo "Installing: nuScenes-panoptic-v1.0-mini.tar.gz"
34+
curl -fsSL https://motional-nuscenes.s3-ap-northeast-1.amazonaws.com/public/nuscenes-panoptic-v1.0/nuScenes-panoptic-v1.0-mini.tar.gz | tar -xzf - --strip-components=1 -C ${NUSCENES}
35+
36+
echo "Installing: nuScenes-map-expansion-v1.3.zip"
37+
curl -fsSL https://motional-nuscenes.s3-ap-northeast-1.amazonaws.com/public/v1.0/nuScenes-map-expansion-v1.3.zip -o nuScenes-map-expansion-v1.3.zip
38+
unzip -q nuScenes-map-expansion-v1.3.zip -d ${NUSCENES}/maps/
39+
40+
echo "Installing: can_bus.zip"
41+
curl -fsSL https://motional-nuscenes.s3-ap-northeast-1.amazonaws.com/public/v1.0/can_bus.zip -o can_bus.zip
42+
unzip -q can_bus.zip -d ${NUSCENES} can_bus/scene-0001_*
43+
44+
echo "Removing zip files . . ."
45+
rm nuScenes-map-expansion-v1.3.zip can_bus.zip
46+
- name: Install dependencies
47+
shell: bash
48+
env:
49+
PYTHON_VERSION: ${{ inputs.python-version }}
50+
run: |
51+
PYTHON_VERSION_UNDERSCORE=${PYTHON_VERSION//./_}
52+
pip install -r setup/requirements_${PYTHON_VERSION_UNDERSCORE}_lock.txt
53+
- name: Run Python unit tests
54+
shell: bash
55+
run: |
56+
python -m unittest discover python-sdk
57+
- name: Run Jupyter notebook tests
58+
shell: bash
59+
run: |
60+
pip install jupyter -q
61+
export PYTHONPATH="${PYTHONPATH}:$(pwd)/python-sdk"
62+
./setup/test_tutorial.sh --ci

.github/workflows/pipeline.yml

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
name: nuscenes-devkit CI pipeline
2+
on: [pull_request]
3+
env:
4+
NUSCENES: data/sets/nuscenes
5+
NUIMAGES: data/sets/nuimages
6+
jobs:
7+
test-in-3-9:
8+
runs-on: ubuntu-latest
9+
steps:
10+
- name: Check out repository code
11+
uses: actions/checkout@v4
12+
- id: run-test-in-3-9
13+
uses: ./.github/actions/run-python-tests
14+
with:
15+
python-version: 3.9
16+
test-in-3-12:
17+
runs-on: ubuntu-latest
18+
steps:
19+
- name: Check out repository code
20+
uses: actions/checkout@v4
21+
- id: run-test-in-3-12
22+
uses: ./.github/actions/run-python-tests
23+
with:
24+
python-version: 3.12

python-sdk/nuscenes/eval/tracking/constants.py

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@
1717
MOT_METRIC_MAP = { # Mapping from motmetrics names to metric names used here.
1818
'num_frames': '', # Used in FAF.
1919
'num_objects': 'gt', # Used in MOTAR computation.
20+
'pred_frequencies': '', # Only needed in background.
2021
'num_predictions': '', # Only printed out.
2122
'num_matches': 'tp', # Used in MOTAR computation and printed out.
2223
'motar': 'motar', # Only used in AMOTA.

python-sdk/nuscenes/eval/tracking/evaluate.py

Lines changed: 9 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -5,10 +5,9 @@
55
import json
66
import os
77
import time
8-
from typing import Tuple, List, Dict, Any
8+
from typing import Any, Dict, List, Tuple
99

1010
import numpy as np
11-
1211
from nuscenes import NuScenes
1312
from nuscenes.eval.common.config import config_factory
1413
from nuscenes.eval.common.loaders import (
@@ -21,9 +20,14 @@
2120
load_prediction_of_sample_tokens,
2221
)
2322
from nuscenes.eval.tracking.algo import TrackingEvaluation
24-
from nuscenes.eval.tracking.constants import AVG_METRIC_MAP, MOT_METRIC_MAP, LEGACY_METRICS
25-
from nuscenes.eval.tracking.data_classes import TrackingMetrics, TrackingMetricDataList, TrackingConfig, TrackingBox, \
26-
TrackingMetricData
23+
from nuscenes.eval.tracking.constants import AVG_METRIC_MAP, LEGACY_METRICS, MOT_METRIC_MAP
24+
from nuscenes.eval.tracking.data_classes import (
25+
TrackingBox,
26+
TrackingConfig,
27+
TrackingMetricData,
28+
TrackingMetricDataList,
29+
TrackingMetrics,
30+
)
2731
from nuscenes.eval.tracking.loaders import create_tracks
2832
from nuscenes.eval.tracking.render import recall_metric_curve, summary_plot
2933
from nuscenes.eval.tracking.utils import print_final_metrics

python-sdk/nuscenes/eval/tracking/metrics.py

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@
77
py-motmetrics at:
88
https://github.com/cheind/py-motmetrics
99
"""
10-
from typing import Any
10+
from typing import Any, Optional
1111

1212
import numpy as np
1313

@@ -109,7 +109,7 @@ def longest_gap_duration(df: DataFrame, obj_frequencies: DataFrame) -> float:
109109

110110

111111
def motar(df: DataFrame, num_matches: int, num_misses: int, num_switches: int, num_false_positives: int,
112-
num_objects: int, alpha: float = 1.0) -> float:
112+
num_objects: int, alpha: float = 1.0, ana: Optional[dict] = None) -> float:
113113
"""
114114
Initializes a MOTAR class which refers to the modified MOTA metric at https://www.nuscenes.org/tracking.
115115
Note that we use the measured recall, which is not identical to the hypothetical recall of the
@@ -121,6 +121,7 @@ def motar(df: DataFrame, num_matches: int, num_misses: int, num_switches: int, n
121121
:param num_false_positives: The number of false positives.
122122
:param num_objects: The total number of objects of this class in the GT.
123123
:param alpha: MOTAR weighting factor (previously 0.2).
124+
:param ana: something for caching, introduced by motmetrics 1.4.0
124125
:return: The MOTAR or nan if there are no GT objects.
125126
"""
126127
recall = num_matches / num_objects

python-sdk/nuscenes/eval/tracking/mot.py

Lines changed: 53 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -6,50 +6,74 @@
66
77
py-motmetrics at:
88
https://github.com/cheind/py-motmetrics
9+
10+
Notes by Michael Hoss:
11+
For Python 3.10, we need to update the version of py-motmetrics to 1.4.0.
12+
Then, to keep this code working, we need to change back the types of OId HId to object because they are
13+
strings in nuscenes-devkit, whereas motmetrics changed these types to float from 1.1.3 to 1.4.0.
914
"""
1015
from collections import OrderedDict
1116
from itertools import count
1217

13-
import motmetrics
1418
import numpy as np
1519
import pandas as pd
20+
from motmetrics import MOTAccumulator
1621

22+
_INDEX_FIELDS = ['FrameId', 'Event']
1723

18-
class MOTAccumulatorCustom(motmetrics.mot.MOTAccumulator):
24+
class MOTAccumulatorCustom(MOTAccumulator):
25+
"""This custom class was created by nuscenes-devkit to use a faster implementation of
26+
`new_event_dataframe_with_data` under compatibility with motmetrics<=1.1.3.
27+
Now that we use motmetrics==1.4.0, we need to use this custom implementation to use
28+
objects instead of strings for OId and HId.
29+
"""
1930
def __init__(self):
2031
super().__init__()
2132

2233
@staticmethod
2334
def new_event_dataframe_with_data(indices, events):
24-
"""
25-
Create a new DataFrame filled with data.
26-
This version overwrites the original in MOTAccumulator achieves about 2x speedups.
35+
"""Create a new DataFrame filled with data.
2736
2837
Params
2938
------
30-
indices: list
31-
list of tuples (frameid, eventid)
32-
events: list
33-
list of events where each event is a list containing
34-
'Type', 'OId', HId', 'D'
39+
indices: dict
40+
dict of lists with fields 'FrameId' and 'Event'
41+
events: dict
42+
dict of lists with fields 'Type', 'OId', 'HId', 'D'
3543
"""
36-
idx = pd.MultiIndex.from_tuples(indices, names=['FrameId', 'Event'])
37-
df = pd.DataFrame(events, index=idx, columns=['Type', 'OId', 'HId', 'D'])
44+
45+
if len(events) == 0:
46+
return MOTAccumulatorCustom.new_event_dataframe()
47+
48+
raw_type = pd.Categorical(
49+
events['Type'],
50+
categories=['RAW', 'FP', 'MISS', 'SWITCH', 'MATCH', 'TRANSFER', 'ASCEND', 'MIGRATE'],
51+
ordered=False)
52+
series = [
53+
pd.Series(raw_type, name='Type'),
54+
pd.Series(events['OId'], dtype=object, name='OId'), # OId is string in nuscenes-devkit
55+
pd.Series(events['HId'], dtype=object, name='HId'), # HId is string in nuscenes-devkit
56+
pd.Series(events['D'], dtype=float, name='D')
57+
]
58+
59+
idx = pd.MultiIndex.from_arrays(
60+
[indices[field] for field in _INDEX_FIELDS],
61+
names=_INDEX_FIELDS)
62+
df = pd.concat(series, axis=1)
63+
df.index = idx
3864
return df
3965

4066
@staticmethod
4167
def new_event_dataframe():
42-
""" Create a new DataFrame for event tracking. """
68+
"""Create a new DataFrame for event tracking."""
4369
idx = pd.MultiIndex(levels=[[], []], codes=[[], []], names=['FrameId', 'Event'])
44-
cats = pd.Categorical([], categories=['RAW', 'FP', 'MISS', 'SWITCH', 'MATCH'])
70+
cats = pd.Categorical([], categories=['RAW', 'FP', 'MISS', 'SWITCH', 'MATCH', 'TRANSFER', 'ASCEND', 'MIGRATE'])
4571
df = pd.DataFrame(
4672
OrderedDict([
47-
('Type', pd.Series(cats)), # Type of event. One of FP (false positive), MISS, SWITCH, MATCH
48-
('OId', pd.Series(dtype=object)),
49-
# Object ID or -1 if FP. Using float as missing values will be converted to NaN anyways.
50-
('HId', pd.Series(dtype=object)),
51-
# Hypothesis ID or NaN if MISS. Using float as missing values will be converted to NaN anyways.
52-
('D', pd.Series(dtype=float)), # Distance or NaN when FP or MISS
73+
('Type', pd.Series(cats)), # Type of event. One of FP (false positive), MISS, SWITCH, MATCH
74+
('OId', pd.Series(dtype=object)), # Object ID or -1 if FP. Using float as missing values will be converted to NaN anyways.
75+
('HId', pd.Series(dtype=object)), # Hypothesis ID or NaN if MISS. Using float as missing values will be converted to NaN anyways.
76+
('D', pd.Series(dtype=float)), # Distance or NaN when FP or MISS
5377
]),
5478
index=idx
5579
)
@@ -63,8 +87,7 @@ def events(self):
6387
return self.cached_events_df
6488

6589
@staticmethod
66-
def merge_event_dataframes(dfs, update_frame_indices=True, update_oids=True, update_hids=True,
67-
return_mappings=False):
90+
def merge_event_dataframes(dfs, update_frame_indices=True, update_oids=True, update_hids=True, return_mappings=False):
6891
"""Merge dataframes.
6992
7093
Params
@@ -104,24 +127,29 @@ def merge_event_dataframes(dfs, update_frame_indices=True, update_oids=True, upd
104127

105128
# Update index
106129
if update_frame_indices:
107-
next_frame_id = max(r.index.get_level_values(0).max() + 1,
108-
r.index.get_level_values(0).unique().shape[0])
130+
# pylint: disable=cell-var-from-loop
131+
next_frame_id = max(r.index.get_level_values(0).max() + 1, r.index.get_level_values(0).unique().shape[0])
109132
if np.isnan(next_frame_id):
110133
next_frame_id = 0
111-
copy.index = copy.index.map(lambda x: (x[0] + next_frame_id, x[1]))
134+
if not copy.index.empty:
135+
copy.index = copy.index.map(lambda x: (x[0] + next_frame_id, x[1]))
112136
infos['frame_offset'] = next_frame_id
113137

114138
# Update object / hypothesis ids
115139
if update_oids:
140+
# pylint: disable=cell-var-from-loop
116141
oid_map = dict([oid, str(next(new_oid))] for oid in copy['OId'].dropna().unique())
117142
copy['OId'] = copy['OId'].map(lambda x: oid_map[x], na_action='ignore')
118143
infos['oid_map'] = oid_map
119144

120145
if update_hids:
146+
# pylint: disable=cell-var-from-loop
121147
hid_map = dict([hid, str(next(new_hid))] for hid in copy['HId'].dropna().unique())
122148
copy['HId'] = copy['HId'].map(lambda x: hid_map[x], na_action='ignore')
123149
infos['hid_map'] = hid_map
124150

151+
# Avoid pandas warning. But is this legit/do we need such a column later on again?
152+
# copy = copy.dropna(axis=1, how='all')
125153
r = pd.concat((r, copy))
126154
mapping_infos.append(infos)
127155

python-sdk/nuscenes/eval/tracking/utils.py

Lines changed: 12 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@
33

44
import unittest
55
import warnings
6-
from typing import Optional, Dict
6+
from typing import Dict, Optional
77

88
import numpy as np
99

@@ -14,8 +14,15 @@
1414
raise unittest.SkipTest('Skipping test as motmetrics was not found!')
1515

1616
from nuscenes.eval.tracking.data_classes import TrackingMetrics
17-
from nuscenes.eval.tracking.metrics import motar, mota_custom, motp_custom, faf, track_initialization_duration, \
18-
longest_gap_duration, num_fragmentations_custom
17+
from nuscenes.eval.tracking.metrics import (
18+
faf,
19+
longest_gap_duration,
20+
mota_custom,
21+
motar,
22+
motp_custom,
23+
num_fragmentations_custom,
24+
track_initialization_duration,
25+
)
1926

2027

2128
def category_to_tracking_name(category_name: str) -> Optional[str]:
@@ -148,8 +155,8 @@ def create_motmetrics() -> MetricsHost:
148155
# Register standard metrics.
149156
fields = [
150157
'num_frames', 'obj_frequencies', 'num_matches', 'num_switches', 'num_false_positives', 'num_misses',
151-
'num_detections', 'num_objects', 'num_predictions', 'mostly_tracked', 'mostly_lost', 'num_fragmentations',
152-
'motp', 'mota', 'precision', 'recall', 'track_ratios'
158+
'num_detections', 'num_objects', 'pred_frequencies', 'num_predictions', 'mostly_tracked', 'mostly_lost',
159+
'num_fragmentations', 'motp', 'mota', 'precision', 'recall', 'track_ratios'
153160
]
154161
for field in fields:
155162
mh.register(getattr(motmetrics.metrics, field), formatter='{:d}'.format)

python-sdk/nuscenes/map_expansion/map_api.py

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -29,7 +29,7 @@
2929
from nuscenes.utils.geometry_utils import view_points
3030

3131
# Recommended style to use as the plots will show grids.
32-
plt.style.use('seaborn-whitegrid')
32+
plt.style.use('seaborn-v0_8-whitegrid')
3333

3434
# Define a map geometry type for polygons and lines.
3535
Geometry = Union[Polygon, LineString]
@@ -1820,8 +1820,8 @@ def mask_for_polygons(polygons: MultiPolygon, mask: np.ndarray) -> np.ndarray:
18201820
def int_coords(x):
18211821
# function to round and convert to int
18221822
return np.array(x).round().astype(np.int32)
1823-
exteriors = [int_coords(poly.exterior.coords) for poly in polygons]
1824-
interiors = [int_coords(pi.coords) for poly in polygons for pi in poly.interiors]
1823+
exteriors = [int_coords(poly.exterior.coords) for poly in polygons.geoms]
1824+
interiors = [int_coords(pi.coords) for poly in polygons.geoms for pi in poly.interiors]
18251825
cv2.fillPoly(mask, exteriors, 1)
18261826
cv2.fillPoly(mask, interiors, 0)
18271827
return mask
@@ -1885,7 +1885,7 @@ def _polygon_geom_to_mask(self,
18851885
[1.0, 0.0, 0.0, 1.0, trans_x, trans_y])
18861886
new_polygon = affinity.scale(new_polygon, xfact=scale_width, yfact=scale_height, origin=(0, 0))
18871887

1888-
if new_polygon.geom_type is 'Polygon':
1888+
if new_polygon.geom_type == 'Polygon':
18891889
new_polygon = MultiPolygon([new_polygon])
18901890
map_mask = self.mask_for_polygons(new_polygon, map_mask)
18911891

@@ -1922,7 +1922,7 @@ def _line_geom_to_mask(self,
19221922

19231923
map_mask = np.zeros(canvas_size, np.uint8)
19241924

1925-
if layer_name is 'traffic_light':
1925+
if layer_name == 'traffic_light':
19261926
return None
19271927

19281928
for line in layer_geom:
@@ -1968,7 +1968,7 @@ def _get_layer_polygon(self,
19681968
origin=(patch_x, patch_y), use_radians=False)
19691969
new_polygon = affinity.affine_transform(new_polygon,
19701970
[1.0, 0.0, 0.0, 1.0, -patch_x, -patch_y])
1971-
if new_polygon.geom_type is 'Polygon':
1971+
if new_polygon.geom_type == 'Polygon':
19721972
new_polygon = MultiPolygon([new_polygon])
19731973
polygon_list.append(new_polygon)
19741974

@@ -1983,7 +1983,7 @@ def _get_layer_polygon(self,
19831983
origin=(patch_x, patch_y), use_radians=False)
19841984
new_polygon = affinity.affine_transform(new_polygon,
19851985
[1.0, 0.0, 0.0, 1.0, -patch_x, -patch_y])
1986-
if new_polygon.geom_type is 'Polygon':
1986+
if new_polygon.geom_type == 'Polygon':
19871987
new_polygon = MultiPolygon([new_polygon])
19881988
polygon_list.append(new_polygon)
19891989

@@ -2003,7 +2003,7 @@ def _get_layer_line(self,
20032003
if layer_name not in self.map_api.non_geometric_line_layers:
20042004
raise ValueError("{} is not a line layer".format(layer_name))
20052005

2006-
if layer_name is 'traffic_light':
2006+
if layer_name == 'traffic_light':
20072007
return None
20082008

20092009
patch_x = patch_box[0]

0 commit comments

Comments
 (0)