Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 9 additions & 22 deletions .github/workflows/appimage.yml
Original file line number Diff line number Diff line change
Expand Up @@ -31,15 +31,13 @@ jobs:
uses: actions/checkout@v3
with:
submodules: recursive
path: viam-cartographer

- name: Check out PR branch code
if: github.event_name == 'pull_request_target'
uses: actions/checkout@v3
with:
ref: ${{ github.event.pull_request.head.sha }}
submodules: recursive
path: viam-cartographer

- name: apt update
run: |
Expand All @@ -48,31 +46,20 @@ jobs:
- name: make clean
run: |
chown -R testbot:testbot .
sudo -u testbot bash -lc 'cd viam-cartographer && make clean'
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[q] Is removing this a benefit of the (TOOL_BIN) additions made to the Makefile?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, it's actually due to removing the "path:" setting above. The normal paradigm is to checkout in the working directory. Only when checking out multiple repos side-by-side do you usually need explicit paths for them. This was just normalizing things to what we do (almost) everywhere else.

sudo -u testbot bash -lc 'make clean'

- name: make bufinstall buf setup
uses: nick-fields/retry@v2
with:
timeout_minutes: 5
max_attempts: 3
retry_on: timeout
shell: bash
command: |
chown -R testbot:testbot .
sudo -u testbot bash -lc 'cd viam-cartographer && make bufinstall buf setup'

- name: make build
- name: make setup
run: |
sudo -u testbot bash -lc 'cd viam-cartographer && make build'
sudo -u testbot bash -lc 'make setup'

- name: make install-lua-files
- name: make build
run: |
sudo -u testbot bash -lc 'cd viam-cartographer && make install-lua-files'
sudo -u testbot bash -lc 'make build'

- name: Build AppImage (PR)
if: contains(github.event.pull_request.labels.*.name, 'appimage') || contains(github.event.pull_request.labels.*.name, 'appimage-ignore-tests')
run: |
sudo -u testbot bash -lc 'cd viam-cartographer && make BUILD_CHANNEL="pr-${{ github.event.pull_request.number }}" appimage'
sudo -u testbot bash -lc 'make BUILD_CHANNEL="pr-${{ github.event.pull_request.number }}" appimage'

- name: Authorize GCP Upload
uses: google-github-actions/auth@v1
Expand All @@ -84,22 +71,22 @@ jobs:
uses: google-github-actions/upload-cloud-storage@v0.10.4
with:
headers: "cache-control: no-cache"
path: 'viam-cartographer/etc/packaging/appimages/deploy/'
path: 'etc/packaging/appimages/deploy/'
destination: 'packages.viam.com/apps/slam-servers/'
glob: '*'
parent: false
gzip: false

- name: Build AppImage (Latest)
if: github.event_name == 'workflow_dispatch' || github.event_name == 'push'
run: sudo -u testbot bash -lc 'cd viam-cartographer && make appimage-ci'
run: sudo -u testbot bash -lc 'make appimage-ci'

- name: Deploy Files (Latest)
if: github.event_name == 'workflow_dispatch' || github.event_name == 'push'
uses: google-github-actions/upload-cloud-storage@v0.10.4
with:
headers: "cache-control: no-cache"
path: 'viam-cartographer/etc/packaging/appimages/deploy/'
path: 'etc/packaging/appimages/deploy/'
destination: 'packages.viam.com/apps/slam-servers/'
glob: '*'
parent: false
Expand Down
34 changes: 11 additions & 23 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -57,59 +57,47 @@ jobs:

- name: apt update
run: |
sudo apt update
apt-get update

- name: Verify no uncommitted changes from make lint
run: |
git init
git add .
chown -R testbot:testbot .
sudo -u testbot bash -lc 'make lint-setup lint'
GEN_DIFF=$(git status -s)
sudo -u testbot bash -lc 'sudo apt-get install -y clang-format && make lint'
GEN_DIFF=$(git status -s --ignore-submodules)

if [ -n "$GEN_DIFF" ]; then
echo '"make lint" resulted in changes not in git' 1>&2
git status
exit 1
fi

- name: make bufinstall buf setup setup-cpp-full-mod
uses: nick-fields/retry@v2
with:
timeout_minutes: 5
max_attempts: 3
retry_on: timeout
shell: bash
command: |
chown -R testbot:testbot .
sudo -u testbot bash -lc 'make bufinstall buf setup setup-cpp-full-mod'
- name: make setup
run: |
sudo -u testbot bash -lc 'make setup'

- name: make build
run: |
sudo -u testbot bash -lc 'make build'

- name: make install-lua-files
- name: make install
run: |
sudo -u testbot bash -lc 'make install-lua-files'
sudo -u testbot bash -lc 'make install'

- name: make test
run: |
sudo -u testbot bash -lc 'make test'

- name: Copy carto_grpc_server binary
if: matrix.platform == 'linux/amd64'
run: |
sudo cp viam-cartographer/build/carto_grpc_server /usr/local/bin/carto_grpc_server

- name: Run viam-cartographer cartographer integration tests
if: matrix.platform == 'linux/amd64'
run: |
sudo -u testbot bash -lc 'sudo go test -v -race -run TestCartographerIntegration'
sudo -u testbot bash -lc 'go test -v -race -run TestCartographerIntegration'

- name: make test-cpp-full-mod-valgrind
- name: make setup-cpp-full-mod test-cpp-full-mod-valgrind
# Currently we only run valgrind on arm64 as x86
# doesn't valgrind clean due to what appears to be a bug in GLOG
# https://github.com/viamrobotics/viam-cartographer/actions/runs/5156238014/jobs/9287413446
if: matrix.platform != 'linux/amd64'
run: |
sudo -u testbot bash -lc 'make test-cpp-full-mod-valgrind'
sudo -u testbot bash -lc 'make setup-cpp-full-mod test-cpp-full-mod-valgrind'
127 changes: 65 additions & 62 deletions Makefile
Original file line number Diff line number Diff line change
@@ -1,35 +1,43 @@
BUILD_CHANNEL?=local
TOOL_BIN = bin/gotools/$(shell uname -s)-$(shell uname -m)
PATH_WITH_TOOLS="`pwd`/$(TOOL_BIN):$(HOME)/go/bin/:${PATH}"
GIT_REVISION = $(shell git rev-parse HEAD | tr -d '\n')
TAG_VERSION?=$(shell git tag --points-at | sort -Vr | head -n1)
GO_BUILD_LDFLAGS = -ldflags "-X 'main.Version=${TAG_VERSION}' -X 'main.GitRevision=${GIT_REVISION}'"
TOOL_BIN := $(shell pwd)/bin/tools/$(shell uname -s)-$(shell uname -m)
GIT_REVISION := $(shell git rev-parse HEAD | tr -d '\n')
TAG_VERSION ?= $(shell git tag --points-at | sort -Vr | head -n1)
GO_BUILD_LDFLAGS := -ldflags "-X 'main.Version=${TAG_VERSION}' -X 'main.GitRevision=${GIT_REVISION}'"
SHELL := /usr/bin/env bash
export PATH := $(TOOL_BIN):$(PATH)
export GOBIN := $(TOOL_BIN)

ifneq (, $(shell which brew))
EXTRA_CMAKE_FLAGS := -DCMAKE_PREFIX_PATH=$(shell brew --prefix) -DQt5_DIR=$(shell brew --prefix qt5)/lib/cmake/Qt5
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[note to self] from build scripts

export PKG_CONFIG_PATH := $(shell brew --prefix openssl@3)/lib/pkgconfig
endif

default: build

ARTIFACT="~/go/bin/artifact"
artifact-pull: $(TOOL_BIN)/artifact
artifact pull

artifact-pull:
PATH=${PATH_WITH_TOOLS} artifact pull
$(TOOL_BIN)/artifact:
go install go.viam.com/utils/artifact/cmd/artifact

bufinstall:
sudo apt-get install -y protobuf-compiler-grpc libgrpc-dev libgrpc++-dev || brew install grpc openssl --quiet
$(TOOL_BIN)/buf:
go install github.com/bufbuild/buf/cmd/buf@v1.8.0

bufsetup:
GOBIN=`pwd`/grpc/bin go install github.com/bufbuild/buf/cmd/buf@v1.8.0
ln -sf `which grpc_cpp_plugin` grpc/bin/protoc-gen-grpc-cpp
$(TOOL_BIN)/protoc-gen-grpc-cpp:
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[q] what is the benefit of moving to this more complex make target name?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Make targets are supposed to be real things. It's the whole reason it's called "make" in the first place, because it makes the thing. E.g. if I type make viam-server the expectation is that a file (and usually a program) called "viam-server" will be created. Because of this, dependencies work on this same principal. A target depends on other targets. If those targets exist and are newer than their dependencies, then they become no-ops. So by correctly making targets to create the "buf" binary, other things can depend on it. If "buf" isn't there, it'll be installed. If it is there, then it just continues on. Thus you can build really complex systems and when you run make, it only has to rebuild things that changed or things that haven't been built yet.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

gotcha, so thats why other repos were updated to use dist/buf and similar setups

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, though in that case, it was using a slightly different pattern where the "dist" folder contains touchfiles (just empty files with specific names) to basically flag when a job had been "done." You can then reset everything by just wiping those files out, causing a clean build to happen. Sometimes when you're linking in more complex tooling (that might have dozens of things it generates, like the buf generate command does) that's an easier way than setting up targets for EVERY output file. Especially if the output files often change or are variable in number.

mkdir -p "$(TOOL_BIN)"
which grpc_cpp_plugin && ln -sf `which grpc_cpp_plugin` $(TOOL_BIN)/protoc-gen-grpc-cpp

buf: bufsetup
PATH="${PATH}:`pwd`/grpc/bin" buf generate --template ./buf/buf.gen.yaml buf.build/viamrobotics/api
PATH="${PATH}:`pwd`/grpc/bin" buf generate --template ./buf/buf.grpc.gen.yaml buf.build/viamrobotics/api
PATH="${PATH}:`pwd`/grpc/bin" buf generate --template ./buf/buf.gen.yaml buf.build/googleapis/googleapis
buf: $(TOOL_BIN)/buf $(TOOL_BIN)/protoc-gen-grpc-cpp
buf generate --template ./buf/buf.gen.yaml buf.build/viamrobotics/api
buf generate --template ./buf/buf.grpc.gen.yaml buf.build/viamrobotics/api
buf generate --template ./buf/buf.gen.yaml buf.build/googleapis/googleapis

clean:
rm -rf grpc
rm -rf bin
rm -rf viam-cartographer/build
rm -rf viam-cartographer/cartographer/build
rm -rf grpc bin viam-cartographer/build

clean-all:
git clean -fxd
cd viam-cartographer/cartographer && git checkout . && git clean -fxd
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[note to self] resets CMakeList


ensure-submodule-initialized:
@if [ ! -d "viam-cartographer/cartographer/cartographer" ]; then \
Expand All @@ -38,21 +46,8 @@ ensure-submodule-initialized:
else \
echo "Submodule found successfully"; \
fi

lint-setup-cpp:
ifeq ("Darwin", "$(shell uname -s)")
brew install clang-format
else
sudo apt-get install -y clang-format
endif

lint-setup-go:
GOBIN=`pwd`/$(TOOL_BIN) go install \
github.com/edaniels/golinters/cmd/combined \
github.com/golangci/golangci-lint/cmd/golangci-lint \
github.com/rhysd/actionlint/cmd/actionlint

lint-setup: lint-setup-cpp lint-setup-go
grep -q viam-patched viam-cartographer/cartographer/CMakeLists.txt || \
(cd viam-cartographer/cartographer && git checkout . && git apply ../cartographer_patches/carto.patch)

lint-cpp:
find . -type f -not -path \
Expand All @@ -63,42 +58,46 @@ lint-cpp:
-and \( -iname '*.h' -o -iname '*.cpp' -o -iname '*.cc' \) \
| xargs clang-format -i --style="{BasedOnStyle: Google, IndentWidth: 4}"

lint-go:
lint-go: $(TOOL_BIN)/combined $(TOOL_BIN)/golangci-lint $(TOOL_BIN)/actionlint
go vet -vettool=$(TOOL_BIN)/combined ./...
GOGC=50 $(TOOL_BIN)/golangci-lint run -v --fix --config=./etc/golangci.yaml
PATH=$(PATH_WITH_TOOLS) actionlint
GOGC=50 golangci-lint run -v --fix --config=./etc/golangci.yaml
actionlint

$(TOOL_BIN)/combined $(TOOL_BIN)/golangci-lint $(TOOL_BIN)/actionlint:
go install \
github.com/edaniels/golinters/cmd/combined \
github.com/golangci/golangci-lint/cmd/golangci-lint \
github.com/rhysd/actionlint/cmd/actionlint

lint: ensure-submodule-initialized lint-cpp lint-go

setup: ensure-submodule-initialized
ifeq ("Darwin", "$(shell uname -s)")
cd viam-cartographer/scripts && ./setup_cartographer_macos.sh
setup: install-dependencies ensure-submodule-initialized artifact-pull

install-dependencies:
ifneq (, $(shell which brew))
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[note to self] from setup scripts

brew update
brew install abseil boost ceres-solver protobuf ninja cairo googletest lua@5.3 pkg-config cmake go@1.20 grpc clang-format
brew link lua@5.3
brew install openssl@3 eigen gflags glog suite-sparse sphinx-doc pcl nlopt-static
else ifneq (, $(shell which apt-get))
$(warning "Installing cartographer external dependencies via APT.")
$(warning "Packages may be too old to work with this project.")
sudo apt-get update
sudo apt-get install -y cmake ninja-build libgmock-dev libboost-iostreams-dev liblua5.3-dev libcairo2-dev python3-sphinx libnlopt-dev \
libabsl-dev libceres-dev libprotobuf-dev protobuf-compiler protobuf-compiler-grpc libpcl-dev libgrpc-dev libgrpc++-dev clang-format
else
cd viam-cartographer/scripts && ./setup_cartographer_linux.sh
endif
@make artifact-pull

build: build-module
ifneq ($(wildcard viam-cartographer/cartographer/build/.),)
cd viam-cartographer && ./scripts/build_viam_cartographer.sh
else
cd viam-cartographer && ./scripts/build_cartographer.sh && ./scripts/build_viam_cartographer.sh
$(error "Unsupported system. Only apt and brew currently supported.")
endif

build-debug: build-module
ifneq ($(wildcard viam-cartographer/cartographer/build/.),)
cd viam-cartographer && ./scripts/build_viam_cartographer_debug.sh
else
cd viam-cartographer && ./scripts/build_cartographer.sh && ./scripts/build_viam_cartographer_debug.sh
endif
build: ensure-submodule-initialized buf build-module
cd viam-cartographer && cmake -Bbuild -G Ninja ${EXTRA_CMAKE_FLAGS} && cmake --build build
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we dont have to run the ninja command anymore?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cmake --build build runs it. (It says run a build in the directory "build", if the double-naming is confusing.) But if you ever switch to another cmake tooling instead of ninja, it'll run that too.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[q] so if we wanted separate build directories for different architectures, we would modify cmake --build build here?

[nit] if so, could we add a comment here for this, as that's something the slam team wants to look into doing in the future?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, in theory you can just change -Bbuild to -Bbuild-`uname -m` (or whatever name scheme you want, and it'll configure the output there. Then you can actually compile with the same thing just changing build to build-`uname -m` in the final command as well.

More realistically, you'd just want to make them variables in Make, then you can set the BUILD_DIR variable once and use everywhere. Do note that some of the cartographer tests in go are written using cgo and depending on explicit paths (which is why I had to change one) so there's not as easy of a way to handle that.

Likewise, it's not just the cmake part that would need to be per-architecture, as the golang stuff would be as well, so it's likely a bit of a more complex change you really want, to completely split out architectures if you really want to be able to build both in the same code space (without just doing "make clean" instead.)


build-debug: EXTRA_CMAKE_FLAGS += -DCMAKE_BUILD_TYPE=Debug -DFORCE_DEBUG_BUILD=True
build-debug: build

build-module:
mkdir -p bin && go build $(GO_BUILD_LDFLAGS) -o bin/cartographer-module module/main.go

install-lua-files:
sudo mkdir -p /usr/local/share/cartographer/lua_files/
sudo cp viam-cartographer/lua_files/* /usr/local/share/cartographer/lua_files/

test-cpp:
viam-cartographer/build/unit_tests -p -l all

Expand All @@ -122,7 +121,11 @@ test-go:

test: test-cpp test-go

install:
install-lua-files:
sudo mkdir -p /usr/local/share/cartographer/lua_files/
sudo cp viam-cartographer/lua_files/* /usr/local/share/cartographer/lua_files/

install: install-lua-files
sudo rm -f /usr/local/bin/carto_grpc_server
sudo rm -f /usr/local/bin/cartographer-module
sudo cp viam-cartographer/build/carto_grpc_server /usr/local/bin/carto_grpc_server
Expand Down
10 changes: 3 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -84,15 +84,11 @@ These containers are set to persist between sessions via the `persistent` parame
#### Setup, build, and run the binary

```bash
# Setup the gRPC files
make bufinstall && make buf
# Install dependencies
# Install dependencies using brew (or apt-get)
make setup
# Build & install the binary
# Build
make build
# Install lua files
make install-lua-files
# Install the cartographer-module binary
# Install lua files and binaries to /usr/local
make install
# Run the binary
cartographer-module
Expand Down
2 changes: 1 addition & 1 deletion cartofacade/cgo_api.go
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ package cartofacade
#cgo CFLAGS: -I../viam-cartographer/src/carto_facade

// the libraries that need to be linked can be derived from line 258 of the build.ninja file that is autogenerated during make build
#cgo LDFLAGS: -L../viam-cartographer/build -L../viam-cartographer/cartographer/build -lviam-cartographer -lcartographer -ldl -lm -labsl_hash -labsl_city -labsl_bad_optional_access -labsl_strerror -labsl_str_format_internal -labsl_synchronization -labsl_strings -labsl_throw_delegate -lcairo -llua5.3 -lstdc++ -lceres -lprotobuf -lglog -lboost_filesystem -lboost_iostreams -lpcl_io -lpcl_common -labsl_raw_hash_set
#cgo LDFLAGS: -L../viam-cartographer/build -L../viam-cartographer/build/cartographer -lviam-cartographer -lcartographer -ldl -lm -labsl_hash -labsl_city -labsl_bad_optional_access -labsl_strerror -labsl_str_format_internal -labsl_synchronization -labsl_strings -labsl_throw_delegate -lcairo -llua5.3 -lstdc++ -lceres -lprotobuf -lglog -lboost_filesystem -lboost_iostreams -lpcl_io -lpcl_common -labsl_raw_hash_set

#include "../viam-cartographer/src/carto_facade/carto_facade.h"

Expand Down
5 changes: 1 addition & 4 deletions etc/packaging/appimages/cartographer-module-aarch64.yml
Original file line number Diff line number Diff line change
Expand Up @@ -7,13 +7,10 @@ script:
- mkdir -p $TARGET_APPDIR/usr/share/cartographer/lua_files
- cp ../../../bin/cartographer-module $TARGET_APPDIR/usr/bin/
- cp ../../../viam-cartographer/build/carto_grpc_server $TARGET_APPDIR/usr/bin/
- cp ../../../viam-cartographer/cartographer/build/libcartographer.a $TARGET_APPDIR/usr/lib/
- cp ../../../viam-cartographer/build/libviam-cartographer.a $TARGET_APPDIR/usr/lib/
- cp ../../../viam-cartographer/lua_files/* $TARGET_APPDIR/usr/share/cartographer/lua_files
- mkdir -p $TARGET_APPDIR/usr/share/icons/viam/256x256/apps/
- cp ./viam-server.png $TARGET_APPDIR/usr/share/icons/viam/256x256/apps/viam-server.png
- go install github.com/Otterverse/aix@latest
- cp `go env GOPATH`/bin/aix $TARGET_APPDIR/usr/bin/
- GOBIN=$TARGET_APPDIR/usr/bin/ go install github.com/Otterverse/aix@latest
- chmod 755 $TARGET_APPDIR/usr/bin/*

AppDir:
Expand Down
5 changes: 1 addition & 4 deletions etc/packaging/appimages/cartographer-module-x86_64.yml
Original file line number Diff line number Diff line change
Expand Up @@ -7,13 +7,10 @@ script:
- mkdir -p $TARGET_APPDIR/usr/share/cartographer/lua_files
- cp ../../../bin/cartographer-module $TARGET_APPDIR/usr/bin/
- cp ../../../viam-cartographer/build/carto_grpc_server $TARGET_APPDIR/usr/bin/
- cp ../../../viam-cartographer/cartographer/build/libcartographer.a $TARGET_APPDIR/usr/lib/
- cp ../../../viam-cartographer/build/libviam-cartographer.a $TARGET_APPDIR/usr/lib/
- cp ../../../viam-cartographer/lua_files/* $TARGET_APPDIR/usr/share/cartographer/lua_files
- mkdir -p $TARGET_APPDIR/usr/share/icons/viam/256x256/apps/
- cp ./viam-server.png $TARGET_APPDIR/usr/share/icons/viam/256x256/apps/viam-server.png
- go install github.com/Otterverse/aix@latest
- cp `go env GOPATH`/bin/aix $TARGET_APPDIR/usr/bin/
- GOBIN=$TARGET_APPDIR/usr/bin/ go install github.com/Otterverse/aix@latest
- chmod 755 $TARGET_APPDIR/usr/bin/*

AppDir:
Expand Down
Loading