Synchronizing a client's clock

[xopxe]

I have an ESP32 microcontroller connected as a client to a router on a Linux host over a serial connection.

Micro-ROS has a rmw_uros_sync_session call that allows getting the time from the agent.

Is there a similar functionality with Pico-ROS? When connected over a network, I can synchronize my microcontroller using NTP, but what should I do when connected over a serial link? (rmw_uros_sync_session works in any link).

[xopxe]

Just a couple of comments.

• Micro-ROS' rmw_uros_sync_session doesn't actually sync the clock; it just updates an offset variable you can use (typically through a rmw_uros_epoch_millis() call).

• I need an up-to-date time to stamp ROS topics.

[g4sp3r]

Hey @xopxe, thanks for raising the issue and providing useful info.

Unfortunately nothing like this currently exists.
We rely on zenoh-pico for networking and system utilities and rmw_zenoh to get connection to ROS. Maybe I missed it but I haven't seen any network time sync features in zenoh.

I read that zenohd fixes wrong timestamps but probably you need the correct timestamp inside ROS Header.

What you suggested could probably be implemented in "application" layer for now. Publish the current time on a topic from ROS host, the ESP32 can subscribe and calculate the offset for current clock. Use this offset when sending other messages.
It would be even "nicer" to make a service call for it but only service server is currently supported for Pico-ROS.

[xopxe]

That's unfortunate, @g4sp3r. In your opinion, such functionality should reside where? I guess this would be useful for all microcontroller-based Zenoh-Pico applications that do not connect via an IP network.

Yes, I thought about implementing this on the application side. My idea is to replicate the NTP mechanism: send an "echo request" and then take half the RTT as the latency. Add this latency to the timestamp on the response, and that's your time. NTP does this from both ends to account for asymmetric RTTs, but I don't think this would be important here.

[g4sp3r]

@xopxe I agree, the feature would be useful in a lot of situations.

In your opinion, such functionality should reside where?

My understanding is that time sync protocol belongs to Application layer of OSI, so it makes sense that zenoh doesn't support any feature like that.
Disregarding the layers for a moment, there is possibility that router timestamp is sent in keep-alive messages. This is just my wild speculation, but then it could make sense to support it in zenoh-pico for "free" since messages are already sent. Zenoh-pico also has a system.c wrapper with access to platform clock functions so it seems like a natural place to add it.

On the other hand, following the model, you could argue that support should come from PicoROS, but then it's not totally clear who will provide the clock sync services. I am not aware of any standard ROS time sync service we could depend on in comparison with micoROS that has it's agent and can control both sides.

One option would be to vendor the time sync ROS node running on the host in Pico-ROS repo. Add support for service call in picoros.c. Add a function like picoros_time_sync(char* topic name, (sync_cb*)(z_clock_t now), (sync_failed_cb*)(void)). Save the current time when function is called and make a service call. If we get reply from above node your proposed RTT/2 is added to the time inside message and callback is called. The callback is needed since zenoh-pico system wrapper has no functions to set the clock, so the actual calling of current platform clock setting function would be delegated to application.

Or alternatively provide

z_clock_t picoros_time_get(void)
void picoros_time_set(z_clock_t now) 
void picoros_time_sync(char* topic) //querry reply comes on another thread
bool picoros_is_time_synced(void)

managing the internal offset variable like you outlined above.

My sense is to first investigate the zenoh-pico option since it could be more elegant.
Also in parallel support service call in PicoROS as first step since it will be needed for other cases.

What are your thoughts?

[xopxe]

I agree with your analysis, @g4sp3r

Having this service integrated inside zenoh-pico would be the most convenient option, but its implementation is not trivial. In the generic case, it will need to work on all links that Zenoh supports, including those with high latency and/or jitter. NTP, which solves this problem over IP networks, has a lot of code handling outliers, drifting, and so on. This complexity makes it more reasonable to place it in the application layer, as you said (NTP is considered an application layer service).

So, implementing it as a standard application first makes sense. It would be easier to tailor to different use cases and experiment a bit. I started prototyping such a version based on notifications, which mimics NTP message exchange. I saw that you have a PR for service clients. I'll try it as soon as possible and also use it to test the simpler "RTT/2" method, which could be good enough on some links.

[g4sp3r]

@xopxe, I think your plan to implement it as standard application at first is good.
For start we could link the time-sync example application in the readme, or add it as part of examples in this repo.
If we can find a good solution I am also very happy to integrate it in Pico-ROS.

When you come to testing the service client PR, please let me know if there are some things with the api you think could be improved, or any other ideas you might come up with.

[xopxe]

Hi @g4sp3r

I did a proof of concept for time synchronization.

The C++ code is Arduino through platformio, and is copied straight from my codebase so it refers to a bunch of stuff that from my environment, but shows the basic idea:

• The microcontroller side sends an empty stamped message, and registers the time.
• The host responds with a stamped message with the arrival time for the request
• the microcontroller receives the response and obtains the emission and reception times for both. Then it computes the RTT and estimates the offset.

On an ESP32 over a TCP on a 115200 serial link I get about 7-8ms in RTT, and consecutive synchronization sessions get under 1ms difference, sometimes spiking to under 10ms differences. I think averaging several synchronizations will be good enough.

Implementing this using services would be cleaner. As it is now for real use you would need to consider responses coming from other instances of the service. As soon as you consider the service request working, I'll port this.

[g4sp3r]

Hey @xopxe, that's great! Thanks for sharing ❤️

I think the new service call api shouldn't change anymore. You could try porting and give some comments in the PR if you hit any difficulties. I wanted to do more testing with different embedded devices before merging and your set-up on esp32 with serial would definitely be another useful data point.

[saikishor]

@xopxe can you share a working example of your setup?
With @g4sp3r we tried to setup the esp32s3 with a PicoROS joystick example over serial, we could establish the connection to the router but when we do ros2 topic list. We don't see any topics at all and neither do the nodes.

@g4sp3r has stm32 working over serial with topics exposed. It would be really helpful for me to know how did you solve the issue or if you share the codebase, I can debug it myself for the roscon2025 workshop

Thank you

[g4sp3r]

Looks like we were were missing "tcp/[::]:7447" in our ZENOH_CONFIG_OVERRIDE when testing with @saikishor . Using DEFAULT_RMW_ZENOH_ROUTER_CONFIG.json5 with added serial port works.

🤦

[xopxe]

Just not to disappoint, I realized in the timing tests I did in my proof of concept I was actually connected through WiFi and not the serial.

Meanwhile, I realized that the topic-based time synchronization runs on standard messages from std_msgs and sensor_msgs, but the service-based version would depend on a custom type, which is annoying and probably overengineering without a proper design.

[g4sp3r]

@xopxe you are right, that is annoying.

I will still merge #8 and propose we leave this issue open for further discussions.

[xopxe]

I have a notification-based synchronization service working. The server is available here, while the client code can be found here. The client code is integrated into a bigger system, but can be easily extracted into a self-contained example.