JA | EN

SOBITS Display

Overview

SOBITS Display is a dashboard that connects ROS 2 and a browser (HTML/JavaScript) using rosbridge_server.

It allows any text or image to be displayed for a specified number of seconds.

It also automatically detects when TTS and STT are being used, and displays a speaker or microphone icon while they are active.

On SOBIT HOME, the face is drawn in real time on a Canvas rather than as a static image, supporting expression switching, a speaking animation, and gaze control (mouse tracking or programmatic specification).

(back to top)

Setup

This section explains how to set up this repository.

(back to top)

Requirements

First, prepare the following environment before proceeding to the installation steps.

System	Version
Ubuntu	24.04 (Noble Numbat)
ROS	Jazzy Jalisco
Python	3.12

(back to top)

Installation

If you want to display on the HSR or SOBIT HOME's display, run the following on the built-in PC (via SSH or AnyDesk).

If you want to display on a local PC, run the following on the local PC.

1. Move to the src folder of your ROS2 workspace.
2. Clone this repository.

   git clone https://github.com/TeamSOBITS/sobits_display.git

3. Move into the repository.

   cd sobits_display/

4. Install the dependent packages. Note that this may take some time.

   bash install.sh

5. Compile the package.

   cd ~/colcon_ws/

   colcon build --symlink-install

   source ~/colcon_ws/install/setup.sh

(back to top)

About UI Changes

After changing the UI code, the changes may not be reflected due to browser caching.

If that happens, clear the cache and reload the page.

On Chrome, you can do this with ctrl + shift + R.

(back to top)

Usage

First, launch sobits_display.launch.py on the built-in PC with the following command.

If you want to display on a PC, run the following on that PC.

ros2 launch sobits_display sobits_display.launch.py

(back to top)

Accessing the Display

Access the following URL in your browser.

※ If launched on the built-in PC, run the browser on that same PC.

http://localhost:8080

If using chromium, you can also launch it with an alias like the following.

# Launch the display with chromium
alias sd='chromium --app=http://localhost:8080'

※ If chromium is installed locally, add the alias to your local .bashrc. ※ To switch to fullscreen, you need to manually press the F11 key.

(back to top)

Microphone / Speaker Icons

It automatically detects when TTS and STT are being used, and displays a speaker or microphone icon while they are active.

(back to top)

Microphone / Speaker Volume

Run the following command to show microphone and speaker volume.

ros2 launch sobits_display volume_publisher.launch.py

The volume shown is the volume of the PC on which it was launched. If launched on the robot's built-in PC, that PC's volume will be shown.

(back to top)

API Reference

You can control the display using the following.

(back to top)

Initialization

To use the SOBITS Display features, you need to run the init() method once.

• display.init()
  • Arguments
    • img_dir (str): Reference directory for images to display. If a relative path is given, it is calculated from cwd. Using an absolute path is recommended.
    • robot_name ('hsrb_robot' | 'sobit_home'): Name of the robot being used
    • (optional) bridge_url (str): URL of the bridge (port is fixed at 8000). Specify the URL of the host that ran sobits_display.launch.py.
      • For local testing → "http://localhost:8000"

(back to top)

Show/Hide Text

Use the show_text() method to display text.

• display.show_text()

  • Arguments
    • text (str): The text to display. Use "" to hide it.
    • display_sec (float | None): Duration to display the text. None displays it permanently.

  # Displays text for 5 seconds, then hides it
  
  from sobits_display import display
  
  display.init(
      img_dir="./imgs"
      robot_name="hsrb_robot"
  )
  
  display.show_text("Is your order apple?", 5)

  

(back to top)

Show Images

Use the show_images() method to display images.

• display.show_images()

  • Arguments
    • file_names (list[str]): List of image file names to display. Use [] to hide them.
    • display_sec (float | None): Duration to display the images. None displays them permanently.

  # Displays images for 5 seconds, then hides them
  
  from sobits_display import display
  
  display.init(
      img_dir="./imgs"
      robot_name="hsrb_robot"
  )
  
  display.show_images(["apple.png", "lemon.png"], 5)

※ Supported extensions: .png, .jpg, .svg, .gif

(back to top)

Expression Display (SOBIT HOME)

Use the show_expression() method to switch SOBIT HOME's facial expression. This is only effective when robot_name="sobit_home" (the face is drawn in real time on a Canvas rather than as a static image).

• display.show_expression()

  • Arguments
    • expression (str): Name of the expression to display. One of "idle", "happy", "sad", "angry", "surprised", "listening", "wink"
    • display_sec (float | None): Duration to display the expression. None maintains it. If specified, it reverts to "idle" after the time elapses.

  # Displays a happy expression for 3 seconds, then reverts to idle
  
  from sobits_display import display
  
  display.init(
      img_dir=".",
      robot_name="sobit_home",
  )
  
  display.show_expression("happy", 3)

※ While idle is displayed, it blinks automatically. Running clear_display() also reverts the expression to idle. ※ For "angry", the eye color turns red.

(back to top)

Speaking Animation (SOBIT HOME)

Use the set_speaking() method to show or hide the mouth's speaking animation. This can be controlled independently of the current expression (show_expression()), so for example you can display a "happy" expression while it appears to be speaking.

• display.set_speaking()

  • Arguments
    • is_speaking (bool): True to move the mouth and display it, False to hide it
    • display_sec (float | None): Reverts to hidden after a set time. None maintains it.

  from sobits_display import display
  
  display.init(
      img_dir=".",
      robot_name="sobit_home",
  )
  
  display.show_expression("happy")
  display.set_speaking(True, 3)  # Speaks for 3 seconds while keeping the happy expression

(back to top)

Gaze Control (SOBIT HOME)

Use the set_gaze() method to programmatically fix the position of the pupils. If not set, it automatically follows the mouse cursor on the browser. set_gaze() takes priority over mouse tracking.

• display.set_gaze()

  • Arguments
    • x (float): Horizontal gaze direction. -1.0 (left edge) to 1.0 (right edge)
    • y (float): Vertical gaze direction. -1.0 (top edge) to 1.0 (bottom edge)

  from sobits_display import display
  
  display.init(
      img_dir=".",
      robot_name="sobit_home",
  )
  
  display.set_gaze(-1.0, 0.0)  # Look to the left

Use the clear_gaze() method to release the fixed gaze and return to mouse tracking.

• display.clear_gaze()

  from sobits_display import display
  
  display.init(
      img_dir=".",
      robot_name="sobit_home",
  )
  
  display.clear_gaze()

※ Running clear_display() also releases the fixed gaze.

(back to top)

Clear Display

Use the clear_display() method to hide the text and images.

• display.clear_display()

  # Displays images for 5 seconds, then hides them
  
  import time
  
  from sobits_display import display
  
  display.init(
      img_dir="./imgs"
      robot_name="hsrb_robot"
  )
  
  display.show_text("Hello")
  display.show_images(["waving_hand.png"])
  
  time.sleep(5)
  
  display.clear_display()

(back to top)

Start Button

Use the show_start_button() method to display a start button on SOBIT HOME's touch display.

• display.show_start_button()

  • Arguments
    • title (str): Title text displayed above the button
    • subtitle (str): Supplementary text displayed above the button
    • button_label (str): Text displayed on the button

  from sobits_display import display
  
  display.init(
      img_dir=".",
      robot_name="sobit_home",
  )
  
  display.show_start_button(
      title="READY FOR GPSR?",
      subtitle="Tap the screen to launch the run",
      button_label="START",
  )

  

Use the hide_start_button() method to hide the currently displayed start button.

• display.hide_start_button()

  from sobits_display import display
  
  display.init(
      img_dir=".",
      robot_name="sobit_home",
  )
  
  display.hide_start_button()

When the start button is pressed, a start notification is published to a ROS 2 topic.

• /competition_start
  • Type: std_msgs/msg/Bool
  • Content: Publishes data: true when the button is pressed

※ The start button is also automatically hidden when show_text(), show_images(), or clear_display() is executed.

(back to top)

Milestones

• SOBIT HOME touch support
• Automatic display launch for SOBIT HOME and HSRB
• Display remaining seconds for TTS/STT
• Display camera footage
• Support for SOBIT HOME's emotional expressions
• Component layout via a configuration file

Please see the Issues page to check current bugs and new feature requests.

(back to top)

References

• Vue.js
• FastAPI

(back to top)