Robot Basic Information Introduction

Control Architecture

This robot uses the architecture of the host device + slave device (Dual Controller architecture), the host device can be Raspberry Pi, Jetson Nano, Orin Nano or other single-board computers with similar 40PIN interfaces, and the slave device uses ESP32 to control robot peripherals, read sensor information, and control motor closed-loop speed (PID controller).

The host device and the slave device use JSON format instructions to communicate through the serial port, the specific communication content can refer to the documentation of the slave device. As a beginner, you don't need to understand those instructions, you only need to follow the tutorial document to call common instructions or encapsulated functions in the early stage.

Advantages of Dual Controller Architecture

1. Jetson or other single-board computers can serve as the host device for processing complex tasks, such as visual information processing, while the ESP32, as the slave device, is responsible for controlling peripherals and sensors. This modular design enhances the system's flexibility and scalability.
2. Jetson or other single-board computers focus on advanced processing and decision-making, while ESP32 handles real-time low-level tasks such as motor control. This division of labor allows each component to concentrate on its area of expertise.
3. This architecture can efficiently allocate processing power and IO resources, reducing the burden on a single system and improving overall efficiency.
4. The JSON format is used for communication through the serial port, which improves the efficiency and readability of data transmission, and is easy to debug and expand.
5. For makers and hobbyists on a budget, this architecture can reduce cost and system complexity while maintaining high performance.

Interactive Development Basics with JupyterLab

What is JupyterLab?

• Interactive Development Environment: JupyterLab is an open-source interactive development environment that provides an easy-to-use interface for writing code, running experiments, and viewing data.
• Ideal for data science and machine learning: While originally designed for data science and machine learning, its flexibility and ease of use make it ideal for robot programming and experimentation.
• Web-based tool: As a browser-based application, JupyterLab does not require a complicated installation process, and users can use it on almost any operating system.

Advantages of developing with JupyterLab

1. User-friendly programming environment: JupyterLab provides a clean, intuitive user interface that makes programming and experimentation more beginner-friendly. Easily write and test code with an interactive notebook for beginners to explore and learn step by step.
2. Immediate feedback and result visualization: Instant feedback, users can immediately see the effect of code changes, which is great for debugging and learning. JupyterLab makes it easy to visualize data to help understand the behavior and performance of robots.
3. Support for multiple programming languages: JupyterLab supports multiple programming languages such as Python, providing flexibility for users of all skill levels.
4. Customization and Extensibility: JupyterLab is highly customizable and extensible, allowing users to add new features or tools according to their needs.
5. Cross-platform and accessibility: JupyterLab is a web-based tool with excellent cross-platform capabilities, it can run on different operating systems, and it can be accessed through a browser.

JupyterLab Basics

• You can refer to the official documentation of JupyterLab to learn: https://jupyterlab.readthedocs.io/en/latest/getting_started/overview.html
• Since our interactive tutorials are all made by Jupter notebooks(.ipynb) file, we'll cover some of the basics here.

What is a Jupyter notebooks(.ipynb) document?

A Jupyter notebooks(.ipynb) is a document that combines ready-to-run code with narrative text (Markdown), equations (LaTeX), images, interactive visualizations, and other rich outputs.

Switch the theme of the document

• Our default theme is the bright Jupyter Dark.
• You can switch to a dark theme according to your preference: click Settings - Theme - JupyterLab Dark at the top of the screen.

COMMAND / EDIT MODE

JupyterLab has two working modes, one is COMMAND mode and the other is EDIT mode

• COMMAND mode

When you are in COMMAND mode, you can quickly perform the overall operations of the notebook, such as adding or removing cells, moving cells, changing cell types, etc. In this mode, the cell border is grayed out. You can enter COMMAND mode by pressing the Esc key.

• EDIT Mode

EDIT mode allows you to enter or modify code or text in a cell. In this mode, the cell border is blue. You can enter EDIT mode by clicking or pressing Enter in the selected cell.

Cell Operations

In JupyterLab, you can do the following:

• In COMMAND mode, use the arrow keys up and down to select a cell.
• Add cells below: You can add new cells below the current cell by clicking the + button in the toolbar or by using the shortcut key B (in COMMAND mode).
• Add cells above: You can add new cells below the current cell by clicking the + button in the toolbar or by using the shortcut key A (in COMMAND mode).
• Delete cells: Use D (press D key twice in a row in COMMAND mode) to delete the currently selected cells.
• Copy cells: (in COMMAND mode) Use the shortcut key C.
• Paste cells: (in COMMAND mode) Use the shortcut key V.
• Cut cells: (in COMMAND mode) Use the shortcut key X.
• Undo: (in COMMAND mode) Use the shortcut key Z.
• Redo: (in COMMAND mode) Use the shortcut Shift + Z.
• Convert the current cell to a code block: (in COMMAND mode) Use the shortcut key Y.
• Convert the current cell to Markdown: (in COMMAND mode) Use the shortcut key M.
• Switch cell type: You can set the cell as a code cell, a Markdown cell, or an original cell. This can be swithed in COMMAND mode via the drop-down menu in the toolbar or the shortcut Y (code cell), M (Markdown cell).
• Run Cells: You can click the button in the toolbar or use the shortcut key Shift + Enter to execute the current cell and run subsequent cells automatically.

Save & Export

• Save notebook: You can save your notebook by clicking the button in the toolbar or by using the shortcut key S (in COMMAND mode).
• Export notebooks: JupyterLab supports exporting notebooks to a variety of formats, including HTML, PDF, Markdown, and more. This can be accessed via File > Export Notebook As... menu options.

What is JupyterLab's Kernel?

• JupyterLab's Kernel is a computing engine that executes code written by users in notebooks.
• Each notebook is associated with a Kernel, which can be a different programming language, such as Python, R, or Julia. The Kernel also has access to resources, such as memory or CPU.

Set the Kernel as the virtual environment for the bot

• When you open the subsequent .ipynb tutorial documentation, you need to manually select the Kernel in the notebook so that the bot-related code blocks can be executed correctly.
• Here's how: Click on the Kernel option next to in the top right corner of the notebook tab and select Python 3 (ipykernel) from the drop-down menu.

Kernel Management

• Startup: When you open a Jupyter notebook, the associated Kernel will start automatically, and a small green dot will appear in front of the corresponding note in the file list.
• Restart: If the Kernel crashes or needs to clear the state of the current session, you can use Kernel -> Restart Kernel... option to restart the Kernel.
• Stop: In the note interface where the Kernel is running, use Kernel -> Shut Down Kernel to stop the Kernel of the current note; It is also possible to stop the Kernel of all notes with Kernel -> Shut Down All Kernels.
• Note: If you call the camera in the Kernel of a tutorial and do not stop the Kernel of the note, it will always occupy this resource, and other tutorials will not be able to use normally, and you need to stop the Kernel of the tutorial before other tutorials can be used normally.

Run Code Block

After selecting the correct Kernel, you can run the code block in the notebook. In JupyterLab, code blocks are the basic building blocks of notebooks. Here's how to run a block:

• Run a single block: Select the block you want to run, then click the button on the toolbar or use the shortcut key Shift + Enter. This will execute the current block of code and select the next block.

print("test text in jupyterlab")

• Run all code blocks: You can also run all code blocks in the entire notebook. To do this, click on the Run menu in the toolbar and select Run All Cells.

for i in range(0, 10): print(i)

• Stop code block running: If you need to stop a running code block, you can click the button on the toolbar.

With these basic how-tos, you can effectively use JupyterLab for a variety of tasks. More advanced features and detailed instructions can be found in the official JupyterLab documentation.

Delete Code Block Output

• Delete a single code block output: Select a code block and click Edit - Clear Cell Output at the top of the code block
• Delete all code block outputs: Click Edit - Clear Outputs of All Cells at the top

More advanced content

• You can refer to the official documentation of JupyterLab to learn: https://jupyterlab.readthedocs.io/en/latest/getting_started/overview.html

Python Chassis Motion Control

In this chapter, we'll provide a Python demo for controlling the motion of a robot's chassis. This approach can be adapted to other programming languages for similar motion control tasks.

Control Mechanism Overview

We utilize code blocks within JupyterLab to compose JSON commands. These commands are then sent to the slave controller via the GPIO serial port on a Jetson (the default baud rate for communication with the slave controller is 115200). Upon receiving these commands, the slave controller executes the specified actions.

Further sections will delve into the variety of commands that can be sent to the slave controller. Alternatively, you might choose to implement this function in a different programming language or develop a dedicated application for the host controller.

Design Advantages

Adopting a dual-controller architecture significantly liberates the resources of the host device. In this setup, the host controller (SBC such as Raspberry Pi or Jetson) acts as the "brain", while the ESP32 slave controller (sub-controller) serves as a "cerebellum-like" entity handling the lower-level motion controls. This division of tasks allows the host to focus on high-level tasks like vision processing and decision-making, while the sub-controller efficiently manages precise movement control and other low-level tasks. Such an arrangement ensures that the sub-controller can maintain accurate wheel rotation speeds through high-frequency PID control, without overburdening the host with computationally intensive tasks.

Main Program (app.py)

The main program of the product, app.py, automatically executes upon booting due to the configuration set by autorun.sh (which is pre-configured in the product). Running app.py occupies the GPIO serial port and the camera, which might lead to conflicts or errors if other tutorials or programs require these resources. Ensure to disable the auto-start of app.py before proceeding with development or learning.

As app.py employs multithreading and uses crontab for autostart, typical commands like sudo killall python won't suffice to terminate it. You would need to comment out the relevant line in crontab and reboot the device.

crontab -e

Upon your first use of this command, you will be prompted to select an editor to open the crontab file. It is recommended to choose nano by entering the corresponding number for nano, followed by pressing the Enter key to confirm.
Use the # symbol to comment out the line containing ...... app.py.

# @reboot ~/ugv_pt_rpi/ugv-env/bin/python ~/ugv_pt_rpi/app.py >> ~/ugv.log 2>&1
@reboot /bin/bash ~/ugv_pt_rpi/start_jupyter.sh >> ~/jupyter_log.log 2>&1

Note: Make sure not to comment out the line containing start_jupyter.sh, as doing so will prevent JupyterLab from launching on startup, disabling your access to interactive tutorials.

To exit and save the changes, after editing the content of crontab, press Ctrl + X to exit nano. Since you have made edits to the crontab file, it will ask if you want to save the changes (Save modified buffer?). Enter the letter Y and then press Enter to exit and save the modifications.

After restarting the device, the main program will no longer run automatically upon boot-up, allowing you to freely use the tutorials within JupyterLab. If you wish to re-enable the automatic execution of the main program at startup in the future, you can reopen the crontab file using the method described above. Simply remove the # symbol in front of the @ symbol, then exit and save the changes. This will restore the automatic startup functionality of the main program.

Chassis Control Demo

In the following demo, it's crucial to use the correct device name and baud rate (default 115200) that matches the sub-controller.
Before executing the code block below, ensure the robot is elevated so that the drive wheels are off the ground. Activating the code will cause the robot to move; take precautions to prevent it from falling off the table.

from base_ctrl import BaseController
import time

base = BaseController('/dev/ttyTHS0', 115200)

# The wheel rotates at 0.2 m/s for 2 seconds and then stops
base.send_command({"T":1,"L":0.2,"R":0.2})
time.sleep(2)
base.send_command({"T":1,"L":0,"R":0})

By invoking the code block mentioned above, the Raspberry Pi will initially send the command {"T":1,"L":0.2,"R":0.2} (the structure of commands will be discussed in more detail in later chapters). This command starts the wheels turning. After a two-second interval, the Raspberry Pi will send another command {"T":1,"L":0,"R":0}, causing the wheels to stop. It's important to note that even if the command to stop the wheels isn't sent, the wheels will still cease turning if no new commands are issued. This is due to a heartbeat function built into the sub-controller. The purpose of this heartbeat function is to halt the current motion command automatically if the host controller hasn't sent any new commands to the sub-controller for an extended period. This function is designed to prevent continuous motion of the sub-controller in case the host encounters a problem that leads to a freeze or crash.

If you want the robot to continue moving indefinitely, the master control unit needs to cyclically send motion control commands every 2 to 4 seconds.

Chassis Model Selection

You may find that the direction or speed of the robot's wheels does not match your expectations after inputting the motion control commands. This could be because, before controlling the chassis, you need to set the type of chassis. Each time you run the main demo (app.py), the chassis type is automatically configured, and the configuration parameters are stored in config.yaml. This ensures that the chassis is controlled according to the correct parameters. By default, the chassis needs to be configured once by the host computer every time it is powered on. You can send the following commands to the chassis to set its type:
"main" value means the chassis type:

• 1: RaspRover (4-wheel 4WD chassis)
• 2: UGV Rover (6-wheel 4WD chassis)
• 3: UGV Beast (off-road tracked chassis)

"module" value means the module type:

• 0: can be set whether install the pan-tile module or not
• 1: robotic arm module (if there is not a robotic arm installed, report errors when the chassis Ping fails.)
• 2: pan-tilt module (if there is not a pan-tilt module installed, report errors when the chassis Ping fails.)

For example, if you use the UGV Rover with the pan-tilt module, you can send the following commands to the slave computer to set the chassis type:

{"T":900,"main":2,"module":0} or {"T":900,"main":2,"module":2}

• If you don't require chassis feedback on the pan-tilt's angle information (intended for user secondary development, not included in the demos), it's recommended to use the former one.

You can modify the JSON parameter to set the chassis model in the following demo block according to your products:

base.send_command({"T":900,"main":2,"module":0})

Then the following code block is executed to control the chassis, and the wheels turn in the right direction and at the right speed.

base.send_command({"T":1,"L":0.2,"R":0.2})
time.sleep(2)
base.send_command({"T":1,"L":0,"R":0})

Chassis Steering Principle

The earlier demo allows you to make the robot move forward for two seconds before stopping. Further adjustments to the parameters can control the direction of the chassis, based on the differential steering principle.

When turning, the inner wheels (on the side towards which the turn is made) travel a shorter distance and thus should rotate slower to maintain stability. The differential gear system achieves this by allowing the drive wheels to rotate at different speeds. Usually, the outer wheels (on the opposite side of the turn) rotate faster than the inner wheels. This variation in speed results in the turning motion of the vehicle, allowing it to steer in the intended direction.

You can control the vehicle's steering by assigning different target linear velocities to each wheel, thus achieving maneuverability and easily adjustable turning radii.

Pan-tilt Control & LED Light Control

Pan-tilt Control

Basics

A product with a pan-tilt contains two servos, a pan servo and a tilt servo. The pan servo controls the rotation of the horizontal plane of the pan-tilt, and the rotation range is ±180° (total range 360°); The tilt servo controls the rotation of the vertical plane of the pan-tilt, and the rotation range is -45°~90° (total range 135°).

For products that do not come with a pan-tilt, you can also expand the pan-tilt on the RaspRover yourself.

# Import the library used to control the chassis
from base_ctrl import BaseController

base = BaseController('/dev/ttyTHS0', 115200)

In the code block above we import and instantiate the library for chassis control, and then control the pan-tilt movement by changing the angle of the pan-tilt pan servo and tilt servo.

Change the value in the following code, in the following code:

• input_x is the rotation angle of the pan-tilt pan servo, and the rotation range of the pan servo plane PAN axis is ±180° (total range 360°);
• input_y is the rotation angle of the pan-tilt tilt servo, and the vertical plane of the tilt servo TILT axis rotation range is -45°~90° (total range 135°);
• input_speed is the speed at which the pan-tilt rotates, and when the speed value is 0, the rotation speed is the fastest;
• input_acc is the acceleration at the beginning and end of the pan-tilt rotation, the lower the value, the smoother the start and stop. When the acceleration value is 0, it rotates according to the maximum acceleration.

Running the following code, you can see that the pan-tilt will move 45° to the right and up and then stop.

input_x = 45 
input_y = 45 
input_speed = 0 
input_acc = 0

base.gimbal_ctrl(input_x, input_y, input_speed, input_acc)

In addition to controlling the pan-tilt movement by changing the rotation angle of the two servos in the pan-tilt, you can also directly control the pan-tilt continuous movement.

Change the value in the following code, in the following code:

• input_x is the rotation mode of the pan-tilt pan servo. When the value is -1, it means that the pan-tilt is continuously rotated to the left (clockwise); When the value is 1, it means that the pan-tilt is continuously rotated to the right (counterclockwise); When the value is 0, it means that the rotation is stopped.
• input_y is the rotation mode of the pan-tilt tilt servo. When the value is -1, it means that the pan-tilt tilt servo continuously rotates downward (head down); When the value is 1, it means that the pan-tilt tilt servo continuously rotates upwards (heads up); When the value is 0, it means that the rotation is stopped.
• input_speed is the speed at which the pan-tilt is rotated.

Only when the value of input_x and input_y is 2, the pan-tilt will automatically return to the median position.

Run the following code and the pan-tilt will move to the left until it stops when the movement reaches 180°.

input_x = -1 
input_y = 0 
input_speed = 0

base.gimbal_base_ctrl(input_x, input_y, input_speed)

Run the following code, the pan-tilt will move upwards and stop when the movement reaches 90°.

input_x = 0 
input_y = 1 
input_speed = 0

base.gimbal_base_ctrl(input_x, input_y, input_speed)

Finally, run the following code to return the pan-tilt to the middle position.

input_x = 2 
input_y = 2 
input_speed = 0

base.gimbal_base_ctrl(input_x, input_y, input_speed)

LED Lighting Control

Basic Introduction

WAVE ROVER and UGV series products, the driver board integrates two 12V switch control interfaces (the actual maximum voltage will change with the battery voltage), respectively, the IO4 and IO5 pins of ESP32 are controlled through the MOS tube, each corresponding to two interfaces, a total of 4 12V switch control interfaces, according to the default assembly method, IO4 controls the chassis headlights (WAVE ROVER does not have chassis headlights), IO5 controls the headlights, you can control the switch of these two interfaces and control the voltage by sending corresponding commands to the lower computer (but due to the MOS tube control itself has a certain delay, the PWM of the IO output of ESP32 is not linear with the voltage output of the actual interface).

For products that are not equipped with LED lights, you can expand the 12.6V withstand voltage LED lights on these two 12V switch control interfaces (under normal circumstances, the 12V withstand voltage can also be used, because in order to safety and protect the battery, the UPS of the product will not charge the battery above 12V), you can also expand other peripherals on the remaining switch control interfaces - such as the 12V water bomb gun gearbox, which is directly connected to the IO5 control interface, and can realize the function of automatic aiming and shooting.

If you need to run a program within a code block, you can select the code block you want to run and press Ctrl+Enter to run the program, or if you are using JupyterLab on your phone or tablet, you can press the triangle play button above to run the code block.

In the above code block we import and instantiate the library used to control the chassis, and then control the switch of the IO4 interface, the variable IO4_PWM is the PWM of the IO4 pin output of ESP32, the variable range is 0-255, when the value of this variable is 0, the interface switch of the IO4 control is turned off; When this variable is 255, the output voltage of the IO4 controlled interface switch is close to the BAT voltage of the UPS (the voltage of the current three lithium batteries in the UPS in series).

Run the following code block to turn on the interface switch for IO4 control (light up the chassis headlights). Note: The WAVE ROVE does not have chassis headlights, so there is no change to run the following code block, you need to run the next code block that turns on the headlights to turn on the headlights, the headlights are located on the camera pan-tilt, if the product is not equipped with a camera pan-tilt, there are no headlights.

IO4_PWM = 255 
IO5_PWM = 0

base.lights_ctrl(IO4_PWM, IO5_PWM)

Run the following code block to turn on the interface switch of IO5 control (light up the pan-tilt headlight). Note: If the product is not equipped with a camera pan-tilt, there is no headlamp.

IO4_PWM = 255 
IO5_PWM = 255

base.lights_ctrl(IO4_PWM, IO5_PWM)

If your product has LEDs that should all be lit up by now, you can run the following code block to reduce the brightness of the LEDs:

IO4_PWM = 64 
IO5_PWM = 64

base.lights_ctrl(IO4_PWM, IO5_PWM)

Finally, run the following code block to turn off the LED lights.

base.lights_ctrl(0, 0)

Here runs a code block integrated with pan-tilt functionality.

import time
base.gimbal_ctrl(0, 0, 0, 0)
base.lights_ctrl(255, 255)
time.sleep(0.3)
base.gimbal_ctrl(45, 0, 0, 0)
base.lights_ctrl(0, 0)
time.sleep(0.3)
base.gimbal_ctrl(-45, 0, 0, 0)
base.lights_ctrl(255, 255)
time.sleep(0.3)
base.gimbal_ctrl(0, 90, 0, 0)
base.lights_ctrl(0, 0)
time.sleep(0.3)
base.gimbal_ctrl(0, 0, 0, 0)

This tutorial demonstrates how to control an OLED display connected to an ESP32 module using JSON commands. OLED displays are widely used for showing various types of information, such as text and images.

OLED Screens Basics

OLED displays communicate with the ESP32 module via the I2C (Inter-Integrated Circuit) interface. These displays are capable of showing custom text content and support multi-line display.

The product comes with an OLED display that communicates with the ESP32 module through the I2C interface. Upon powering up, the display automatically shows some basic information about the slave device. The content displayed on the screen can be altered by sending JSON commands from the host device.

OLED Screen Control JSON Commands

• {"T":3,"lineNum":0,"Text":"putYourTextHere"}
  • Controls the display to show custom content.
  • lineNum refers to the line number, and a single JSON command can change the content of one line. For the commonly used 0.91-inch OLED displays, the value of lineNum can be 0, 1, 2, or 3, allowing for four lines in total.
    
  • Text is the content you wish to display on that line. If the content exceeds the line length, it will automatically wrap to the next line, potentially overwriting the last line's content.

lineNum refers to the line number. A single JSON command can modify the content of one line. When the subordinate machine receives a new command, the default OLED display screen at startup will disappear, replaced by the content you've added. For most products that use a 0.91-inch OLED display, the value of lineNum can be 0, 1, 2, or 3, allowing for a total of four lines. Text is the textual content you wish to display on that line. If the content for a line is too long, it will automatically wrap to the next line, potentially overwriting the content on the last line.

from base_ctrl import BaseController

base = BaseController('/dev/ttyTHS0', 115200)

#Modify the OLED display content
base.send_command({"T":3,"lineNum":0,"Text":"this is line0"})
base.send_command({"T":3,"lineNum":1,"Text":"this is line1"})
base.send_command({"T":3,"lineNum":2,"Text":"this is line2"})
base.send_command({"T":3,"lineNum":3,"Text":"this is line3"})

Running the provided code block will display four lines of text on the OLED:

this is line0

this is line1

this is line2

this is line3

Displaying Dynamic Information on OLED

The tutorial above outlined a method for displaying simple text on the OLED screen. We will now proceed with a slightly more complex example. Running the following code block will display the current time on the OLED screen. Note that the time displayed might not be accurate due to potential discrepancies with the Raspberry Pi's clock. This example serves to demonstrate how to update the screen content in the main program, where we employ this method to display real-time information such as the device's IP address and operational status on the OLED screen.

# Import the datetime class from the datetime module to fetch and manipulate the current date and time.
from datetime import datetime
#  Import the time module, primarily used for delay processing within the program.
import time

# Create an infinite loop using while True to allow the program to run continuously.
while True:
    # Use datetime.now().strftime("%H:%M:%S") to obtain the current time and format it as "hour:minute:second".
    current_time = datetime.now().strftime("%H:%M:%S")
    # Utilize the base.send_command method to send a command that includes the current time.
    base.send_command({"T":3,"lineNum":0,"Text":current_time})
    # Use time.sleep(1) to pause the program for 1 second, ensuring that the time is updated and a command is sent every second.
    time.sleep(1)

Running the last code block, you'll observe the first line of the OLED screen updating to show the current time, refreshing every second. This function runs in an infinite loop, which can be terminated by clicking the stop button(■) above.

Build UI interfaces in JupyterLab

Building UI interfaces in JupyterLab typically uses the ipywidgets library, which provides a simple yet powerful way to create interactive user interfaces. Here are the detailed steps:

Import the required libraries

The ipywidgets library has been installed in our product. If you can't find the library when you run the block, you can install the library you need in the UI by pip install ipywidgets.

Select the following code block and press Ctrl + Enter to run the code.

import ipywidgets as widgets 
from IPython.display import display

Create UI Components

We can use various components from the ipywidgets library to build our UI interface, such as text boxes, buttons, output boxes, etc. For example:

# Create a text box 
text = widgets. Text(description='Please enter a name:')

# Create a button 
button = widgets. Button(description="Say hello")

# Create an output box 
output = widgets. Output()

Define Event Handler

We need to define a handler that handles user interaction events. In this example, we'll define a function to handle the click event of the button and display a greeting in the output box.

# Define a function greet_user that takes a sender argument that represents the object that triggers the event, such as a button 
def greet_user(sender): 
   # Use the with statement and the output object to capture the output of the print function so that it appears in the expected output area 
   # output is the output object that has been defined 
   with output: 
       # Use the print function to output a greeting where the format method is used to insert the current value of the text control into the string 
       # "{}" is a placeholder that the format function replaces with the value of text.value 
       print("Hello, {}".format(text.value))

# Use the on_click method to associate the button's click event with the greet_user function 
# When the user clicks the button, the greet_user function is called
button.on_click(greet_user)

Show UI interface

Finally, we put all the UI components in a layout and display them through the display function.

# Put all the components in a vertical layout 
ui = widgets. VBox([text, button, output])

# Display UI 
display(ui)

With these steps, we can build a simple UI in JupyterLab. The user can enter the content in the text box, and after clicking the button, the program will display the corresponding greeting in the output box according to the input content.

Use JSON Instructions to Control the Slave Device

This product is developed using the dual controller architecture, and the host controller sends JSON format commands to the slave device through the serial port (Jetson through the GPIO serial port). Note: This chapter is a precursor to the introduction of the JSON instruction set of the slave device, and the content is similar to the content of the previous Python chassis motion control chapter.

Advantages of JSON Data Format

JSON (JavaScript Object Notation) is a lightweight data exchange format, which has become one of the standards for data transmission on the Internet. Here are some of the advantages of JSON:

• Readable: JSON uses a text format that is easy for humans to understand and write, using key-value pairs to organize data, which makes the data easier to read and understand when it is transmitted and stored.

• Lightweight: Compared with other data formats such as XML, JSON's syntax is more concise and compact, so it is more lightweight, which can reduce the size of data transmission and the occupation of network bandwidth, and improve transmission efficiency.

• Easy to parse: JSON data structure is simple and clear, easy to parse and serialize, almost all programming languages provide JSON parsing and generation libraries, so that developers can easily process JSON data.

• Good compatibility with various languages: JSON is supported in almost all programming languages, so it is easy to exchange and communicate data in different platforms and systems.

• Support for multiple data types: JSON supports a variety of data types, including strings, numbers, booleans, arrays, and objects, etc., which makes it flexible to represent various types of data structures.

• Seamless integration with web technologies: JSON was originally developed from JavaScript, so it is very tightly integrated with web technologies, and it is very compatible with the JavaScript language, which can be easily used in web applications. Convenient to use in web applications.

Simple JSON command to control the slave device demo

In the following demo, you need to use the correct GPIO device name and use the same baud rate as the slave device (115200 by default).

Before running the following code block, you need to raise the product and keep the drive wheels all off the ground, after calling the following code block, the robot will start to move, be careful not to let the robot fall off the desktop.

from base_ctrl import BaseController 
import time

base = BaseController('/dev/ttyTHS0', 115200)

# The wheel rotates at a speed of 0.2m/s for 2 seconds and then stops 
base.send_command({"T":1,"L":0.2,"R":0.2}) 
time.sleep(2) 
base.send_command({"T":1,"L":0,"R":0})

By calling the code block above, Jetson will first send {"T":1,"L":0.2,"R":0.2} (we will explain the composition of the command in detail in the following chapter), the wheel will start to turn, and after two seconds Jetson will send {"T":1,"L":0,"R":0} This command, the wheel will stop rotating, and it should be noted here that even if you don't send the later command to stop the wheel turning, if you don't send a new command, The wheel will still stop rotating, this is because the slave device contains a heartbeat function, the heartbeat function is used when the host controller has no new instructions sent to the slave device for a long time, the slave device automatically stops the current moving instructions, and the purpose of changing the function is to avoid the host controller from crashing for some reasons and causing the slave device to continue to move.

If you want the robot to continue to move continuously, the host computer needs to send motion control instructions every 2-4 seconds.

Sub-controller JSON instruction set

In the previous chapter, we introduced a simple demo, in which we send motion control commands to the sub-controller through the host controller, and the sub-controller can receive a lot of instructions, in this chapter we will introduce these instructions.

Composition of JSON instructions

In the previous section, we sent {"T":1,"L":0.2,"R":0.2} instruction as an example, the T value in this JSON data represents the type (Type) of the instruction, the L value represents the target linear velocity of the left (LEFT) wheel, the R value represents the target linear velocity of the right (RIGHT) wheel, and the unit of linear velocity is m/s by default. The parameters of the motion control are the target linear velocity of the left and right wheels, respectively.

All subsequent JSON directives will contain a T-value that defines the type of instruction, but the specific instruction parameters will vary depending on the instruction type.

JSON instruction set

You can view the definitions of these instructions in the json_cmd.h file of our open-source sub-controller demos, or add new sub-controller features to them yourself.

Motion Control Instructions

These instructions are the most basic instructions for mobile robots, which are used for motion-related function control.

Each of the following directives consists of three parts: a case, a short introduction, and a detailed introduction.

CMD_SPEED_CTRL

• {"T":1,"L":0.5,"R":0.5}
• Set the target linear velocity of the wheels on both sides (speed closed-loop control)

L and R represent the target linear velocity of the left and right wheels respectively, the unit is m/s, the negative value is the reverse rotation of the substitution, and 0 is the stop. The value range of the target linear speed depends on the diameter of the motor/reducer/wheel used in the product, and the relevant calculation formula can be found in the open source sub-controller demo. It should be noted here that for chassis using carbon brush DC motors, when the absolute value of a given target speed is very small (but not 0), the speed of the product may fluctuate greatly during movement due to the low speed performance of carbon brush DC motors is usually poor. 

CMD_PWM_INPUT

• {"T":11,"L":164,"R":164}
• Set the PWM value of the driving wheels on both sides (speed open-loop control)

L and R represent the PWM values of the left and right wheels respectively, the range of the values is -255 ~ 255, the negative value represents the reverse, when the absolute value of the value is 255, it means that the PWM is 100%, which means to let the wheels on this side run at full power. 

CMD_ROS_CTRL

• {"T":13,"X":0.1,"Z":0.3}
• ROS control (speed closed-loop control)

 This instruction is used for the ROS host computer to control the command of chassis movement, X represents the linear speed of movement, the unit is m/s can be negative; Z stands for angular velocity of steering in rad/s and can be negative. 

CMD_SET_MOTOR_PID

• {"T":2,"P":20,"I":2000,"D":0,"L":255}
• PID controller settings

This command is used to tune the parameters of the PID controller, the PID parameter in the above JSON example is the default parameter of the product, where L stands for WINDUP_LIMITS, which is the reserved interface, This parameter is not currently used in the current product. 

OLED Display Control Command

An OLED display is installed on the product, which communicates with the ESP32 module of the host computer of the sub-controller through I2C, and the host computer can change the content displayed on the display by sending JSON commands.

CMD_OLED_CTRL

• {"T":3,"lineNum":0,"Text":"putYourTextHere"}
• Control the display display to display custom content

lineNum is the number of lines, a JSON command can change the content of a line, after the sub-controller receives a new instruction, the default OLED interface will disappear when booted, and it will be replaced by your newly added content, For the 0.91-inch OLED display used in most products, the lineNum value can be 0, 1, 2, 3, a total of four lines; Text is the text you want to display on this line, and if you have too much content in this line, it will automatically wrap, but it will also squeeze out the last line. 

CMD_OLED_DEFAULT

• {"T":-3}
• Control the display to display custom content

 Use this command to make the OLED display display display the default screen when powered on. 

Module Type

Different types of modules will be installed on the mobile chassis (none/robotic arm/pan-tilt),Use this command to tell the sub-controller the type of module currently installed,This instruction is usually automatically sent to the sub-controller when the host controller is turned on,This part will be introduced in the following chapters。

CMD_MODULE_TYPE

• {"T":4,"cmd":0}
• Set the module type

cmd value represents the type of module, there are currently three types to choose from, 0: install nothing, 1: robotic arm, 2: pan-tilt

IMU related functions

The chassis is equipped with an IMU sensor, you can get the data of the IMU sensor through the following commands, it should be noted here that the chassis information continuous feedback function (which contains the IMU information) will be turned on by default after the product is turned on, and the IMU related functions here are only necessary to use when the chassis information continuous feedback function is turned off.

CMD_GET_IMU_DATA

• {"T":126}
• Get IMU data

 Send this command to get the data of the IMU. 

CMD_CALI_IMU_STEP

• {"T":127}
• IMU Calibration (Reserved Interface)

Current product program does not need to perform calibration, this command is reserved interface. 

CMD_GET_IMU_OFFSET

• {"T":128}
• Get the offset of the current IMU (reserved interface)

 Use this command to feedback the current offset of each axis of the IMU. 

CMD_SET_IMU_OFFSET

• {"T":129,"x":-12,"y":0,"z":0}
• Set the offset of the IMU (reserved interface)

 Use this command to set the offset of each axis of the IMU, this command is a reserved interface, and the current product does not need to execute this command. 

Chassis information feedback

CMD_BASE_FEEDBACK

• {"T":130}
• Chassis information feedback

The product usually turns on chassis information feedback by default after booting, which is automatic, if the chassis information continuous feedback function is turned off, you need to obtain the chassis information at a time, you can use this command to obtain the basic information of the chassis. 

CMD_BASE_FEEDBACK_FLOW

• {"T":131,"cmd":1}
• Continuous chassis information feedback

The value of cmd is set to 1, and the function is turned on by default, and the chassis information will be continuously fed; When the value of cmd is set to 0, turn off the function, and after the function is turned off, the host computer can get the chassis information through CMD_BASE_FEEDBACK. 

CMD_FEEDBACK_FLOW_INTERVAL

• {"T":142,"cmd":0}
• Set the extra interval time for continuous feedback

The value of cmd is the extra interval that needs to be set, the unit is ms, and the frequency of chassis feedback can be adjusted through this command. 

CMD_UART_ECHO_MODE

• {"T":143,"cmd":0}
• Set the command echo mode

When the value of cmd is set to 0, turn off echo; When the value of cmd is set to 1, the echo is enabled, and when the command echo mode is enabled, the sub-controller will output the received command. 

WIFI related settings

CMD_WIFI_ON_BOOT

• {"T":401,"cmd":3}
• Set the WIFI mode when powered on.

When cmd is 0, turn off the WIFI function. 1-ap； 2-sta； 3-ap+sta。 

CMD_SET_AP

• {"T":402,"ssid":"UGV","password":"12345678"}
• Set the name and password of the AP mode. (ESP32 builds hotspot)

CMD_SET_STA

• {"T":403,"ssid":"WIFI_NAME","password":"WIFI_PASSWORD"}
• Set the name and password of the STA mode. (ESP32 goes to connect to known hotspots)

CMD_WIFI_APSTA

• {"T":404,"ap_ssid":"UGV","ap_password":"12345678","sta_ssid":"WIFI_NAME","sta_password":"WIFI_PASSWORD"}
• Set the name and password of the AP and STA modes. (AP+STA mode)

CMD_WIFI_INFO

• {"T":405}
• Get the information of the current WIFI.

CMD_WIFI_CONFIG_CREATE_BY_STATUS

• {"T":406}
• Create a new WIFI profile with the current settings.

CMD_WIFI_CONFIG_CREATE_BY_INPUT

• {"T":407,"mode":3,"ap_ssid":"UGV","ap_password":"12345678","sta_ssid":"WIFI_NAME","sta_password":"WIFI_PASSWORD"}
• Create a new WIFI profile with the settings you entered.

CMD_WIFI_STOP

• {"T":408}
• Disconnect WIFI.

12V switch and pan-tilt settings

CMD_LED_CTRL

• {"T":132,"IO4":255,"IO5":255}
• 12V switch output setting

The sub-controller of the product has two 12V switch interfaces, each with 2 interfaces for a total of four interfaces, you can set the output voltage of these interfaces through this command, when the value is 255, it is 3S battery voltage. 
By default, the product uses these interfaces to control the LED lights, and you can control the brightness of the LED lights through this command. 

CMD_GIMBAL_CTRL_SIMPLE

• {"T":133,"X":0,"Y":0,"SPD":0,"ACC":0}
• Basic Pan-tilt Control Command

This command is used to control the orientation of the pan-tilt. 
X is the horizontal direction in angle, positive values to the right, negative values to the left, and the value range is -180 to 180. 
Y is the direction of the value in angle, positive values are up, negative values are down, and the values range from -30 to 90. 
SPD is the velocity, ACC is the acceleration, and when the value is 0, it is the fastest speed/acceleration. 

CMD_GIMBAL_CTRL_MOVE

• {"T":134,"X":45,"Y":45,"SX":300,"SY":300}
• Pan-tilt Continuous Control Command

This command is used to continuously control the pan-tilt orientation. 
X is the horizontal direction in angle, positive values to the right, negative values to the left, and the value range is -180 to 180. 
Y is the direction of the value in angle, positive values are up, negative values are down, and the values range from -30 to 90. 
SX and SY are the speeds of the X and Y axes, respectively. 

CMD_GIMBAL_CTRL_STOP

• {"T":135}
• Pan-tilt Stop Command

When using the above command to make the pan-tilt move, you can use this command to stop the pan-tilt at any time. 

CMD_GIMBAL_STEADY

• {"T":137,"s":0,"y":0}
• Pan-tilt self-stabilization function

When s is 0, turn off this function when s is 1, after this function is enabled, the pan-tilt will automatically adjust the vertical angle of the pan-tilt through the IMU data, and y is the target angle between the pan-tilt and the ground (even if the pan-tilt self-stabilization function is turned on, the camera can also look up and down). 

CMD_GIMBAL_USER_CTRL

• {"T":141,"X":0,"Y":0,"SPD":300}
• Pan-tilt UI Control

This command is used to control the pan-tilt in the UI interface, the X value can be -1, 0 and 1, -1 is to turn left, 0 is to stop, and 1 is to turn right. 
The Y value can be -1, 0 and 1, -1 for downward rotation, 0 for stopping, and 1 for upward rotation. 
SPD is velocity 

Robotic Arm Control

CMD_MOVE_INIT

• {"T":100}
• The robotic arm moves to the initial attitude

 Under normal circumstances, the robotic arm will automatically rotate to the initial position when it is turned on. This directive causes process blocking. 

CMD_SINGLE_JOINT_CTRL

• {"T":101,"joint":0,"rad":0,"spd":0,"acc":10}
• Single-axis motion control of the robotic arm
• joint: joint serial number.
  • 1: BASE_JOINT the basal joint.
  • 2: SHOULDER_JOINT shoulder joint.
  • 3: ELBOW_JOINT elbow joint.
  • 4: EOAT_JOINT wrist/jaw joints.
• rad: The angle to be rotated to (displayed in radians), based on the initial position of each joint, the default angle and direction of rotation of each joint are as follows:
  • The default angle of the initial position of the BASE_JOINT is 0, and the angle rotation range is between 3.14 and -3.14. As the angle increases, the basal joint rotates to the left; When the angle is reduced, the base joint is rotated to the right.
  • The default angle of the initial position of the SHOULDER_JOINT is 0, and the angle rotation range is between 1.57 and -1.57. When the angle increases, the shoulder joint rotates forward; When the angle is reduced, the shoulder joint rotates backwards.
  • The default angle of the ELBOW_JOINT's initial position is 1.570796, and the angle rotation range is between 3.14 and -1.11. When the angle increases, the elbow rotates downward; When the angle is reduced, the elbow rotates in the opposite direction.
  • The default angle of the initial position of the EOAT_JOINT is 3.141593. The default jaw joint is the jaw joint, which has an angle rotation range between 1.08 and 3.14, and when the angle is reduced, the jaw joint opens. If the wrist joint is replaced, the angle rotation range is between 1.08 and 5.20, and when the angle increases, the wrist joint rotates downward; When the angle is reduced, the wrist joint is turned upward.
• SPD: The speed of rotation, the speed unit is steps per second, one revolution of the servo is 4096 steps, the higher the value, the faster the speed, when the speed value is 0, it rotates at the maximum speed.
• ACC: Acceleration at the beginning and end of the rotation, the smaller the value, the smoother the start and stop, the value can be 0-254, the unit is 100 steps/second^2. If set to 10, the speed is varied at a square acceleration of 1000 steps per second. When the acceleration value is 0, the maximum acceleration is followed.

CMD_JOINTS_RAD_CTRL

• {"T":102,"base":0,"shoulder":0,"elbow":1.57,"hand":1.57,"spd":0,"acc":10}
• Control the rotation of all joints of the robotic arm in the form of radian system
• base: the angle of the basic joint, the angle rotation range is shown above CMD_SINGLE_ JOINT_CTRL" command.
• shoulder: The angle of the shoulder joint.
• elbow: The angle of the elbow joint.
• Hand: The angle of the gripper/wrist joint.
• SPD: The speed of rotation, the speed unit is steps/second, the rotation of the servo is 4096 steps, the higher the value, the faster the speed, when the speed value is 0, it rotates at the maximum speed.
• ACC: Acceleration at the beginning and end of rotation, the smaller the value, the smoother the start and stop, the value can be 0-254, the unit is 100 steps/sec^2. If it is set to 10, the speed will be changed according to the square acceleration and deceleration of 1000 steps per second. When the acceleration value is 0, the maximum acceleration is followed.

CMD_SINGLE_AXIS_CTRL

• {"T":103,"axis":2,"pos":0,"spd":0.25}
• Single-axis coordinate control
• axis,1-x,2-y,3-z,4-t. Except for the T-axis, the pos parameter of the other axes is mm, and the unit of the T-axis is rad. SPD is the velocity coefficient, and the higher the value, the faster the speed.

CMD_XYZT_GOAL_CTRL

• {"T":104,"x":235,"y":0,"z":234,"t":3.14,"spd":0.25}
• Robotic arm coordinate motion control (inverse kinematics control)

 This function will cause a block

CMD_XYZT_DIRECT_CTRL

• {"T":1041,"x":235,"y":0,"z":234,"t":3.14}
• Robotic Arm Coordinate Motion Control (Inverse Kinematics Control)

 This function does not cause blocking

CMD_SERVO_RAD_FEEDBACK

• {"T":105}
• Feedback the coordinate information of the robotic arm

CMD_EOAT_HAND_CTRL

• {"T":106,"cmd":1.57,"spd":0,"acc":0}
• End joint control (radian)
• cmd: The angle to be rotated (displayed in radians). The default angle of the initial position of the EOAT_JOINT is 3.141593.
  • The default of the product is the jaw joint, the angle rotation range is between 1.08 and 3.14, when the angle is reduced, the jaw joint will be opened.
  • If the wrist joint is replaced, the angle rotation range is between 1.08 and 5.20, and the wrist joint rotates downward when the angle increases, and the wrist joint turns upward when the angle decreases.
• SPD: The speed of rotation, the speed unit is steps/second, the rotation of the servo is 4096 steps, the higher the value, the faster the speed, when the speed value is 0, it rotates at the maximum speed.
• ACC: Acceleration at the beginning and end of rotation, the smaller the value, the smoother the start and stop, the value can be 0-254, the unit is 100 steps/sec^2. If it is set to 10, the speed will be changed according to the square acceleration and deceleration of 1000 steps per second. When the acceleration value is 0, the maximum acceleration is followed

CMD_EOAT_GRAB_TORQUE

*{"T":107,"tor":200} 

• Jaw Force Control

tor's value can be up to 1000, representing 100% strength. 

CMD_SET_JOINT_PID

• {"T":108,"joint":3,"p":16,"i":0}
• Joint PID settings

CMD_RESET_PID

• {"T":109}
• Reset the joint PID

CMD_SET_NEW_X

• {"T":110,"xAxisAngle":0}
• Set the new X-axis direction

CMD_DYNAMIC_ADAPTATION

• {"T":112,"mode":0,"b":1000,"s":1000,"e":1000,"h":1000}
• Adaptive control of dynamic external forces

Other settings

CMD_HEART_BEAT_SET

• {"T":136,"cmd":3000}
• Set the heartbeat function time

cmudd unit is ms, you can use this command to set the heartbeat function time, if the sub-controller does not receive a new exercise command within the time, it will automatically stop moving, It is used to avoid the host controller crashing during use, causing the sub-controller to move all the time. During the process, the host controller crashed, causing the sub-controller to keep moving. 

CMD_SET_SPD_RATE

• {"T":138,"L":1,"R":1}
• Set the left and right speed ratio

The product uses the principle of differential steering, when the left and right wheels of the product give the same target speed, the product may not go straight due to encoder error or tire grip error, you can use this command to fine-tune the speed of the left and right wheels, for example, the left wheel needs to rotate slower, you can change the value of L to 0.98. Try not to set values greater than one for L and R values. 

CMD_GET_SPD_RATE

• {"T":139}
• Get the current velocity ratio

Use this command to get the current velocity ratio

ESP-NOW related settings

CMD_BROADCAST_FOLLOWER

• {"T":300,"mode":1}
• {"T":300,"mode":0,"mac":"CC:DB:A7:5B:E4:1C"}
• Set the mode in which ESP-NOW is controlled by broadcast
  • When mode is set to 1, other devices can be controlled by broadcast commands; When the mode is 0, it can only be controlled by the device with the MAC address.

CMD_GET_MAC_ADDRESS

• {"T":302}
• Get the MAC address of the current device

CMD_ESP_NOW_ADD_FOLLOWER

• {"T":303,"mac":"FF:FF:FF:FF"}
• Add MAC address to the controlled device (PEER)

CMD_ESP_NOW_REMOVE_FOLLOWER

• {"T":304,"mac":"FF:FF:FF:FF:FF:FF:FF"}
• Remove MAC address from PEER

CMD_ESP_NOW_GROUP_CTRL

• {"T":305,"dev":0,"b":0,"s":0,"e":1.57,"h":1.57,"cmd":0,"megs":"hello!"}
• ESP-NOW multicast control

CMD_ESP_NOW_SINGLE

• {"T":306,"mac":"FF:FF:FF","dev":0,"b":0,"s":0,"e":1.57,"h":1.57,"cmd":0,"megs":"hello!"}
• ESP-NOW unicast/multicast control

Task file related functions

This function belongs to the high-level functions of the sub-controller, and the following operations are usually not required when used with the host controller.

CMD_SCAN_FILES

• {"T":200}
• Scan the current task file

CMD_CREATE_FILE

• {"T":201,"name":"file.txt","content":"inputContentHere."}
• Create a task file

CMD_READ_FILE

• {"T":202,"name":"file.txt"}
• Read the task file

CMD_DELETE_FILE

• {"T":203,"name":"file.txt"}
• Delete the task file

CMD_APPEND_LINE

• {"T":204,"name":"file.txt","content":"inputContentHere."}
• Add a new command at the end of the task file

CMD_INSERT_LINE

• {"T":205,"name":"file.txt","lineNum":3,"content":"content"}
• Inserts a new command in the middle of the task file

CMD_REPLACE_LINE

• {"T":206,"name":"file.txt","lineNum":3,"content":"Content"}
• Replace a command in the task file

Bus Servo Settings

CMD_SET_SERVO_ID

• {"T":501,"raw":1,"new":11}
• Change the servo ID

raw is the original ID of the servo (the new servos are all 1), new is the ID to be changed, the maximum is not more than 254, it cannot be negative, and 255 is the broadcast ID. 

CMD_SET_MIDDLE

• {"T":502,"id":11}
• Set the current position of the servo to the servo center (only ST series servos are valid).

CMD_SET_SERVO_PID

• {"T":503,"id":14,"p":16}
• Set the P value of the servo PID.

ESP32 Related Functions

CMD_REBOOT

• {"T":600}
• Restart.

CMD_FREE_FLASH_SPACE

• {"T":601}
• GET THE AMOUNT OF SPACE LEFT FOR FLASH.

CMD_BOOT_MISSION_INFO

• {"T":602}
• Outputs the current boot task file.

CMD_RESET_BOOT_MISSION

• {"T":603}
• Reset the boot task file.

CMD_NVS_CLEAR

• {"T":604}
• Clean up the NVS area of ESP32, if the WIFI connection cannot be successful, you can try to call this command and then restart.

CMD_INFO_PRINT

• {"T":605,"cmd":1}
• Set the feedback mode.
• When cmd is set to 1, the debugging information is printed. 2. Continuous chassis information feedback; 0, nothing feedback.

Automatically send instructions when booted

This chapter tutorial is used to introduce the host controller will automatically execute instructions and send some instructions to the sub-controller every time it is turned on, the code blocks in this chapter of the tutorial do not need to be executed (and cannot be executed), only used to understand some automatic operations of the product after booting, if you have the need, change or add these instructions.

cmd_on_boot() function

cmd_on_boot() function is located in the main program app.py of the product, this function will be called every time you boot up, you can edit this function to adjust the parameters/add instructions to the instructions that run automatically at boot.

def cmd_on_boot(): 
      # Define the list of commands to be executed at startup 
      cmd_list = [ 
             'base -c {"T":142,"cmd":50}', # set feedback interval 
             'base -c {"T":131,"cmd":1}', # serial feedback flow on 
             'base -c {"T":143,"cmd":0}', # serial echo off 
             'base -c {"T":4,"cmd":2}', # select the module - 0:None 1:RoArm-M2-S 2:Gimbal 
             'base -c {"T":300,"mode":0,"mac":"EF:EF:EF:EF:EF:EF"}', # the base won't be ctrl by esp-now broadcast cmd, but it can still recv broadcast megs. 
             'send -a -b' # add broadcast mac addr to peer 
      ] 
     # traversal command list 
     for i in range(0, len(cmd_list)): 
          camera.cmd_process(cmd_list[i])

The host controller of the product can control some functions through the command line instruction, similar to the base -c instruction above, which is used to directly pass the JSON instruction written later to the sub-controller through Jetson's GPIO serial port, and we will explain in detail what the default instruction here means to run automatically at boot.

• base -c {"T":142,"cmd":50}

Used to set the extra interval time of continuous feedback information from the sub-controller, the unit of cmd value is ms, this function is used to reduce the frequency of feedback information from the sub-controller, and the purpose is to reduce the computing power pressure of the host controller to process the feedback information of the sub-controller.

• base -c {"T":131,"cmd":1}

Turn on the continuous information feedback function of the sub-controller, after the function is turned on, there is no need for the host controller to ask and answer to get the information of the sub-controller, the sub-controller will normally turn on the function by default, but we still send another command to turn on the function, which is safer.

• base -c {"T":143,"cmd":0}

Turn off the serial port command echo, so that when the host computer sends instructions to the sub-controller, the sub-controller will no longer feed back the received instructions to the host computer, so as to avoid the host computer from processing useless information.

• base -c {"T":4,"cmd":2}

Set the type of the external module, if the value of cmd is 0, it means that there is no external module; 1. a mechanical arm; 2. Gimbal, if your product does not have a gimbal or robotic arm installed, you need to change the value here to 0.

• base -c {"T":300,"mode":0,"mac":"EF:EF:EF:EF:EF:EF:EF"}

Avoid the chassis being controlled by the ESP-NOW broadcast of other devices, but you can make up a MAC address by yourself in addition to the device with the mac address, or you can use the MAC address of your own ESP32 remote control.

• send -a -b

Add the broadcast address (FF:FF) to the peer, so that you can then send the broadcast message directly to other devices through the broadcast signal.

You can learn about other host controller command line instructions through the WEB Command Line Application chapter later.

Play Audio Files

For security reasons, you don't have direct access to audio devices through JupyterLab (environmental limitations), and we don't have a code block for users to run.

The program here comes from the audio_ctrl.py of the main program of the product, and you can refer to the code here to understand how the main program of the product implements the audio file playback function.

Functions related to audio playback in the main program of the product

There is a folder called sounds in the main folder of the product, and there are many subfolders in this folder: connected, others, recv_new_cmd, robot_started, searching_for_target, target_detected, target_locked.

In the default program we provide, there is only one audio file in connected and one in each robot_started.

When the main program of the robot is running, it will automatically play an audio file within a robot_started.

When a client uses a browser to connect to this WEB application, it will automatically randomly play an audio file within the connected.

You can put custom audio files in these folders as voice packs to customize your product.

import pygame # import pygame library for audio playback 
import random # import random library for random selection of audio files 
import yaml # import yaml library for reading configuration files 
import os # import os library for file operations 
import threading # Import the threading library for multithreading

# Get the configuration file 
curpath = os.path.realpath(__file__) # Get the absolute path of the current script 
thisPath = os.path.dirname(curpath) # Get the directory where the current script is located 
with open(thisPath + '/config.yaml', 'r') as yaml_file: # Open the configuration file 
    config = yaml.safe_load(yaml_file) # Load configuration files using yaml library 

# Initialize pygame.mixer and set the default volume for audio output 
pygame.mixer.init() 
pygame.mixer.music.set_volume(config['audio_config']['default_volume'])

# Create an event object that controls audio playback 
play_audio_event = threading. Event()

# Read the minimum playback interval from the config file 
min_time_bewteen_play = config['audio_config']['min_time_bewteen_play']

# Define the function to play audio 
def play_audio(input_audio_file): 
    try: 
       pygame.mixer.music.load(input_audio_file) # Load the audio file 
       pygame.mixer.music.play() # Play audio 
    except: 
       play_audio_event.clear() # Clear the event flag when an error occurs 
       return 
    while pygame.mixer.music.get_busy(): # Wait for audio playback to complete 
       pass 
    time.sleep(min_time_bewteen_play) # Wait for minimum playback interval
    play_audio_event.clear() # Clear event

# Define the function to play random audio 
def play_random_audio(input_dirname, force_flag): 
    if play_audio_event.is_set() and not force_flag: 
       return 
    # Get all audio files in the specified directory 
    audio_files = [f for f in os.listdir(current_ path + "/sounds/" + input_dirname) if f.endswith((".mp3", ".wav")))] 
    # Randomly select an audio file from the list of audio files 
    audio_file = random.choice(audio_files) 
    play_audio_event.set() # Set up the event 
    # Create a thread to play the audio 
    audio_ thread = threading. Thread(target=play_audio, args=(current_path + "/sounds/" + input_dirname + "/" + audio_file,)) 
    audio_thread.start() # Start the thread

# Define the thread function to play audio 
def play_audio_thread(input_file): 
    if play_audio_event.is_set(): # Return if the event has already been set
       return 
    play_audio_event.set()  # Set the event 
    # Create a thread to play audio 
    audio_thread = threading. Thread(target=play_audio, args=(input_file,)) 
    audio_thread.start() # Start the thread

# Define the function to play the specified file 
def play_file(audio_file): 
    audio_file = current_path + "/sounds/" + audio_file # Build the full path of the audio file 
    play_audio_thread(audio_file) # Play the audio in a new thread

# Define the function to set the audio volume 
def set_audio_volume(input_volume): 
    input_volume = float(input_volume) # Convert the input volume to floating-point number 
    if input_volume > 1: # If the volume is greater than 1, set it to 1 
       input_volume = 1 
    elif input_volume < 0: # If the volume is less than 0, set to 0 
       input_volume = 0 
    pygame.mixer.music.set_volume(input_volume) # Set the volume

# Define a function to set the minimum playback interval 
def set_min_time_between(input_time): 
    global min_time_bewteen_play # Use the global variable 
    min_time_bewteen_play = input_time # Set the minimum playback interval

Text-to-speech (TTS)

For security reasons, you don't have direct access to audio devices through JupyterLab (environmental limitations), and we don't have code blocks here for users to run.

The program here comes from the audio_ctrl.py of the main program of the product, and you can refer to the code here to understand how the main program of the product performs the text-to-speech function.

import pyttsx3 # import pyttsx3 library for text-to-speech functionality 
import threading # import threading module for creating threads

# Initialize pyttsx3 engine 
engine = pyttsx3.init()

# Create an event object to control the synchronization of voice playback 
play_audio_event = threading. Event()

# Set the engine property, here is the speed of voice playback, the higher the value, the faster the speed of speech 
engine.setProperty('rate', 180)

# Define a function to play the speech 
def play_speech(input_text): 
    engine.say(input_text) # Enter the text into the engine 
    engine.runAndWait() # Wait for the voice output to complete 
    play_audio_event.clear() # Clear the event to indicate that the voice playback is complete

# Define a function to play a voice in a new thread 
def play_speech_thread(input_text): 
    if play_audio_event.is_set(): # If there is already a voice in playback, return it directly without repeating it 
       return 
    play_audio_event.set() # Set an event to indicate that a new voice playback task starts 
    # Create a new thread and call play_ speech function to play speech 
    speech_thread = threading. Thread(target=play_speech, args=(input_text,)) 
    speech_thread.start() # Start a new thread and start playing voice

This code uses the pyttsx3 library to implement text-to-speech functionality, and uses the threading module to create a thread to play speech asynchronously. The play_speech() function is used to play the speech of the specified text in the main thread, while the play_speech_thread() function is used to play the voice in the new thread to avoid blocking the main thread. At the same time, the synchronization of voice playback is controlled by play_audio_event to ensure that only one voice is playing at the same time.

Use Flask to achieve low-latency video transmission

This section introduces how to use Flask to build a web application to display the real-time picture of the robot camera, because the web application has the cross-platform feature, users can watch the real-time picture of the camera through the browser on mobile phones/PCs/tablets and other devices, and realize the wireless video transmission function.

What is Flask?

Flask is a lightweight web application framework for building web applications quickly using Python.

• Lightweight: Flask is a lightweight framework with a relatively small core library, but it is flexible and extensible enough for developers to choose to add as many extensions and libraries as they need.
• Simple and easy to use: Flask has a simple design and is easy to use. Its API is clear and well-documented, allowing developers to get started and build web applications quickly.
• Routing system: Flask uses decorators to define URL routes, mapping requests to the appropriate handlers. This makes it intuitive and simple to create different pages and handle different requests.
• Templating Engine: Flask integrates with the Jinja2 templating engine, making it easier to build dynamic content in your app. The templating engine allows you to embed dynamically generated content in HTML.
• IDV Server: Flask comes with a simple IDV server for easy development and debugging. However, in a production environment, it is advisable to use a more powerful web server such as Gunicorn or uWSGI.
• Plugins and extensions: Flask supports many plugins and extensions in order to add extra functionality such as database integration, authentication, form processing, and more.
• RESTful support: Flask provides good support for RESTful APIs, making it easy to build and design RESTful APIs.
• WSGI compatible: Flask is based on WSGI (Web Server Gateway Interface), which allows it to run on many WSGI compliant web servers.
• Active community: Flask has a large and active community, which means you can easily find tons of documentation, tutorials, and third-party extensions, as well as get support.

Preparation

Since the main program will automatically run by default when the product is booted up, the main program will occupy the camera resources, in this case, this tutorial cannot be used, you need to end the main program or prohibit the main program from running automatically before restarting the robot.

It should be noted here that the regular sudo killall python method usually does not work due to the multi-threading used in the main bot program and the automatic operation is configured by crontab, so we will introduce the method of disabling the automatic running of the main program.

If you have disabled the automatic startup of the bot's main program, you do not need to perform the End Main Program section below.

End Main Program

1. Clicking on the + sign next to the tab on this page above will open a new tab called Launcher.

2. Click Terminal in Other to open the Terminal window.

3. Type bash in the terminal window and press enter.

4. Now you can use the Bash Shell to control the robot.

5. Enter the command: sudo killall -9 python

Web Application demo

Note that the following code block cannot be run in Jupyter Lab

The following code cannot be run in Jupyter Lab because the Flask application conflicts with Jupyter Lab over the use of port numbers, and the following programs are stored in folders named 12 in tutorial_cn and tutorial_en, and there is also a file named in folder 12 template's folder is used to store web page resources, and here's how the demo runs.

1. Use the method described above to open the terminal, at this time pay attention to the folder path on the left, the default path of the newly opened terminal is the same as the file path on the left, you need to browse to the tutorial_cn or tutorial_en folder, open the terminal and enter cd 12 to browse to the 12 folder.

2. Use the following command to start the Flask web application server: python flask_camera.py

3. Then open the browser on the device in the same LAN (or this device can open a new tab in the browser), enter the IP of Jetson: 5000 (for example, if the IP address of Jetson is 192.168.10.104, open the address 192.168.10.104:5000), note that : needs to be a colon in English.

4. Use Ctrl + C in the terminal to end the run.

Introduction to Flask program

from flask import Flask, render_template, Response  # Import the Flask class from the flask library, the render_template function is used to render the HTML template, and the Response class is used to generate the response object
from picamera2 import Picamera2 # Import Picamera2 class from picamera2 library for accessing and controlling the camera 
import time # Import time module, which can be used to handle time-related tasks 
import cv2 # Import OpenCV library for image processing

app = Flask(__name__) # Create an instance of the Flask application

def gen_frames(): # Define a generator function that generates the image captured by the camera on a frame-by-frame basis 
    picam2 = Picamera2() # Create an instance of Picamera2

    # Configure camera parameters and set the format and size of the video 
    picam2.configure(picam2.create_video_configuration(main={"format": 'XRGB8888', "size": (640, 480)}))

    picam2.start() # Start the camera 
    while True: 
        frame = picam2.capture_array() # Capture a frame from the camera 
        frame = cv2.cvtColor(frame, cv2. COLOR_BGR2RGB) 
        ret, buffer = cv2.imencode('.jpg', frame) # Encodes the captured image frame to JPEG format

        frame = buffer.tobytes() # Convert JPEG images to byte streams

        # Use yield to return an image byte stream, so that you can send video frames continuously to form a video stream 
        yield (b'--frame\r\n' 
            b'Content-Type: image/jpeg\r\n\r\n' + frame + b'\r\n')

@app.route('/') # Define the root route 
def index(): 
    return render_template('index.html') # Return index.html page

@app.route('/video_feed') # Define the video stream route 
def video_feed(): 
    # Returns the response object, the content is the video stream, and the content type is multipart/x-mixed-replace 
    return Response(gen_frames(), mimetype='multipart/x-mixed-replace; boundary=frame')

if __name__ == '__main__': 
    app.run(host='0.0.0.0', port=5000, debug=True) # Start the Flask application, listen on port 5000 on all network interfaces, and enable debug mode

Descriptions of some key parts of the code

gen_frames(): This is a generator function that continuously captures frames from the camera, encodes them into JPEG format, and generates frame bytes as part of a multi-part response. The resulting frames are transmitted to the client in real time.

@app.route('/'): This decorator associates the index() function with the root URL(/). When a user accesses the root URL, it will render an HTML template named '12_index.html'.

@app.route('/video_feed'): This decorator associates the video_feed() function with the '/video_feed' URL. This route is used for real-time video transmission, and frames are sent as multipart responses.

app.run(host='0.0.0.0', port=5000, debug=True): This line starts the Flask development server and listens for all available network interfaces (0.0.0.0) on port 5000. debug=True option to enable the server's debug mode. The debug mode of the server.

Introduction to the Web Section

Notes: 
<!doctype html>: Declares the HTML document type. 
<html lang="en">: The root element of the HTML document, specifying that the page language is English. 
<head>: Contains meta information about the document, such as character set and page title. 
<-- Required meta tags -->: HTML annotation, reminding that these are some required meta tags. 
<meta charset="utf-8">: Specifies that the document uses the UTF-8 character set. 
<title>Live Video Based on Flask</title>: Set page title. 
<body>: Contains the visible part of the document. 
<-- The image tag below is dynamically updated with the video feed from Flask -->: HTML annotation stating that the image tag below dynamically updates to show the video stream from Flask. 
<img src="{{ url_for('video_feed') }}">: Image label that uses the video_feed route defined in Flask to get a live video stream.

In the previous chapter, we used Flask to display the real-time camera feed, a method that required opening a new tab in the browser or accessing it from another device. In this chapter, we'll explore a solution for viewing the real-time video stream directly in Jupyter Lab.

Preparation

Since the product automatically runs the main program at startup, which occupies the camera resource, this tutorial cannot be used in such situations. You need to terminate the main program or disable its automatic startup before restarting the robot.
It's worth noting that because the robot's main program uses multi-threading and is configured to run automatically at startup through crontab, the usual method sudo killall python typically doesn't work. Therefore, we'll introduce the method of disabling the automatic startup of the main program here.
If you have already disabled the automatic startup of the robot's main demo, you do not need to proceed with the section on Terminate the Main Demo.

Terminate the Main Demo

1. Click the + icon next to the tab for this page to open a new tab called "Launcher."

2. Click on Terminal under Other to open a terminal window.

3. Type bash into the terminal window and press Enter.

4. Now you can use the Bash Shell to control the robot.

5. Enter the command: sudo killall -9 python.

Demo

The following code block can be run directly:

1. Select the code block below.

2. Press Shift + Enter to run the code block.

3. Watch the real-time video window.

4. Press STOP to close the real-time video and release the camera resources.

If you cannot see the real-time camera feed when running

• Click on Kernel -> Shut down all kernels above.
• Close the current section tab and open it again.
• Click STOP to release the camera resources, then run the code block again.
• Reboot the device.

import matplotlib.pyplot as plt  # Import the matplotlib library for plotting
import cv2  # Import the OpenCV library for image processing
from picamera2 import Picamera2  # Import the Picamera2 library for accessing the Raspberry Pi Camera
import numpy as np  # Import the NumPy library for mathematical computations
from IPython.display import display, Image  # Import IPython display functionality
import ipywidgets as widgets  # Import the ipywidgets library for creating interactive widgets
import threading  # Import the threading library for multithreading

# Create a toggle button as a stop button
stopButton = widgets.ToggleButton(
    value=False,  # The initial state of the button is unselected
    description='Stop',  # Text displayed on the button
    disabled=False,  # The button is initially enabled
    button_style='danger',  # The button style is red
    tooltip='Description',  # Tooltip displayed when hovering over the button
    icon='square'  # Icon displayed on the button
)

# Define a function for displaying the video stream
def view(button):
    # If you are using a CSI camera you need to comment out the picam2 code and the camera code.
    # Since the latest versions of OpenCV no longer support CSI cameras (4.9.0.80), you need to use picamera2 to get the camera footage
    # picam2 = Picamera2()  # Create an instance of Picamera2
    # Configure camera parameters, set video format and size
    # picam2.configure(picam2.create_video_configuration(main={"format": 'XRGB8888', "size": (640, 480)}))
    # picam2.start()  # Start the camera
    camera = cv2.VideoCapture(-1) #Create camera example
    #Set resolution
    camera.set(cv2.CAP_PROP_FRAME_WIDTH, 640)
    camera.set(cv2.CAP_PROP_FRAME_HEIGHT, 480)
    
    display_handle=display(None, display_id=True)  # Create a display handle for updating the displayed content
    while True:
        # frame = picam2.capture_array()  # Capture a frame from the camera
        _, frame = camera.read()
        
        # You can perform frame processing here if needed (e.g., flipping, color conversion, etc.)

        _, frame = cv2.imencode('.jpeg', frame)  # Encode the frame as JPEG format
        display_handle.update(Image(data=frame.tobytes()))  # Update the displayed image
        if stopButton.value==True:  # Check if the stop button is pressed
            #picam2.close()  # If yes, close the camera
            cv2.release() # if yes, close the camera
            display_handle.update(None)  # Clear the displayed content

# Display the stop button
display(stopButton)
# Create and start a thread, with the target function as view and the stop button as the argument
thread = threading.Thread(target=view, args=(stopButton,))
thread.start()  # Start the thread

This chapter builds upon the previous tutorial, capturing frames from the camera at regular intervals and saving them in the /templates/pictures/ folder within the ugv_jetson directory.

Preparation

Since the product automatically runs the main program at startup, which occupies the camera resource, this tutorial cannot be used in such situations. You need to terminate the main program or disable its automatic startup before restarting the robot.
It's worth noting that because the robot's main program uses multi-threading and is configured to run automatically at startup through crontab, the usual method sudo killall python typically doesn't work. Therefore, we'll introduce the method of disabling the automatic startup of the main program here.
If you have already disabled the automatic startup of the robot's main demo, you do not need to proceed with the section on Terminate the Main Demo.

Terminate the Main Demo

1. Click the + icon next to the tab for this page to open a new tab called "Launcher."

2. Click on "Terminal" under "Other" to open a terminal window.

3. Type bash into the terminal window and press Enter.

4. Now you can use the Bash Shell to control the robot.

5. Enter the command: sudo killall -9 python.

Demo

The following code block can be run directly:

1. Select the code block below.

2. Press Shift + Enter to run the code block.

3. Watch the real-time video window.

4. Press STOP to close the real-time video and release the camera resources.

If you cannot see the real-time camera feed when running:

• Click on Kernel -> Shut down all kernels above.
• Close the current section tab and open it again.
• Click STOP to release the camera resources, then run the code block again.
• Reboot the device.

Notes

If you are using a USB camera, you need to uncomment the line frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB).

Differences from the Previous Chapter

You can adjust the value of time_intervel to change the interval between photos, measured in seconds.
The photos you capture will be stored in the `/ugv_jetson/templates/pictures/` folder..

import cv2  # Import the OpenCV library for image processing
from picamera2 import Picamera2  # Import the Picamera2 library to access the Raspberry Pi Camera
import numpy as np
from IPython.display import display, Image  # Import IPython display functionality
import ipywidgets as widgets  # Import the ipywidgets library for creating cloud interactive widgets
import threading  # Import the threading library for multithreading

import os, time  # Import the os and time libraries for file operations and time-related functionalities

# Change the interval time for taking photos here (in seconds)
time_intervel = 3  # Take a photo every 3 seconds

# Set the path for saving the images
# You can change the save path here
photo_path = '/home/ws/ugv_pt_rpi/static/'

# Create a toggle button as a stop button
# ================
stopButton = widgets.ToggleButton(
    value=False,  # The initial state of the button is unselected
    description='Stop',  # Text displayed on the button
    disabled=False,  # The button is initially enabled
    button_style='danger',  # 'success', 'info', 'warning', 'danger' or ''
    tooltip='Description',  # Tooltip displayed when hovering over the button
    icon='square'  # Button icon (FontAwesome name without the `fa-` prefix)
)


# Define a function for displaying the video stream and taking photos at regular intervals
# ================
def view(button):
    last_picture_time = time.time()  # Record the time of the last photo taken
    num_count = 0  # Initialize the photo counter
    # If you are using a CSI camera you need to comment out the picam2 code and the camera code
    # Since the latest versions of OpenCV no longer support CSI cameras (4.9.0.80), you need to use picamera2 to get the camera footage
    # picam2 = Picamera2()  # Create an instance of Picamera2
    # Configure camera parameters, set video format and size
    # picam2.configure(picam2.create_video_configuration(main={"format": 'XRGB8888', "size": (640, 480)}))
    # picam2.start()  # Start the camera
    
    camera = cv2.VideoCapture(-1) #Create camera example
    #Set resolution
    camera.set(cv2.CAP_PROP_FRAME_WIDTH, 640)
    camera.set(cv2.CAP_PROP_FRAME_HEIGHT, 480)
    
    display_handle=display(None, display_id=True)  # Create a display handle for updating the displayed content
    
    i = 0
    while True:
        # frame = picam2.capture_array()
        # frame = cv2.flip(frame, 1) # flip the image
        _, frame = camera.read() #Capturing a frame from the camera


        # Take a photo every few seconds
        if time.time() - last_picture_time >= time_intervel:
            num_count = num_count + 1  # Update the photo counter
            photo_filename = f'{photo_path}photo_{num_count}.jpg'  # Define the file name for the photo
            cv2.imwrite(photo_filename, frame)  # Save the photo to the specified path
            last_picture_time = time.time()  # Update the time of the last photo taken
            print(f'{num_count} photos saved. new photo: {photo_filename}')  # Print information about the saved photo
            
        _, frame = cv2.imencode('.jpeg', frame)  # Encode the frame as JPEG format
        display_handle.update(Image(data=frame.tobytes()))  # Update the displayed image
        if stopButton.value==True:  # Check if the stop button is pressed
            #picam2.close()  # If yes, close the camera
            cv2.release() # If yes, close the camera
            display_handle.update(None)

            
# Display the stop button and start the video stream display thread
# ================
display(stopButton)
thread = threading.Thread(target=view, args=(stopButton,))
thread.start()

This tutorial utilizes OpenCV to detect changes in the scene. You can set a threshold for how much change is detected, and adjusting this threshold allows you to modify the sensitivity of the motion detection.

This chapter requires an understanding of the preceding chapters.

Preparation

Since the product automatically runs the main program at startup, which occupies the camera resource, this tutorial cannot be used in such situations. You need to terminate the main program or disable its automatic startup before restarting the robot.
It's worth noting that because the robot's main program uses multi-threading and is configured to run automatically at startup through crontab, the usual method sudo killall python typically doesn't work. Therefore, we'll introduce the method of disabling the automatic startup of the main program here.
If you have already disabled the automatic startup of the robot's main demo, you do not need to proceed with the section on Terminate the Main Demo.

Terminate the Main Demo

1. Click the + icon next to the tab for this page to open a new tab called "Launcher."

2. Click on Terminal under Other to open a terminal window.

3. Type bash into the terminal window and press Enter.

4. Now you can use the Bash Shell to control the robot.

5. Enter the command: sudo killall -9 python.

Example

The following code block can be run directly:

1. Select the code block below.

2. Press Shift + Enter to run the code block.

3. Watch the real-time video window.

4. Press STOP to close the real-time video and release the camera resources.

If you cannot see the real-time camera feed when running:

• Click on Kernel -> Shut down all kernels above.
• Close the current section tab and open it again.
• Click STOP to release the camera resources, then run the code block again.
• Reboot the device.

Notes

If you are using a USB camera, you need to uncomment the line frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB).

Features of This Chapter

You need to adjust some parameters to increase the threshold (sensitivity) of OpenCV for detecting changes in the scene. The lower the threshold value, the more sensitive OpenCV is to changes in the scene.

Running

When you run the code block, you can see the real-time feed from the camera. You can wave your hand in front of the camera, and the program will automatically outline the areas of change with green boxes.

import cv2
from picamera2 import Picamera2
import numpy as np
from IPython.display import display, Image
import ipywidgets as widgets
import threading

import imutils  # Library for simplifying image processing tasks

threshold = 2000  # Set the threshold for motion detection

# Create a "Stop" button to control the process
# ===================================================
stopButton = widgets.ToggleButton(
    value=False,
    description='Stop',
    disabled=False,
    button_style='danger',  # 'success', 'info', 'warning', 'danger' or ''
    tooltip='Description',
    icon='square'  # Button icon (FontAwesome name without the `fa-` prefix)
)


# Display function definition, used to capture and process video frames, while performing motion detection
# ===================================================
def view(button):
    # If you are using a CSI camera you need to comment out the picam2 code and the camera code
    # Since the latest versions of OpenCV no longer support CSI cameras (4.9.0.80), you need to use picamera2 to get the camera footage
    # picam2 = Picamera2()  # Create a Picamera2 instance
    # picam2.configure(picam2.create_video_configuration(main={"format": 'XRGB8888', "size": (640, 480)}))  # Configure camera parameters
    # picam2.start()  # Start the camera
    
    camera = cv2.VideoCapture(-1)  #Create camera example
    #Set resolution
    camera.set(cv2.CAP_PROP_FRAME_WIDTH, 640)
    camera.set(cv2.CAP_PROP_FRAME_HEIGHT, 480)
    
    display_handle = display(None, display_id=True)
    i = 0

    avg = None  # Used to store the average frame

    while True:
        # frame = picam2.capture_array()  # Capture a frame from the camera
        # frame = cv2.flip(frame, 1) # if your camera reverses your image
        _, frame = camera.read() # Capture a frame image from camera 

        img = cv2.cvtColor(frame, cv2.COLOR_RGB2BGR)  # Convert frame color from RGB to BGR
        gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)  # Convert the frame to grayscale
        gray = cv2.GaussianBlur(gray, (21, 21), 0)  # Apply Gaussian blur to the grayscale image
        if avg is None:  # If the average frame does not exist, create it
            avg = gray.copy().astype("float")
            continue

        try:
            cv2.accumulateWeighted(gray, avg, 0.5)  # Update the average frame
        except:
            continue

        frameDelta = cv2.absdiff(gray, cv2.convertScaleAbs(avg))  # Calculate the difference between the current frame and the average frame

        # Apply a threshold to find contours in the difference image
        thresh = cv2.threshold(frameDelta, 5, 255, cv2.THRESH_BINARY)[1]
        thresh = cv2.dilate(thresh, None, iterations=2)
        cnts = cv2.findContours(thresh.copy(), cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
        cnts = imutils.grab_contours(cnts)
        # Iterate through contours
        for c in cnts:
            # Ignore contours that are too small
            if cv2.contourArea(c) < threshold:
                continue
            # Calculate the bounding box of the contour and draw a rectangle around it
            (mov_x, mov_y, mov_w, mov_h) = cv2.boundingRect(c)
            cv2.rectangle(frame, (mov_x, mov_y), (mov_x + mov_w, mov_y + mov_h), (128, 255, 0), 1)  # Draw a rectangle around the moving area

        _, frame = cv2.imencode('.jpeg', frame)  # Encode the processed frame in JPEG format
        display_handle.update(Image(data=frame.tobytes()))  # Update the displayed image
        if stopButton.value == True:  # Check if the "Stop" button is pressed
            #picam2.close()  # If yes, close the camera
            cv2.release() # if yes, close the camera
            display_handle.update(None)  # Clear the displayed image


# Display the stop button and start the video stream display thread
# ===================================================
display(stopButton)
thread = threading.Thread(target=view, args=(stopButton,))
thread.start()

This tutorial teaches how to control the camera for taking photos and recording videos by adding buttons to the page. Similar to previous tutorials, images are by default saved in the /ugv_jetson/templates/pictures/ folder, and videos are saved in the /ugv_jetson/templates/videos folder.

Preparation

Since the product automatically runs the main program at startup, which occupies the camera resource, this tutorial cannot be used in such situations. You need to terminate the main program or disable its automatic startup before restarting the robot.
It's worth noting that because the robot's main demo uses multi-threading and is configured to run automatically at startup through crontab, the usual method sudo killall python typically doesn't work. Therefore, we'll introduce the method of disabling the automatic startup of the main program here.
If you have already disabled the automatic startup of the robot's main demo, you do not need to proceed with the section on Terminate the Main Demo.

Terminate the Main Demo

1. Click the + icon next to the tab for this page to open a new tab called "Launcher."

2. Click on Terminal under Other to open a terminal window.

3. Type bash into the terminal window and press Enter.

4. Now you can use the Bash Shell to control the robot.

5. Enter the command: sudo killall -9 python.

Demo

The following code block can be run directly:

1. Select the code block below.

2. Press Shift + Enter to run the code block.

3. Watch the real-time video window.

4. Press STOP to close the real-time video and release the camera resources.

If you cannot see the real-time camera feed when running:

• Click on Kernel -> Shut down all kernels above.
• Close the current section tab and open it again.
• Click STOP to release the camera resources, then run the code block again.
• Reboot the device.

Notes

If you are using a USB camera, you need to uncomment the line frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB).

Running

When the code block is executed, you can take a photo by clicking on PHOTO.

import cv2  # Import the OpenCV library for image processing
from picamera2 import Picamera2  # Import the library to access the Raspberry Pi Camera
import numpy as np  # Import the library for mathematical calculations
from IPython.display import display, Image  # Import to display images in Jupyter Notebook
import ipywidgets as widgets  # Import to create interactive interface widgets like buttons
import threading  # Import to create new threads for asynchronous task execution

import os, time  # Import for file and directory operations and time-related functions

time_interval = 3  # Set the time interval for taking photos (seconds)

photo_path = '/home/ws/ugv_jetson/static/'  # Set the directory path to store photos and videos

# Create a "Stop" button for users to stop video capture and photo taking
# ================
stopButton = widgets.ToggleButton(
    value=False,
    description='Stop',
    disabled=False,
    button_style='danger',  # Set button style: 'success', 'info', 'warning', 'danger' or ''
    tooltip='Description',
    icon='square'  # Set button icon (FontAwesome names without the `fa-` prefix)
)

# Create a "Photo" button for users to instantly take a photo by clicking it
# ================
photoButton = widgets.ToggleButton(
    value=False,
    description='Photo',
    disabled=False,
    button_style='danger',  # 'success', 'info', 'warning', 'danger' or ''
    tooltip='Description',
    icon='square'  # (FontAwesome names without the `fa-` prefix)
)

# Set the time interval for continuous shooting (seconds)
time_interval = 3
photo_num_count = 0  # Initialize the photo counter
capture_lock = threading.Lock()
last_photo_time = time.time()  # Record the last time a photo was taken


def photo_button_clicked(change, frame):
    global photo_num_count
    if change['new']:  # When the "Photo" button is clicked
        photo_num_count = 1  # Increment the photo counter
        photo_filename = f'{photo_path}photo_{photo_num_count}.jpg'  # Set the path and filename for saving the photo
        cv2.imwrite(photo_filename, frame)  # Save the photo
        print(f'{photo_num_count} photos saved. New photo: {photo_filename}')  # Print photo save information
        photoButton.value = False  # Reset the status of the "Photo" button


# Define a display function to capture and display video frames and respond to photo capture requests
# ================
def view(stop_button, photo_button):
    # If you are using a CSI camera you need to comment out the picam2 code and the camera code
    # Since the latest versions of OpenCV no longer support CSI cameras (4.9.0.80), you need to use picamera2 to get the camera footage
    # picam2 = Picamera2()  # Create an instance of Picamera2
    # picam2.configure(picam2.create_video_configuration(main={"format": 'XRGB8888', "size": (640, 480)}))  # Configure camera parameters
    # picam2.start()

    camera = cv2.VideoCapture(-1)  #Create camera example
    #Set resolution
    camera.set(cv2.CAP_PROP_FRAME_WIDTH, 640)
    camera.set(cv2.CAP_PROP_FRAME_HEIGHT, 480)
    
    display_handle = display(None, display_id=True)  # Create a display handle to update the displayed image
    i = 0
    while True:
        #frame = picam2.capture_array()  # Capture a frame from the camera
        _, frame = camera.read()

        photoButton.observe(lambda change: photo_button_clicked(change, frame), names='value')  # Listen for clicks on the "Photo" button

        _, frame = cv2.imencode('.jpeg', frame)  # Encode the frame as JPEG format
        display_handle.update(Image(data=frame.tobytes()))
        if stopButton.value==True:
            #picam2.close()  # If yes, close the camera
            cv2.release() # if yes, close the camera
            display_handle.update(None)


# Display the "Stop" and "Photo" buttons and start a new thread to execute the display function
# ================
display(stopButton)
display(photoButton)
thread = threading.Thread(target=view, args=(stopButton, photoButton,))
thread.start()

Here's something to note: due to stability issues with JupyterLab components used in this example, pressing the "Photo" button may result in saving multiple photos. You can navigate to /ugv_jetson/templates/pictures/ in the left sidebar of JupyterLab to view the captured photos.

This chapter introduces how to use OpenCV to compare feature databases and achieve face recognition. Although this method is not as efficient as MediaPipe's solution, it allows for the detection of other objects by replacing the feature database file.

Preparation

Since the product automatically runs the main program at startup, which occupies the camera resource, this tutorial cannot be used in such situations. You need to terminate the main program or disable its automatic startup before restarting the robot.
It's worth noting that because the robot's main program uses multi-threading and is configured to run automatically at startup through crontab, the usual method sudo killall python typically doesn't work. Therefore, we'll introduce the method of disabling the automatic startup of the main program here.
If you have already disabled the automatic startup of the robot's main demo, you do not need to proceed with the section on Terminate the Main Demo.

Terminate the Main Demo

1. Click the + icon next to the tab for this page to open a new tab called "Launcher."

2. Click on Terminal under Other to open a terminal window.

3. Type bash into the terminal window and press Enter.

4. Now you can use the Bash Shell to control the robot.

5. Enter the command: sudo killall -9 python.

Demo

The following code block can be run directly:

1. Select the code block below.

2. Press Shift + Enter to run the code block.

3. Watch the real-time video window.

4. Press STOP to close the real-time video and release the camera resources.

If you cannot see the real-time camera feed when running:

• Click on Kernel -> Shut down all kernels above.
• Close the current section tab and open it again.
• Click STOP to release the camera resources, then run the code block again.
• Reboot the device.

Features of This Chapter

The face feature database file is located in the same path as this .ipynb file. You can change the faceCascade variable to modify what needs to be detected. You'll need to replace the current haarcascade_frontalface_default.xml file with other feature files.
When the code block runs successfully, you can position the robot's camera on a face, and the area containing the face will be automatically highlighted on the screen.

import cv2  # Import the OpenCV library for image processing
from picamera2 import Picamera2  # Library for accessing the Raspberry Pi Camera
import numpy as np  # Library for mathematical calculations
from IPython.display import display, Image  # Library for displaying images in Jupyter Notebook
import ipywidgets as widgets  # Library for creating interactive widgets like buttons
import threading  # Library for creating new threads to execute tasks asynchronously

# Load the Haar cascade classifier for face detection
faceCascade = cv2.CascadeClassifier('haarcascade_frontalface_default.xml')

# Create a "Stop" button for users to stop the video stream by clicking on it
# ================
stopButton = widgets.ToggleButton(
    value=False,
    description='Stop',
    disabled=False,
    button_style='danger', # 'success', 'info', 'warning', 'danger' or ''
    tooltip='Description',
    icon='square' # (FontAwesome names without the `fa-` prefix)
)


# Define a display function to process video frames and perform face detection
# ================
def view(button):
    # If you are using a CSI camera you need to comment out the picam2 code and the camera code.
    # Since the latest versions of OpenCV no longer support CSI cameras (4.9.0.80), you need to use picamera2 to get the camera footage
    # picam2 = Picamera2()  # Create an instance of Picamera2
    # picam2.configure(picam2.create_video_configuration(main={"format": 'XRGB8888', "size": (640, 480)}))  # Configure camera parameters
    # picam2.start()  # Start the camera

    camera = cv2.VideoCapture(-1) #Create camera example 
    #Set resolution
    camera.set(cv2.CAP_PROP_FRAME_WIDTH, 640)
    camera.set(cv2.CAP_PROP_FRAME_HEIGHT, 480)
    
    display_handle=display(None, display_id=True)  # Create a display handle to update the displayed image
    i = 0
    
    avg = None
    
    while True:
        # frame = picam2.capture_array()
        _, frame = camera.read()
        # frame = cv2.flip(frame, 1) # if your camera reverses your image

        img = cv2.cvtColor(frame, cv2.COLOR_RGB2BGR)  # Convert the image from RGB to BGR because OpenCV defaults to BGR
        gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)  # Convert the image to grayscale because face detection is typically performed on grayscale images

        # Perform face detection using the cascade classifier
        faces = faceCascade.detectMultiScale(
                gray,     
                scaleFactor=1.2,
                minNeighbors=5,     
                minSize=(20, 20)
            )

        if len(faces):
            for (x,y,w,h) in faces: # Loop through all detected faces
                cv2.rectangle(frame,(x,y),(x+w,y+h),(64,128,255),1) # Draw a rectangle around the detected face
        
        _, frame = cv2.imencode('.jpeg', frame) # Encode the frame as JPEG format
        display_handle.update(Image(data=frame.tobytes()))
        if stopButton.value==True:
            #picam2.close()  # If yes, close the camera
            cv2.release() # if yes, close the camera
            display_handle.update(None)

            
# Display the "Stop" button and start a thread to execute the display function
# ================
display(stopButton)
thread = threading.Thread(target=view, args=(stopButton,))
thread.start()

This section introduces how to implement common object recognition using DNN (Deep Neural Network) + OpenCV.

Preparation

Since the product automatically runs the main program at startup, which occupies the camera resource, this tutorial cannot be used in such situations. You need to terminate the main program or disable its automatic startup before restarting the robot.
It's worth noting that because the robot's main program uses multi-threading and is configured to run automatically at startup through crontab, the usual method sudo killall python typically doesn't work. Therefore, we'll introduce the method of disabling the automatic startup of the main program here.
If you have already disabled the automatic startup of the robot's main demo, you do not need to proceed with the section on Terminate the Main Demo.

Terminate the Main Demo

1. Click the + icon next to the tab for this page to open a new tab called "Launcher."

2. Click on Terminal under Other to open a terminal window.

3. Type bash into the terminal window and press Enter.

4. Now you can use the Bash Shell to control the robot.

5. Enter the command: sudo killall -9 python.

Demo

The following code block can be run directly:

1. Select the code block below.

2. Press Shift + Enter to run the code block.

3. Watch the real-time video window.

4. Press STOP to close the real-time video and release the camera resources.

If you cannot see the real-time camera feed when running:

• Click on Kernel -> Shut down all kernels above.
• Close the current section tab and open it again.
• Click STOP to release the camera resources, then run the code block again.
• Reboot the device.

Features of This Section

The deploy.prototxt file and mobilenet_iter_73000.caffemodel file are in the same path as this .ipynb file.
When the code blocks run successfully, you can point the camera at common objects such as: "background", "aeroplane", "bicycle", "bird", "boat", "bottle", "bus", "car", "cat", "chair", "cow", "diningtable", "dog", "horse", "motorbike", "person", "pottedplant", "sheep", "sofa", "train", "tvmonitor".
The program will annotate the objects it recognizes in the image and label them with their names.

import cv2  # Import the OpenCV library for image processing
from picamera2 import Picamera2  # Library for accessing Raspberry Pi Camera
import numpy as np  # Library for mathematical calculations
from IPython.display import display, Image  # Library for displaying images in Jupyter Notebook
import ipywidgets as widgets  #Library for creating interactive widgets like buttons
import threading  # Library for creating new threads for asynchronous task execution

# Pre-defined class names based on the Caffe model
class_names = ["background", "aeroplane", "bicycle", "bird", "boat",
               "bottle", "bus", "car", "cat", "chair", "cow", "diningtable",
               "dog", "horse", "motorbike", "person", "pottedplant", "sheep",
               "sofa", "train", "tvmonitor"]

# Load Caffe model
net = cv2.dnn.readNetFromCaffe('deploy.prototxt', 'mobilenet_iter_73000.caffemodel')

# Create a "Stop" button for users to stop the video stream by clicking it
# ================
stopButton = widgets.ToggleButton(
    value=False,
    description='Stop',
    disabled=False,
    button_style='danger', # 'success', 'info', 'warning', 'danger' or ''
    tooltip='Description',
    icon='square' # (FontAwesome names without the `fa-` prefix)
)


# Define the display function to process video frames and perform object detection
# ================
def view(button):
    # If you are using a CSI camera you need to comment out the picam2 code and the camera code.
    # Since the latest versions of OpenCV no longer support CSI cameras (4.9.0.80), you need to use picamera2 to get the camera footage

    
    # picam2 = Picamera2()  # Create Picamera2 example
    # picam2.configure(picam2.create_video_configuration(main={"format": 'XRGB8888', "size": (640, 480)}))  # Configure camera parameters
    # picam2.start()  # Start camera
    
    camera = cv2.VideoCapture(-1) # Create camera example
    #Set resolution
    camera.set(cv2.CAP_PROP_FRAME_WIDTH, 640)
    camera.set(cv2.CAP_PROP_FRAME_HEIGHT, 480)
    
    display_handle=display(None, display_id=True)  # Create a display handle for updating displayed images
    i = 0
    
    avg = None
    
    while True:
        # frame = picam2.capture_array() # Capture a frame from the camera
        _, frame = camera.read() # Capture a frame from the camerav
        # frame = cv2.flip(frame, 1) # if your camera reverses your image

        img = cv2.cvtColor(frame, cv2.COLOR_RGB2BGR)  # Convert the image from RGB to BGR because OpenCV uses BGR by default
        (h, w) = img.shape[:2]  # Get the height and width of the image
        # Generate the input blob for the network
        blob = cv2.dnn.blobFromImage(cv2.resize(img, (300, 300)), 0.007843, (300, 300), 127.5)
        net.setInput(blob)  # Set the blob as the input to the network
        detections = net.forward()  # Perform forward pass to get the detection results

        # Loop over the detected objects
        for i in range(0, detections.shape[2]):
            confidence = detections[0, 0, i, 2]  # Get the confidence of the detected object
            if confidence > 0.2:  # If the confidence is above the threshold, process the detected object
                idx = int(detections[0, 0, i, 1])  # Get the class index
                box = detections[0, 0, i, 3:7] * np.array([w, h, w, h])  # Get the bounding box of the object
                (startX, startY, endX, endY) = box.astype("int")  # Convert the bounding box to integers

                # Annotate the object and confidence on the image
                label = "{}: {:.2f}%".format(class_names[idx], confidence * 100)
                cv2.rectangle(frame, (startX, startY), (endX, endY), (0, 255, 0), 2)
                y = startY - 15 if startY - 15 > 15 else startY + 15
                cv2.putText(frame, label, (startX, y), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 255, 0), 2)
        
        _, frame = cv2.imencode('.jpeg', frame)  #  Encode the frame as JPEG format
        display_handle.update(Image(data=frame.tobytes()))  # Update the displayed image
        if stopButton.value==True:  # Check if the "Stop" button is pressed
            # picam2.close()  # If yes, close the camera
            cv2.release() # If yes, close the camera
            display_handle.update(None)  # Clear the displayed content

            
#Display the "Stop" button and start the display function's thread
# ================
display(stopButton)
thread = threading.Thread(target=view, args=(stopButton,))
thread.start()

In this tutorial, we'll integrate some functions to modify frame images within OpenCV, such as blurring, color space conversion, erosion, and dilation.

Preparation

Since the product automatically runs the main program at startup, which occupies the camera resource, this tutorial cannot be used in such situations. You need to terminate the main program or disable its automatic startup before restarting the robot.
It's worth noting that because the robot's main program uses multi-threading and is configured to run automatically at startup through crontab, the usual method sudo killall python typically doesn't work. Therefore, we'll introduce the method of disabling the automatic startup of the main program here.
If you have already disabled the automatic startup of the robot's main demo, you do not need to proceed with the section on Terminate the Main Demo.

Terminate the Main Demo

1. Click the + icon next to the tab for this page to open a new tab called "Launcher."

2. Click on Terminal under Other to open a terminal window.

3. Type bash into the terminal window and press Enter.

4. Now you can use the Bash Shell to control the robot.

5. Enter the command: sudo killall -9 python.

Demo

The following code block can be run directly:

1. Select the code block below.

2. Press Shift + Enter to run the code block.

3. Watch the real-time video window.

4. Press STOP to close the real-time video and release the camera resources.

If you cannot see the real-time camera feed when running:

• Click on Kernel -> Shut down all kernels above.
• Close the current section tab and open it again.
• Click STOP to release the camera resources, then run the code block again.
• Reboot the device.

Execution

By default, we detect blue balls in the example. Ensure that there are no blue objects in the background to avoid interfering with the color recognition function. You can also modify the detection color (in the HSV color space) through secondary development.

import cv2
import imutils
from picamera2 import Picamera2  # Library for accessing Raspberry Pi Camera
import numpy as np  # Library for mathematical calculations
from IPython.display import display, Image  # Library for displaying images in Jupyter Notebook
import ipywidgets as widgets  # Library for creating interactive widgets such as buttons
import threading  # Library for creating new threads to execute tasks asynchronously

# Create a "Stop" button that users can click to stop the video stream
# ================================================================
stopButton = widgets.ToggleButton(
    value=False,
    description='Stop',
    disabled=False,
    button_style='danger', # 'success', 'info', 'warning', 'danger' or ''
    tooltip='Description',
    icon='square' # (FontAwesome names without the `fa-` prefix)
)

# Define the display function to process video frames and recognize objects of specific colors
def view(button):
    # If you are using a CSI camera you need to comment out the picam2 code and the camera code.
    # Since the latest versions of OpenCV no longer support CSI cameras (4.9.0.80), you need to use picamera2 to get the camera footage
    # picam2 = Picamera2()  # Create an instance of Picamera2
    # Configure camera parameters, set the format and size of the video
    # picam2.configure(picam2.create_video_configuration(main={"format": 'XRGB8888', "size": (640, 480)}))  
    # picam2.start()  # Start the camera
    
    camera = cv2.VideoCapture(-1) #Create camera example
    #Set resolution
    camera.set(cv2.CAP_PROP_FRAME_WIDTH, 640)
    camera.set(cv2.CAP_PROP_FRAME_HEIGHT, 480)
    
    display_handle=display(None, display_id=True)  # Create a display handle to update displayed images
    i = 0
    
    # Define the color range to be detected
    color_upper = np.array([120, 255, 220])
    color_lower = np.array([90, 120, 90])
    min_radius = 12  # Define the minimum radius for detecting objects
    
    while True:
        # img = picam2.capture_array() # Capture a frame from the camera
        _, img = camera.read()
        # frame = cv2.flip(frame, 1) # if your camera reverses your image
        
        blurred = cv2.GaussianBlur(img, (11, 11), 0)  # Apply Gaussian blur to the image to remove noise
        hsv = cv2.cvtColor(blurred, cv2.COLOR_BGR2HSV)  # Convert the image from BGR to HSV color space
        mask = cv2.inRange(hsv, color_lower, color_upper)  # Create a mask to retain only objects within a specific color range
        mask = cv2.erode(mask, None, iterations=5)  # Apply erosion to the mask to remove small white spots
        mask = cv2.dilate(mask, None, iterations=5)  # Apply dilation to the mask to highlight the object regions

        # Find contours in the mask
        cnts = cv2.findContours(mask.copy(), cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
        cnts = imutils.grab_contours(cnts)  # Extract contours
        center = None  # Initialize the center of the object

        if len(cnts) > 0:
            # Find the largest contour in the mask, then use
            # it to compute the minimum enclosing circle and
            # centroid
            c = max(cnts, key=cv2.contourArea)  # Find the largest contour
            ((x, y), radius) = cv2.minEnclosingCircle(c)  # Compute the minimum enclosing circle of the contour
            M = cv2.moments(c)  # Compute the moments of the contour
            center = (int(M["m10"] / M["m00"]), int(M["m01"] / M["m00"]))  # Compute the center of the contour based on moments

            if radius > min_radius:  # If the radius of the minimum enclosing circle is greater than the predefined minimum radius, draw circles and center points
                cv2.circle(img, (int(x), int(y)), int(radius), (128, 255, 255), 1)  # Draw the minimum enclosing circle
        
        _, frame = cv2.imencode('.jpeg', img)  # Encode the frame to JPEG format
        display_handle.update(Image(data=frame.tobytes()))  # Update the displayed image
        if stopButton.value==True:  # Check if the "Stop" button has been pressed
            #picam2.close()  # If yes, close the camera
            cv2.release() # if yes, close the camera
            display_handle.update(None)  # Clear the displayed content


# Display the "Stop" button and start a thread to execute the display function
# ================================================================
display(stopButton)
thread = threading.Thread(target=view, args=(stopButton,))
thread.start()

This section introduces how to implement gesture recognition using MediaPipe + OpenCV.

What is MediaPipe?

MediaPipe is an open-source framework developed by Google for building machine learning-based multimedia processing applications. It provides a set of tools and libraries for processing video, audio, and image data, and applies machine learning models to achieve various functionalities such as pose estimation, gesture recognition, and face detection. MediaPipe is designed to offer efficient, flexible, and easy-to-use solutions, enabling developers to quickly build a variety of multimedia processing applications.

Preparation

Since the product automatically runs the main program at startup, which occupies the camera resource, this tutorial cannot be used in such situations. You need to terminate the main program or disable its automatic startup before restarting the robot.
It's worth noting that because the robot's main program uses multi-threading and is configured to run automatically at startup through crontab, the usual method sudo killall python typically doesn't work. Therefore, we'll introduce the method of disabling the automatic startup of the main program here.
If you have already disabled the automatic startup of the robot's main demo, you do not need to proceed with the section on Terminate the Main Demo.

Terminate the Main Demo

1. Click the + icon next to the tab for this page to open a new tab called "Launcher."

2. Click on Terminal under Other to open a terminal window.

3. Type bash into the terminal window and press Enter.

4. Now you can use the Bash Shell to control the robot.

5. Enter the command: sudo killall -9 python.

Demo

The following code block can be run directly:

1. Select the code block below.

2. Press Shift + Enter to run the code block.

3. Watch the real-time video window.

4. Press STOP to close the real-time video and release the camera resources.

If you cannot see the real-time camera feed when running:

• Click on Kernel -> Shut down all kernels above.
• Close the current section tab and open it again.
• Click STOP to release the camera resources, then run the code block again.
• Reboot the device.

Running

In this tutorial, the camera pan-tilt will rotate to make sure your hand or other fragile objects are away from the camera pan-tilt's rotation radius.

We detect blue balls by default in the demo to ensure that there are no blue objects in the background of the picture that affect the color recognition function, and you can also change the detection color (HSV color space) through secondary development.

import matplotlib.pyplot as plt
import cv2
from picamera2 import Picamera2
import numpy as np
from IPython.display import display, Image
import ipywidgets as widgets
import threading

# Stop button
# ================
stopButton = widgets.ToggleButton(
    value=False,
    description='Stop',
    disabled=False,
    button_style='danger', # 'success', 'info', 'warning', 'danger' or ''
    tooltip='Description',
    icon='square' # (FontAwesome names without the `fa-` prefix)
)


def gimbal_track(fx, fy, gx, gy, iterate):
    global gimbal_x, gimbal_y
    distance = math.sqrt((fx - gx) ** 2 + (gy - fy) ** 2)
    gimbal_x += (gx - fx) * iterate
    gimbal_y += (fy - gy) * iterate
    if gimbal_x > 180:
        gimbal_x = 180
    elif gimbal_x < -180:
        gimbal_x = -180
    if gimbal_y > 90:
        gimbal_y = 90
    elif gimbal_y < -30:
        gimbal_y = -30
    gimbal_spd = int(distance * track_spd_rate)
    gimbal_acc = int(distance * track_acc_rate)
    if gimbal_acc < 1:
        gimbal_acc = 1
    if gimbal_spd < 1:
        gimbal_spd = 1
    base.base_json_ctrl({"T":self.CMD_GIMBAL,"X":gimbal_x,"Y":gimbal_y,"SPD":gimbal_spd,"ACC":gimbal_acc})
    return distance


# Display function
# ================
def view(button):
    picam2 = Picamera2()
    picam2.configure(picam2.create_video_configuration(main={"format": 'XRGB8888', "size": (640, 480)}))
    picam2.start()
    display_handle=display(None, display_id=True)

    color_upper = np.array([120, 255, 220])
    color_lower = np.array([ 90, 120,  90])
    min_radius = 12
    track_color_iterate = 0.023
    
    while True:
        frame = picam2.capture_array()
        # frame = cv2.flip(frame, 1) # if your camera reverses your image

        # uncomment this line if you are using USB camera
        # frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)

        img = cv2.cvtColor(frame, cv2.COLOR_RGB2BGR)
        blurred = cv2.GaussianBlur(img, (11, 11), 0)
        hsv = cv2.cvtColor(blurred, cv2.COLOR_BGR2HSV)
        mask = cv2.inRange(hsv, color_lower, color_upper)
        mask = cv2.erode(mask, None, iterations=5)
        mask = cv2.dilate(mask, None, iterations=5)

        cnts = cv2.findContours(mask.copy(), cv2.RETR_EXTERNAL,
            cv2.CHAIN_APPROX_SIMPLE)
        cnts = imutils.grab_contours(cnts)
        center = None

        height, width = img.shape[:2]
        center_x, center_y = width // 2, height // 2

        if len(cnts) > 0:
            # find the largest contour in the mask, then use
            # it to compute the minimum enclosing circle and
            # centroid
            c = max(cnts, key=cv2.contourArea)
            ((x, y), radius) = cv2.minEnclosingCircle(c)
            M = cv2.moments(c)
            center = (int(M["m10"] / M["m00"]), int(M["m01"] / M["m00"]))

            # only proceed if the radius meets a minimum size
            if radius > min_radius:
                distance = gimbal_track(center_x, center_y, center[0], center[1], track_color_iterate) #
                cv2.circle(overlay_buffer, (int(x), int(y)), int(radius), (128, 255, 255), 1)
        
        
        _, frame = cv2.imencode('.jpeg', frame)
        display_handle.update(Image(data=frame.tobytes()))
        if stopButton.value==True:
            picam2.close()
            display_handle.update(None)
            
            
# Run
# ================
display(stopButton)
thread = threading.Thread(target=view, args=(stopButton,))
thread.start()

In this tutorial, we'll use the basic functionalities of OpenCV to detect yellow lines (default color) in the image and control the direction of the chassis based on the position of these lines. Please note that in this example, the chassis won't move. Instead, we'll only showcase the algorithms using OpenCV on the image. For safety reasons, we won't integrate motion control in this tutorial, as it's heavily influenced by external factors. Users should fully understand the code's functionality before adding corresponding motion control features.

If you want to control the robot's movement through this example, please refer to the "Python Chassis Motion Control" section to add the relevant motion control functions (our open-source example is located in robot_ctrl.py).

Preparation

Since the product automatically runs the main program at startup, which occupies the camera resource, this tutorial cannot be used in such situations. You need to terminate the main program or disable its automatic startup before restarting the robot.
It's worth noting that because the robot's main program uses multi-threading and is configured to run automatically at startup through crontab, the usual method sudo killall python typically doesn't work. Therefore, we'll introduce the method of disabling the automatic startup of the main program here.
If you have already disabled the automatic startup of the robot's main demo, you do not need to proceed with the section on Terminate the Main Demo.

Terminate the Main Demo

1. Click the + icon next to the tab for this page to open a new tab called "Launcher."

2. Click on Terminal under Other to open a terminal window.

3. Type bash into the terminal window and press Enter.

4. Now you can use the Bash Shell to control the robot.

5. Enter the command: sudo killall -9 python.

Demo

The following code block can be run directly:

1. Select the code block below.

2. Press Shift + Enter to run the code block.

3. Watch the real-time video window.

4. Press STOP to close the real-time video and release the camera resources.

If you cannot see the real-time camera feed when running:

• Click on Kernel -> Shut down all kernels above.
• Close the current section tab and open it again.
• Click STOP to release the camera resources, then run the code block again.
• Reboot the device.

Note

If you use the USB camera you need to uncomment frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB).

Running

After running the following code block, you can place a yellow tape in front of the camera and observe if there are contours of the yellow tape in the black screen. Try to detect the yellow tape using two target detection lines.

import cv2  # Import OpenCV library for image process  
import imutils, math  # Library to aid image processing and mathematical operations 
from picamera2 import Picamera2 # Library to access Raspberry Pi Camera 
import numpy as np
from IPython.display import display, Image # Display images on Jupyter Notebook 
import ipywidgets as widgets # Widgets for creating interactive interfaces, such as buttons
import threading # Used to create new threads for asynchronous execution of tasks

# Stop button
# ================
stopButton = widgets.ToggleButton(
    value=False,
    description='Stop',
    disabled=False,
    button_style='danger', # 'success', 'info', 'warning', 'danger' or ''
    tooltip='Description',
    icon='square' # (FontAwesome names without the `fa-` prefix)
)


# findline autodrive

# Upper sampling line, 0.6 for position, higher values
sampling_line_1 = 0.6

# Lower sampling line, the value needs to be greater than sampling_line_1 and less than 1."
sampling_line_2 = 0.9

# Detect slope impact  
slope_impact = 1.5

# The effect of line position detected by the lower detection line on turning
base_impact = 0.005

# The speed impact on turning 
speed_impact = 0.5

# Line tracking speed
line_track_speed = 0.3

# Effect of slope on patrol speed
slope_on_speed = 0.1

# Color of target line, HSV color space
line_lower = np.array([25, 150, 70])
line_upper = np.array([42, 255, 255])

def view(button):
    # If you are using a CSI camera you need to comment out the picam2 code and the camera code.
    # Since the latest versions of OpenCV no longer support CSI cameras (4.9.0.80), you need to use picamera2 to get the camera footage
    # picam2 = Picamera2() #Create Picamera2 example 
    # Set camera parameters and video format and size
    # picam2.configure(picam2.create_video_configuration(main={"format": 'XRGB8888', "size": (640, 480)}))
    # picam2.start() #Start camera 

    camera = cv2.VideoCapture(-1)  #Create camera example
    #Set resolution
    camera.set(cv2.CAP_PROP_FRAME_WIDTH, 640)
    camera.set(cv2.CAP_PROP_FRAME_HEIGHT, 480)
    
    display_handle=display(None, display_id=True)
    
    while True:
        # img = picam2.capture_array()
        _, img = camera.read() #Capture one frame from the camera 
        # frame = cv2.flip(frame, 1) # if your camera reverses your image

        height, width = img.shape[:2]
        center_x, center_y = width // 2, height // 2
        # Image preprocessing includes color space conversion, Gaussian blur, and color range filtering, etc.
        hsv = cv2.cvtColor(img, cv2.COLOR_BGR2HSV)

        line_mask = cv2.inRange(hsv, line_lower, line_upper)  # Filter out target lines based on color ranges
        line_mask = cv2.erode(line_mask, None, iterations=1)  # Eroding operation to remove noise
        line_mask = cv2.dilate(line_mask, None, iterations=1)  # Expansion operation enhances the target line

        # Detect the target line based on the positions of the upper and lower sampling lines, and calculate steering and velocity control signals according to the detection results
        sampling_h1 = int(height * sampling_line_1)
        sampling_h2 = int(height * sampling_line_2)
        
        get_sampling_1 = line_mask[sampling_h1]
        get_sampling_2 = line_mask[sampling_h2]

        # Calculate the width of the target line at the upper and lower sampling lines
        sampling_width_1 = np.sum(get_sampling_1 == 255)
        sampling_width_2 = np.sum(get_sampling_2 == 255)

        if sampling_width_1:
            sam_1 = True
        else:
            sam_1 = False
        if sampling_width_2:
            sam_2 = True
        else:
            sam_2 = False

        # Get the edge index of the target line at the upper and lower sampling lines
        line_index_1 = np.where(get_sampling_1 == 255)
        line_index_2 = np.where(get_sampling_2 == 255)

        # If the target line is detected at the upper sampling line, calculate the center position of the target line
        if sam_1:
            sampling_1_left  = line_index_1[0][0]  # Index of the leftmost index of the upper sampling line target line
            sampling_1_right = line_index_1[0][sampling_width_1 - 1]  # Index to the far right of the upper sampling line target line
            sampling_1_center= int((sampling_1_left + sampling_1_right) / 2)  # Index of the center of the upper sampling line target line
        # If a target line is detected at the lower sampling line, calculate the target line center position
        if sam_2:
            sampling_2_left  = line_index_2[0][0]
            sampling_2_right = line_index_2[0][sampling_width_2 - 1]
            sampling_2_center= int((sampling_2_left + sampling_2_right) / 2)

        # Initialize steering and speed control signals
        line_slope = 0
        input_speed = 0
        input_turning = 0
        
        # If the target line is detected at both sampling lines, calculate the slope of the line, and then calculate velocity and steering control signals based on the slope and the position of the target line.
        if sam_1 and sam_2:
            line_slope = (sampling_1_center - sampling_2_center) / abs(sampling_h1 - sampling_h2) # Calculate the slope of the line
            impact_by_slope = slope_on_speed * abs(line_slope) # Calculate the effect on velocity based on the slope
            input_speed = line_track_speed - impact_by_slope # Calculated speed control signal
            input_turning = -(line_slope * slope_impact + (sampling_2_center - center_x) * base_impact) #+ (speed_impact * input_speed) # Calculating steering control signals
        elif not sam_1 and sam_2: # If the target line is detected only at the lower sampling line
            input_speed = 0 # Set speed to 0
            input_turning = (sampling_2_center - center_x) * base_impact # Calculating steering control signals
        elif sam_1 and not sam_2: # If the target line is detected only at the upper sample line
            input_speed = (line_track_speed / 3) # slow down
            input_turning = 0 # No steering
        else: # If neither sampling line detects the target line
            input_speed = - (line_track_speed / 3) # backward
            input_turning = 0 # No turning

        # base.base_json_ctrl({"T":13,"X":input_speed,"Z":input_turning})

        cv2.putText(line_mask, f'X: {input_speed:.2f}, Z: {input_turning:.2f}', (center_x+50, center_y+0), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (255, 255, 255), 1)
        # Visualization operations include drawing lines at the positions of the sampling lines, marking the sampling results, and displaying steering and velocity control signals
        cv2.line(line_mask, (0, sampling_h1), (img.shape[1], sampling_h1), (255, 0, 0), 2)
        cv2.line(line_mask, (0, sampling_h2), (img.shape[1], sampling_h2), (255, 0, 0), 2)

        if sam_1:
            # Draw green marker lines at the ends of the target line at the upper sample line
            cv2.line(line_mask, (sampling_1_left, sampling_h1+20), (sampling_1_left, sampling_h1-20), (0, 255, 0), 2)
            cv2.line(line_mask, (sampling_1_right, sampling_h1+20), (sampling_1_right, sampling_h1-20), (0, 255, 0), 2)
        if sam_2:
            # Draw green marker lines at the ends of the target line at the lower sampling line
            cv2.line(line_mask, (sampling_2_left, sampling_h2+20), (sampling_2_left, sampling_h2-20), (0, 255, 0), 2)
            cv2.line(line_mask, (sampling_2_right, sampling_h2+20), (sampling_2_right, sampling_h2-20), (0, 255, 0), 2)
        if sam_1 and sam_2:
            # If the target line is detected at both the upper and lower sample lines, draw a red line from the center of the upper sample line to the center of the lower sample line.
            cv2.line(line_mask, (sampling_1_center, sampling_h1), (sampling_2_center, sampling_h2), (255, 0, 0), 2)
        
        _, frame = cv2.imencode('.jpeg', line_mask)
        display_handle.update(Image(data=frame.tobytes()))
        if stopButton.value==True:
            #picam2.close()  # If yes, close the camera
            cv2.release() # if yes, close the camera
            display_handle.update(None)


# Display the "Stop" button and start the thread that displays the function
# ================
display(stopButton)
thread = threading.Thread(target=view, args=(stopButton,))
thread.start()

This section introduces how to implement gesture recognition using MediaPipe + OpenCV.

What is MediaPipe?

MediaPipe is an open-source framework developed by Google for building machine learning-based multimedia processing applications. It provides a set of tools and libraries for processing video, audio, and image data, and applies machine learning models to achieve various functionalities such as pose estimation, gesture recognition, and face detection. MediaPipe is designed to offer efficient, flexible, and easy-to-use solutions, enabling developers to quickly build a variety of multimedia processing applications.

Preparation

Since the product automatically runs the main program at startup, which occupies the camera resource, this tutorial cannot be used in such situations. You need to terminate the main program or disable its automatic startup before restarting the robot.
It's worth noting that because the robot's main program uses multi-threading and is configured to run automatically at startup through crontab, the usual method sudo killall python typically doesn't work. Therefore, we'll introduce the method of disabling the automatic startup of the main program here.
If you have already disabled the automatic startup of the robot's main demo, you do not need to proceed with the section on Terminate the Main Demo.

Terminate the Main Demo

1. Click the + icon next to the tab for this page to open a new tab called "Launcher."

2. Click on Terminal under Other to open a terminal window.

3. Type bash into the terminal window and press Enter.

4. Now you can use the Bash Shell to control the robot.

5. Enter the command: sudo killall -9 python.

Example

The following code block can be run directly:

1. Select the code block below.

2. Press Shift + Enter to run the code block.

3. Watch the real-time video window.

4. Press STOP to close the real-time video and release the camera resources.

If you cannot see the real-time camera feed when running:

• Click on Kernel -> Shut down all kernels above.
• Close the current section tab and open it again.
• Click STOP to release the camera resources, then run the code block again.
• Reboot the device.

Note

If you use the USB camera you need to uncomment frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB).

Features of this Section

When the code block is running normally, you can put your hand in front of the camera, the real-time video screen will be labeled with the joints of your hand, the labeled joints will change with the change of your hand, and the position of each joint will also be output, which is convenient for the secondary development of gesture control.
MediaPipe's gesture recognition process uses different names to correspond to different joints, and you can call the corresponding number to get the position information of the joint.
MediaPipe Hand

• WRIST
• THUMB_CMC
• THUMB_MCP
• THUMB_IP
• THUMB_TIP
• INDEX_FINGER_MCP
• INDEX_FINGER_PIP
• INDEX_FINGER_DIP
• INDEX_FINGER_TIP
• MIDDLE_FINGER_MCP
• MIDDLE_FINGER_PIP
• MIDDLE_FINGER_DIP
• MIDDLE_FINGER_TIP
• RING_FINGER_MCP
• RING_FINGER_PIP
• RING_FINGER_DIP
• RING_FINGER_TIP
• PINKY_MCP
• PINKY_PIP
• PINKY_DIP
• PINKY_TIP

import cv2
import imutils, math
from picamera2 import Picamera2  # for accessing Raspberry Pi Camera library
from IPython.display import display, Image  # for displaying images on Jupyter Notebook  
import ipywidgets as widgets  # for creating interactive widgets like button  
import threading  #  Library for creating new threads for asynchronous task execution
import mediapipe as mp  # Import the MediaPipe library for detecting hand gesture key points  


# Create a "STOP" button that users can click to stop the video stream  
# ================
stopButton = widgets.ToggleButton(
    value=False,
    description='Stop',
    disabled=False,
    button_style='danger', # 'success', 'info', 'warning', 'danger' or ''
    tooltip='Description',
    icon='square' # (FontAwesome names without the `fa-` prefix)
)

# Initialize MediaPipe drawing tool and hand keypoint detection model  
mpDraw = mp.solutions.drawing_utils

mpHands = mp.solutions.hands
hands = mpHands.Hands(max_num_hands=1) # Initialize the detection model of hand keypoint, up to one hand  

# Define display functions to process video frames and perform hand keypoint detection
def view(button):
    # If you use the CSI camera, you need to comment out picam2 and camera 
    # Since the latest version OpenCV does not support CSI camera (4.9.0.80) anymore, you need to use picamera2 to get camera image
    
    # picam2 = Picamera2()  #Create Picamera2 example
    # Configure camera parameters and set the format and size of video
    # picam2.configure(picam2.create_video_configuration(main={"format": 'XRGB8888', "size": (640, 480)}))
    # picam2.start()  # Start camera

    camera = cv2.VideoCapture(-1) # Create camera example
    #Set resolution
    camera.set(cv2.CAP_PROP_FRAME_WIDTH, 640)
    camera.set(cv2.CAP_PROP_FRAME_HEIGHT, 480)
    
    display_handle=display(None, display_id=True)  # Creates a display handle for updating the displayed image
    
    while True:
        # frame = picam2.capture_array()
        _, frame = camera.read() # Capture a frame from the camera
        # frame = cv2.flip(frame, 1) # if your camera reverses your image

        img = cv2.cvtColor(frame, cv2.COLOR_RGB2BGR)
        results = hands.process(img)

        # If the hand key is detected
        if results.multi_hand_landmarks:
            for handLms in results.multi_hand_landmarks: # Iterate over each hand detected
                # Draw hand keypoints
                for id, lm in enumerate(handLms.landmark):
                    h, w, c = img.shape
                    cx, cy = int(lm.x * w), int(lm.y * h)  # Calculate the position of the keypoint in the image
                    cv2.circle(img, (cx, cy), 5, (255, 0, 0), -1)  # Drawing dots at keypoint locations

                
                frame = cv2.cvtColor(frame, cv2.COLOR_RGB2BGR)
                mpDraw.draw_landmarks(frame, handLms, mpHands.HAND_CONNECTIONS) # Drawing hand skeleton connecting lines
                frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB) 

                target_pos = handLms.landmark[mpHands.HandLandmark.INDEX_FINGER_TIP]

        _, frame = cv2.imencode('.jpeg', frame)
        display_handle.update(Image(data=frame.tobytes()))
        if stopButton.value==True:
            # picam2.close() # if yes, close camera
            cv2.release() # if yes, close camera
            display_handle.update(None)

# Display the "Stop" button and start the thread that displays the function
# ================
display(stopButton)
thread = threading.Thread(target=view, args=(stopButton,))
thread.start()

This section introduces how to implement face recognition using MediaPipe + OpenCV.

What is MediaPipe?

MediaPipe is an open-source framework developed by Google for building machine learning-based multimedia processing applications. It provides a set of tools and libraries for processing video, audio, and image data, and applies machine learning models to achieve various functionalities such as pose estimation, gesture recognition, and face detection. MediaPipe is designed to offer efficient, flexible, and easy-to-use solutions, enabling developers to quickly build a variety of multimedia processing applications.

Preparation

Since the product automatically runs the main program at startup, which occupies the camera resource, this tutorial cannot be used in such situations. You need to terminate the main program or disable its automatic startup before restarting the robot.
It's worth noting that because the robot's main program uses multi-threading and is configured to run automatically at startup through crontab, the usual method sudo killall python typically doesn't work. Therefore, we'll introduce the method of disabling the automatic startup of the main program here.
If you have already disabled the automatic startup of the robot's main demo, you do not need to proceed with the section on Terminate the Main Demo.

Terminate the Main Demo

1. Click the + icon next to the tab for this page to open a new tab called "Launcher."

2. Click on Terminal under Other to open a terminal window.

3. Type bash into the terminal window and press Enter.

4. Now you can use the Bash Shell to control the robot.

5. Enter the command: sudo killall -9 python.

Demo

The following code block can be run directly:

1. Select the code block below.

2. Press Shift + Enter to run the code block.

3. Watch the real-time video window.

4. Press STOP to close the real-time video and release the camera resources.

If you cannot see the real-time camera feed when running:

• Click on Kernel -> Shut down all kernels above.
• Close the current section tab and open it again.
• Click STOP to release the camera resources, then run the code block again.
• Reboot the device.

Note

If you use the USB camera you need to uncomment frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB).

Features of this Section

When the code block runs correctly, MediaPipe will automatically detect faces in the video feed, outlining the position of the face and marking the facial features.

import cv2  # Import the OpenCV library for image processing
import imutils, math  # Libraries for auxiliary image processing and mathematical operations
from picamera2 import Picamera2  # Library for accessing the Raspberry Pi Camera
from IPython.display import display, Image  # Library for displaying images in Jupyter Notebook
import ipywidgets as widgets  # Library for creating interactive interface widgets such as buttons
import threading  # Library for creating new threads for asynchronous task execution
import mediapipe as mp  # Import the MediaPipe library for face detection

# Create a "STOP" button that users can click to stop the video stream
# ================
stopButton = widgets.ToggleButton(
    value=False,
    description='Stop',
    disabled=False,
    button_style='danger', # 'success', 'info', 'warning', 'danger' or ''
    tooltip='Description',
    icon='square' # (FontAwesome names without the `fa-` prefix)
)

# Initialize the face detection model of MediaPipe
mpDraw = mp.solutions.drawing_utils


# MediaPipe Hand GS
mp_face_detection = mp.solutions.face_detection
face_detection = mp_face_detection.FaceDetection(model_selection=0, min_detection_confidence=0.5)

#Define a display function to process video frames and perform face detection
def view(button):
    picam2 = Picamera2()
    picam2.configure(picam2.create_video_configuration(main={"format": 'XRGB8888', "size": (640, 480)}))
    picam2.start()
    display_handle=display(None, display_id=True)
    
    while True:
        frame = picam2.capture_array() # Capture one frame from the camera 
        # frame = cv2.flip(frame, 1) # if your camera reverses your image

        # uncomment this line if you are using USB camera
        # frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)

        img = cv2.cvtColor(frame, cv2.COLOR_RGB2BGR)
        
        results = face_detection.process(img)

        # If a face is detected
        if results.detections:
            for detection in results.detections: # Iterate through each detected face
                frame = cv2.cvtColor(frame, cv2.COLOR_RGB2BGR)
                mpDraw.draw_detection(frame, detection) # Use MediaPipe's drawing tools to draw face landmarks
                frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)

        _, frame = cv2.imencode('.jpeg', frame)  # Encode the processed frame to JPEG format
        display_handle.update(Image(data=frame.tobytes()))  # Update the displayed image
        if stopButton.value==True:  # Check if the "STOP" button is pressed
            picam2.close()  # If yes, close the camera
            display_handle.update(None)  # Clear the displayed content

#  Display the "STOP" button and start a thread to run the display function
# ================
display(stopButton)
thread = threading.Thread(target=view, args=(stopButton,))
thread.start()

This section describes how to implement pose detection using MediaPipe + OpenCV.

What is MediaPipe?

MediaPipe is an open-source framework developed by Google for building machine learning-based multimedia processing applications. It provides a set of tools and libraries for processing video, audio, and image data, and applies machine learning models to achieve various functionalities such as pose estimation, gesture recognition, and face detection. MediaPipe is designed to offer efficient, flexible, and easy-to-use solutions, enabling developers to quickly build a variety of multimedia processing applications.

Preparation

Since the product automatically runs the main program at startup, which occupies the camera resource, this tutorial cannot be used in such situations. You need to terminate the main program or disable its automatic startup before restarting the robot.
It's worth noting that because the robot's main program uses multi-threading and is configured to run automatically at startup through crontab, the usual method sudo killall python typically doesn't work. Therefore, we'll introduce the method of disabling the automatic startup of the main program here.
If you have already disabled the automatic startup of the robot's main demo, you do not need to proceed with the section on Terminate the Main Demo.

Terminate the Main Demo

1. Click the + icon next to the tab for this page to open a new tab called "Launcher."

2. Click on Terminal under Other to open a terminal window.

3. Type bash into the terminal window and press Enter.

4. Now you can use the Bash Shell to control the robot.

5. Enter the command: sudo killall -9 python.

Demo

The following code block can be executed directly:

1. Select the code block below.

2. Press Shift + Enter to run the code block.

3. Watch the real-time video window.

4. Press STOP to close the real-time video and release the camera resources.

If the real-time camera view is not visible during execution

• Click on Kernel - Shut down all kernels above.
• Close the current chapter tab and reopen it.
• Click STOP to release the camera resources, then run the code block again.
• Reboot the device.

Note

If you use the USB camera you need to uncomment frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB).

Features

When the code block runs normally, MediaPipe will automatically mark the joints of the human body when there is a face in the frame.

import cv2  # Import the OpenCV library for image processing
import imutils, math  # Auxiliary libraries for image processing and mathematical operations
from picamera2 import Picamera2  # Library to access the Raspberry Pi Camera
from IPython.display import display, Image  # Library to display images in Jupyter Notebook
import ipywidgets as widgets  # Library for creating interactive widgets, such as buttons
import threading  # Library for creating new threads for asynchronous execution of tasks
import mediapipe as mp  # Import the MediaPipe library for pose detection

# Create a "Stop" button that users can click to stop the video stream
# ================
stopButton = widgets.ToggleButton(
    value=False,
    description='Stop',
    disabled=False,
    button_style='danger', # 'success', 'info', 'warning', 'danger' or ''
    tooltip='Description',
    icon='square' # (FontAwesome names without the `fa-` prefix)
)

# Initialize MediaPipe's drawing tools and pose detection model
mpDraw = mp.solutions.drawing_utils


# MediaPipe Hand GS
mp_pose = mp.solutions.pose
pose = mp_pose.Pose(static_image_mode=False, 
                    model_complexity=1, 
                    smooth_landmarks=True, 
                    min_detection_confidence=0.5, 
                    min_tracking_confidence=0.5)

#  Define the display function to process video frames and perform pose detection
def view(button):
    picam2 = Picamera2()  # Create an instance of Picamera2
    picam2.configure(picam2.create_video_configuration(main={"format": 'XRGB8888', "size": (640, 480)}))  # Configure camera parameters
    picam2.start()  # Start the camera
    display_handle=display(None, display_id=True)  # Create a display handle to update the displayed image
    
    while True:
        frame = picam2.capture_array()
        # frame = cv2.flip(frame, 1) # if your camera reverses your image

        # uncomment this line if you are using USB camera
        # frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)

        img = cv2.cvtColor(frame, cv2.COLOR_RGB2BGR)
        
        results = pose.process(img) # Use MediaPipe to process the image and get pose detection results

         # If pose landmarks are detected
        if results.pose_landmarks:
            frame = cv2.cvtColor(frame, cv2.COLOR_RGB2BGR)  #  Convert the image from RGB to BGR for drawing
            mpDraw.draw_landmarks(frame, results.pose_landmarks, mp_pose.POSE_CONNECTIONS)  # Use MediaPipe's drawing tools to draw pose landmarks and connections
            frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)  # Convert the image back from BGR to RGB for display
            
        _, frame = cv2.imencode('.jpeg', frame)  # Encode the processed frame into JPEG format
        display_handle.update(Image(data=frame.tobytes()))  # Update the displayed image
        if stopButton.value==True:  # Check if the "Stop" button is pressed
            picam2.close()  # If yes, close the camera
            display_handle.update(None)  # Clear the displayed content

# Display the "Stop" button and start a thread to run the display function
display(stopButton)
thread = threading.Thread(target=view, args=(stopButton,))
thread.start()  # Start the thread

In the previous chapters, we introduced how to use Flask to achieve low-latency image transmission, which is used to transfer the camera feed to the web application interface. Here, we will discuss how to pass the information entered on the web application interface to the backend of the web application. This functionality is used to control the robot using the web application.

from flask import Flask, request  # Import the Flask class and request object from the flask package

app = Flask(__name__)  # Create a Flask app instance

@app.route('/', methods=['GET', 'POST'])  # Define the root route, and allow GET and POST methods
def index():
    if request.method == 'POST':   # Check if the current request is POST
        # Get form data
        form_data = request.form
        # Print form data
        print(form_data)
        # Return a simple response
        return 'Received data: {}'.format(form_data)
    else:
        # If it's a GET request, return a simple form page
        return '''
        <form method="post">
            <label for="input_data">Input data:</label><br>
            <input type="text" id="input_data" name="input_data"><br>
            <input type="submit" value="Submit">
        </form>
        '''

if __name__ == '__main__':
    app.run(host='0.0.0.0')

You can select the code block above and press Ctrl + Enter to run it. If you encounter a port conflict, it means you've previously run this code block. You'll need to click Kernel in the JupyterLab menu bar and then select Shut Down All Kernels to release the resources, including network port resources, occupied by the previously run code block. After that, you can rerun the code block to run this Flask application.

Once you run the code block, you'll see messages like "Running on http://127.0.0.1:5000" and "Running on http://[IP]:5000". Usually, the [IP] here refers to the IP address assigned to your Raspberry Pi by your router. You can open a web browser on any device within the same local network and visit [IP]:5000. Note that the : symbol here must be the English colon, representing the port number 5000 of the IP address you're accessing.

Upon visiting this page, you'll see an input box and a Submit button. You can enter some content into the input box and then click the Submit button. After that, you can see the content you entered on the web page, and the backend will display the content it received below the code block in JupyterLab.

Introduction to Main Program Architecture

File Structure and Functionality Introduction

• ugv_jetson
  • [file folder] media (README.MD material)
  • [file folder] models (pre-training model)
  • [file folder] sounds (Used to store audio files, where voice packs can be configured)
  • [file folder] templates (The information, photos and videos related to the web application are also stored in this source)
  • [file folder] tutorial_cn (Interactive tutorial in Chinese)
  • [file folder] tutorial_en (Interactive tutorial in English)
  • LICENSE (GPL-3.0 open-source tutorial)
  • app.py (main demo, including webRTC, web-socket and flask related functions)
  • asound.conf (sound card config file)
  • audio_ctrl.py (Audio function related libraries)
  • autorun.sh (Configure the product master and jupyterlab boot autorun scripts)
  • base_camera.py (flask real-time video of the underlying multi-threaded capture library, the original project for flask-video-streaming)
  • base_ctrl.py (the library to communicate with the slave, communicate with the slave through the serial port)
  • config.yaml (Configuration file to configure some parameters)
  • cv_ctrl.py (the related OpenCV functions)
  • os_info.py (for obtaining system information)
  • requirements.txt (python project dependency library)
  • setup.sh (auto-installation script)
  • start_jupyter.sh (enable jupyterlab server)
  • sunny (sunny-ngrok demo)

Install Script

There is a file named setup.sh in the project folder, written in a shell, that helps to automatically configure the robotics product's uploader, including setting up the serial port, setting up the camera, creating the project's virtual environment, and installing dependency libraries. All these steps are already configured in the image on the TF card we shipped.

Installation script usage, the installation process needs to download and install a lot of dependent libraries from the network, for areas with special network environment we recommend you to use the method of downloading the image file directly from our official website to install the product.

setup.sh should run with root permission.

cd ugv_jetson/
sudo chmod +x setup.sh
sudo ./setup.sh

Auto-running Demo

autorun.sh in the project folder is used to configure the product main program (app.py) and jupyterlab (start_jupyter.sh) to autorun on power-up (as user instead of root), as well as to generate the configuration file for jupyterlab.

autorun.sh should run with user permission.

sudo chmod +x autorun.sh ./autorun.sh

Update Project

This project will be updated for a long time, when you need to update it, just navigate to the project folder and use the git pull command to update it, note that this operation will replace the changes you made before, after updating it, you need to change the parameters in config.yaml, set the type of the chassis and the type of the module, and you can also use the command line tool in the WEB control interface to configure the type of the product.

cd ugv_jetson/
git pull

After restarting the device, open the WEB control interface and enter the command used to set the product type, for example: s 20

The first digit 2 represents chassis type, the second digit 0 represents no module or pan-tilt module.

The first digit:

• 1: RaspRover
• 2: UGV Rover
• 3: UGV Beast

The second digit:

• 0: None/PT
• 1: RoArm
• 2: Pan-Tilt

This configuration needs to be configured only once after an update or reinstallation and will be saved in config.yaml.

Open-source Project

This project is open source under the GPL-3.0 agreement , you can realize your project according to our project, we will also continue to update more features.

Project address: https://github.com/waveshareteam/ugv_jetson

What is YAML?

YAML (YAML Ain't Markup Language) is a human-readable data serialization format used to represent complex data structures. Its main purposes include configuration files, data exchange and storage, and passing data to programs.

Advantages of YAML

Human Language Readability

YAML uses indentation and a well-structured format, making files easy to read and understand. It is less verbose compared to XML or JSON and closer to natural language.

Ease of Writing and Editing

YAML syntax is clear and concise, requiring no additional markup symbols (such as XML tags or JSON braces), making it easier to write and edit.

Support for Complex Data Structures

YAML supports nesting, lists, dictionaries, and other complex data structures, making it easy to represent various types of data.

Extensibility

YAML allows the use of tags and anchors to represent relationships between objects, enabling data reuse and referencing, thereby enhancing extensibility.

Language Independence

YAML is a universal data serialization format that is not tied to any specific programming language, making it easy to parse and generate using multiple programming languages.

Configuration File config.yaml for this Product

In the config.yaml of this product, we configure some key parameters related to the robot:

audio_config - Audio-related settings

• audio_output: Whether to use audio output
• default_volume: Default volume level
• min_time_between_play: Minimum interval between audio playback
• speed_rate: TTS speech rate

base_config - Basic information

• robot_name: Product name
• module_type: Module type (0 - No module, 1 - Robotic arm, 2 - Gimbal)
• sbc_version: Host version

sbc_config - Upper device settings

• feedback_interval: Interval for receiving feedback information
• disabled_http_log: Disable HTTP server log information

args_config - Robot parameter settings

• max_speed: Maximum speed
• slow_speed: Low-speed speed
• max_rate: Maximum speed rate
• mid_rate: Medium speed rate
• min_rate: Low-speed speed rate

cv - OpenCV parameter settings

• default_color: Default target color for color recognition
• color_lower: HSV LOWER value of target color
• color_upper: HSV UPPER value of target color
• min_radius: Radius threshold of target area
• sampling_rad: Sampling area radius
• track_color_iterate: Color tracking (pan-tilt) speed rate
• track_faces_iterate: Face tracking (pan-tilt) speed rate
• track_spd_rate: Pan-tilt rotation speed
• track_acc_rate: Pan-tilt rotation acceleration
• aimed_error: Aiming lock judgment threshold

cmd_config - Instruction type codes

These codes are related to the definitions of instructions in the lower computer program and are used for communication debugging between the upper and lower computers. Changing these involves changing the lower computer program.

code - Function codes

These codes correspond to functions, and the frontend page also needs to load this .yaml file to obtain these configurations. This way, when the frontend web application communicates with the backend, different buttons correspond to different functions. Change only if necessary.

fb - Feedback information codes

These codes correspond to types of feedback information. Some of this feedback is from the chassis to the host controller, and some is from the backend to the frontend. Using the same .yaml file for both the backend and frontend ensures uniformity of these codes and feedback information types. Change only if necessary.s and feedback information types. Change only if necessary.

In previous tutorials, we briefly introduced how to disable the automatic startup of the product's main program. The method used is to comment out the command to run the product's main program in the Crontab file. In this chapter, you will learn more about Crontab and why we use Crontab instead of Services for automatic startup.
Crontab is a tool in the Linux system used for scheduling periodic tasks. Through Crontab, users can set specific times, dates, or intervals to execute specific commands or scripts. Here are some important concepts and usages of Crontab:

Crontab File

The Crontab file is where scheduling information for periodic tasks is stored. Each user has their own Crontab file for storing their own scheduling information. Crontab files are usually stored in the /var/spool/cron directory, named after the user's username.

Crontab Format

Each line in the Crontab file represents a scheduled task. Each line consists of five fields representing minutes, hours, date, month, and day of the week. You can use the # symbol to comment out a line to disable the corresponding task scheduling.

Usage

To edit the Crontab file, you can use the crontab -e command. This command opens a text editor, allowing the user to edit their Crontab file. After editing, save and exit the editor, and the Crontab file will be updated. Common options:

• -e: Edit the user's Crontab file.
• -l: List the contents of the user's Crontab file.
• -r: Remove the user's Crontab file.
• -u: Specify the user to operate on.

Comparison

In contrast, using services to achieve automatic startup at boot time is typically done by executing a series of predefined services or scripts when the system starts up. These services can be found in the /etc/init.d/ directory and started, stopped, or restarted using system service management tools like systemctl.

Advantages of Crontab

• Flexibility: Tasks can be scheduled very flexibly, including specifying minutes, hours, dates, months, and days of the week.
• Simplicity: Crontab configuration is relatively simple, making it convenient for scheduling simple periodic tasks.
• User Independence: Each user has their own Crontab file, allowing them to manage their own tasks without affecting other users.

Advantages of Services

• Reliability: Services implemented through services are often more stable and reliable as they are system-level services that are automatically loaded and run when the system starts.
• Management: System administrators can more easily manage services, including starting, stopping, restarting, and checking status.
• Control Permissions: For tasks requiring privileged execution, using services provides better control over permissions to ensure security.

Special Advantages of Crontab in this Product

Lower resource usage: Through our testing and comparison, the CPU resource usage of the same Python script using Crontab is 1/4 of that using services. For applications like complex robot main programs, using Crontab for automatic startup is a better choice. Services are more suitable for important services or applications that need to be executed at system startup. to be executed at system startup.

To facilitate the secondary development of the product, we have added a command-line input window in the WEB application. You can input commands in this window, and after clicking the SEND button, the command will be sent to the upper computer application. The upper computer application will execute corresponding functionalities or parameter adjustments based on the command you send.
We already have some ready-made commands that you can refer to in the following sections of the WEB Command Line Application to learn about those commands. In this section, we will introduce how to implement custom command-line functionality while explaining how this feature is implemented, making it easier for you to understand the subsequent sections.

Adding Functionality

The example demo for command-line functionality is written in the main program robot_ctrl.py, and they are handled by the cmd_process() function. Below is our default command-line instruction processing function. This function is incomplete because the content afterward deals with other functionalities, which are omitted here without affecting the understanding of the function itself.
Note: The code block below cannot be executed in JupyterLab and is only used for illustration purposes.

def cmd_process(self, args_str):
    global show_recv_flag, show_info_flag, info_update_time, mission_flag
    global track_color_iterate, track_faces_iterate, track_spd_rate, track_acc_rate
    # Split the input parameter string into a list: args
    args = args_str.split()
    if args[0] == 'base':
        self.info_update("CMD:" + args_str, (0,255,255), 0.36)
        if args[1] == '-c' or args[1] == '--cmd':
            base.base_json_ctrl(json.loads(args[2]))
        elif args[1] == '-r' or args[1] == '--recv':
            if args[2] == 'on':
                show_recv_flag = True
            elif args[2] == 'off':
                show_recv_flag = False

    elif args[0] == 'info':
        info_update_time = time.time()
        show_info_flag = True

    elif args[0] == 'audio':
        self.info_update("CMD:" + args_str, (0,255,255), 0.36)
        if args[1] == '-s' or args[1] == '--say':
            audio_ctrl.play_speech_thread(' '.join(args[2:]))
        elif args[1] == '-v' or args[1] == '--volume':
            audio_ctrl.set_audio_volume(args[2])
        elif args[1] == '-p' or args[1] == '--play_file':
            audio_ctrl.play_file(args[2])

Let's take audio -s hey hi hello as an example. This command is used for text-to-speech functionality, where audio represents an audio-related function, `-s` or `--say` indicates text-to-speech, and the following parameters are the content you want it to say. After sending this command, the robot will say "hey hi hello".
Firstly, when this function receives a command-line instruction since the command-line instruction is a string, we need to use args = args_str.split() to convert this string into a list. Then, we can check each value in the list to execute the corresponding functionality. If you need to extend other custom functionalities, you just need to add another elif args[0] == 'newCmd'.

To make the configuration of product parameters easier and allow users to add custom functionalities conveniently, we have designed command-line parameter functionality for the product. You can achieve corresponding functionalities by inputting commands into the command-line tool on the web page. This chapter will detail these functionalities.

Sending Information via ESP-NOW

Add Broadcast MAC Address to Peer

• send -a -b
• send --add --broadcast
• send -a FF:FF:FF:FF:FF:FF
• send --add FF:FF:FF:FF:FF:FF

Add Specific MAC Address to Peer

• send -a AA:BB:CC:DD:EE:FF
• send -add AA:BB:CC:DD:EE:FF

Remove Broadcast MAC Address from Peer

• send -rm -b
• send --remove --boardcast
• send -rm FF:FF:FF:FF:FF:FF
• send --remove FF:FF:FF:FF:FF:FF

Remove Specific MAC Address from Peer

• send -rm AA:BB:CC:DD:EE:FF
• send --remove AA:BB:CC:DD:EE:FF

Broadcast Sending (Broadcast MAC should be added to peer before first use)

• send -b what's up bro
• send --broadcast what's up bro

Unicast Sending (Target MAC should be added to peer before first use)

• send AA:BB:CC:DD:EE:FF what's up bro

Multicast Sending (Target MAC should be added to peer before first use, multiple targets can be added, and cannot include broadcast MAC address: FF:FF:FF:FF:FF:FF)

• send -g what's up bro
• send --group what's up bro

Audio Playback

Text-to-Speech (TTS, initialization takes longer for the first use)

• audio -s what's up bro
• audio --say what's up bro

Set Volume Level (Range between 0 and 1.0)

• audio -v 0.9
• audio --volume 0.9

Play Audio File from the 'sounds' Directory (.mp3 .wav formats, can play files from other directories)

• audio -p file.mp3
• audio --play_file file.mp3
• audio -p others/file.mp3

Chassis

Send JSON Commands Directly to Chassis, Specific commands should be referred to related WIKI

• base -c {'T':1,'L':0,'R':0}
• base --cmd {'T':1,'L':0,'R':0}

Enable Displaying Information from Chassis on Screen

• base -r on
• base --recv on

Disable Displaying Information from Chassis on Screen

• base -r off
• base --recv off

Default Display Time for Command Line/ESP-NOW Information is 10 seconds

• info

OpenCV

Set Target Color Range

• cv -r [90,120,90] [120,255,200]
• cv -range [90,120,90] [120,255,200]

Select Color (Default colors are red, green, and blue only)

• cv -s red
• cv --select red

Set Pan-tilt Tracking Parameters

Color Tracking Iteration Ratio

• track -c 0.023

Face/Gesture Tracking Iteration Ratio

• track -f 0.068

Tracking Speed

• track -s 60

Action Acceleration Ratio

• track -a 0.4