Google Chat: zj734465502@gmail.com
+86-0755-88291180
sales01@spotpear.com
dragon_manager@163.com
services01@spotpear.com
manager01@spotpear.com
WhatsApp:13246739196
This robot uses the architecture of the host device + slave device (Dual Controller architecture), the host device can be Raspberry Pi, Jetson Nano, Orin Nano or other single-board computers with similar 40PIN interfaces, and the slave device uses ESP32 to control robot peripherals, read sensor information, and control motor closed-loop speed (PID controller).
The host device and the slave device use JSON format instructions to communicate through the serial port, the specific communication content can refer to the documentation of the slave device. As a beginner, you don't need to understand those instructions, you only need to follow the tutorial document to call common instructions or encapsulated functions in the early stage.
A Jupyter notebooks(.ipynb) is a document that combines ready-to-run code with narrative text (Markdown), equations (LaTeX), images, interactive visualizations, and other rich outputs.
JupyterLab has two working modes, one is COMMAND mode and the other is EDIT mode
When you are in COMMAND mode, you can quickly perform the overall operations of the notebook, such as adding or removing cells, moving cells, changing cell types, etc. In this mode, the cell border is grayed out. You can enter COMMAND mode by pressing the Esc key.
EDIT mode allows you to enter or modify code or text in a cell. In this mode, the cell border is blue. You can enter EDIT mode by clicking or pressing Enter in the selected cell.
In JupyterLab, you can do the following:
After selecting the correct Kernel, you can run the code block in the notebook. In JupyterLab, code blocks are the basic building blocks of notebooks. Here's how to run a block:
print("test text in jupyterlab")
for i in range(0, 10): print(i)
With these basic how-tos, you can effectively use JupyterLab for a variety of tasks. More advanced features and detailed instructions can be found in the official JupyterLab documentation.
In this chapter, we'll provide a Python demo for controlling the motion of a robot's chassis. This approach can be adapted to other programming languages for similar motion control tasks.
We utilize code blocks within JupyterLab to compose JSON commands. These commands are then sent to the slave controller via the GPIO serial port on a Jetson (the default baud rate for communication with the slave controller is 115200). Upon receiving these commands, the slave controller executes the specified actions.
Further sections will delve into the variety of commands that can be sent to the slave controller. Alternatively, you might choose to implement this function in a different programming language or develop a dedicated application for the host controller.
Adopting a dual-controller architecture significantly liberates the resources of the host device. In this setup, the host controller (SBC such as Raspberry Pi or Jetson) acts as the "brain", while the ESP32 slave controller (sub-controller) serves as a "cerebellum-like" entity handling the lower-level motion controls. This division of tasks allows the host to focus on high-level tasks like vision processing and decision-making, while the sub-controller efficiently manages precise movement control and other low-level tasks. Such an arrangement ensures that the sub-controller can maintain accurate wheel rotation speeds through high-frequency PID control, without overburdening the host with computationally intensive tasks.
The main program of the product, app.py, automatically executes upon booting due to the configuration set by autorun.sh (which is pre-configured in the product). Running app.py occupies the GPIO serial port and the camera, which might lead to conflicts or errors if other tutorials or programs require these resources. Ensure to disable the auto-start of app.py before proceeding with development or learning.
As app.py employs multithreading and uses crontab for autostart, typical commands like sudo killall python won't suffice to terminate it. You would need to comment out the relevant line in crontab and reboot the device.
crontab -e
Upon your first use of this command, you will be prompted to select an editor to open the crontab file. It is recommended to choose nano by entering the corresponding number for nano, followed by pressing the Enter key to confirm.
Use the # symbol to comment out the line containing ...... app.py.
# @reboot ~/ugv_pt_rpi/ugv-env/bin/python ~/ugv_pt_rpi/app.py >> ~/ugv.log 2>&1
@reboot /bin/bash ~/ugv_pt_rpi/start_jupyter.sh >> ~/jupyter_log.log 2>&1
Note: Make sure not to comment out the line containing start_jupyter.sh, as doing so will prevent JupyterLab from launching on startup, disabling your access to interactive tutorials.
To exit and save the changes, after editing the content of crontab, press Ctrl + X to exit nano. Since you have made edits to the crontab file, it will ask if you want to save the changes (Save modified buffer?). Enter the letter Y and then press Enter to exit and save the modifications.
After restarting the device, the main program will no longer run automatically upon boot-up, allowing you to freely use the tutorials within JupyterLab. If you wish to re-enable the automatic execution of the main program at startup in the future, you can reopen the crontab file using the method described above. Simply remove the # symbol in front of the @ symbol, then exit and save the changes. This will restore the automatic startup functionality of the main program.
In the following demo, it's crucial to use the correct device name and baud rate (default 115200) that matches the sub-controller.
Before executing the code block below, ensure the robot is elevated so that the drive wheels are off the ground. Activating the code will cause the robot to move; take precautions to prevent it from falling off the table.
from base_ctrl import BaseController
import time
base = BaseController('/dev/ttyTHS0', 115200)
# The wheel rotates at 0.2 m/s for 2 seconds and then stops
base.send_command({"T":1,"L":0.2,"R":0.2})
time.sleep(2)
base.send_command({"T":1,"L":0,"R":0})
By invoking the code block mentioned above, the Raspberry Pi will initially send the command {"T":1,"L":0.2,"R":0.2} (the structure of commands will be discussed in more detail in later chapters). This command starts the wheels turning. After a two-second interval, the Raspberry Pi will send another command {"T":1,"L":0,"R":0}, causing the wheels to stop. It's important to note that even if the command to stop the wheels isn't sent, the wheels will still cease turning if no new commands are issued. This is due to a heartbeat function built into the sub-controller. The purpose of this heartbeat function is to halt the current motion command automatically if the host controller hasn't sent any new commands to the sub-controller for an extended period. This function is designed to prevent continuous motion of the sub-controller in case the host encounters a problem that leads to a freeze or crash.
If you want the robot to continue moving indefinitely, the master control unit needs to cyclically send motion control commands every 2 to 4 seconds.
You may find that the direction or speed of the robot's wheels does not match your expectations after inputting the motion control commands. This could be because, before controlling the chassis, you need to set the type of chassis. Each time you run the main demo (app.py), the chassis type is automatically configured, and the configuration parameters are stored in config.yaml. This ensures that the chassis is controlled according to the correct parameters. By default, the chassis needs to be configured once by the host computer every time it is powered on. You can send the following commands to the chassis to set its type:
"main" value means the chassis type:
"module" value means the module type:
For example, if you use the UGV Rover with the pan-tilt module, you can send the following commands to the slave computer to set the chassis type:
{"T":900,"main":2,"module":0} or {"T":900,"main":2,"module":2}
You can modify the JSON parameter to set the chassis model in the following demo block according to your products:
base.send_command({"T":900,"main":2,"module":0})
Then the following code block is executed to control the chassis, and the wheels turn in the right direction and at the right speed.
base.send_command({"T":1,"L":0.2,"R":0.2})
time.sleep(2)
base.send_command({"T":1,"L":0,"R":0})
The earlier demo allows you to make the robot move forward for two seconds before stopping. Further adjustments to the parameters can control the direction of the chassis, based on the differential steering principle.
When turning, the inner wheels (on the side towards which the turn is made) travel a shorter distance and thus should rotate slower to maintain stability. The differential gear system achieves this by allowing the drive wheels to rotate at different speeds. Usually, the outer wheels (on the opposite side of the turn) rotate faster than the inner wheels. This variation in speed results in the turning motion of the vehicle, allowing it to steer in the intended direction.
You can control the vehicle's steering by assigning different target linear velocities to each wheel, thus achieving maneuverability and easily adjustable turning radii.
A product with a pan-tilt contains two servos, a pan servo and a tilt servo. The pan servo controls the rotation of the horizontal plane of the pan-tilt, and the rotation range is ±180° (total range 360°); The tilt servo controls the rotation of the vertical plane of the pan-tilt, and the rotation range is -45°~90° (total range 135°).
For products that do not come with a pan-tilt, you can also expand the pan-tilt on the RaspRover yourself.
# Import the library used to control the chassis
from base_ctrl import BaseController
base = BaseController('/dev/ttyTHS0', 115200)
In the code block above we import and instantiate the library for chassis control, and then control the pan-tilt movement by changing the angle of the pan-tilt pan servo and tilt servo.
Change the value in the following code, in the following code:
Running the following code, you can see that the pan-tilt will move 45° to the right and up and then stop.
input_x = 45 input_y = 45 input_speed = 0 input_acc = 0 base.gimbal_ctrl(input_x, input_y, input_speed, input_acc)
In addition to controlling the pan-tilt movement by changing the rotation angle of the two servos in the pan-tilt, you can also directly control the pan-tilt continuous movement.
Change the value in the following code, in the following code:
Only when the value of input_x and input_y is 2, the pan-tilt will automatically return to the median position.
Run the following code and the pan-tilt will move to the left until it stops when the movement reaches 180°.
input_x = -1 input_y = 0 input_speed = 0 base.gimbal_base_ctrl(input_x, input_y, input_speed)
Run the following code, the pan-tilt will move upwards and stop when the movement reaches 90°.
input_x = 0 input_y = 1 input_speed = 0 base.gimbal_base_ctrl(input_x, input_y, input_speed)
Finally, run the following code to return the pan-tilt to the middle position.
input_x = 2 input_y = 2 input_speed = 0 base.gimbal_base_ctrl(input_x, input_y, input_speed)
WAVE ROVER and UGV series products, the driver board integrates two 12V switch control interfaces (the actual maximum voltage will change with the battery voltage), respectively, the IO4 and IO5 pins of ESP32 are controlled through the MOS tube, each corresponding to two interfaces, a total of 4 12V switch control interfaces, according to the default assembly method, IO4 controls the chassis headlights (WAVE ROVER does not have chassis headlights), IO5 controls the headlights, you can control the switch of these two interfaces and control the voltage by sending corresponding commands to the lower computer (but due to the MOS tube control itself has a certain delay, the PWM of the IO output of ESP32 is not linear with the voltage output of the actual interface).
For products that are not equipped with LED lights, you can expand the 12.6V withstand voltage LED lights on these two 12V switch control interfaces (under normal circumstances, the 12V withstand voltage can also be used, because in order to safety and protect the battery, the UPS of the product will not charge the battery above 12V), you can also expand other peripherals on the remaining switch control interfaces - such as the 12V water bomb gun gearbox, which is directly connected to the IO5 control interface, and can realize the function of automatic aiming and shooting.
If you need to run a program within a code block, you can select the code block you want to run and press Ctrl+Enter to run the program, or if you are using JupyterLab on your phone or tablet, you can press the triangle play button above to run the code block.
In the above code block we import and instantiate the library used to control the chassis, and then control the switch of the IO4 interface, the variable IO4_PWM is the PWM of the IO4 pin output of ESP32, the variable range is 0-255, when the value of this variable is 0, the interface switch of the IO4 control is turned off; When this variable is 255, the output voltage of the IO4 controlled interface switch is close to the BAT voltage of the UPS (the voltage of the current three lithium batteries in the UPS in series).
Run the following code block to turn on the interface switch for IO4 control (light up the chassis headlights). Note: The WAVE ROVE does not have chassis headlights, so there is no change to run the following code block, you need to run the next code block that turns on the headlights to turn on the headlights, the headlights are located on the camera pan-tilt, if the product is not equipped with a camera pan-tilt, there are no headlights.
IO4_PWM = 255 IO5_PWM = 0 base.lights_ctrl(IO4_PWM, IO5_PWM)
Run the following code block to turn on the interface switch of IO5 control (light up the pan-tilt headlight). Note: If the product is not equipped with a camera pan-tilt, there is no headlamp.
IO4_PWM = 255 IO5_PWM = 255 base.lights_ctrl(IO4_PWM, IO5_PWM)
If your product has LEDs that should all be lit up by now, you can run the following code block to reduce the brightness of the LEDs:
IO4_PWM = 64 IO5_PWM = 64 base.lights_ctrl(IO4_PWM, IO5_PWM)
Finally, run the following code block to turn off the LED lights.
base.lights_ctrl(0, 0)
Here runs a code block integrated with pan-tilt functionality.
import time
base.gimbal_ctrl(0, 0, 0, 0)
base.lights_ctrl(255, 255)
time.sleep(0.3)
base.gimbal_ctrl(45, 0, 0, 0)
base.lights_ctrl(0, 0)
time.sleep(0.3)
base.gimbal_ctrl(-45, 0, 0, 0)
base.lights_ctrl(255, 255)
time.sleep(0.3)
base.gimbal_ctrl(0, 90, 0, 0)
base.lights_ctrl(0, 0)
time.sleep(0.3)
base.gimbal_ctrl(0, 0, 0, 0)
This tutorial demonstrates how to control an OLED display connected to an ESP32 module using JSON commands. OLED displays are widely used for showing various types of information, such as text and images.
OLED displays communicate with the ESP32 module via the I2C (Inter-Integrated Circuit) interface. These displays are capable of showing custom text content and support multi-line display.
The product comes with an OLED display that communicates with the ESP32 module through the I2C interface. Upon powering up, the display automatically shows some basic information about the slave device. The content displayed on the screen can be altered by sending JSON commands from the host device.
lineNum refers to the line number. A single JSON command can modify the content of one line. When the subordinate machine receives a new command, the default OLED display screen at startup will disappear, replaced by the content you've added. For most products that use a 0.91-inch OLED display, the value of lineNum can be 0, 1, 2, or 3, allowing for a total of four lines. Text is the textual content you wish to display on that line. If the content for a line is too long, it will automatically wrap to the next line, potentially overwriting the content on the last line.
from base_ctrl import BaseController
base = BaseController('/dev/ttyTHS0', 115200)
#Modify the OLED display content
base.send_command({"T":3,"lineNum":0,"Text":"this is line0"})
base.send_command({"T":3,"lineNum":1,"Text":"this is line1"})
base.send_command({"T":3,"lineNum":2,"Text":"this is line2"})
base.send_command({"T":3,"lineNum":3,"Text":"this is line3"})
Running the provided code block will display four lines of text on the OLED:
this is line0 this is line1 this is line2 this is line3
The tutorial above outlined a method for displaying simple text on the OLED screen. We will now proceed with a slightly more complex example. Running the following code block will display the current time on the OLED screen. Note that the time displayed might not be accurate due to potential discrepancies with the Raspberry Pi's clock. This example serves to demonstrate how to update the screen content in the main program, where we employ this method to display real-time information such as the device's IP address and operational status on the OLED screen.
# Import the datetime class from the datetime module to fetch and manipulate the current date and time.
from datetime import datetime
# Import the time module, primarily used for delay processing within the program.
import time
# Create an infinite loop using while True to allow the program to run continuously.
while True:
# Use datetime.now().strftime("%H:%M:%S") to obtain the current time and format it as "hour:minute:second".
current_time = datetime.now().strftime("%H:%M:%S")
# Utilize the base.send_command method to send a command that includes the current time.
base.send_command({"T":3,"lineNum":0,"Text":current_time})
# Use time.sleep(1) to pause the program for 1 second, ensuring that the time is updated and a command is sent every second.
time.sleep(1)
Running the last code block, you'll observe the first line of the OLED screen updating to show the current time, refreshing every second. This function runs in an infinite loop, which can be terminated by clicking the stop button(■) above.
Building UI interfaces in JupyterLab typically uses the ipywidgets library, which provides a simple yet powerful way to create interactive user interfaces. Here are the detailed steps:
The ipywidgets library has been installed in our product. If you can't find the library when you run the block, you can install the library you need in the UI by pip install ipywidgets.
Select the following code block and press Ctrl + Enter to run the code.
import ipywidgets as widgets
from IPython.display import display
We can use various components from the ipywidgets library to build our UI interface, such as text boxes, buttons, output boxes, etc. For example:
# Create a text box
text = widgets. Text(description='Please enter a name:')
# Create a button
button = widgets. Button(description="Say hello")
# Create an output box
output = widgets. Output()
We need to define a handler that handles user interaction events. In this example, we'll define a function to handle the click event of the button and display a greeting in the output box.
# Define a function greet_user that takes a sender argument that represents the object that triggers the event, such as a button
def greet_user(sender):
# Use the with statement and the output object to capture the output of the print function so that it appears in the expected output area
# output is the output object that has been defined
with output:
# Use the print function to output a greeting where the format method is used to insert the current value of the text control into the string
# "{}" is a placeholder that the format function replaces with the value of text.value
print("Hello, {}".format(text.value))
# Use the on_click method to associate the button's click event with the greet_user function
# When the user clicks the button, the greet_user function is called
button.on_click(greet_user)
Finally, we put all the UI components in a layout and display them through the display function.
# Put all the components in a vertical layout
ui = widgets. VBox([text, button, output])
# Display UI
display(ui)
With these steps, we can build a simple UI in JupyterLab. The user can enter the content in the text box, and after clicking the button, the program will display the corresponding greeting in the output box according to the input content.
After the product is turned on, the slave device will continue to feedback all kinds of information to the upper computer by default, and you can get the current working status of the product through these feedback information.
Normally, you need to continuously get the feedback from the slave device, but in this example, we only get a JSON message that is fed back by the slave device (comment out or delete the break line to get the feedback continuously).
Select the following code block, run the code block with Ctrl + Enter, when it gets the first complete JSON information with a T value of 1001, it will jump out of the loop and output feedback information, including the current wheel speed, IMU, pan-tilt angle (if installed), robot arm angle (if installed), power supply voltage, etc.
from base_ctrl
import BaseController import json
base = BaseController('/dev/ttyTHS0', 115200)
# Use an infinite loop to continuously listen to serial data
while True:
try:
# Read a line of data from the serial port, decode it into a string in 'utf-8' format, and try to convert it to a JSON object
data_recv_buffer = json.loads(base.rl.readline(). decode('utf-8'))
# Check whether the parsed data contains the 'T' key
if 'T' in data_recv_buffer:
# If the value of 'T' is 1001, then print the received data and jump out of the loop
if data_recv_buffer['T'] == 1001:
print(data_recv_buffer)
break
# If an exception occurs while reading or processing data, Ignore the exception and continue to listen for the next row of
except:
pass
The following code is only used to understand the principle of reading JSON information from the underlying serial port, and the following code blocks cannot be executed.
class ReadLine:
# Constructor to initialize an instance of the ReadLine class
# s: The serial port object passed in to communicate with the serial port.
def __init__(self, s):
self.buf = bytearray() # Initialize a byte array to store the data read from the serial port but not yet processed
self.s = s # Save the incoming serial port object, which will be used to read the serial port data later
def readline(self):
i = self.buf.find(b"\n") # Find if there is a line break in the buffer
if i >= 0:
r = self.buf[:i+1] # If there is a line break, extract the data before the line break
self.buf = self.buf[i+1:] # Update the buffer to remove the processed data
return r
while True:
i = max(1, min( 512, self.s.in_waiting)) # Get the number of bytes that can be read, up to 512 bytes
data = self.s.read(i) # Read data from the serial port
i = data.find(b"\n") # Find a line break
if i >= 0:
r = self.buf + data[:i+1] # If a line break is found, merge the read data with the data in the buffer
self.buf[0:] = data[i+1:] # Update buffer to remove processed data
return r
else:
self.buf.extend(data) # If no newline is found, add data to buffer
This product is developed using the dual controller architecture, and the host controller sends JSON format commands to the slave device through the serial port (Jetson through the GPIO serial port). Note: This chapter is a precursor to the introduction of the JSON instruction set of the slave device, and the content is similar to the content of the previous Python chassis motion control chapter.
JSON (JavaScript Object Notation) is a lightweight data exchange format, which has become one of the standards for data transmission on the Internet. Here are some of the advantages of JSON:
In the following demo, you need to use the correct GPIO device name and use the same baud rate as the slave device (115200 by default).
Before running the following code block, you need to raise the product and keep the drive wheels all off the ground, after calling the following code block, the robot will start to move, be careful not to let the robot fall off the desktop.
from base_ctrl import BaseController
import time
base = BaseController('/dev/ttyTHS0', 115200)
# The wheel rotates at a speed of 0.2m/s for 2 seconds and then stops
base.send_command({"T":1,"L":0.2,"R":0.2})
time.sleep(2)
base.send_command({"T":1,"L":0,"R":0})
By calling the code block above, Jetson will first send {"T":1,"L":0.2,"R":0.2} (we will explain the composition of the command in detail in the following chapter), the wheel will start to turn, and after two seconds Jetson will send {"T":1,"L":0,"R":0} This command, the wheel will stop rotating, and it should be noted here that even if you don't send the later command to stop the wheel turning, if you don't send a new command, The wheel will still stop rotating, this is because the slave device contains a heartbeat function, the heartbeat function is used when the host controller has no new instructions sent to the slave device for a long time, the slave device automatically stops the current moving instructions, and the purpose of changing the function is to avoid the host controller from crashing for some reasons and causing the slave device to continue to move.
If you want the robot to continue to move continuously, the host computer needs to send motion control instructions every 2-4 seconds.
In the previous chapter, we introduced a simple demo, in which we send motion control commands to the sub-controller through the host controller, and the sub-controller can receive a lot of instructions, in this chapter we will introduce these instructions.
In the previous section, we sent {"T":1,"L":0.2,"R":0.2} instruction as an example, the T value in this JSON data represents the type (Type) of the instruction, the L value represents the target linear velocity of the left (LEFT) wheel, the R value represents the target linear velocity of the right (RIGHT) wheel, and the unit of linear velocity is m/s by default. The parameters of the motion control are the target linear velocity of the left and right wheels, respectively.
All subsequent JSON directives will contain a T-value that defines the type of instruction, but the specific instruction parameters will vary depending on the instruction type.
You can view the definitions of these instructions in the json_cmd.h file of our open-source sub-controller demos, or add new sub-controller features to them yourself.
These instructions are the most basic instructions for mobile robots, which are used for motion-related function control.
Each of the following directives consists of three parts: a case, a short introduction, and a detailed introduction.
L and R represent the target linear velocity of the left and right wheels respectively, the unit is m/s, the negative value is the reverse rotation of the substitution, and 0 is the stop. The value range of the target linear speed depends on the diameter of the motor/reducer/wheel used in the product, and the relevant calculation formula can be found in the open source sub-controller demo. It should be noted here that for chassis using carbon brush DC motors, when the absolute value of a given target speed is very small (but not 0), the speed of the product may fluctuate greatly during movement due to the low speed performance of carbon brush DC motors is usually poor.
L and R represent the PWM values of the left and right wheels respectively, the range of the values is -255 ~ 255, the negative value represents the reverse, when the absolute value of the value is 255, it means that the PWM is 100%, which means to let the wheels on this side run at full power.
This instruction is used for the ROS host computer to control the command of chassis movement, X represents the linear speed of movement, the unit is m/s can be negative; Z stands for angular velocity of steering in rad/s and can be negative.
This command is used to tune the parameters of the PID controller, the PID parameter in the above JSON example is the default parameter of the product, where L stands for WINDUP_LIMITS, which is the reserved interface, This parameter is not currently used in the current product.
An OLED display is installed on the product, which communicates with the ESP32 module of the host computer of the sub-controller through I2C, and the host computer can change the content displayed on the display by sending JSON commands.
lineNum is the number of lines, a JSON command can change the content of a line, after the sub-controller receives a new instruction, the default OLED interface will disappear when booted, and it will be replaced by your newly added content, For the 0.91-inch OLED display used in most products, the lineNum value can be 0, 1, 2, 3, a total of four lines; Text is the text you want to display on this line, and if you have too much content in this line, it will automatically wrap, but it will also squeeze out the last line.
Use this command to make the OLED display display display the default screen when powered on.
Different types of modules will be installed on the mobile chassis (none/robotic arm/pan-tilt),Use this command to tell the sub-controller the type of module currently installed,This instruction is usually automatically sent to the sub-controller when the host controller is turned on,This part will be introduced in the following chapters。
cmd value represents the type of module, there are currently three types to choose from, 0: install nothing, 1: robotic arm, 2: pan-tilt
The chassis is equipped with an IMU sensor, you can get the data of the IMU sensor through the following commands, it should be noted here that the chassis information continuous feedback function (which contains the IMU information) will be turned on by default after the product is turned on, and the IMU related functions here are only necessary to use when the chassis information continuous feedback function is turned off.
Send this command to get the data of the IMU.
Current product program does not need to perform calibration, this command is reserved interface.
Use this command to feedback the current offset of each axis of the IMU.
Use this command to set the offset of each axis of the IMU, this command is a reserved interface, and the current product does not need to execute this command.
The product usually turns on chassis information feedback by default after booting, which is automatic, if the chassis information continuous feedback function is turned off, you need to obtain the chassis information at a time, you can use this command to obtain the basic information of the chassis.
The value of cmd is set to 1, and the function is turned on by default, and the chassis information will be continuously fed; When the value of cmd is set to 0, turn off the function, and after the function is turned off, the host computer can get the chassis information through CMD_BASE_FEEDBACK.
The value of cmd is the extra interval that needs to be set, the unit is ms, and the frequency of chassis feedback can be adjusted through this command.
When the value of cmd is set to 0, turn off echo; When the value of cmd is set to 1, the echo is enabled, and when the command echo mode is enabled, the sub-controller will output the received command.
When cmd is 0, turn off the WIFI function. 1-ap; 2-sta; 3-ap+sta。
The sub-controller of the product has two 12V switch interfaces, each with 2 interfaces for a total of four interfaces, you can set the output voltage of these interfaces through this command, when the value is 255, it is 3S battery voltage. By default, the product uses these interfaces to control the LED lights, and you can control the brightness of the LED lights through this command.
This command is used to control the orientation of the pan-tilt. X is the horizontal direction in angle, positive values to the right, negative values to the left, and the value range is -180 to 180. Y is the direction of the value in angle, positive values are up, negative values are down, and the values range from -30 to 90. SPD is the velocity, ACC is the acceleration, and when the value is 0, it is the fastest speed/acceleration.
This command is used to continuously control the pan-tilt orientation. X is the horizontal direction in angle, positive values to the right, negative values to the left, and the value range is -180 to 180. Y is the direction of the value in angle, positive values are up, negative values are down, and the values range from -30 to 90. SX and SY are the speeds of the X and Y axes, respectively.
When using the above command to make the pan-tilt move, you can use this command to stop the pan-tilt at any time.
When s is 0, turn off this function when s is 1, after this function is enabled, the pan-tilt will automatically adjust the vertical angle of the pan-tilt through the IMU data, and y is the target angle between the pan-tilt and the ground (even if the pan-tilt self-stabilization function is turned on, the camera can also look up and down).
This command is used to control the pan-tilt in the UI interface, the X value can be -1, 0 and 1, -1 is to turn left, 0 is to stop, and 1 is to turn right. The Y value can be -1, 0 and 1, -1 for downward rotation, 0 for stopping, and 1 for upward rotation. SPD is velocity
Under normal circumstances, the robotic arm will automatically rotate to the initial position when it is turned on. This directive causes process blocking.
This function will cause a block
This function does not cause blocking
*{"T":107,"tor":200}
tor's value can be up to 1000, representing 100% strength.
cmudd unit is ms, you can use this command to set the heartbeat function time, if the sub-controller does not receive a new exercise command within the time, it will automatically stop moving, It is used to avoid the host controller crashing during use, causing the sub-controller to move all the time. During the process, the host controller crashed, causing the sub-controller to keep moving.
The product uses the principle of differential steering, when the left and right wheels of the product give the same target speed, the product may not go straight due to encoder error or tire grip error, you can use this command to fine-tune the speed of the left and right wheels, for example, the left wheel needs to rotate slower, you can change the value of L to 0.98. Try not to set values greater than one for L and R values.
Use this command to get the current velocity ratio
This function belongs to the high-level functions of the sub-controller, and the following operations are usually not required when used with the host controller.
raw is the original ID of the servo (the new servos are all 1), new is the ID to be changed, the maximum is not more than 254, it cannot be negative, and 255 is the broadcast ID.
This chapter tutorial is used to introduce the host controller will automatically execute instructions and send some instructions to the sub-controller every time it is turned on, the code blocks in this chapter of the tutorial do not need to be executed (and cannot be executed), only used to understand some automatic operations of the product after booting, if you have the need, change or add these instructions.
cmd_on_boot() function is located in the main program app.py of the product, this function will be called every time you boot up, you can edit this function to adjust the parameters/add instructions to the instructions that run automatically at boot.
def cmd_on_boot():
# Define the list of commands to be executed at startup
cmd_list = [
'base -c {"T":142,"cmd":50}', # set feedback interval
'base -c {"T":131,"cmd":1}', # serial feedback flow on
'base -c {"T":143,"cmd":0}', # serial echo off
'base -c {"T":4,"cmd":2}', # select the module - 0:None 1:RoArm-M2-S 2:Gimbal
'base -c {"T":300,"mode":0,"mac":"EF:EF:EF:EF:EF:EF"}', # the base won't be ctrl by esp-now broadcast cmd, but it can still recv broadcast megs.
'send -a -b' # add broadcast mac addr to peer
]
# traversal command list
for i in range(0, len(cmd_list)):
camera.cmd_process(cmd_list[i])
The host controller of the product can control some functions through the command line instruction, similar to the base -c instruction above, which is used to directly pass the JSON instruction written later to the sub-controller through Jetson's GPIO serial port, and we will explain in detail what the default instruction here means to run automatically at boot.
Used to set the extra interval time of continuous feedback information from the sub-controller, the unit of cmd value is ms, this function is used to reduce the frequency of feedback information from the sub-controller, and the purpose is to reduce the computing power pressure of the host controller to process the feedback information of the sub-controller.
Turn on the continuous information feedback function of the sub-controller, after the function is turned on, there is no need for the host controller to ask and answer to get the information of the sub-controller, the sub-controller will normally turn on the function by default, but we still send another command to turn on the function, which is safer.
Turn off the serial port command echo, so that when the host computer sends instructions to the sub-controller, the sub-controller will no longer feed back the received instructions to the host computer, so as to avoid the host computer from processing useless information.
Set the type of the external module, if the value of cmd is 0, it means that there is no external module; 1. a mechanical arm; 2. Gimbal, if your product does not have a gimbal or robotic arm installed, you need to change the value here to 0.
Avoid the chassis being controlled by the ESP-NOW broadcast of other devices, but you can make up a MAC address by yourself in addition to the device with the mac address, or you can use the MAC address of your own ESP32 remote control.
Add the broadcast address (FF:FF) to the peer, so that you can then send the broadcast message directly to other devices through the broadcast signal.
You can learn about other host controller command line instructions through the WEB Command Line Application chapter later.
For security reasons, you don't have direct access to audio devices through JupyterLab (environmental limitations), and we don't have a code block for users to run.
The program here comes from the audio_ctrl.py of the main program of the product, and you can refer to the code here to understand how the main program of the product implements the audio file playback function.
There is a folder called sounds in the main folder of the product, and there are many subfolders in this folder: connected, others, recv_new_cmd, robot_started, searching_for_target, target_detected, target_locked.
In the default program we provide, there is only one audio file in connected and one in each robot_started.
When the main program of the robot is running, it will automatically play an audio file within a robot_started.
When a client uses a browser to connect to this WEB application, it will automatically randomly play an audio file within the connected.
You can put custom audio files in these folders as voice packs to customize your product.
import pygame # import pygame library for audio playback
import random # import random library for random selection of audio files
import yaml # import yaml library for reading configuration files
import os # import os library for file operations
import threading # Import the threading library for multithreading
# Get the configuration file
curpath = os.path.realpath(__file__) # Get the absolute path of the current script
thisPath = os.path.dirname(curpath) # Get the directory where the current script is located
with open(thisPath + '/config.yaml', 'r') as yaml_file: # Open the configuration file
config = yaml.safe_load(yaml_file) # Load configuration files using yaml library
# Initialize pygame.mixer and set the default volume for audio output
pygame.mixer.init()
pygame.mixer.music.set_volume(config['audio_config']['default_volume'])
# Create an event object that controls audio playback
play_audio_event = threading. Event()
# Read the minimum playback interval from the config file
min_time_bewteen_play = config['audio_config']['min_time_bewteen_play']
# Define the function to play audio
def play_audio(input_audio_file):
try:
pygame.mixer.music.load(input_audio_file) # Load the audio file
pygame.mixer.music.play() # Play audio
except:
play_audio_event.clear() # Clear the event flag when an error occurs
return
while pygame.mixer.music.get_busy(): # Wait for audio playback to complete
pass
time.sleep(min_time_bewteen_play) # Wait for minimum playback interval
play_audio_event.clear() # Clear event
# Define the function to play random audio
def play_random_audio(input_dirname, force_flag):
if play_audio_event.is_set() and not force_flag:
return
# Get all audio files in the specified directory
audio_files = [f for f in os.listdir(current_ path + "/sounds/" + input_dirname) if f.endswith((".mp3", ".wav")))]
# Randomly select an audio file from the list of audio files
audio_file = random.choice(audio_files)
play_audio_event.set() # Set up the event
# Create a thread to play the audio
audio_ thread = threading. Thread(target=play_audio, args=(current_path + "/sounds/" + input_dirname + "/" + audio_file,))
audio_thread.start() # Start the thread
# Define the thread function to play audio
def play_audio_thread(input_file):
if play_audio_event.is_set(): # Return if the event has already been set
return
play_audio_event.set() # Set the event
# Create a thread to play audio
audio_thread = threading. Thread(target=play_audio, args=(input_file,))
audio_thread.start() # Start the thread
# Define the function to play the specified file
def play_file(audio_file):
audio_file = current_path + "/sounds/" + audio_file # Build the full path of the audio file
play_audio_thread(audio_file) # Play the audio in a new thread
# Define the function to set the audio volume
def set_audio_volume(input_volume):
input_volume = float(input_volume) # Convert the input volume to floating-point number
if input_volume > 1: # If the volume is greater than 1, set it to 1
input_volume = 1
elif input_volume < 0: # If the volume is less than 0, set to 0
input_volume = 0
pygame.mixer.music.set_volume(input_volume) # Set the volume
# Define a function to set the minimum playback interval
def set_min_time_between(input_time):
global min_time_bewteen_play # Use the global variable
min_time_bewteen_play = input_time # Set the minimum playback interval
For security reasons, you don't have direct access to audio devices through JupyterLab (environmental limitations), and we don't have code blocks here for users to run.
The program here comes from the audio_ctrl.py of the main program of the product, and you can refer to the code here to understand how the main program of the product performs the text-to-speech function.
import pyttsx3 # import pyttsx3 library for text-to-speech functionality
import threading # import threading module for creating threads
# Initialize pyttsx3 engine
engine = pyttsx3.init()
# Create an event object to control the synchronization of voice playback
play_audio_event = threading. Event()
# Set the engine property, here is the speed of voice playback, the higher the value, the faster the speed of speech
engine.setProperty('rate', 180)
# Define a function to play the speech
def play_speech(input_text):
engine.say(input_text) # Enter the text into the engine
engine.runAndWait() # Wait for the voice output to complete
play_audio_event.clear() # Clear the event to indicate that the voice playback is complete
# Define a function to play a voice in a new thread
def play_speech_thread(input_text):
if play_audio_event.is_set(): # If there is already a voice in playback, return it directly without repeating it
return
play_audio_event.set() # Set an event to indicate that a new voice playback task starts
# Create a new thread and call play_ speech function to play speech
speech_thread = threading. Thread(target=play_speech, args=(input_text,))
speech_thread.start() # Start a new thread and start playing voice
This code uses the pyttsx3 library to implement text-to-speech functionality, and uses the threading module to create a thread to play speech asynchronously. The play_speech() function is used to play the speech of the specified text in the main thread, while the play_speech_thread() function is used to play the voice in the new thread to avoid blocking the main thread. At the same time, the synchronization of voice playback is controlled by play_audio_event to ensure that only one voice is playing at the same time.
This section introduces how to use Flask to build a web application to display the real-time picture of the robot camera, because the web application has the cross-platform feature, users can watch the real-time picture of the camera through the browser on mobile phones/PCs/tablets and other devices, and realize the wireless video transmission function.
Flask is a lightweight web application framework for building web applications quickly using Python.
Since the main program will automatically run by default when the product is booted up, the main program will occupy the camera resources, in this case, this tutorial cannot be used, you need to end the main program or prohibit the main program from running automatically before restarting the robot.
It should be noted here that the regular sudo killall python method usually does not work due to the multi-threading used in the main bot program and the automatic operation is configured by crontab, so we will introduce the method of disabling the automatic running of the main program.
If you have disabled the automatic startup of the bot's main program, you do not need to perform the End Main Program section below.
1. Clicking on the + sign next to the tab on this page above will open a new tab called Launcher.
2. Click Terminal in Other to open the Terminal window.
3. Type bash in the terminal window and press enter.
4. Now you can use the Bash Shell to control the robot.
5. Enter the command: sudo killall -9 python
The following code cannot be run in Jupyter Lab because the Flask application conflicts with Jupyter Lab over the use of port numbers, and the following programs are stored in folders named 12 in tutorial_cn and tutorial_en, and there is also a file named in folder 12 template's folder is used to store web page resources, and here's how the demo runs.
1. Use the method described above to open the terminal, at this time pay attention to the folder path on the left, the default path of the newly opened terminal is the same as the file path on the left, you need to browse to the tutorial_cn or tutorial_en folder, open the terminal and enter cd 12 to browse to the 12 folder.
2. Use the following command to start the Flask web application server: python flask_camera.py
3. Then open the browser on the device in the same LAN (or this device can open a new tab in the browser), enter the IP of Jetson: 5000 (for example, if the IP address of Jetson is 192.168.10.104, open the address 192.168.10.104:5000), note that : needs to be a colon in English.
4. Use Ctrl + C in the terminal to end the run.
from flask import Flask, render_template, Response # Import the Flask class from the flask library, the render_template function is used to render the HTML template, and the Response class is used to generate the response object
from picamera2 import Picamera2 # Import Picamera2 class from picamera2 library for accessing and controlling the camera
import time # Import time module, which can be used to handle time-related tasks
import cv2 # Import OpenCV library for image processing
app = Flask(__name__) # Create an instance of the Flask application
def gen_frames(): # Define a generator function that generates the image captured by the camera on a frame-by-frame basis
picam2 = Picamera2() # Create an instance of Picamera2
# Configure camera parameters and set the format and size of the video
picam2.configure(picam2.create_video_configuration(main={"format": 'XRGB8888', "size": (640, 480)}))
picam2.start() # Start the camera
while True:
frame = picam2.capture_array() # Capture a frame from the camera
frame = cv2.cvtColor(frame, cv2. COLOR_BGR2RGB)
ret, buffer = cv2.imencode('.jpg', frame) # Encodes the captured image frame to JPEG format
frame = buffer.tobytes() # Convert JPEG images to byte streams
# Use yield to return an image byte stream, so that you can send video frames continuously to form a video stream
yield (b'--frame\r\n'
b'Content-Type: image/jpeg\r\n\r\n' + frame + b'\r\n')
@app.route('/') # Define the root route
def index():
return render_template('index.html') # Return index.html page
@app.route('/video_feed') # Define the video stream route
def video_feed():
# Returns the response object, the content is the video stream, and the content type is multipart/x-mixed-replace
return Response(gen_frames(), mimetype='multipart/x-mixed-replace; boundary=frame')
if __name__ == '__main__':
app.run(host='0.0.0.0', port=5000, debug=True) # Start the Flask application, listen on port 5000 on all network interfaces, and enable debug mode
gen_frames(): This is a generator function that continuously captures frames from the camera, encodes them into JPEG format, and generates frame bytes as part of a multi-part response. The resulting frames are transmitted to the client in real time.
@app.route('/'): This decorator associates the index() function with the root URL(/). When a user accesses the root URL, it will render an HTML template named '12_index.html'.
@app.route('/video_feed'): This decorator associates the video_feed() function with the '/video_feed' URL. This route is used for real-time video transmission, and frames are sent as multipart responses.
app.run(host='0.0.0.0', port=5000, debug=True): This line starts the Flask development server and listens for all available network interfaces (0.0.0.0) on port 5000. debug=True option to enable the server's debug mode. The debug mode of the server.
Notes:
<!doctype html>: Declares the HTML document type.
<html lang="en">: The root element of the HTML document, specifying that the page language is English.
<head>: Contains meta information about the document, such as character set and page title.
<-- Required meta tags -->: HTML annotation, reminding that these are some required meta tags.
<meta charset="utf-8">: Specifies that the document uses the UTF-8 character set.
<title>Live Video Based on Flask</title>: Set page title.
<body>: Contains the visible part of the document.
<-- The image tag below is dynamically updated with the video feed from Flask -->: HTML annotation stating that the image tag below dynamically updates to show the video stream from Flask.
<img src="{{ url_for('video_feed') }}">: Image label that uses the video_feed route defined in Flask to get a live video stream.
In the previous chapter, we used Flask to display the real-time camera feed, a method that required opening a new tab in the browser or accessing it from another device. In this chapter, we'll explore a solution for viewing the real-time video stream directly in Jupyter Lab.
Since the product automatically runs the main program at startup, which occupies the camera resource, this tutorial cannot be used in such situations. You need to terminate the main program or disable its automatic startup before restarting the robot.
It's worth noting that because the robot's main program uses multi-threading and is configured to run automatically at startup through crontab, the usual method sudo killall python typically doesn't work. Therefore, we'll introduce the method of disabling the automatic startup of the main program here.
If you have already disabled the automatic startup of the robot's main demo, you do not need to proceed with the section on Terminate the Main Demo.
The following code block can be run directly:
import matplotlib.pyplot as plt # Import the matplotlib library for plotting
import cv2 # Import the OpenCV library for image processing
from picamera2 import Picamera2 # Import the Picamera2 library for accessing the Raspberry Pi Camera
import numpy as np # Import the NumPy library for mathematical computations
from IPython.display import display, Image # Import IPython display functionality
import ipywidgets as widgets # Import the ipywidgets library for creating interactive widgets
import threading # Import the threading library for multithreading
# Create a toggle button as a stop button
stopButton = widgets.ToggleButton(
value=False, # The initial state of the button is unselected
description='Stop', # Text displayed on the button
disabled=False, # The button is initially enabled
button_style='danger', # The button style is red
tooltip='Description', # Tooltip displayed when hovering over the button
icon='square' # Icon displayed on the button
)
# Define a function for displaying the video stream
def view(button):
# If you are using a CSI camera you need to comment out the picam2 code and the camera code.
# Since the latest versions of OpenCV no longer support CSI cameras (4.9.0.80), you need to use picamera2 to get the camera footage
# picam2 = Picamera2() # Create an instance of Picamera2
# Configure camera parameters, set video format and size
# picam2.configure(picam2.create_video_configuration(main={"format": 'XRGB8888', "size": (640, 480)}))
# picam2.start() # Start the camera
camera = cv2.VideoCapture(-1) #Create camera example
#Set resolution
camera.set(cv2.CAP_PROP_FRAME_WIDTH, 640)
camera.set(cv2.CAP_PROP_FRAME_HEIGHT, 480)
display_handle=display(None, display_id=True) # Create a display handle for updating the displayed content
while True:
# frame = picam2.capture_array() # Capture a frame from the camera
_, frame = camera.read()
# You can perform frame processing here if needed (e.g., flipping, color conversion, etc.)
_, frame = cv2.imencode('.jpeg', frame) # Encode the frame as JPEG format
display_handle.update(Image(data=frame.tobytes())) # Update the displayed image
if stopButton.value==True: # Check if the stop button is pressed
#picam2.close() # If yes, close the camera
cv2.release() # if yes, close the camera
display_handle.update(None) # Clear the displayed content
# Display the stop button
display(stopButton)
# Create and start a thread, with the target function as view and the stop button as the argument
thread = threading.Thread(target=view, args=(stopButton,))
thread.start() # Start the thread
This chapter builds upon the previous tutorial, capturing frames from the camera at regular intervals and saving them in the /templates/pictures/ folder within the ugv_jetson directory.
Since the product automatically runs the main program at startup, which occupies the camera resource, this tutorial cannot be used in such situations. You need to terminate the main program or disable its automatic startup before restarting the robot.
It's worth noting that because the robot's main program uses multi-threading and is configured to run automatically at startup through crontab, the usual method sudo killall python typically doesn't work. Therefore, we'll introduce the method of disabling the automatic startup of the main program here.
If you have already disabled the automatic startup of the robot's main demo, you do not need to proceed with the section on Terminate the Main Demo.
The following code block can be run directly:
If you are using a USB camera, you need to uncomment the line frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB).
You can adjust the value of time_intervel to change the interval between photos, measured in seconds.
The photos you capture will be stored in the `/ugv_jetson/templates/pictures/` folder..
import cv2 # Import the OpenCV library for image processing
from picamera2 import Picamera2 # Import the Picamera2 library to access the Raspberry Pi Camera
import numpy as np
from IPython.display import display, Image # Import IPython display functionality
import ipywidgets as widgets # Import the ipywidgets library for creating cloud interactive widgets
import threading # Import the threading library for multithreading
import os, time # Import the os and time libraries for file operations and time-related functionalities
# Change the interval time for taking photos here (in seconds)
time_intervel = 3 # Take a photo every 3 seconds
# Set the path for saving the images
# You can change the save path here
photo_path = '/home/ws/ugv_pt_rpi/static/'
# Create a toggle button as a stop button
# ================
stopButton = widgets.ToggleButton(
value=False, # The initial state of the button is unselected
description='Stop', # Text displayed on the button
disabled=False, # The button is initially enabled
button_style='danger', # 'success', 'info', 'warning', 'danger' or ''
tooltip='Description', # Tooltip displayed when hovering over the button
icon='square' # Button icon (FontAwesome name without the `fa-` prefix)
)
# Define a function for displaying the video stream and taking photos at regular intervals
# ================
def view(button):
last_picture_time = time.time() # Record the time of the last photo taken
num_count = 0 # Initialize the photo counter
# If you are using a CSI camera you need to comment out the picam2 code and the camera code
# Since the latest versions of OpenCV no longer support CSI cameras (4.9.0.80), you need to use picamera2 to get the camera footage
# picam2 = Picamera2() # Create an instance of Picamera2
# Configure camera parameters, set video format and size
# picam2.configure(picam2.create_video_configuration(main={"format": 'XRGB8888', "size": (640, 480)}))
# picam2.start() # Start the camera
camera = cv2.VideoCapture(-1) #Create camera example
#Set resolution
camera.set(cv2.CAP_PROP_FRAME_WIDTH, 640)
camera.set(cv2.CAP_PROP_FRAME_HEIGHT, 480)
display_handle=display(None, display_id=True) # Create a display handle for updating the displayed content
i = 0
while True:
# frame = picam2.capture_array()
# frame = cv2.flip(frame, 1) # flip the image
_, frame = camera.read() #Capturing a frame from the camera
# Take a photo every few seconds
if time.time() - last_picture_time >= time_intervel:
num_count = num_count + 1 # Update the photo counter
photo_filename = f'{photo_path}photo_{num_count}.jpg' # Define the file name for the photo
cv2.imwrite(photo_filename, frame) # Save the photo to the specified path
last_picture_time = time.time() # Update the time of the last photo taken
print(f'{num_count} photos saved. new photo: {photo_filename}') # Print information about the saved photo
_, frame = cv2.imencode('.jpeg', frame) # Encode the frame as JPEG format
display_handle.update(Image(data=frame.tobytes())) # Update the displayed image
if stopButton.value==True: # Check if the stop button is pressed
#picam2.close() # If yes, close the camera
cv2.release() # If yes, close the camera
display_handle.update(None)
# Display the stop button and start the video stream display thread
# ================
display(stopButton)
thread = threading.Thread(target=view, args=(stopButton,))
thread.start()
This tutorial utilizes OpenCV to detect changes in the scene. You can set a threshold for how much change is detected, and adjusting this threshold allows you to modify the sensitivity of the motion detection.
This chapter requires an understanding of the preceding chapters.
Since the product automatically runs the main program at startup, which occupies the camera resource, this tutorial cannot be used in such situations. You need to terminate the main program or disable its automatic startup before restarting the robot.
It's worth noting that because the robot's main program uses multi-threading and is configured to run automatically at startup through crontab, the usual method sudo killall python typically doesn't work. Therefore, we'll introduce the method of disabling the automatic startup of the main program here.
If you have already disabled the automatic startup of the robot's main demo, you do not need to proceed with the section on Terminate the Main Demo.
The following code block can be run directly:
If you are using a USB camera, you need to uncomment the line frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB).
You need to adjust some parameters to increase the threshold (sensitivity) of OpenCV for detecting changes in the scene. The lower the threshold value, the more sensitive OpenCV is to changes in the scene.
When you run the code block, you can see the real-time feed from the camera. You can wave your hand in front of the camera, and the program will automatically outline the areas of change with green boxes.
import cv2
from picamera2 import Picamera2
import numpy as np
from IPython.display import display, Image
import ipywidgets as widgets
import threading
import imutils # Library for simplifying image processing tasks
threshold = 2000 # Set the threshold for motion detection
# Create a "Stop" button to control the process
# ===================================================
stopButton = widgets.ToggleButton(
value=False,
description='Stop',
disabled=False,
button_style='danger', # 'success', 'info', 'warning', 'danger' or ''
tooltip='Description',
icon='square' # Button icon (FontAwesome name without the `fa-` prefix)
)
# Display function definition, used to capture and process video frames, while performing motion detection
# ===================================================
def view(button):
# If you are using a CSI camera you need to comment out the picam2 code and the camera code
# Since the latest versions of OpenCV no longer support CSI cameras (4.9.0.80), you need to use picamera2 to get the camera footage
# picam2 = Picamera2() # Create a Picamera2 instance
# picam2.configure(picam2.create_video_configuration(main={"format": 'XRGB8888', "size": (640, 480)})) # Configure camera parameters
# picam2.start() # Start the camera
camera = cv2.VideoCapture(-1) #Create camera example
#Set resolution
camera.set(cv2.CAP_PROP_FRAME_WIDTH, 640)
camera.set(cv2.CAP_PROP_FRAME_HEIGHT, 480)
display_handle = display(None, display_id=True)
i = 0
avg = None # Used to store the average frame
while True:
# frame = picam2.capture_array() # Capture a frame from the camera
# frame = cv2.flip(frame, 1) # if your camera reverses your image
_, frame = camera.read() # Capture a frame image from camera
img = cv2.cvtColor(frame, cv2.COLOR_RGB2BGR) # Convert frame color from RGB to BGR
gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY) # Convert the frame to grayscale
gray = cv2.GaussianBlur(gray, (21, 21), 0) # Apply Gaussian blur to the grayscale image
if avg is None: # If the average frame does not exist, create it
avg = gray.copy().astype("float")
continue
try:
cv2.accumulateWeighted(gray, avg, 0.5) # Update the average frame
except:
continue
frameDelta = cv2.absdiff(gray, cv2.convertScaleAbs(avg)) # Calculate the difference between the current frame and the average frame
# Apply a threshold to find contours in the difference image
thresh = cv2.threshold(frameDelta, 5, 255, cv2.THRESH_BINARY)[1]
thresh = cv2.dilate(thresh, None, iterations=2)
cnts = cv2.findContours(thresh.copy(), cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
cnts = imutils.grab_contours(cnts)
# Iterate through contours
for c in cnts:
# Ignore contours that are too small
if cv2.contourArea(c) < threshold:
continue
# Calculate the bounding box of the contour and draw a rectangle around it
(mov_x, mov_y, mov_w, mov_h) = cv2.boundingRect(c)
cv2.rectangle(frame, (mov_x, mov_y), (mov_x + mov_w, mov_y + mov_h), (128, 255, 0), 1) # Draw a rectangle around the moving area
_, frame = cv2.imencode('.jpeg', frame) # Encode the processed frame in JPEG format
display_handle.update(Image(data=frame.tobytes())) # Update the displayed image
if stopButton.value == True: # Check if the "Stop" button is pressed
#picam2.close() # If yes, close the camera
cv2.release() # if yes, close the camera
display_handle.update(None) # Clear the displayed image
# Display the stop button and start the video stream display thread
# ===================================================
display(stopButton)
thread = threading.Thread(target=view, args=(stopButton,))
thread.start()
This tutorial teaches how to control the camera for taking photos and recording videos by adding buttons to the page. Similar to previous tutorials, images are by default saved in the /ugv_jetson/templates/pictures/ folder, and videos are saved in the /ugv_jetson/templates/videos folder.
Since the product automatically runs the main program at startup, which occupies the camera resource, this tutorial cannot be used in such situations. You need to terminate the main program or disable its automatic startup before restarting the robot.
It's worth noting that because the robot's main demo uses multi-threading and is configured to run automatically at startup through crontab, the usual method sudo killall python typically doesn't work. Therefore, we'll introduce the method of disabling the automatic startup of the main program here.
If you have already disabled the automatic startup of the robot's main demo, you do not need to proceed with the section on Terminate the Main Demo.
The following code block can be run directly:
If you are using a USB camera, you need to uncomment the line frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB).
When the code block is executed, you can take a photo by clicking on PHOTO.
import cv2 # Import the OpenCV library for image processing
from picamera2 import Picamera2 # Import the library to access the Raspberry Pi Camera
import numpy as np # Import the library for mathematical calculations
from IPython.display import display, Image # Import to display images in Jupyter Notebook
import ipywidgets as widgets # Import to create interactive interface widgets like buttons
import threading # Import to create new threads for asynchronous task execution
import os, time # Import for file and directory operations and time-related functions
time_interval = 3 # Set the time interval for taking photos (seconds)
photo_path = '/home/ws/ugv_jetson/static/' # Set the directory path to store photos and videos
# Create a "Stop" button for users to stop video capture and photo taking
# ================
stopButton = widgets.ToggleButton(
value=False,
description='Stop',
disabled=False,
button_style='danger', # Set button style: 'success', 'info', 'warning', 'danger' or ''
tooltip='Description',
icon='square' # Set button icon (FontAwesome names without the `fa-` prefix)
)
# Create a "Photo" button for users to instantly take a photo by clicking it
# ================
photoButton = widgets.ToggleButton(
value=False,
description='Photo',
disabled=False,
button_style='danger', # 'success', 'info', 'warning', 'danger' or ''
tooltip='Description',
icon='square' # (FontAwesome names without the `fa-` prefix)
)
# Set the time interval for continuous shooting (seconds)
time_interval = 3
photo_num_count = 0 # Initialize the photo counter
capture_lock = threading.Lock()
last_photo_time = time.time() # Record the last time a photo was taken
def photo_button_clicked(change, frame):
global photo_num_count
if change['new']: # When the "Photo" button is clicked
photo_num_count = 1 # Increment the photo counter
photo_filename = f'{photo_path}photo_{photo_num_count}.jpg' # Set the path and filename for saving the photo
cv2.imwrite(photo_filename, frame) # Save the photo
print(f'{photo_num_count} photos saved. New photo: {photo_filename}') # Print photo save information
photoButton.value = False # Reset the status of the "Photo" button
# Define a display function to capture and display video frames and respond to photo capture requests
# ================
def view(stop_button, photo_button):
# If you are using a CSI camera you need to comment out the picam2 code and the camera code
# Since the latest versions of OpenCV no longer support CSI cameras (4.9.0.80), you need to use picamera2 to get the camera footage
# picam2 = Picamera2() # Create an instance of Picamera2
# picam2.configure(picam2.create_video_configuration(main={"format": 'XRGB8888', "size": (640, 480)})) # Configure camera parameters
# picam2.start()
camera = cv2.VideoCapture(-1) #Create camera example
#Set resolution
camera.set(cv2.CAP_PROP_FRAME_WIDTH, 640)
camera.set(cv2.CAP_PROP_FRAME_HEIGHT, 480)
display_handle = display(None, display_id=True) # Create a display handle to update the displayed image
i = 0
while True:
#frame = picam2.capture_array() # Capture a frame from the camera
_, frame = camera.read()
photoButton.observe(lambda change: photo_button_clicked(change, frame), names='value') # Listen for clicks on the "Photo" button
_, frame = cv2.imencode('.jpeg', frame) # Encode the frame as JPEG format
display_handle.update(Image(data=frame.tobytes()))
if stopButton.value==True:
#picam2.close() # If yes, close the camera
cv2.release() # if yes, close the camera
display_handle.update(None)
# Display the "Stop" and "Photo" buttons and start a new thread to execute the display function
# ================
display(stopButton)
display(photoButton)
thread = threading.Thread(target=view, args=(stopButton, photoButton,))
thread.start()
Here's something to note: due to stability issues with JupyterLab components used in this example, pressing the "Photo" button may result in saving multiple photos. You can navigate to /ugv_jetson/templates/pictures/ in the left sidebar of JupyterLab to view the captured photos.
This chapter introduces how to use OpenCV to compare feature databases and achieve face recognition. Although this method is not as efficient as MediaPipe's solution, it allows for the detection of other objects by replacing the feature database file.
Since the product automatically runs the main program at startup, which occupies the camera resource, this tutorial cannot be used in such situations. You need to terminate the main program or disable its automatic startup before restarting the robot.
It's worth noting that because the robot's main program uses multi-threading and is configured to run automatically at startup through crontab, the usual method sudo killall python typically doesn't work. Therefore, we'll introduce the method of disabling the automatic startup of the main program here.
If you have already disabled the automatic startup of the robot's main demo, you do not need to proceed with the section on Terminate the Main Demo.
The following code block can be run directly:
The face feature database file is located in the same path as this .ipynb file. You can change the faceCascade variable to modify what needs to be detected. You'll need to replace the current haarcascade_frontalface_default.xml file with other feature files.
When the code block runs successfully, you can position the robot's camera on a face, and the area containing the face will be automatically highlighted on the screen.
import cv2 # Import the OpenCV library for image processing
from picamera2 import Picamera2 # Library for accessing the Raspberry Pi Camera
import numpy as np # Library for mathematical calculations
from IPython.display import display, Image # Library for displaying images in Jupyter Notebook
import ipywidgets as widgets # Library for creating interactive widgets like buttons
import threading # Library for creating new threads to execute tasks asynchronously
# Load the Haar cascade classifier for face detection
faceCascade = cv2.CascadeClassifier('haarcascade_frontalface_default.xml')
# Create a "Stop" button for users to stop the video stream by clicking on it
# ================
stopButton = widgets.ToggleButton(
value=False,
description='Stop',
disabled=False,
button_style='danger', # 'success', 'info', 'warning', 'danger' or ''
tooltip='Description',
icon='square' # (FontAwesome names without the `fa-` prefix)
)
# Define a display function to process video frames and perform face detection
# ================
def view(button):
# If you are using a CSI camera you need to comment out the picam2 code and the camera code.
# Since the latest versions of OpenCV no longer support CSI cameras (4.9.0.80), you need to use picamera2 to get the camera footage
# picam2 = Picamera2() # Create an instance of Picamera2
# picam2.configure(picam2.create_video_configuration(main={"format": 'XRGB8888', "size": (640, 480)})) # Configure camera parameters
# picam2.start() # Start the camera
camera = cv2.VideoCapture(-1) #Create camera example
#Set resolution
camera.set(cv2.CAP_PROP_FRAME_WIDTH, 640)
camera.set(cv2.CAP_PROP_FRAME_HEIGHT, 480)
display_handle=display(None, display_id=True) # Create a display handle to update the displayed image
i = 0
avg = None
while True:
# frame = picam2.capture_array()
_, frame = camera.read()
# frame = cv2.flip(frame, 1) # if your camera reverses your image
img = cv2.cvtColor(frame, cv2.COLOR_RGB2BGR) # Convert the image from RGB to BGR because OpenCV defaults to BGR
gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY) # Convert the image to grayscale because face detection is typically performed on grayscale images
# Perform face detection using the cascade classifier
faces = faceCascade.detectMultiScale(
gray,
scaleFactor=1.2,
minNeighbors=5,
minSize=(20, 20)
)
if len(faces):
for (x,y,w,h) in faces: # Loop through all detected faces
cv2.rectangle(frame,(x,y),(x+w,y+h),(64,128,255),1) # Draw a rectangle around the detected face
_, frame = cv2.imencode('.jpeg', frame) # Encode the frame as JPEG format
display_handle.update(Image(data=frame.tobytes()))
if stopButton.value==True:
#picam2.close() # If yes, close the camera
cv2.release() # if yes, close the camera
display_handle.update(None)
# Display the "Stop" button and start a thread to execute the display function
# ================
display(stopButton)
thread = threading.Thread(target=view, args=(stopButton,))
thread.start()
This section introduces how to implement common object recognition using DNN (Deep Neural Network) + OpenCV.
Since the product automatically runs the main program at startup, which occupies the camera resource, this tutorial cannot be used in such situations. You need to terminate the main program or disable its automatic startup before restarting the robot.
It's worth noting that because the robot's main program uses multi-threading and is configured to run automatically at startup through crontab, the usual method sudo killall python typically doesn't work. Therefore, we'll introduce the method of disabling the automatic startup of the main program here.
If you have already disabled the automatic startup of the robot's main demo, you do not need to proceed with the section on Terminate the Main Demo.
The following code block can be run directly:
The deploy.prototxt file and mobilenet_iter_73000.caffemodel file are in the same path as this .ipynb file.
When the code blocks run successfully, you can point the camera at common objects such as: "background", "aeroplane", "bicycle", "bird", "boat", "bottle", "bus", "car", "cat", "chair", "cow", "diningtable", "dog", "horse", "motorbike", "person", "pottedplant", "sheep", "sofa", "train", "tvmonitor".
The program will annotate the objects it recognizes in the image and label them with their names.
import cv2 # Import the OpenCV library for image processing
from picamera2 import Picamera2 # Library for accessing Raspberry Pi Camera
import numpy as np # Library for mathematical calculations
from IPython.display import display, Image # Library for displaying images in Jupyter Notebook
import ipywidgets as widgets #Library for creating interactive widgets like buttons
import threading # Library for creating new threads for asynchronous task execution
# Pre-defined class names based on the Caffe model
class_names = ["background", "aeroplane", "bicycle", "bird", "boat",
"bottle", "bus", "car", "cat", "chair", "cow", "diningtable",
"dog", "horse", "motorbike", "person", "pottedplant", "sheep",
"sofa", "train", "tvmonitor"]
# Load Caffe model
net = cv2.dnn.readNetFromCaffe('deploy.prototxt', 'mobilenet_iter_73000.caffemodel')
# Create a "Stop" button for users to stop the video stream by clicking it
# ================
stopButton = widgets.ToggleButton(
value=False,
description='Stop',
disabled=False,
button_style='danger', # 'success', 'info', 'warning', 'danger' or ''
tooltip='Description',
icon='square' # (FontAwesome names without the `fa-` prefix)
)
# Define the display function to process video frames and perform object detection
# ================
def view(button):
# If you are using a CSI camera you need to comment out the picam2 code and the camera code.
# Since the latest versions of OpenCV no longer support CSI cameras (4.9.0.80), you need to use picamera2 to get the camera footage
# picam2 = Picamera2() # Create Picamera2 example
# picam2.configure(picam2.create_video_configuration(main={"format": 'XRGB8888', "size": (640, 480)})) # Configure camera parameters
# picam2.start() # Start camera
camera = cv2.VideoCapture(-1) # Create camera example
#Set resolution
camera.set(cv2.CAP_PROP_FRAME_WIDTH, 640)
camera.set(cv2.CAP_PROP_FRAME_HEIGHT, 480)
display_handle=display(None, display_id=True) # Create a display handle for updating displayed images
i = 0
avg = None
while True:
# frame = picam2.capture_array() # Capture a frame from the camera
_, frame = camera.read() # Capture a frame from the camerav
# frame = cv2.flip(frame, 1) # if your camera reverses your image
img = cv2.cvtColor(frame, cv2.COLOR_RGB2BGR) # Convert the image from RGB to BGR because OpenCV uses BGR by default
(h, w) = img.shape[:2] # Get the height and width of the image
# Generate the input blob for the network
blob = cv2.dnn.blobFromImage(cv2.resize(img, (300, 300)), 0.007843, (300, 300), 127.5)
net.setInput(blob) # Set the blob as the input to the network
detections = net.forward() # Perform forward pass to get the detection results
# Loop over the detected objects
for i in range(0, detections.shape[2]):
confidence = detections[0, 0, i, 2] # Get the confidence of the detected object
if confidence > 0.2: # If the confidence is above the threshold, process the detected object
idx = int(detections[0, 0, i, 1]) # Get the class index
box = detections[0, 0, i, 3:7] * np.array([w, h, w, h]) # Get the bounding box of the object
(startX, startY, endX, endY) = box.astype("int") # Convert the bounding box to integers
# Annotate the object and confidence on the image
label = "{}: {:.2f}%".format(class_names[idx], confidence * 100)
cv2.rectangle(frame, (startX, startY), (endX, endY), (0, 255, 0), 2)
y = startY - 15 if startY - 15 > 15 else startY + 15
cv2.putText(frame, label, (startX, y), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 255, 0), 2)
_, frame = cv2.imencode('.jpeg', frame) # Encode the frame as JPEG format
display_handle.update(Image(data=frame.tobytes())) # Update the displayed image
if stopButton.value==True: # Check if the "Stop" button is pressed
# picam2.close() # If yes, close the camera
cv2.release() # If yes, close the camera
display_handle.update(None) # Clear the displayed content
#Display the "Stop" button and start the display function's thread
# ================
display(stopButton)
thread = threading.Thread(target=view, args=(stopButton,))
thread.start()
In this tutorial, we'll integrate some functions to modify frame images within OpenCV, such as blurring, color space conversion, erosion, and dilation.
Since the product automatically runs the main program at startup, which occupies the camera resource, this tutorial cannot be used in such situations. You need to terminate the main program or disable its automatic startup before restarting the robot.
It's worth noting that because the robot's main program uses multi-threading and is configured to run automatically at startup through crontab, the usual method sudo killall python typically doesn't work. Therefore, we'll introduce the method of disabling the automatic startup of the main program here.
If you have already disabled the automatic startup of the robot's main demo, you do not need to proceed with the section on Terminate the Main Demo.
The following code block can be run directly:
By default, we detect blue balls in the example. Ensure that there are no blue objects in the background to avoid interfering with the color recognition function. You can also modify the detection color (in the HSV color space) through secondary development.
import cv2
import imutils
from picamera2 import Picamera2 # Library for accessing Raspberry Pi Camera
import numpy as np # Library for mathematical calculations
from IPython.display import display, Image # Library for displaying images in Jupyter Notebook
import ipywidgets as widgets # Library for creating interactive widgets such as buttons
import threading # Library for creating new threads to execute tasks asynchronously
# Create a "Stop" button that users can click to stop the video stream
# ================================================================
stopButton = widgets.ToggleButton(
value=False,
description='Stop',
disabled=False,
button_style='danger', # 'success', 'info', 'warning', 'danger' or ''
tooltip='Description',
icon='square' # (FontAwesome names without the `fa-` prefix)
)
# Define the display function to process video frames and recognize objects of specific colors
def view(button):
# If you are using a CSI camera you need to comment out the picam2 code and the camera code.
# Since the latest versions of OpenCV no longer support CSI cameras (4.9.0.80), you need to use picamera2 to get the camera footage
# picam2 = Picamera2() # Create an instance of Picamera2
# Configure camera parameters, set the format and size of the video
# picam2.configure(picam2.create_video_configuration(main={"format": 'XRGB8888', "size": (640, 480)}))
# picam2.start() # Start the camera
camera = cv2.VideoCapture(-1) #Create camera example
#Set resolution
camera.set(cv2.CAP_PROP_FRAME_WIDTH, 640)
camera.set(cv2.CAP_PROP_FRAME_HEIGHT, 480)
display_handle=display(None, display_id=True) # Create a display handle to update displayed images
i = 0
# Define the color range to be detected
color_upper = np.array([120, 255, 220])
color_lower = np.array([90, 120, 90])
min_radius = 12 # Define the minimum radius for detecting objects
while True:
# img = picam2.capture_array() # Capture a frame from the camera
_, img = camera.read()
# frame = cv2.flip(frame, 1) # if your camera reverses your image
blurred = cv2.GaussianBlur(img, (11, 11), 0) # Apply Gaussian blur to the image to remove noise
hsv = cv2.cvtColor(blurred, cv2.COLOR_BGR2HSV) # Convert the image from BGR to HSV color space
mask = cv2.inRange(hsv, color_lower, color_upper) # Create a mask to retain only objects within a specific color range
mask = cv2.erode(mask, None, iterations=5) # Apply erosion to the mask to remove small white spots
mask = cv2.dilate(mask, None, iterations=5) # Apply dilation to the mask to highlight the object regions
# Find contours in the mask
cnts = cv2.findContours(mask.copy(), cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
cnts = imutils.grab_contours(cnts) # Extract contours
center = None # Initialize the center of the object
if len(cnts) > 0:
# Find the largest contour in the mask, then use
# it to compute the minimum enclosing circle and
# centroid
c = max(cnts, key=cv2.contourArea) # Find the largest contour
((x, y), radius) = cv2.minEnclosingCircle(c) # Compute the minimum enclosing circle of the contour
M = cv2.moments(c) # Compute the moments of the contour
center = (int(M["m10"] / M["m00"]), int(M["m01"] / M["m00"])) # Compute the center of the contour based on moments
if radius > min_radius: # If the radius of the minimum enclosing circle is greater than the predefined minimum radius, draw circles and center points
cv2.circle(img, (int(x), int(y)), int(radius), (128, 255, 255), 1) # Draw the minimum enclosing circle
_, frame = cv2.imencode('.jpeg', img) # Encode the frame to JPEG format
display_handle.update(Image(data=frame.tobytes())) # Update the displayed image
if stopButton.value==True: # Check if the "Stop" button has been pressed
#picam2.close() # If yes, close the camera
cv2.release() # if yes, close the camera
display_handle.update(None) # Clear the displayed content
# Display the "Stop" button and start a thread to execute the display function
# ================================================================
display(stopButton)
thread = threading.Thread(target=view, args=(stopButton,))
thread.start()
This section introduces how to implement gesture recognition using MediaPipe + OpenCV.
MediaPipe is an open-source framework developed by Google for building machine learning-based multimedia processing applications. It provides a set of tools and libraries for processing video, audio, and image data, and applies machine learning models to achieve various functionalities such as pose estimation, gesture recognition, and face detection. MediaPipe is designed to offer efficient, flexible, and easy-to-use solutions, enabling developers to quickly build a variety of multimedia processing applications.
Since the product automatically runs the main program at startup, which occupies the camera resource, this tutorial cannot be used in such situations. You need to terminate the main program or disable its automatic startup before restarting the robot.
It's worth noting that because the robot's main program uses multi-threading and is configured to run automatically at startup through crontab, the usual method sudo killall python typically doesn't work. Therefore, we'll introduce the method of disabling the automatic startup of the main program here.
If you have already disabled the automatic startup of the robot's main demo, you do not need to proceed with the section on Terminate the Main Demo.
The following code block can be run directly:
In this tutorial, the camera pan-tilt will rotate to make sure your hand or other fragile objects are away from the camera pan-tilt's rotation radius.
We detect blue balls by default in the demo to ensure that there are no blue objects in the background of the picture that affect the color recognition function, and you can also change the detection color (HSV color space) through secondary development.
import matplotlib.pyplot as plt
import cv2
from picamera2 import Picamera2
import numpy as np
from IPython.display import display, Image
import ipywidgets as widgets
import threading
# Stop button
# ================
stopButton = widgets.ToggleButton(
value=False,
description='Stop',
disabled=False,
button_style='danger', # 'success', 'info', 'warning', 'danger' or ''
tooltip='Description',
icon='square' # (FontAwesome names without the `fa-` prefix)
)
def gimbal_track(fx, fy, gx, gy, iterate):
global gimbal_x, gimbal_y
distance = math.sqrt((fx - gx) ** 2 + (gy - fy) ** 2)
gimbal_x += (gx - fx) * iterate
gimbal_y += (fy - gy) * iterate
if gimbal_x > 180:
gimbal_x = 180
elif gimbal_x < -180:
gimbal_x = -180
if gimbal_y > 90:
gimbal_y = 90
elif gimbal_y < -30:
gimbal_y = -30
gimbal_spd = int(distance * track_spd_rate)
gimbal_acc = int(distance * track_acc_rate)
if gimbal_acc < 1:
gimbal_acc = 1
if gimbal_spd < 1:
gimbal_spd = 1
base.base_json_ctrl({"T":self.CMD_GIMBAL,"X":gimbal_x,"Y":gimbal_y,"SPD":gimbal_spd,"ACC":gimbal_acc})
return distance
# Display function
# ================
def view(button):
picam2 = Picamera2()
picam2.configure(picam2.create_video_configuration(main={"format": 'XRGB8888', "size": (640, 480)}))
picam2.start()
display_handle=display(None, display_id=True)
color_upper = np.array([120, 255, 220])
color_lower = np.array([ 90, 120, 90])
min_radius = 12
track_color_iterate = 0.023
while True:
frame = picam2.capture_array()
# frame = cv2.flip(frame, 1) # if your camera reverses your image
# uncomment this line if you are using USB camera
# frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
img = cv2.cvtColor(frame, cv2.COLOR_RGB2BGR)
blurred = cv2.GaussianBlur(img, (11, 11), 0)
hsv = cv2.cvtColor(blurred, cv2.COLOR_BGR2HSV)
mask = cv2.inRange(hsv, color_lower, color_upper)
mask = cv2.erode(mask, None, iterations=5)
mask = cv2.dilate(mask, None, iterations=5)
cnts = cv2.findContours(mask.copy(), cv2.RETR_EXTERNAL,
cv2.CHAIN_APPROX_SIMPLE)
cnts = imutils.grab_contours(cnts)
center = None
height, width = img.shape[:2]
center_x, center_y = width // 2, height // 2
if len(cnts) > 0:
# find the largest contour in the mask, then use
# it to compute the minimum enclosing circle and
# centroid
c = max(cnts, key=cv2.contourArea)
((x, y), radius) = cv2.minEnclosingCircle(c)
M = cv2.moments(c)
center = (int(M["m10"] / M["m00"]), int(M["m01"] / M["m00"]))
# only proceed if the radius meets a minimum size
if radius > min_radius:
distance = gimbal_track(center_x, center_y, center[0], center[1], track_color_iterate) #
cv2.circle(overlay_buffer, (int(x), int(y)), int(radius), (128, 255, 255), 1)
_, frame = cv2.imencode('.jpeg', frame)
display_handle.update(Image(data=frame.tobytes()))
if stopButton.value==True:
picam2.close()
display_handle.update(None)
# Run
# ================
display(stopButton)
thread = threading.Thread(target=view, args=(stopButton,))
thread.start()
In this tutorial, we'll use the basic functionalities of OpenCV to detect yellow lines (default color) in the image and control the direction of the chassis based on the position of these lines. Please note that in this example, the chassis won't move. Instead, we'll only showcase the algorithms using OpenCV on the image. For safety reasons, we won't integrate motion control in this tutorial, as it's heavily influenced by external factors. Users should fully understand the code's functionality before adding corresponding motion control features.
If you want to control the robot's movement through this example, please refer to the "Python Chassis Motion Control" section to add the relevant motion control functions (our open-source example is located in robot_ctrl.py).
Since the product automatically runs the main program at startup, which occupies the camera resource, this tutorial cannot be used in such situations. You need to terminate the main program or disable its automatic startup before restarting the robot.
It's worth noting that because the robot's main program uses multi-threading and is configured to run automatically at startup through crontab, the usual method sudo killall python typically doesn't work. Therefore, we'll introduce the method of disabling the automatic startup of the main program here.
If you have already disabled the automatic startup of the robot's main demo, you do not need to proceed with the section on Terminate the Main Demo.
The following code block can be run directly:
If you use the USB camera you need to uncomment frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB).
After running the following code block, you can place a yellow tape in front of the camera and observe if there are contours of the yellow tape in the black screen. Try to detect the yellow tape using two target detection lines.
import cv2 # Import OpenCV library for image process
import imutils, math # Library to aid image processing and mathematical operations
from picamera2 import Picamera2 # Library to access Raspberry Pi Camera
import numpy as np
from IPython.display import display, Image # Display images on Jupyter Notebook
import ipywidgets as widgets # Widgets for creating interactive interfaces, such as buttons
import threading # Used to create new threads for asynchronous execution of tasks
# Stop button
# ================
stopButton = widgets.ToggleButton(
value=False,
description='Stop',
disabled=False,
button_style='danger', # 'success', 'info', 'warning', 'danger' or ''
tooltip='Description',
icon='square' # (FontAwesome names without the `fa-` prefix)
)
# findline autodrive
# Upper sampling line, 0.6 for position, higher values
sampling_line_1 = 0.6
# Lower sampling line, the value needs to be greater than sampling_line_1 and less than 1."
sampling_line_2 = 0.9
# Detect slope impact
slope_impact = 1.5
# The effect of line position detected by the lower detection line on turning
base_impact = 0.005
# The speed impact on turning
speed_impact = 0.5
# Line tracking speed
line_track_speed = 0.3
# Effect of slope on patrol speed
slope_on_speed = 0.1
# Color of target line, HSV color space
line_lower = np.array([25, 150, 70])
line_upper = np.array([42, 255, 255])
def view(button):
# If you are using a CSI camera you need to comment out the picam2 code and the camera code.
# Since the latest versions of OpenCV no longer support CSI cameras (4.9.0.80), you need to use picamera2 to get the camera footage
# picam2 = Picamera2() #Create Picamera2 example
# Set camera parameters and video format and size
# picam2.configure(picam2.create_video_configuration(main={"format": 'XRGB8888', "size": (640, 480)}))
# picam2.start() #Start camera
camera = cv2.VideoCapture(-1) #Create camera example
#Set resolution
camera.set(cv2.CAP_PROP_FRAME_WIDTH, 640)
camera.set(cv2.CAP_PROP_FRAME_HEIGHT, 480)
display_handle=display(None, display_id=True)
while True:
# img = picam2.capture_array()
_, img = camera.read() #Capture one frame from the camera
# frame = cv2.flip(frame, 1) # if your camera reverses your image
height, width = img.shape[:2]
center_x, center_y = width // 2, height // 2
# Image preprocessing includes color space conversion, Gaussian blur, and color range filtering, etc.
hsv = cv2.cvtColor(img, cv2.COLOR_BGR2HSV)
line_mask = cv2.inRange(hsv, line_lower, line_upper) # Filter out target lines based on color ranges
line_mask = cv2.erode(line_mask, None, iterations=1) # Eroding operation to remove noise
line_mask = cv2.dilate(line_mask, None, iterations=1) # Expansion operation enhances the target line
# Detect the target line based on the positions of the upper and lower sampling lines, and calculate steering and velocity control signals according to the detection results
sampling_h1 = int(height * sampling_line_1)
sampling_h2 = int(height * sampling_line_2)
get_sampling_1 = line_mask[sampling_h1]
get_sampling_2 = line_mask[sampling_h2]
# Calculate the width of the target line at the upper and lower sampling lines
sampling_width_1 = np.sum(get_sampling_1 == 255)
sampling_width_2 = np.sum(get_sampling_2 == 255)
if sampling_width_1:
sam_1 = True
else:
sam_1 = False
if sampling_width_2:
sam_2 = True
else:
sam_2 = False
# Get the edge index of the target line at the upper and lower sampling lines
line_index_1 = np.where(get_sampling_1 == 255)
line_index_2 = np.where(get_sampling_2 == 255)
# If the target line is detected at the upper sampling line, calculate the center position of the target line
if sam_1:
sampling_1_left = line_index_1[0][0] # Index of the leftmost index of the upper sampling line target line
sampling_1_right = line_index_1[0][sampling_width_1 - 1] # Index to the far right of the upper sampling line target line
sampling_1_center= int((sampling_1_left + sampling_1_right) / 2) # Index of the center of the upper sampling line target line
# If a target line is detected at the lower sampling line, calculate the target line center position
if sam_2:
sampling_2_left = line_index_2[0][0]
sampling_2_right = line_index_2[0][sampling_width_2 - 1]
sampling_2_center= int((sampling_2_left + sampling_2_right) / 2)
# Initialize steering and speed control signals
line_slope = 0
input_speed = 0
input_turning = 0
# If the target line is detected at both sampling lines, calculate the slope of the line, and then calculate velocity and steering control signals based on the slope and the position of the target line.
if sam_1 and sam_2:
line_slope = (sampling_1_center - sampling_2_center) / abs(sampling_h1 - sampling_h2) # Calculate the slope of the line
impact_by_slope = slope_on_speed * abs(line_slope) # Calculate the effect on velocity based on the slope
input_speed = line_track_speed - impact_by_slope # Calculated speed control signal
input_turning = -(line_slope * slope_impact + (sampling_2_center - center_x) * base_impact) #+ (speed_impact * input_speed) # Calculating steering control signals
elif not sam_1 and sam_2: # If the target line is detected only at the lower sampling line
input_speed = 0 # Set speed to 0
input_turning = (sampling_2_center - center_x) * base_impact # Calculating steering control signals
elif sam_1 and not sam_2: # If the target line is detected only at the upper sample line
input_speed = (line_track_speed / 3) # slow down
input_turning = 0 # No steering
else: # If neither sampling line detects the target line
input_speed = - (line_track_speed / 3) # backward
input_turning = 0 # No turning
# base.base_json_ctrl({"T":13,"X":input_speed,"Z":input_turning})
cv2.putText(line_mask, f'X: {input_speed:.2f}, Z: {input_turning:.2f}', (center_x+50, center_y+0), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (255, 255, 255), 1)
# Visualization operations include drawing lines at the positions of the sampling lines, marking the sampling results, and displaying steering and velocity control signals
cv2.line(line_mask, (0, sampling_h1), (img.shape[1], sampling_h1), (255, 0, 0), 2)
cv2.line(line_mask, (0, sampling_h2), (img.shape[1], sampling_h2), (255, 0, 0), 2)
if sam_1:
# Draw green marker lines at the ends of the target line at the upper sample line
cv2.line(line_mask, (sampling_1_left, sampling_h1+20), (sampling_1_left, sampling_h1-20), (0, 255, 0), 2)
cv2.line(line_mask, (sampling_1_right, sampling_h1+20), (sampling_1_right, sampling_h1-20), (0, 255, 0), 2)
if sam_2:
# Draw green marker lines at the ends of the target line at the lower sampling line
cv2.line(line_mask, (sampling_2_left, sampling_h2+20), (sampling_2_left, sampling_h2-20), (0, 255, 0), 2)
cv2.line(line_mask, (sampling_2_right, sampling_h2+20), (sampling_2_right, sampling_h2-20), (0, 255, 0), 2)
if sam_1 and sam_2:
# If the target line is detected at both the upper and lower sample lines, draw a red line from the center of the upper sample line to the center of the lower sample line.
cv2.line(line_mask, (sampling_1_center, sampling_h1), (sampling_2_center, sampling_h2), (255, 0, 0), 2)
_, frame = cv2.imencode('.jpeg', line_mask)
display_handle.update(Image(data=frame.tobytes()))
if stopButton.value==True:
#picam2.close() # If yes, close the camera
cv2.release() # if yes, close the camera
display_handle.update(None)
# Display the "Stop" button and start the thread that displays the function
# ================
display(stopButton)
thread = threading.Thread(target=view, args=(stopButton,))
thread.start()
This section introduces how to implement gesture recognition using MediaPipe + OpenCV.
MediaPipe is an open-source framework developed by Google for building machine learning-based multimedia processing applications. It provides a set of tools and libraries for processing video, audio, and image data, and applies machine learning models to achieve various functionalities such as pose estimation, gesture recognition, and face detection. MediaPipe is designed to offer efficient, flexible, and easy-to-use solutions, enabling developers to quickly build a variety of multimedia processing applications.
Since the product automatically runs the main program at startup, which occupies the camera resource, this tutorial cannot be used in such situations. You need to terminate the main program or disable its automatic startup before restarting the robot.
It's worth noting that because the robot's main program uses multi-threading and is configured to run automatically at startup through crontab, the usual method sudo killall python typically doesn't work. Therefore, we'll introduce the method of disabling the automatic startup of the main program here.
If you have already disabled the automatic startup of the robot's main demo, you do not need to proceed with the section on Terminate the Main Demo.
The following code block can be run directly:
If you use the USB camera you need to uncomment frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB).
When the code block is running normally, you can put your hand in front of the camera, the real-time video screen will be labeled with the joints of your hand, the labeled joints will change with the change of your hand, and the position of each joint will also be output, which is convenient for the secondary development of gesture control.
MediaPipe's gesture recognition process uses different names to correspond to different joints, and you can call the corresponding number to get the position information of the joint.
MediaPipe Hand
import cv2
import imutils, math
from picamera2 import Picamera2 # for accessing Raspberry Pi Camera library
from IPython.display import display, Image # for displaying images on Jupyter Notebook
import ipywidgets as widgets # for creating interactive widgets like button
import threading # Library for creating new threads for asynchronous task execution
import mediapipe as mp # Import the MediaPipe library for detecting hand gesture key points
# Create a "STOP" button that users can click to stop the video stream
# ================
stopButton = widgets.ToggleButton(
value=False,
description='Stop',
disabled=False,
button_style='danger', # 'success', 'info', 'warning', 'danger' or ''
tooltip='Description',
icon='square' # (FontAwesome names without the `fa-` prefix)
)
# Initialize MediaPipe drawing tool and hand keypoint detection model
mpDraw = mp.solutions.drawing_utils
mpHands = mp.solutions.hands
hands = mpHands.Hands(max_num_hands=1) # Initialize the detection model of hand keypoint, up to one hand
# Define display functions to process video frames and perform hand keypoint detection
def view(button):
# If you use the CSI camera, you need to comment out picam2 and camera
# Since the latest version OpenCV does not support CSI camera (4.9.0.80) anymore, you need to use picamera2 to get camera image
# picam2 = Picamera2() #Create Picamera2 example
# Configure camera parameters and set the format and size of video
# picam2.configure(picam2.create_video_configuration(main={"format": 'XRGB8888', "size": (640, 480)}))
# picam2.start() # Start camera
camera = cv2.VideoCapture(-1) # Create camera example
#Set resolution
camera.set(cv2.CAP_PROP_FRAME_WIDTH, 640)
camera.set(cv2.CAP_PROP_FRAME_HEIGHT, 480)
display_handle=display(None, display_id=True) # Creates a display handle for updating the displayed image
while True:
# frame = picam2.capture_array()
_, frame = camera.read() # Capture a frame from the camera
# frame = cv2.flip(frame, 1) # if your camera reverses your image
img = cv2.cvtColor(frame, cv2.COLOR_RGB2BGR)
results = hands.process(img)
# If the hand key is detected
if results.multi_hand_landmarks:
for handLms in results.multi_hand_landmarks: # Iterate over each hand detected
# Draw hand keypoints
for id, lm in enumerate(handLms.landmark):
h, w, c = img.shape
cx, cy = int(lm.x * w), int(lm.y * h) # Calculate the position of the keypoint in the image
cv2.circle(img, (cx, cy), 5, (255, 0, 0), -1) # Drawing dots at keypoint locations
frame = cv2.cvtColor(frame, cv2.COLOR_RGB2BGR)
mpDraw.draw_landmarks(frame, handLms, mpHands.HAND_CONNECTIONS) # Drawing hand skeleton connecting lines
frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
target_pos = handLms.landmark[mpHands.HandLandmark.INDEX_FINGER_TIP]
_, frame = cv2.imencode('.jpeg', frame)
display_handle.update(Image(data=frame.tobytes()))
if stopButton.value==True:
# picam2.close() # if yes, close camera
cv2.release() # if yes, close camera
display_handle.update(None)
# Display the "Stop" button and start the thread that displays the function
# ================
display(stopButton)
thread = threading.Thread(target=view, args=(stopButton,))
thread.start()
This section introduces how to implement face recognition using MediaPipe + OpenCV.
MediaPipe is an open-source framework developed by Google for building machine learning-based multimedia processing applications. It provides a set of tools and libraries for processing video, audio, and image data, and applies machine learning models to achieve various functionalities such as pose estimation, gesture recognition, and face detection. MediaPipe is designed to offer efficient, flexible, and easy-to-use solutions, enabling developers to quickly build a variety of multimedia processing applications.
Since the product automatically runs the main program at startup, which occupies the camera resource, this tutorial cannot be used in such situations. You need to terminate the main program or disable its automatic startup before restarting the robot.
It's worth noting that because the robot's main program uses multi-threading and is configured to run automatically at startup through crontab, the usual method sudo killall python typically doesn't work. Therefore, we'll introduce the method of disabling the automatic startup of the main program here.
If you have already disabled the automatic startup of the robot's main demo, you do not need to proceed with the section on Terminate the Main Demo.
The following code block can be run directly:
If you use the USB camera you need to uncomment frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB).
When the code block runs correctly, MediaPipe will automatically detect faces in the video feed, outlining the position of the face and marking the facial features.
import cv2 # Import the OpenCV library for image processing
import imutils, math # Libraries for auxiliary image processing and mathematical operations
from picamera2 import Picamera2 # Library for accessing the Raspberry Pi Camera
from IPython.display import display, Image # Library for displaying images in Jupyter Notebook
import ipywidgets as widgets # Library for creating interactive interface widgets such as buttons
import threading # Library for creating new threads for asynchronous task execution
import mediapipe as mp # Import the MediaPipe library for face detection
# Create a "STOP" button that users can click to stop the video stream
# ================
stopButton = widgets.ToggleButton(
value=False,
description='Stop',
disabled=False,
button_style='danger', # 'success', 'info', 'warning', 'danger' or ''
tooltip='Description',
icon='square' # (FontAwesome names without the `fa-` prefix)
)
# Initialize the face detection model of MediaPipe
mpDraw = mp.solutions.drawing_utils
# MediaPipe Hand GS
mp_face_detection = mp.solutions.face_detection
face_detection = mp_face_detection.FaceDetection(model_selection=0, min_detection_confidence=0.5)
#Define a display function to process video frames and perform face detection
def view(button):
picam2 = Picamera2()
picam2.configure(picam2.create_video_configuration(main={"format": 'XRGB8888', "size": (640, 480)}))
picam2.start()
display_handle=display(None, display_id=True)
while True:
frame = picam2.capture_array() # Capture one frame from the camera
# frame = cv2.flip(frame, 1) # if your camera reverses your image
# uncomment this line if you are using USB camera
# frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
img = cv2.cvtColor(frame, cv2.COLOR_RGB2BGR)
results = face_detection.process(img)
# If a face is detected
if results.detections:
for detection in results.detections: # Iterate through each detected face
frame = cv2.cvtColor(frame, cv2.COLOR_RGB2BGR)
mpDraw.draw_detection(frame, detection) # Use MediaPipe's drawing tools to draw face landmarks
frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
_, frame = cv2.imencode('.jpeg', frame) # Encode the processed frame to JPEG format
display_handle.update(Image(data=frame.tobytes())) # Update the displayed image
if stopButton.value==True: # Check if the "STOP" button is pressed
picam2.close() # If yes, close the camera
display_handle.update(None) # Clear the displayed content
# Display the "STOP" button and start a thread to run the display function
# ================
display(stopButton)
thread = threading.Thread(target=view, args=(stopButton,))
thread.start()
This section describes how to implement pose detection using MediaPipe + OpenCV.
MediaPipe is an open-source framework developed by Google for building machine learning-based multimedia processing applications. It provides a set of tools and libraries for processing video, audio, and image data, and applies machine learning models to achieve various functionalities such as pose estimation, gesture recognition, and face detection. MediaPipe is designed to offer efficient, flexible, and easy-to-use solutions, enabling developers to quickly build a variety of multimedia processing applications.
Since the product automatically runs the main program at startup, which occupies the camera resource, this tutorial cannot be used in such situations. You need to terminate the main program or disable its automatic startup before restarting the robot.
It's worth noting that because the robot's main program uses multi-threading and is configured to run automatically at startup through crontab, the usual method sudo killall python typically doesn't work. Therefore, we'll introduce the method of disabling the automatic startup of the main program here.
If you have already disabled the automatic startup of the robot's main demo, you do not need to proceed with the section on Terminate the Main Demo.
The following code block can be executed directly:
If you use the USB camera you need to uncomment frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB).
When the code block runs normally, MediaPipe will automatically mark the joints of the human body when there is a face in the frame.
import cv2 # Import the OpenCV library for image processing
import imutils, math # Auxiliary libraries for image processing and mathematical operations
from picamera2 import Picamera2 # Library to access the Raspberry Pi Camera
from IPython.display import display, Image # Library to display images in Jupyter Notebook
import ipywidgets as widgets # Library for creating interactive widgets, such as buttons
import threading # Library for creating new threads for asynchronous execution of tasks
import mediapipe as mp # Import the MediaPipe library for pose detection
# Create a "Stop" button that users can click to stop the video stream
# ================
stopButton = widgets.ToggleButton(
value=False,
description='Stop',
disabled=False,
button_style='danger', # 'success', 'info', 'warning', 'danger' or ''
tooltip='Description',
icon='square' # (FontAwesome names without the `fa-` prefix)
)
# Initialize MediaPipe's drawing tools and pose detection model
mpDraw = mp.solutions.drawing_utils
# MediaPipe Hand GS
mp_pose = mp.solutions.pose
pose = mp_pose.Pose(static_image_mode=False,
model_complexity=1,
smooth_landmarks=True,
min_detection_confidence=0.5,
min_tracking_confidence=0.5)
# Define the display function to process video frames and perform pose detection
def view(button):
picam2 = Picamera2() # Create an instance of Picamera2
picam2.configure(picam2.create_video_configuration(main={"format": 'XRGB8888', "size": (640, 480)})) # Configure camera parameters
picam2.start() # Start the camera
display_handle=display(None, display_id=True) # Create a display handle to update the displayed image
while True:
frame = picam2.capture_array()
# frame = cv2.flip(frame, 1) # if your camera reverses your image
# uncomment this line if you are using USB camera
# frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
img = cv2.cvtColor(frame, cv2.COLOR_RGB2BGR)
results = pose.process(img) # Use MediaPipe to process the image and get pose detection results
# If pose landmarks are detected
if results.pose_landmarks:
frame = cv2.cvtColor(frame, cv2.COLOR_RGB2BGR) # Convert the image from RGB to BGR for drawing
mpDraw.draw_landmarks(frame, results.pose_landmarks, mp_pose.POSE_CONNECTIONS) # Use MediaPipe's drawing tools to draw pose landmarks and connections
frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB) # Convert the image back from BGR to RGB for display
_, frame = cv2.imencode('.jpeg', frame) # Encode the processed frame into JPEG format
display_handle.update(Image(data=frame.tobytes())) # Update the displayed image
if stopButton.value==True: # Check if the "Stop" button is pressed
picam2.close() # If yes, close the camera
display_handle.update(None) # Clear the displayed content
# Display the "Stop" button and start a thread to run the display function
display(stopButton)
thread = threading.Thread(target=view, args=(stopButton,))
thread.start() # Start the thread
In the previous chapters, we introduced how to use Flask to achieve low-latency image transmission, which is used to transfer the camera feed to the web application interface. Here, we will discuss how to pass the information entered on the web application interface to the backend of the web application. This functionality is used to control the robot using the web application.
from flask import Flask, request # Import the Flask class and request object from the flask package
app = Flask(__name__) # Create a Flask app instance
@app.route('/', methods=['GET', 'POST']) # Define the root route, and allow GET and POST methods
def index():
if request.method == 'POST': # Check if the current request is POST
# Get form data
form_data = request.form
# Print form data
print(form_data)
# Return a simple response
return 'Received data: {}'.format(form_data)
else:
# If it's a GET request, return a simple form page
return '''
<form method="post">
<label for="input_data">Input data:</label><br>
<input type="text" id="input_data" name="input_data"><br>
<input type="submit" value="Submit">
</form>
'''
if __name__ == '__main__':
app.run(host='0.0.0.0')
You can select the code block above and press Ctrl + Enter to run it. If you encounter a port conflict, it means you've previously run this code block. You'll need to click Kernel in the JupyterLab menu bar and then select Shut Down All Kernels to release the resources, including network port resources, occupied by the previously run code block. After that, you can rerun the code block to run this Flask application.
Once you run the code block, you'll see messages like "Running on http://127.0.0.1:5000" and "Running on http://[IP]:5000". Usually, the [IP] here refers to the IP address assigned to your Raspberry Pi by your router. You can open a web browser on any device within the same local network and visit [IP]:5000. Note that the : symbol here must be the English colon, representing the port number 5000 of the IP address you're accessing.
Upon visiting this page, you'll see an input box and a Submit button. You can enter some content into the input box and then click the Submit button. After that, you can see the content you entered on the web page, and the backend will display the content it received below the code block in JupyterLab.
There is a file named setup.sh in the project folder, written in a shell, that helps to automatically configure the robotics product's uploader, including setting up the serial port, setting up the camera, creating the project's virtual environment, and installing dependency libraries. All these steps are already configured in the image on the TF card we shipped.
Installation script usage, the installation process needs to download and install a lot of dependent libraries from the network, for areas with special network environment we recommend you to use the method of downloading the image file directly from our official website to install the product.
setup.sh should run with root permission.
cd ugv_jetson/
sudo chmod +x setup.sh
sudo ./setup.sh
autorun.sh in the project folder is used to configure the product main program (app.py) and jupyterlab (start_jupyter.sh) to autorun on power-up (as user instead of root), as well as to generate the configuration file for jupyterlab.
autorun.sh should run with user permission.
sudo chmod +x autorun.sh ./autorun.sh
This project will be updated for a long time, when you need to update it, just navigate to the project folder and use the git pull command to update it, note that this operation will replace the changes you made before, after updating it, you need to change the parameters in config.yaml, set the type of the chassis and the type of the module, and you can also use the command line tool in the WEB control interface to configure the type of the product.
cd ugv_jetson/
git pull
After restarting the device, open the WEB control interface and enter the command used to set the product type, for example: s 20
The first digit 2 represents chassis type, the second digit 0 represents no module or pan-tilt module.
The first digit:
The second digit:
This configuration needs to be configured only once after an update or reinstallation and will be saved in config.yaml.
This project is open source under the GPL-3.0 agreement , you can realize your project according to our project, we will also continue to update more features.
Project address: https://github.com/waveshareteam/ugv_jetson
YAML (YAML Ain't Markup Language) is a human-readable data serialization format used to represent complex data structures. Its main purposes include configuration files, data exchange and storage, and passing data to programs.
YAML uses indentation and a well-structured format, making files easy to read and understand. It is less verbose compared to XML or JSON and closer to natural language.
YAML syntax is clear and concise, requiring no additional markup symbols (such as XML tags or JSON braces), making it easier to write and edit.
YAML supports nesting, lists, dictionaries, and other complex data structures, making it easy to represent various types of data.
YAML allows the use of tags and anchors to represent relationships between objects, enabling data reuse and referencing, thereby enhancing extensibility.
YAML is a universal data serialization format that is not tied to any specific programming language, making it easy to parse and generate using multiple programming languages.
In the config.yaml of this product, we configure some key parameters related to the robot:
These codes are related to the definitions of instructions in the lower computer program and are used for communication debugging between the upper and lower computers. Changing these involves changing the lower computer program.
These codes correspond to functions, and the frontend page also needs to load this .yaml file to obtain these configurations. This way, when the frontend web application communicates with the backend, different buttons correspond to different functions. Change only if necessary.
These codes correspond to types of feedback information. Some of this feedback is from the chassis to the host controller, and some is from the backend to the frontend. Using the same .yaml file for both the backend and frontend ensures uniformity of these codes and feedback information types. Change only if necessary.s and feedback information types. Change only if necessary.
In previous tutorials, we briefly introduced how to disable the automatic startup of the product's main program. The method used is to comment out the command to run the product's main program in the Crontab file. In this chapter, you will learn more about Crontab and why we use Crontab instead of Services for automatic startup.
Crontab is a tool in the Linux system used for scheduling periodic tasks. Through Crontab, users can set specific times, dates, or intervals to execute specific commands or scripts. Here are some important concepts and usages of Crontab:
The Crontab file is where scheduling information for periodic tasks is stored. Each user has their own Crontab file for storing their own scheduling information. Crontab files are usually stored in the /var/spool/cron directory, named after the user's username.
Each line in the Crontab file represents a scheduled task. Each line consists of five fields representing minutes, hours, date, month, and day of the week. You can use the # symbol to comment out a line to disable the corresponding task scheduling.
To edit the Crontab file, you can use the crontab -e command. This command opens a text editor, allowing the user to edit their Crontab file. After editing, save and exit the editor, and the Crontab file will be updated. Common options:
In contrast, using services to achieve automatic startup at boot time is typically done by executing a series of predefined services or scripts when the system starts up. These services can be found in the /etc/init.d/ directory and started, stopped, or restarted using system service management tools like systemctl.
Lower resource usage: Through our testing and comparison, the CPU resource usage of the same Python script using Crontab is 1/4 of that using services. For applications like complex robot main programs, using Crontab for automatic startup is a better choice. Services are more suitable for important services or applications that need to be executed at system startup. to be executed at system startup.
To facilitate the secondary development of the product, we have added a command-line input window in the WEB application. You can input commands in this window, and after clicking the SEND button, the command will be sent to the upper computer application. The upper computer application will execute corresponding functionalities or parameter adjustments based on the command you send.
We already have some ready-made commands that you can refer to in the following sections of the WEB Command Line Application to learn about those commands. In this section, we will introduce how to implement custom command-line functionality while explaining how this feature is implemented, making it easier for you to understand the subsequent sections.
The example demo for command-line functionality is written in the main program robot_ctrl.py, and they are handled by the cmd_process() function. Below is our default command-line instruction processing function. This function is incomplete because the content afterward deals with other functionalities, which are omitted here without affecting the understanding of the function itself.
Note: The code block below cannot be executed in JupyterLab and is only used for illustration purposes.
def cmd_process(self, args_str):
global show_recv_flag, show_info_flag, info_update_time, mission_flag
global track_color_iterate, track_faces_iterate, track_spd_rate, track_acc_rate
# Split the input parameter string into a list: args
args = args_str.split()
if args[0] == 'base':
self.info_update("CMD:" + args_str, (0,255,255), 0.36)
if args[1] == '-c' or args[1] == '--cmd':
base.base_json_ctrl(json.loads(args[2]))
elif args[1] == '-r' or args[1] == '--recv':
if args[2] == 'on':
show_recv_flag = True
elif args[2] == 'off':
show_recv_flag = False
elif args[0] == 'info':
info_update_time = time.time()
show_info_flag = True
elif args[0] == 'audio':
self.info_update("CMD:" + args_str, (0,255,255), 0.36)
if args[1] == '-s' or args[1] == '--say':
audio_ctrl.play_speech_thread(' '.join(args[2:]))
elif args[1] == '-v' or args[1] == '--volume':
audio_ctrl.set_audio_volume(args[2])
elif args[1] == '-p' or args[1] == '--play_file':
audio_ctrl.play_file(args[2])
Let's take audio -s hey hi hello as an example. This command is used for text-to-speech functionality, where audio represents an audio-related function, `-s` or `--say` indicates text-to-speech, and the following parameters are the content you want it to say. After sending this command, the robot will say "hey hi hello".
Firstly, when this function receives a command-line instruction since the command-line instruction is a string, we need to use args = args_str.split() to convert this string into a list. Then, we can check each value in the list to execute the corresponding functionality. If you need to extend other custom functionalities, you just need to add another elif args[0] == 'newCmd'.
To make the configuration of product parameters easier and allow users to add custom functionalities conveniently, we have designed command-line parameter functionality for the product. You can achieve corresponding functionalities by inputting commands into the command-line tool on the web page. This chapter will detail these functionalities.