Lesson01_ESP32-P4_AI_Voice_Chat: CrowPanel 5-inch AI Voice Chat Robot¶
1. Course Introduction¶
This lesson uses ESP-IDF 5.4.2 to drive the CrowPanel Advanced 5-inch ESP32-P4 HMI. The project uses the ESP32-P4 to handle 16-bit RGB LCD display, PDM microphone capture, I2S speaker output, button interaction, and MIPI CSI camera image processing, and establishes a Wi-Fi connection through the ESP32-C6 coprocessor, ultimately forming a complete pipeline of "voice input—network AI service—voice and interface output."
After the firmware is flashed and the device is reset, the system sequentially initializes the STC8 amplifier control, the 800 × 480 RGB display, the function buttons, camera power, and the camera data link; it then waits for the ESP32-C6 communication to be ready and connects to Wi-Fi. Users can switch the conversation state via the function buttons, speak through the microphone, and have replies played through the speaker; when visual functionality is needed, the camera still captures 1024 × 600 RGB565 images, which can be compressed into JPEG for upload and analysis.
2. Learning Objectives¶
- Be able to explain the division of responsibilities among the ESP32-P4, ESP32-C6, RGB LCD, MIPI CSI, PDM, and I2S in this project.
- Be able to identify the key configurations for the 800 × 480 RGB display, 16 kHz audio, buttons, STC8 backlight control, and camera.
- Be able to complete the checks for the ESP32-P4 target, 16 MB Flash, Hex PSRAM, and project partition configuration.
- Be able to explain the initialization sequence in the board constructor and the corresponding screen, audio, network, and camera behaviors.
- Be able to locate display, audio, camera, or network link faults based on build logs, serial logs, and device behavior.
3. Prerequisites¶
3.1 Hardware¶
- One CrowPanel Advanced 5-inch ESP32-P4 HMI AI development board.
- One USB data cable that supports data transfer; do not use a power-only cable.
- On-board or matching PDM microphone, speaker, and MIPI CSI camera.
- Stable USB power supply; power must be cut off before connecting externally or hot-swapping the camera flex cable.
- An available 2.4 GHz Wi-Fi network.
3.2 Software and Project¶
- A Windows PC, VS Code, and the Espressif IDF extension.
- ESP-IDF v5.4.2; the project README requires 5.4 or above, and this lesson uses the actually verified environment 5.4.2 as the baseline.
- Fully preserve the project's
main,managed_components,partitions,dependencies.lock,sdkconfig, andsdkconfig.defaults*. - Target chip:
esp32p4. - Flash: 16 MB; custom partition table:
partitions/v2/16m.csv. - PSRAM: Hex mode, 200 MHz; already configured in the project via
sdkconfig.
4. Software Operation Steps¶
Open the ESP-IDF extension panel in VS Code, click Open ESP-IDF Project, and select the 5inch_P4_HMI_AI folder. After the project opens, confirm that the Explorer root directory contains CMakeLists.txt, main, managed_components, partitions, and sdkconfig.
Do not open only the main subdirectory, otherwise ESP-IDF will be unable to load the full component dependencies and project configuration.
First, select the ESP-IDF v5.4.2 code execution environment in the bottom status bar of VS Code, set the flashing method to UART, and then select the serial port that actually corresponds to the development board. Next, click Set Espressif Device Target and select esp32p4. After configuration is complete, the status bar should simultaneously display ESP-IDF v5.4.2, UART, the corresponding COM port, and ESP32-P4.
The SDK configuration has already been set up for you, so you do not need to configure it again; you can use it directly.
However, during use, you need to change the network in this project to your own network environment.
Open main->boards->elecrow-p4-board->config.h
Just modify the Wi-Fi name and password inside to your own.
After the first build, or after switching the project configuration or modifying SDK parameters, click Full Clean in the status bar to clear the old build cache. Wait for the cleanup to finish before building, to avoid the old CMake cache, target chip, or Hosted parameters affecting this build result. 
Click Build to compile the project. Under normal circumstances, dependency resolution, CMake configuration, and Ninja compilation will complete in sequence, finally outputting Project build complete. If esp_hosted reports Unknown Slave Target, return to steps 4 and 5 to check the ESP32-C6 slave target and Hosted/SDIO configuration; if a message indicates that the ESP-IDF environment variable is missing, re-export the environment—do not directly modify the display, audio, or camera code.
Connect the development board using a USB cable that supports data transfer, click Select Port to Use to select the correct serial port, and then click Flash to flash the firmware. Do not unplug the USB cable or cut off power during flashing; after completion, the development board should reset automatically.
Click Monitor to open the serial monitor. After the device resets, you should sequentially see logs related to Initializing Elecrow P4 Board..., LCD, buttons, camera power, camera initialization, and network startup. Use Ctrl + ] to exit the monitor. Course screenshots and public logs must not contain Wi-Fi passwords, access tokens, or device unique identifiers.
After the project configuration, serial port, and code have all been confirmed correct, you can click the one-click operation button in the ESP-IDF status bar to sequentially complete building, flashing, and opening the serial monitor. When problems occur, you should still execute steps 7–10 separately, so as to determine whether the fault occurs during the cleanup, compilation, flashing, or running phase.
5. Hardware Operation Steps¶
Before connecting the USB power, confirm that the development board is powered off.
Check that the MIPI CSI camera flex cable is oriented correctly, that the gold fingers are fully inserted into the connector and the latch is firmly secured, and at the same time confirm that the LCD flex cable, microphone, speaker, and Wi-Fi antenna (if required by the current model) are securely connected. Hot-swapping the camera flex cable while powered on is prohibited.
Connect the ESP32-P4 development board to the PC using a USB cable that supports data transfer, and confirm that the power indicator lights up and the serial port can be recognized by VS Code.
This project requires a lot of hardware, so be sure to provide the ESP32-P4 development board with sufficient power.
After the firmware flashing completes and the device resets automatically, observe whether the display backlight lights up and whether the 800 × 480 interface is displayed completely. A normal screen should not show persistent black screen, color distortion, obvious misalignment, or partial missing content. If there is only backlight but no image, you should prioritize checking the RGB data lines, the PCLK/HSYNC/VSYNC/DE timing, and the RGB LCD initialized: 800x480 log; if the interface is normal but the backlight brightness is abnormal, then check the STC8 backlight power and PWM control.
After the device resets, wait about 2 seconds to allow the ESP32-P4 and the on-board ESP32-C6 to establish communication via SDIO. Check the serial monitor; the program should output the number of saved SSIDs and begin connecting to Wi-Fi; confirm that Hosted/SDIO initialization failures do not persistently appear in the log. The network connection time will be affected by the router and signal strength.
After the network connection succeeds, speak a clear, short sentence into the PDM microphone. The interface should display changes in the recording, processing, or reply status; after the AI service responds, the speaker should play the voice reply. During testing, keep an appropriate distance between the microphone and the speaker to avoid howling that affects voice recognition.
After connecting to Wi-Fi, it will connect to the xiaozhi platform.
For the connection steps, please refer to:
After completing the steps above, you will be able to have a conversation with xiaozhi.
6. Key Code Explanation¶
6.1 Board Function Parameters: Audio, Display, and Peripheral Pins¶
File: main/boards/elecrow-p4-board/config.h
#define AUDIO_INPUT_SAMPLE_RATE 16000
#define AUDIO_OUTPUT_SAMPLE_RATE 16000
#define AUDIO_I2S_GPIO_WS GPIO_NUM_21
#define AUDIO_I2S_GPIO_BCLK GPIO_NUM_22
#define AUDIO_I2S_GPIO_DOUT GPIO_NUM_23
#define AUDIO_PDM_MIC_CLK GPIO_NUM_24
#define AUDIO_PDM_MIC_DIN GPIO_NUM_25
#define DISPLAY_WIDTH 800
#define DISPLAY_HEIGHT 480
#define RGB_LCD_PIXEL_CLOCK_HZ (18 * 1000 * 1000)
#define RGB_PIN_NUM_PCLK GPIO_NUM_3
#define RGB_PIN_NUM_HSYNC GPIO_NUM_40
#define RGB_PIN_NUM_VSYNC GPIO_NUM_41
#define RGB_PIN_NUM_DE GPIO_NUM_2
These macros are read during the board object construction and when the audio service starts. Both input and output are 16 kHz, which keeps the voice link at a unified sample rate; the speaker uses the standard I2S WS, BCLK, and DOUT, while the microphone uses independent PDM clock and data input. When these two groups of pins are confused, the program may start normally but have no recording or no sound.
The 5-inch panel uses an 800 × 480, 16-bit RGB565 parallel display. The resolution, 18 MHz pixel clock, sync timing, 16 data lines, and LVGL display object must be consistent; errors in width/height or timing will cause cropping, jitter, color distortion, or refresh out-of-bounds. This version's backlight does not use the ESP32-P4's ordinary PWM GPIO; instead, the STC8 controls the backlight power and duty cycle. Therefore, when troubleshooting a black screen, you must separately confirm the RGB image link and the STC8 backlight link.
6.2 Board Initialization Sequence Determines Power-On Behavior¶
File: main/boards/elecrow-p4-board/elecrow_board.cc
InitializeAudioCtrl();
InitializeLCD();
InitializeButtons();
GetBacklight()->RestoreBrightness();
bsp_enable_camera_power();
InitializeCamera();
This code executes during the construction of ElecrowP4Board. It first establishes the STC8 amplifier control, then the RGB display link; after LCD initialization completes, the backlight is restored, so the normal behavior is that the display object is prepared first, and then the STC8 turns on the backlight and displays the interface. The camera must first be supplied with 3.3 V by LDO4 before accessing SCCB/CSI; if the order is swapped, sensor detection may fail.
If the screen does not light up, first check whether RGB LCD initialized: 800x480 appears on the serial port, then check the STC8 LCD backlight power and PWM. If the log stalls during camera initialization, check the flex cable, 3.3 V power supply, and SCCB pins, rather than suspecting the AI service directly.
6.3 16-bit RGB LCD Display Link¶
esp_lcd_rgb_panel_config_t panel_conf = {};
panel_conf.data_width = 16;
panel_conf.bits_per_pixel = 16;
panel_conf.de_gpio_num = RGB_PIN_NUM_DE;
panel_conf.pclk_gpio_num = RGB_PIN_NUM_PCLK;
panel_conf.vsync_gpio_num = RGB_PIN_NUM_VSYNC;
panel_conf.hsync_gpio_num = RGB_PIN_NUM_HSYNC;
panel_conf.timings.pclk_hz = RGB_LCD_PIXEL_CLOCK_HZ;
panel_conf.timings.h_res = DISPLAY_WIDTH;
panel_conf.timings.v_res = DISPLAY_HEIGHT;
panel_conf.flags.fb_in_psram = 1;
panel_conf.num_fbs = 2;
ESP_ERROR_CHECK(esp_lcd_new_rgb_panel(&panel_conf, &panel));
ESP_ERROR_CHECK(esp_lcd_panel_reset(panel));
ESP_ERROR_CHECK(esp_lcd_panel_init(panel));
The code first builds the 16-bit RGB panel configuration, handing the DE, PCLK, VSYNC, HSYNC, RGB565 data lines, and timing parameters to the ESP-IDF driver. Two frame buffers are placed in PSRAM to reduce internal SRAM pressure; after the panel is initialized, RgbLcdDisplay hands the LVGL refresh content to the RGB peripheral. When the backlight is on but there is no image, you should check the sync signals, pixel clock, data line order, and panel timing; when the interface is normal but the brightness cannot be adjusted, then check the STC8 backlight control.
6.4 Function Button and Chat State¶
boot_button_.OnClick([this]() {
auto& app = Application::GetInstance();
if (app.GetDeviceState() == kDeviceStateStarting &&
!WifiStation::GetInstance().IsConnected()) {
ResetWifiConfiguration();
}
app.ToggleChatState();
});
The callback executes when a button click event occurs on GPIO35. When the device is still in the starting state and not connected to the network, the button first triggers a Wi-Fi configuration reset; afterward, regardless of the current state, it calls ToggleChatState() to toggle the chat state. Therefore, the same button serves both network recovery and chat control.
Do not perform unverified experiments by changing the active level, otherwise it may cause continuous pressing or abnormal boot mode. If the button does not respond, check GPIO35, the button callback registration log, and the application state machine in sequence, rather than only checking the interface.
6.5 PDM Microphone and I2S Speaker Adaptation¶
static NoAudioCodecSimplexPdm audio_codec(
AUDIO_INPUT_SAMPLE_RATE,
AUDIO_OUTPUT_SAMPLE_RATE,
AUDIO_I2S_GPIO_BCLK,
AUDIO_I2S_GPIO_WS,
AUDIO_I2S_GPIO_DOUT,
AUDIO_PDM_MIC_CLK,
AUDIO_PDM_MIC_DIN);
This board has no external audio codec chip, so it uses NoAudioCodecSimplexPdm: the speaker uses standard I2S simplex output, and the microphone uses PDM input. This static object is created when the audio service first calls GetAudioCodec() and is reused throughout the entire program lifecycle.
If voice can be recognized but nothing is played, prioritize checking GPIO21/22/23 and the STC8 AUDIO_SD amplifier enable; if it can play but cannot be recognized, prioritize checking GPIO24/25 and the microphone orientation. The two links are independent; you cannot use "the speaker has sound" to prove that the microphone is normal.
6.6 Camera Capture: Sensor → CSI → ISP → PSRAM¶
File: bsp_camera.c
camera_buffer_size = 1024 * 600 * ((BITS_PER_PIXEL + 7) / 8);
camera_buffer = heap_caps_aligned_calloc(
cache_line_size, 1, camera_buffer_size,
MALLOC_CAP_SPIRAM | MALLOC_CAP_8BIT);
err = camera_sensor_init();
err = camera_csi_init();
err = isp_init();
err = esp_cam_ctlr_start(cam_handle);
RGB565 occupies 2 bytes per pixel; one frame requires about 1.17 MiB, so the frame buffer must be placed in PSRAM and aligned to the cache line. The sensor output mode is MIPI_2lane_24Minput_RAW8_1024x600_30fps; CSI receives RAW8, and ISP then converts it to RGB565. When PSRAM is not enabled or fragmentation is severe, the first symptom to appear is usually buffer allocation failure, rather than a network error.
The resolution, pixel format, and buffer size must be synchronized. Modifying only one of them will cause color abnormality, screen tearing, out-of-bounds access, or restart.
6.7 Image Preview and Memory Ownership¶
File: p4_bsp_camera.cc
esp_err_t err = camera_refresh();
auto img_dsc = (lv_img_dsc_t*)heap_caps_calloc(
1, sizeof(lv_img_dsc_t), MALLOC_CAP_8BIT);
img_dsc->header.cf = LV_COLOR_FORMAT_RGB565;
img_dsc->header.w = 1024;
img_dsc->header.h = 600;
img_dsc->header.stride = 1024 * 2;
img_dsc->data_size = 1024 * 600 * 2;
img_dsc->data = (uint8_t*)heap_caps_malloc(
img_dsc->data_size, MALLOC_CAP_SPIRAM);
memcpy((void*)img_dsc->data, my_trans.buffer, img_dsc->data_size);
display->SetPreviewImage(img_dsc);
Capture() first waits for a new frame, then creates an independent LVGL image descriptor and copies the frame into a new PSRAM region. The purpose of the copy is not to improve speed, but to prevent the CSI from overwriting the shared capture buffer while the display is still reading the old image. If the descriptor or image memory allocation fails, the function returns false, and the interface will not get a new preview image.
6.8 JPEG Streaming Encoding and Visual Explanation Request¶
QueueHandle_t jpeg_queue = xQueueCreate(40, sizeof(JpegChunk));
encoder_thread_ = std::thread([this, jpeg_queue, &fb]() {
frame2jpg_cb(&fb, 80, /* chunk callback */, jpeg_queue);
});
http->SetHeader("Content-Type", "multipart/form-data; boundary=" + boundary);
http->SetHeader("Transfer-Encoding", "chunked");
Explain() does not first allocate a complete large JPEG buffer; instead, it lets the encoding thread produce data in chunks, which are handed to the HTTP writer through a FreeRTOS queue. This reduces the pressure of large contiguous memory. The JPEG quality parameter is 80, striking a compromise between clarity and upload size; if http->Open() fails to establish a connection, the code will wait for the encoding thread to finish, release the JPEG data chunks already generated but not yet sent in the queue, then delete the queue and throw an exception.
When the interface address explain_url_ is empty, the function throws an exception; when the token explain_token_ is empty, it does not abort the request, but simply does not add the Authorization request header. When the camera is not initialized, the frame buffer is unavailable, JPEG queue creation fails, HTTP connection fails, or the HTTP status code is not 200, the corresponding failure handling is also entered. When troubleshooting, you should distinguish among the four stages: "camera or frame buffer unavailable," "JPEG queue creation failed," "HTTP connection failed," and "server returned a non-200 status code."
6.9 ESP32-C6 Network Startup and Configuration Risks¶
vTaskDelay(pdMS_TO_TICKS(2000));
auto& ssid_manager = SsidManager::GetInstance();
auto ssid_list = ssid_manager.GetSsidList();
if (ssid_list.empty()) {
ssid_manager.AddSsid(DEFAULT_WIFI_SSID, DEFAULT_WIFI_PASSWORD);
} else {
bool found = false;
for (const auto& item : ssid_list) {
if (item.ssid == DEFAULT_WIFI_SSID) {
found = true;
ssid_manager.AddSsid(DEFAULT_WIFI_SSID, DEFAULT_WIFI_PASSWORD);
break;
}
}
if (!found) {
ssid_manager.AddSsid(DEFAULT_WIFI_SSID, DEFAULT_WIFI_PASSWORD);
}
}
WifiBoard::StartNetwork();
StartNetwork() first waits a fixed 2 seconds to allow the ESP32-C6 to boot and the SDIO link to be established; it then reads the saved network list via SsidManager. If the list is empty, the code adds the default Wi-Fi from config.h; if the default SSID already exists, it updates that entry with the password defined in the macro; if the default SSID is not present in the list, it appends the default network while preserving the other networks. Once the credentials are prepared, WifiBoard::StartNetwork() is called to enter the common network connection flow. 7. Experimental Observations¶
- After the device resets, the serial port displays
Initializing Elecrow P4 Board.... - After the RGB LCD initialization completes,
RGB LCD initialized: 800x480appears; the STC8 turns on the backlight and displays the full interface. - The serial port displays
Buttons initialized; after a short press of the function button, the chat status changes. - After the camera is successfully powered,
Camera power enabledappears; when the camera is successfully initialized,P4 BSP Camera initialized successfullyappears. - During network startup, it first waits approximately 2 seconds, then outputs the number of saved SSIDs and attempts to connect.
- When the network and AI services are functioning normally, after speaking into the microphone, the device enters a processing state and finally plays the response through the speaker.