Troubleshooting

Q: Can aBMC be accessed remotely?

A: Yes. aBMC is an independent out-of-band management unit. When the device is powered on and the management network port is connected, it supports remote access to the Web management interface and the Redfish/IPMI interfaces, enabling remote power on/off, KVM console, hardware monitoring, virtual media, and other operations and maintenance tasks. Local LAN access: In the same intranet environment, the computer and the aBMC management network are interconnected; enter the aBMC intranet IP to access it directly;

• Cross-region access over the Internet: Ports cannot be exposed directly to the public network. There are two compliant solutions:

1. The terminal connects to the device's local LAN through a VPN, and then accesses aBMC;
2. Deploy a cloud proxy server; the intranet aBMC proactively establishes a long connection outward, and remote operations and maintenance over the Internet are achieved through the proxy.

Q: aBMC becomes inaccessible while the server is running?

A: First distinguish between the two fault scenarios and troubleshoot:

• Scenario 1: The aBMC management IP cannot be pinged First troubleshoot the physical link and Layer 2/Layer 3 network: network cable, switch port status, IP subnet, VLAN division, and access control policies.

• Scenario 2: The aBMC IP can be pinged, but the Web management interface cannot be accessed

  1. If an external monitor can be connected on site for troubleshooting If the monitor normally outputs the terminal/system screen, the underlying operating system is running normally. You can run an access test on the local machine: https://127.0.0.1 1.1 If the local machine can open the aBMC page normally: the aBMC business service is running normally, and the fault lies in the network between the O&M terminal and the device. Troubleshooting directions: subnet connectivity, switch ACL policies, VPN, proxy, etc.; 1.2 If the local machine also cannot access the aBMC page: determine that the aBMC service is abnormal, and troubleshoot in the following order 1.2.1 Check the service running status systemctl status aBMC 1.2.2 If the service status is active (running): check the business configuration cat /etc/aBMC/config.ini 1.2.2.1 Confirm that the access protocol is enabled (HTTPS by default) 1.2.2.2 Confirm whether the Web listening port has been modified 1.2.2.3 Confirm the Redis communication port configuration 1.2.3 If the service is abnormal or not started: troubleshoot the cause of the service startup failure based on the logs.

  2. If an external monitor cannot be connected on site, or the monitor is black or distorted with no output

     2.1 Use a Console debug cable (USB to RJ45 serial, baud rate 115200) to connect to the serial terminal and log in to the command-line system;
     2.2 After successful login, reuse the 1.2 procedure above to troubleshoot the aBMC service issue.

  3. If neither the monitor terminal login nor the Console login responds, it is most likely that the BMC program is stuck or the system has crashed. Common causes: 3.1 A user business process running with root privileges causes a kernel exception 3.1.1 First check the system logs and kernel logs to locate the application BUG; 3.1.2 If a kernel crash Oops/panic occurs, the logs often cannot be persisted to disk. It is recommended to continuously capture serial logs to reproduce the issue, so that development can locate the root cause.

     3.2 The BMC system partition disk is full (a frequent customer issue) This is mostly caused by users deploying their own business applications that generate a large amount of debug logs and temporary files without periodic cleanup, exhausting the disk and causing system function anomalies.

     3.3 Improper mount configuration in /etc/fstab modified by the user blocks system startup If the nofail parameter is not added when configuring automatic disk mounting at boot in /etc/fstab, the system will fall into a startup blocking state waiting for a timeout once an anomaly occurs during mounting. Typical scenarios that commonly cause this fault include: 3.3.1 Mismatched resource identifiers: The disk UUID fixed during development and debugging does not match the actual production environment, and was not corrected before deployment. 3.3.2 Underlying hardware failure: The disk has a physical fault, causing the kernel to fail to get a device response and the mount request to be blocked permanently. 3.3.3 Device naming drift: Mounting that depends on device paths (such as /dev/sda1) becomes invalid because the kernel enumeration order changes after a system restart. 3.3.4 Unprepared storage media: The newly added disk was not formatted or partitioned, so the system cannot recognize a valid file system structure when mounting.

Q: How to use Redis on aBMC?

A: Redis 7.2 is built into the aBMC system by default. If a business service requires Redis, it can directly use the instance shipped with the system; no reinstallation or deployment is needed. To customize the Redis port, database number, or access password:

1. Modify the Redis-related parameters in the configuration file /etc/aBMC/config.ini;
2. Restart the aBMC service for the new configuration to take effect.

Q: How to change the LOGO displayed after a session login?

A: By default, after a user logs in to aBMC through SSH, the serial console, or another method, the terminal displays the default system LOGO. To change the LOGO displayed after a session login, edit the following file

/etc/update-motd.d/00-header

After saving the file, establish a new session to see the changed display.

Q: May IP addresses be changed or added on the BMC usb_net network card and the sub-board eth0 network card?

A: No.

• What it is

  The usb_net network card of the BMC (192.168.150.100/24) and the eth0 network card of the sub-board (192.168.150.xx/24) form the SSH communication link between the BMC and the sub-board.

  This link is the private communication link defined by aBMC, and it is used for internal communication between the BMC and the sub-board.

• Why it is not allowed

  aBMC relies on this link for communication between the BMC and the sub-board. Adding, changing, or deleting the IP addresses of the network cards on this link prevents the BMC and the sub-board from communicating normally, which makes the sub-board unreachable and affects the normal use of aBMC.

• What to do if communication fails after a change

  If the related network configuration was changed by mistake and the BMC can no longer communicate with the sub-board, restore it as follows:

  1. BMC

     Run the following commands to clear the Netplan network configuration and reboot the BMC:

     rm -rf /etc/netplan/*
     reboot

  2. Sub-board

     Open the aBMC firmware upgrade page and run the firmware upgrade again.

     For the exact steps, see: Firmware Upgrade

Q: What should I do if the kernel log shows bm-sophon0 api process fail or Tpu hang?

A: These messages indicate abnormal task processing in the Sophon TPU (AI accelerator), which may be caused by driver or firmware issues, prolonged high TPU load, or hardware failure. Proceed as follows:

1. Capture the logs:

   dmesg -T | grep -iE "bm-sophon|tpu"
   journalctl -k --no-pager | grep -iE "bm-sophon|tpu"

2. Record the sub-board number and the services running at the time, then restart the corresponding sub-board or the whole server and check whether the issue recurs.

3. If the issue does not recur after the restart, continue to monitor it. If it recurs, stop the TPU services on that sub-board, collect the logs above, and contact technical support.

Q: The pci_net NIC disappears after the BMC powers on. How to troubleshoot?

A: If the pci_net NIC is not visible in the system after the BMC powers on, check the following:

1. Confirm whether the NIC is actually missing:

   ip link show
   ls /sys/class/net
   lspci -nn | grep -i net

2. Use lsblk to check whether the /boot partition is mounted:

   lsblk
   findmnt /boot

3. If the /boot partition is not mounted, the system boot or partition may be abnormal. Do not format the partition manually, and do not modify /etc/fstab at will. Contact technical support.

4. If /boot is mounted properly but pci_net is still missing, collect the following information and contact technical support:

   dmesg -T | grep -iE "pci|net"
   lspci -nn

Q: The kernel log shows mmc0 errors. How can I tell whether the eMMC is healthy?

A: mmc0 usually corresponds to the on-board eMMC. When mmc0-related errors occur, first check the error messages, then read the eMMC life time information:

1. Check kernel errors:

   dmesg -T | grep -i mmc
   journalctl -k --no-pager | grep -i mmc

2. Read the eMMC life time:

   cat /sys/block/mmcblk0/device/life_time
   cat /sys/block/mmcblk0/device/pre_eol_info

3. Interpretation:

   • life_time contains two hexadecimal values, Type A and Type B in order (the main storage area usually refers to Type B): 0x01 means 0–10% used, 0x02 means 10–20% used, and so on; 0x0A means 90–100% used, and 0x0B means the life time has been exceeded;
   • pre_eol_info: 0x01 means normal, 0x02 means warning (about 80% used), and 0x03 means urgent (about 90% used).

4. If pre_eol_info is 0x01 and life_time is low, the eMMC is healthy and you can continue troubleshooting as a normal I/O error. If 0x02, 0x03, or 0x0B appears, or the kernel keeps reporting I/O errors, back up the data first and contact technical support.

Q: The SATA drive has become read-only. What should I do?

A: A SATA drive becoming read-only is usually caused by the drive entering failure protection mode, or by file system or link issues. Check as follows:

1. Use lsblk to identify the drive letter of the affected disk (represented as /dev/sdX below).

2. Confirm the read-only state:

   mount | grep " ro,"
   cat /sys/block/sdX/ro

3. Check kernel errors and drive health information:

   dmesg -T | grep -iE "ata|sata|I/O error|read-only"
   smartctl -a /dev/sdX   # if smartctl is installed

4. If a hard disk drive (HDD) is used, it is recommended to replace it with a solid-state drive (SSD). HDDs are more likely to enter read-only protection when they age, experience vibration, or have power supply issues, and continued use may cause data loss.

5. Back up the data before replacing the drive. If the read-only issue persists after replacing it with an SSD, collect the logs above and contact technical support.