Determine wakeup cause (ACPI)
I had a problem with my laptop to enter suspend mode, when I ran systemctl suspend it would just woke up immediately.
Let's find out what causes this.
Advanced Configuration and Power Interface (ACPI)
First a few words about ACPI.
The embedded Linux systems I used to work with usually not supports ACPI, even if support for ACPI starts to show up on some ARM64 and Risc-V boards. But still, ACPI is more of a x86/x86_64 thing, so whenever I have to deal with ACPI, it is on my laptop or desktop computer.
I actually had a case a few years ago where I had to debug ACPI tables for a (x86) system, but that is pretty much all I've messed with it.
In short, ACPI is an open standard for device configuration and power management by the operating system. It is an interface between the operation system and the firmware (BIOS or UEFI) that allows the OS to control device configurations (such as discover new hardware) and power management (such as suspend/resume).
Embedded systems are usually quite fixed in their hardware configuration, so the device configuration part of ACPI is not that important. But I think the power management part is welcome.
Wakeup causes
When a system enters sleep mode, it can be woken up by several events. They are either enabled or disabled by the firmware (BIOS/UEFI) and/or the operating system via ACPI.

You will find a list of available wakeup sources by examine /proc/acpi/wakeup.
For example, on my laptop I have these:
Device S-state Status Sysfs node PEG0 S4 *enabled pci:0000:00:06.0 PEGP S4 *disabled pci:0000:02:00.0 PEG1 S4 *enabled pci:0000:00:01.0 PEGP S4 *disabled pci:0000:01:00.0 PEG2 S4 *disabled PEGP S4 *disabled PEG3 S4 *disabled PEGP S4 *disabled GLAN S4 *enabled pci:0000:00:1f.6 XHCI S0 *enabled pci:0000:00:14.0 XDCI S4 *disabled HDAS S4 *disabled pci:0000:00:1f.3 RP01 S4 *disabled PXSX S4 *disabled RP02 S4 *disabled PXSX S4 *disabled RP03 S4 *enabled pci:0000:00:1c.0 PXSX S4 *disabled pci:0000:73:00.0 *disabled platform:rtsx_pci_sdmmc.0 RP04 S4 *disabled PXSX S4 *disabled RP05 S4 *disabled PXSX S4 *disabled RP06 S4 *disabled PXSX S4 *disabled RP07 S4 *disabled PXSX S4 *disabled RP08 S4 *disabled PXSX S4 *disabled RP09 S4 *disabled PXSX S4 *disabled RP10 S4 *disabled PXSX S4 *disabled RP11 S4 *disabled PXSX S4 *disabled RP12 S4 *disabled PXSX S4 *disabled RP13 S4 *disabled PXSX S4 *disabled RP14 S4 *disabled PXSX S4 *disabled RP15 S4 *disabled PXSX S4 *disabled RP16 S4 *disabled PXSX S4 *disabled RP17 S4 *disabled PXSX S4 *disabled RP18 S4 *disabled PXSX S4 *disabled RP19 S4 *disabled PXSX S4 *disabled RP20 S4 *disabled PXSX S4 *disabled RP21 S4 *disabled PXSX S4 *disabled RP22 S4 *disabled PXSX S4 *disabled RP23 S4 *disabled PXSX S4 *disabled RP24 S4 *disabled PXSX S4 *disabled CNVW S4 *disabled pci:0000:00:14.3 TXHC S4 *enabled pci:0000:00:0d.0 TDM0 S4 *enabled pci:0000:00:0d.2 TDM1 S4 *disabled TRP0 S4 *enabled pci:0000:00:07.0 PXSX S4 *disabled TRP1 S4 *enabled pci:0000:00:07.1 PXSX S4 *disabled TRP2 S4 *disabled PXSX S4 *disabled TRP3 S4 *disabled PXSX S4 *disabled AWAC S4 *enabled platform:ACPI000E:00 LID0 S3 *enabled platform:PNP0C0D:00 PBTN S3 *enabled platform:PNP0C0C:00
Most of them are disabled though.
The list contains all devices capable of waking up the system from sleep mode. The first column is the device name, the second column is the sleep state (S0, S3, S4), the third column is the status (enabled/disabled) and the last column is the sysfs node if any.
To toggle the state of a certain device, you can write the device name to the /proc/acpi/wakeup entry. For example, to disable the LID0 device (the lid switch), you can run:
1$ echo LID0 | sudo tee /proc/acpi/wakeup
Get more details
So, for my case, I still need to know which of all these events that wakes up my system.
To find out, I examined the wakeup_sources file exposed by debugfs. If you don't have debugfs mounted, you can do mount it with:
1$ sudo mount -t debugfs none /sys/kernel/debug
Assuming you have a kernel compiled with CONFIG_DEBUG_FS (most distributions have).
Then you can check statistics for all wakeup sources with:
1$ cat /sys/kernel/debug/wakeup_sources/wakeup_sources
On my laptop, I get the following output:
name active_count event_count wakeup_count expire_count active_since total_time max_time last_change prevent_suspend_time ucsi-source-psy-USBC000:002 4 4 0 0 0 0 0 6789 0 2-2.4 0 0 0 0 0 0 0 0 0 2-2.2 0 0 0 0 0 0 0 0 0 ucsi-source-psy-USBC000:001 4 4 0 0 0 1 0 5076 0 0000:00:0d.2 0 0 0 0 0 0 0 0 0 domain0 0 0 0 0 0 0 0 0 0 0-0 0 0 0 0 0 0 0 0 0 DELL0A68:00 0 0 0 0 0 0 0 0 0 i2c-DELL0A68:00 0 0 0 0 0 0 0 0 0 0000:00:1f.6 0 0 0 0 0 0 0 0 0 GTCH7503:00 0 0 0 0 0 0 0 0 0 USBC000:00 0 0 0 0 0 0 0 0 0 INTC1043:03 0 0 0 0 0 0 0 0 0 INTC1043:02 0 0 0 0 0 0 0 0 0 INT33A1:00 0 0 0 0 0 0 0 0 0 INTC1043:01 0 0 0 0 0 0 0 0 0 INTC1043:00 0 0 0 0 0 0 0 0 0 INTC1051:00 0 0 0 0 0 0 0 0 0 INTC1051:00 0 0 0 0 0 0 0 0 0 INTC1040:00 0 0 0 0 0 0 0 0 0 INT34C6:00 0 0 0 0 0 0 0 0 0 ACPI000E:00 0 0 0 0 0 0 0 0 0 ACPI000E:00 0 0 0 0 0 0 0 0 0 ACPI000C:00 0 0 0 0 0 0 0 0 0 3-8.3.3.1 3 3 0 0 0 0 0 5438280 0 mmc0 2 2 0 2 0 10520 5426 5442148 0 serio0 0 0 0 0 0 0 0 0 0 PNP0C14:06 0 0 0 0 0 0 0 0 0 PNP0C14:05 0 0 0 0 0 0 0 0 0 PNP0C14:04 0 0 0 0 0 0 0 0 0 PNP0C14:03 0 0 0 0 0 0 0 0 0 PNP0C14:02 0 0 0 0 0 0 0 0 0 PNP0C14:01 0 0 0 0 0 0 0 0 0 PNP0C14:00 0 0 0 0 0 0 0 0 0 alarmtimer.0.auto 0 0 0 0 0 0 0 0 0 00:01 0 0 0 0 0 0 0 0 0 0000:00:14.0 3 3 0 3 0 307 102 5438332 0 0000:00:0d.0 0 0 0 0 0 0 0 0 0 PNP0C0A:00 0 0 0 0 0 0 0 0 0 STM0125:00 0 0 0 0 0 0 0 0 0 PNP0C0E:00 0 0 0 0 0 0 0 0 0 AC 1 1 0 0 0 2 2 590 0 ACPI0003:00 0 0 0 0 0 0 0 0 0 PNP0C0C:00 0 0 0 0 0 0 0 0 0 PNP0C0D:00 0 0 0 0 0 0 0 0 0 PNP0A08:00 0 0 0 0 0 0 0 0 0 device:43 0 0 0 0 0 0 0 0 0 device:01 0 0 0 0 0 0 0 0 0 LNXVIDEO:00 0 0 0 0 0 0 0 0 0 device:14 0 0 0 0 0 0 0 0 0 device:13 0 0 0 0 0 0 0 0 0 device:34 0 0 0 0 0 0 0 0 0 device:12 0 0 0 0 0 0 0 0 0 0000:00:1c.0 0 0 0 0 0 0 0 0 0 device:42 0 0 0 0 0 0 0 0 0 device:6e 0 0 0 0 0 0 0 0 0 device:9a 0 0 0 0 0 0 0 0 0 device:83 0 0 0 0 0 0 0 0 0 device:82 0 0 0 0 0 0 0 0 0 device:9c 0 0 0 0 0 0 0 0 0 device:15 3 3 0 0 0 0 0 5438229 0 device:91 0 0 0 0 0 0 0 0 0 device:a5 0 0 0 0 0 0 0 0 0 device:9e 0 0 0 0 0 0 0 0 0 device:11 0 0 0 0 0 0 0 0 0 0000:00:07.1 0 0 0 0 0 0 0 0 0 device:a9 0 0 0 0 0 0 0 0 0 0000:00:07.0 0 0 0 0 0 0 0 0 0 device:a7 0 0 0 0 0 0 0 0 0 0000:00:06.0 0 0 0 0 0 0 0 0 0 device:00 0 0 0 0 0 0 0 0 0 device:0f 0 0 0 0 0 0 0 0 0 LNXVIDEO:01 0 0 0 0 0 0 0 0 0 0000:00:01.0 0 0 0 0 0 0 0 0 0 device:02 0 0 0 0 0 0 0 0 0 deleted 0 0 0 0 0 0 0 0 0
Here we have all statistics for all wakup sources.
When comparing the ouput before and after suspending, I can see that this line increments the active_count and wakeup_count values:
0000:00:14.0 3 3 0 3 0 307 102 5438332 0
By consulting the /proc/acpi/wakeup file, I can see that the device 0000:00:14.0 is the XHCI (USB controller), which is enabled to wake up the system:
1$ grep "00:14" /proc/acpi/wakeup
2XHCI S0 *disabled pci:0000:00:14.0
So lets disable it:
1$ echo XHCI | sudo tee /proc/acpi/wakeup
And try to put the system to sleep again:
1$ systemctl suspend
Great, now it works. The system is not waking up by itself and I can use the power button to wake it up from suspend.