[dgryski]

Summary

With -scheduler=cores on the ESP32-S3, an interrupt handler that wakes a goroutine can hang the core.

scheduleTask takes schedulerLock. On the S3 that lock is a plain spinlock and taking it does not mask interrupts. If the interrupt lands while the code it interrupted on the same core holds the lock, the handler spins on a lock it can never get. The interrupted code cannot run until the handler returns.

Peripheral interrupts run on core 0 only, so core 0 is where this happens. A handler that sends on a channel with a parked receiver is enough to trigger it.

This PR adds schedulerSpinLock. Lock disables interrupts and Unlock restores the previous state, saved per core. A handler on the same core now runs after Unlock, so its scheduleTask gets the lock.

schedulerUnlockAndWait needs one change. It relies on interrupts staying off between the unlock and waiti so a wake is not lost. It now releases the lock without restoring interrupts and takes it again with the saved state.

This is the same approach as #6069 on riscv-qemu. The S3 has two cores, so the deadlock that #6069 fixes on three or more harts does not apply here.

How this was found

I wrote Spin models of the cores scheduler while debugging a hang on riscv-qemu (#6047). The model of the rp2 scheduler had a variant where an interrupt handler wakes a goroutine through scheduleTask. It deadlocked. I first treated that as hypothetical, because I found no runtime or driver handler that does it. A user callback can, for example a GPIO callback that sends on a channel.

I then wrote a reproducer for the S3 and ran it in esp-emu (see #6055). It stalls on the first interrupt.

To check the cause, I added a temporary bounded spin in scheduleTaskNoWake for interrupt context. The handler stopped with scheduleTask in an interrupt spun on schedulerLock in 3 of 3 runs. I reverted it. A build where core 0 loops without taking schedulerLock does not stall.

Testing

All runs are on esp-emu 0.51.0 with -scheduler=cores -serial=uart, using a compiler built from dev.

The reproducer has core 1 raise a software interrupt (FROM_CPU_2) at varying points. The handler on core 0 sends on a channel to a parked receiver. Core 0 loops on runtime.Gosched(). Core 1 reports the result.

Program	dev	dev + this PR
reproducer	stalls after 1 interrupt	OK, 5000 interrupts, 3964 wakes
control (handler does not use the scheduler)	OK	OK
core 0 never takes schedulerLock	OK	OK
t/timers.go, mixed and allocgc stress programs	OK	OK
timers and channel stress programs	panic from #6047	same panic, same addresses

With #6060 applied, the timers stress program passes with and without this PR. The channel stress program hangs with and without it. That hang is the one #6106 fixes. With #6060 and #6106 both applied, both stress programs pass with and without this PR, and the reproducer still stalls without this PR and passes with it.

Each result is one run per build. The emulator is deterministic.

Reproducer

main.go

package main

import (
	"device/esp"
	"machine"
	"runtime"
	"runtime/interrupt"
	"sync/atomic"
)

// The runtime uses CPU interrupt line 3 for the cross-core interrupt.
const cpuInt = 4

const (
	rounds    = 5000
	spinLimit = 5000000
)

var (
	ch      = make(chan struct{}, 1)
	entered atomic.Uint32
	woken   atomic.Uint32
	ready   atomic.Uint32
)

func handler(interrupt.Interrupt) {
	// The source is level triggered, so clear it first.
	esp.SYSTEM.CPU_INTR_FROM_CPU_2.Set(0)
	entered.Add(1)
	select {
	case ch <- struct{}{}:
	default:
	}
}

func main() {
	go func() {
		machine.AssignCore(0)
		for {
			<-ch
			woken.Add(1)
		}
	}()

	go func() {
		machine.AssignCore(0)
		esp.INTERRUPT_CORE0.SetCPU_INTR_FROM_CPU_2_MAP(cpuInt)
		if err := interrupt.New(cpuInt, handler).Enable(); err != nil {
			println("ISRWAKE-RESULT: SETUP FAILED", err.Error())
			return
		}
		ready.Store(1)
		for {
			runtime.Gosched()
		}
	}()

	machine.AssignCore(1)
	for ready.Load() == 0 {
	}

	var seed uint32 = 1
	for i := 0; i < rounds; i++ {
		before := entered.Load()
		esp.SYSTEM.CPU_INTR_FROM_CPU_2.Set(1)
		for n := 0; entered.Load() == before; n++ {
			if n > spinLimit {
				println("ISRWAKE-RESULT: STALLED after", i, "interrupts")
				return
			}
		}
		seed = seed*1664525 + 1013904223
		for n := uint32(0); n < (seed>>20)%400; n++ {
			entered.Load()
		}
	}
	println("ISRWAKE-RESULT: OK", rounds, "interrupts,", woken.Load(), "wakes")
}

Build and run

tinygo build -target=esp32s3-generic -scheduler=cores -serial=uart -o isrwake.bin .
esp-emu --chip esp32s3 --firmware isrwake.bin --exit-on "ISRWAKE-RESULT"

Not done

• This has only run on the emulator. I have not tried real hardware.
• I did not measure interrupt latency. Interrupts are now off while the scheduler lock is held.
• The same problem exists on rp2, where schedulerLock is a hardware spinlock that does not mask interrupts. I modelled the fix there and found no new deadlock. I have not tested it on a Pico.
• I did not add a test. It needs an esp-emu target (Add esp-emu emulator targets for ESP32-C3 and ESP32-S3 #6055).
• I did not run the full test suite.

[davecheney]

I tested this on an ESP32-S3-WROOM-1 devkit with -scheduler=cores, using the reproducer from the description, with output over each of its two USB connectors. In the reproducer I replaced time.Sleep with a busy loop on time.Now, because time.Sleep blocks on schedulerLock after the stall.

Serial	parent b517644e	this PR
usb	5 of 5 STALLED after 1 interrupts	6 of 6 OK 5000 interrupts
uart	5 of 5 STALLED after 2 interrupts	5 of 5 OK 5000 interrupts