subprocess on Windows returns `None` instead of raising `UnicodeDecodeError`

[kuotunyu]

Bug description:

On Windows, Popen.communicate() reads stdout and stderr in background threads when a timeout is given, or when more than one standard stream is a pipe. In text mode, the decoding also happens in these threads, in _readerthread():

cpython/Lib/subprocess.py

Lines 1780 to 1782 in 16b357b

	def _readerthread(self, fh, buffer):
	buffer.append(fh.read())
	fh.close()

If decoding fails, the UnicodeDecodeError stays in the thread. It is only printed as "Exception in thread ... (_readerthread)". The caller never gets it, and communicate() returns None for that stream:

cpython/Lib/subprocess.py

Lines 1849 to 1850 in 16b357b

	stdout = stdout[0] if stdout else None
	stderr = stderr[0] if stderr else None

So subprocess.run(..., capture_output=True, text=True) returns normally with stdout=None. The program then fails later with a confusing error, like TypeError: ... 'NoneType'.

The other code paths raise the error. On POSIX, _communicate() decodes in the calling thread. On Windows with one pipe and no timeout, communicate() also reads in the calling thread:

cpython/Lib/subprocess.py

Lines 1372 to 1387 in 16b357b

	# Optimization: If we are not worried about timeouts, we haven't
	# started communicating, and we have one or zero pipes, using select()
	# or threads is unnecessary.
	if (timeout is None and not self._communication_started and
	[self.stdin, self.stdout, self.stderr].count(None) >= 2):
	stdout = None
	stderr = None
	if self.stdin:
	self._stdin_write(input)
	elif self.stdout:
	stdout = self.stdout.read()
	self.stdout.close()
	elif self.stderr:
	stderr = self.stderr.read()
	self.stderr.close()
	self.wait()

Minimal example:

import subprocess, sys

# The child process writes bytes that are not valid UTF-8.
cmd = [sys.executable, "-c", "import sys; sys.stdout.buffer.write(b'ok \\xff\\n')"]

def run(label, **kwargs):
    try:
        r = subprocess.run(cmd, encoding="utf-8", **kwargs)
        print(label, "-> returned, stdout =", repr(r.stdout), flush=True)
    except UnicodeDecodeError as e:
        print(label, "-> raised", repr(e), flush=True)

run("stdout=PIPE", stdout=subprocess.PIPE)
run("stdout=PIPE, timeout=10", stdout=subprocess.PIPE, timeout=10)
run("capture_output=True", capture_output=True)

Output on Windows 11, Python 3.14.5:

stdout=PIPE -> raised UnicodeDecodeError('utf-8', b'ok \xff\n', 3, 4, 'invalid start byte')
Exception in thread Thread-1 (_readerthread):
Traceback (most recent call last):
  File "...\Lib\threading.py", line 1082, in _bootstrap_inner
    self._context.run(self.run)
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^
  File "...\Lib\threading.py", line 1024, in run
    self._target(*self._args, **self._kwargs)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "...\Lib\subprocess.py", line 1614, in _readerthread
    buffer.append(fh.read())
                  ~~~~~~~^^
  File "<frozen codecs>", line 325, in decode
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 3: invalid start byte
stdout=PIPE, timeout=10 -> returned, stdout = None
Exception in thread Thread-2 (_readerthread):
Traceback (most recent call last):
  File "...\Lib\threading.py", line 1082, in _bootstrap_inner
    self._context.run(self.run)
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^
  File "...\Lib\threading.py", line 1024, in run
    self._target(*self._args, **self._kwargs)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "...\Lib\subprocess.py", line 1614, in _readerthread
    buffer.append(fh.read())
                  ~~~~~~~^^
  File "<frozen codecs>", line 325, in decode
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 3: invalid start byte
capture_output=True -> returned, stdout = None

3.12.13, 3.13.13 and 3.15.0b1 give the same three results.

Output on Linux (Ubuntu on WSL 2), Python 3.12.3:

stdout=PIPE -> raised UnicodeDecodeError('utf-8', b'ok \xff\n', 3, 4, 'invalid start byte')
stdout=PIPE, timeout=10 -> raised UnicodeDecodeError('utf-8', b'ok \xff\n', 3, 4, 'invalid start byte')
capture_output=True -> raised UnicodeDecodeError('utf-8', b'ok \xff\n', 3, 4, 'invalid start byte')

I expected all three calls to raise UnicodeDecodeError on Windows too, like on Linux.

This will happen more often with 3.15. Python 3.15 uses UTF-8 as the default encoding (What's New), so text=True now decodes as UTF-8 on Windows. But many Windows programs still write in the console code page. For example, on my Traditional Chinese Windows (code page 950):

import subprocess
r = subprocess.run(["where", "no_such_program"], capture_output=True, text=True)
print("returncode =", r.returncode, "stderr =", repr(r.stderr))

Python 3.14.5:

returncode = 1 stderr = '資訊: 找不到提供模式的檔案。\n'

Python 3.15.0b1:

Exception in thread Thread-2 (_readerthread):
Traceback (most recent call last):
  File "...\Lib\threading.py", line 1218, in _bootstrap_inner
    self._context.run(self.run)
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^
  File "...\Lib\threading.py", line 1160, in run
    self._target(*self._args, **self._kwargs)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "...\Lib\subprocess.py", line 1774, in _readerthread
    buffer.append(fh.read())
                  ~~~~~~~^^
  File "<frozen codecs>", line 325, in decode
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb8 in position 0: invalid start byte
returncode = 1 stderr = None

With a UnicodeDecodeError, the user would know that they need encoding= or errors=. With None, they don't.

This has been seen before:

• Subprocess IndexError possible in _communicate #87589 / bpo-43423 Fix IndexError in subprocess _communicate function #24777 (2021) fixed an IndexError on the same lines by returning None. In the review, it was noted that this only happens when a reader thread fails (bpo-43423 Fix IndexError in subprocess _communicate function #24777 (comment)). A decode error in text mode is an easy way for the reader thread to fail.
• When installing certain libraries using pip in Python 3.15.0a2, pip reports an error #141877: a user of 3.15.0a2 saw these _readerthread tracebacks when pip built numpy. It was closed for another reason.
• In the PR that made UTF-8 mode the default (GH-133711: Enable UTF-8 mode by default (PEP 686) #133712), a Windows buildbot failed in test_regrtest with TypeError: expected string or bytes-like object, got 'NoneType' after the same thread traceback (GH-133711: Enable UTF-8 mode by default (PEP 686) #133712 (comment)). The test was fixed with errors="backslashreplace" in gh-133711: Fix test_regrtest for PYTHONUTF8=1 #134839.

The open PR GH-155409 (for gh-87512) changes the reader threads to read bytes, and decodes in _communicate() like the POSIX code. I ran my example with Lib/subprocess.py from that PR (without its C changes, which this path does not use), and all three calls raised UnicodeDecodeError. So that PR would also fix this on main. But it is a bigger change, and it has no test for this case.

A smaller fix: in _readerthread(), catch the exception and put it in the buffer. Then _communicate() re-raises it in the calling thread after the threads are joined. I tried this change (about 10 lines) on a copy of Lib/subprocess.py from main, with 3.14.5 and 3.15.0b1. With it, all three calls in the example raise UnicodeDecodeError, and bytes mode and the errors= handlers work as before.

I'd be happy to open a PR with this fix and a test, if this approach is acceptable. If GH-155409 is the preferred way, I could send a test for this case instead.

English is not my first language. I used an AI assistant to help write this issue and the reproduction script. The outputs above are from real runs on my machine.

CPython versions tested on:

3.12, 3.13, 3.14, 3.15

Operating systems tested on:

Windows

[SuspiciousChip]

please assign this issue to me

I changed the Windows subprocess reader path so decode failures are surfaced to the caller instead of being lost in a background thread, and I verified it with the subprocess test file.

What I changed
I fixed the bug in the Windows implementation of Popen.communicate() in subprocess.py:1780-1859.

i will work further on it if you can assign this issue to me @kuotunyu

[aisk]

please assign this issue to me

I changed the Windows subprocess reader path so decode failures are surfaced to the caller instead of being lost in a background thread, and I verified it with the subprocess test file.

What I changed
I fixed the bug in the Windows implementation of Popen.communicate() in subprocess.py:1780-1859.

i will work further on it if you can assign this issue to me @kuotunyu

Hi @SuspiciousChip usually we don't assign issue to someone unless he is a project member and an expert on the issue or he is the original reporter.

@kuotunyu said in the original issue he want to work on this, so please find another issue if you want to contribute. :)

[aisk]

CC @serhiy-storchaka as the author of PR GH-155409

[SuspiciousChip]

@aisk thanks for telling me about that. as of i recently started contributing in opensource can you tell me which type of issues should i pick and what to should i learn to get assign to issues.

[aisk]

@aisk thanks for telling me about that. as of i recently started contributing in opensource can you tell me which type of issues should i pick and what to should i learn to get assign to issues.

In CPython we usually don't assign issue to someone unless there are some specified reasons. You can just open a PR for an issue if no one else is working on it, the way to fix it is clear, and the reporter hasn't said that they're working on it.

It's usually best to wait a little while after an issue is created, though. Sometimes contributors have already finished a PR and are just waiting for an issue number to use in the NEWS entry.

Some issues are also labeled "easy", which can be a good place to start. See the dev guide: https://devguide.python.org/getting-started/fixing-issues/