Kernel drops subshells messages if a subshell no longer exists.

[krassowski]

Description

A shell message addressed to a subshell that no longer exists is dropped. The kernel logs Invalid message with a KeyError and sends nothing back (no reply nor error). A client that waits for the message to complete waits forever.

This is easy to hit because delete_subshell_request travels on the control channel while the messages it invalidates travel on the shell channel. A client that sends a shell message on a subshell and then deletes that subshell has no way to order the two, so the delete often wins.

The routing happens in Kernel.shell_channel_thread_main, where the except Exception around the send swallows the message:

ipykernel/ipykernel/kernelbase.py

Lines 585 to 595 in 5dbd5ce

	try:
	msg3 = self.session.deserialize(msg2, content=False, copy=False)
	subshell_id = msg3["header"].get("subshell_id")
	# Find inproc pair socket to use to send message to correct subshell.
	subshell_manager = self.shell_channel_thread.manager
	socket = subshell_manager.get_shell_channel_to_subshell_socket(subshell_id)
	assert socket is not None
	socket.send_multipart(msg, copy=False)
	except Exception:
	self.log.error("Invalid message", exc_info=True) # noqa: G201

The get_shell_channel_to_subshell_socket call ends at a cache lookup, which raises KeyError once the subshell is gone:

ipykernel/ipykernel/subshell_manager.py

Lines 86 to 92 in 5dbd5ce

	def get_shell_channel_to_subshell_pair(self, subshell_id: str | None) -> SocketPair:
	"""Return the inproc socket pair used to send messages from the shell channel
	to a particular subshell or main shell."""
	if subshell_id is None:
	return self._shell_channel_to_main
	with self._lock_cache:
	return self._cache[subshell_id].shell_channel_to_subshell

JupyterLab hits this with comms. Comm messages go to a per comm target subshell, and closing the last comm on a target sends comm_close on that subshell and then deletes it. The comm_close is lost about 70% of the time, and the front end hangs on a promise that never settles. See jupyterlab/jupyterlab#19722.

Reproduce

The script below opens a subshell, sends comm_close on it, then deletes the subshell. It reports how many times the kernel never published the matching idle status. On ipykernel 7.3.0 with Python 3.12.14 it lost 12 and 13 of 20 rounds on two runs. I used Claude Opus 5 to generate and run this reporducer.

import queue, sys, time
from jupyter_client.manager import KernelManager


def send(kc, channel, msg_type, content, subshell_id=None):
    msg = kc.session.msg(msg_type, content)
    if subshell_id is not None:
        msg["header"]["subshell_id"] = subshell_id
    kc.session.send(channel, msg)
    return msg["header"]["msg_id"]


def wait_for(get_msg, msg_id, predicate, timeout):
    deadline = time.time() + timeout
    while time.time() < deadline:
        try:
            msg = get_msg(timeout=max(0.05, deadline - time.time()))
        except queue.Empty:
            return None
        if msg["parent_header"].get("msg_id") == msg_id and predicate(msg):
            return msg
    return None


def is_idle(msg):
    return msg["msg_type"] == "status" and msg["content"]["execution_state"] == "idle"


km = KernelManager(kernel_name="python3")
km.start_kernel()
kc = km.client()
kc.start_channels()
kc.wait_for_ready(timeout=60)

lost = 0
for i in range(20):
    mid = send(kc, kc.control_channel.socket, "create_subshell_request", {})
    reply = wait_for(kc.get_control_msg, mid, lambda m: True, 10)
    subshell_id = reply["content"]["subshell_id"]

    mid = send(kc, kc.shell_channel.socket, "comm_open",
               {"comm_id": f"comm-{i}", "target_name": "test", "data": {}}, subshell_id)
    assert wait_for(kc.get_iopub_msg, mid, is_idle, 5) is not None

    close_id = send(kc, kc.shell_channel.socket, "comm_close",
                    {"comm_id": f"comm-{i}", "data": {}}, subshell_id)
    send(kc, kc.control_channel.socket, "delete_subshell_request",
         {"subshell_id": subshell_id})

    answered = wait_for(kc.get_iopub_msg, close_id, is_idle, 5) is not None
    lost += 0 if answered else 1
    print(f"round {i}: {'ok' if answered else 'lost'}", flush=True)

print(f"{lost}/20 messages never reported idle")
kc.stop_channels()
km.shutdown_kernel(now=True)

The kernel log shows one of these per lost message:

[IPKernelApp] ERROR | Invalid message
Traceback (most recent call last):
  File ".../ipykernel/kernelbase.py", line 591, in shell_channel_thread_main
    socket = subshell_manager.get_shell_channel_to_subshell_socket(subshell_id)
  File ".../ipykernel/subshell_manager.py", line 107, in get_shell_channel_to_subshell_socket
    return self.get_shell_channel_to_subshell_pair(subshell_id).from_socket
  File ".../ipykernel/subshell_manager.py", line 92, in get_shell_channel_to_subshell_pair
    return self._cache[subshell_id].shell_channel_to_subshell
KeyError: '...'

Expected behavior

The client should be able to tell that the message will not be answered. Two options, either one closes the hole:

1. Answer the message. Publish busy and idle for it and send an error reply with status: "error" naming the missing subshell, in the same way as an unknown message type.
2. Drain before deleting. Make delete_subshell_request wait for the messages already queued for that subshell, so a message that reached the kernel before the delete is still processed.

Option 2 alone still leaves a window for a message that arrives after the delete, so option 1 is needed in any case.

Related: _process_control_request catches the KeyError from _delete_subshell for an unknown subshell and returns an error reply, so deletes report the problem and shell messages do not.

ipykernel/ipykernel/subshell_manager.py

Lines 214 to 224 in 5dbd5ce

	else:
	msg = f"Unrecognised message type {type!r}"
	raise RuntimeError(msg)
	except BaseException as err:
	reply = {
	"status": "error",
	"evalue": str(err),
	}
	# Return the reply to the control thread.
	self.control_to_shell_channel.to_socket.send_json(reply)

Context

• ipykernel: 7.3.0
• jupyter_client: 8.9.1
• Python: 3.12.14
• Operating system: Linux

Related:

• Example Tests are flaky because of subshell issues in ipykernel v7 jupyterlab/jupyterlab#19722
• ipykernel 7.0.0 release with subshells #1438

[ianthomas23]

An explicit reply message (option 1) does seem like a good idea. I am not sure if the busy and idle messages are really necessary, but I would defer to your opinion on whether it would cause other problems if they are not issued.

[krassowski]

I am not sure if the busy and idle messages are really necessary

They are in the sense that JupyterLab currently expects idle to resolve future as done. I am not sure if it should.

[krassowski]

Technically:

Though the idle/busy pair is only documented as required for the shell channel here https://jupyter-client.readthedocs.io/en/stable/messaging.html#request-reply and not at all mentioned for the control channel https://jupyter-client.readthedocs.io/en/stable/messaging.html#messages-on-the-control-router-dealer-channel.

The problematic message is sent on the shell channel so maybe that's case shut.