Debugger breakpoints using incorrect URL for a remote debugging session and therefore failing to pause the process

[treffynnon]

Symptoms

This is related to the closed ticket #99, which describes the symptoms of this very problem.

You set a breakpoint, but the process isn't paused when that breakpoint is encountered and carries merrily on.

Environment

This is a remote debugging session between a Windows client (running Chrome devtools) and a remote Linux server (running node with --inspect=0.0.0.0).

What's wrong

By watching the websocket communication with Wireshark it is apparent that the Chrome Dev tools are using an incorrect filepath/URL when calling Debugger.setBreakpointByUrl.

What I am seeing

Request

{
  "id": 17,
  "method": "Debugger.setBreakpointByUrl",
  "params": {
    "lineNumber": 67,
    "url": "EFS\\DEV-efsbastion\\username\\application\\projects\\service\\module\\index.js",
    "columnNumber": 18,
    "condition": ""
  }
}

Response

{
  "id": 17,
  "result": {
    "breakpointId": "EFS\\DEV-efsbastion\\username\\application\\projects\\service\\module\\index.js:67:18",
    "locations": []
  }
}

Note the empty locations array. I am no node debugger expert, but it feels like this locations array should be checked to ensure it is not empty after setting a new breakpoint marker before then showing the marker as active in the UI.

What I should be seeing

Request

{
  "id": 17,
  "method": "Debugger.setBreakpointByUrl",
  "params": {
    "lineNumber": 67,
    "url": "/EFS/DEV-efsbastion/username/application/projects/service/module/index.js",
    "columnNumber": 18,
    "condition": ""
  }
}

Response

{
  "id": 17,
  "result": {
    "breakpointId": "/EFS/DEV-efsbastion/username/application/projects/service/module/index.js:67:18",
    "locations": [{
      "scriptId":"104",
      "lineNumber":67,
      "columnNumber":18
    }]
  }
}

Note the Linux path separators instead of Windows and the initial root slash on the path.

But isn't this a node bug?

No, it is not from what I can tell because of the following two discoveries.

PHPStorm

PHPStorm/WebStorm debugger continues to work just fine using it's URL regexes

Request

{
  "id": 17,
  "method": "Debugger.setBreakpointByUrl",
  "params": {
    "lineNumber": 67,
    "urlRegex": "[/\\\\][iI][nN][dD][eE][xX]\\.[jJ][sS]([;?#!].*)?$"
  }
}

Response

{
  "id": 17,
  "result": {
    "breakpointId": "/[/\\\\][iI][nN][dD][eE][xX]\\.[jJ][sS]([;?#!].*)?$/:67:18",
    "locations": [{
      "scriptId":"104",
      "lineNumber":67,
      "columnNumber":18
    }]
  }
}

Manually recreating the request successfully

If I manually set the breakpoint myself I see a success response back over the websocket. For example consider the following code run from the Chrome console:

let id, key, socket

id = 1000
key = '<debugger-key-goes-here>'

const connect = userSuppliedKey => {
	socket = new WebSocket(`ws://<ip-address-goes-here>:9229/${userSuppliedKey || key}`)
	socket.addEventListener('message', e => {
		const d = JSON.parse(e.data)
		if (d) {
			if (d.method === 'Debugger.scriptParsed') return console.log('scriptParsed', JSON.stringify({ scriptId: d.params.scriptId, url: d.params.url }))
			if (d.method === 'Debugger.breakpointResolved') return console.log('breakpointResolved', JSON.stringify(Object.assign({ breakpointId: d.params.breakpointId }, d.params.location)))
		}
		console.info(`${e.data}`)
	})
}
const send = (method, x) => socket.send(JSON.stringify({ id: ++id, method: `Debugger.${method}`, params: x }))
const _getPossibleBps = (start, end) => send('getPossibleBreakpoints', { start, end })
const getPossibleBps = (scriptId, start, end) => _getPossibleBps({ scriptId, lineNumber: start }, { scriptId, lineNumber: end })
const setBp = (lineNumber, url) => send('setBreakpointByUrl', { lineNumber, url, })
const removeBp = (lineNumber, url, columnNumber) => send('removeBreakpoint', { breakpointId: `${url}:${lineNumber}:${columnNumber || 0}` })
const enable = () => send('enable')

Which gives us this when run in the Chrome console

connect('<debugger-key-goes-here>')
// '<debugger-key-goes-here>'

enable()
// ...
// scriptParsed {"scriptId":"104","url":"/EFS/DEV-efsbastion/username/application/projects/service/module/index.js"}
// ...

getPossibleBps('104', 9, 15)
// {"id":1016,"result":{"locations":[{"scriptId":"104","lineNumber":9,"columnNumber":15},{"scriptId":"104","lineNumber":11,"columnNumber":15},{"scriptId":"104","lineNumber":12,"columnNumber":16},{"scriptId":"104","lineNumber":14,"columnNumber":4},{"scriptId":"104","lineNumber":15,"columnNumber":0},{"scriptId":"104","lineNumber":17,"columnNumber":4},{"scriptId":"104","lineNumber":18,"columnNumber":14}]}}

setBp(11, '/EFS/DEV-efsbastion/username/application/projects/service/module/index.js')
// {"id":1024,"result":{"breakpointId":"/EFS/DEV-efsbastion/username/application/projects/service/module/index.js:11:0","locations":[{"scriptId":"104","lineNumber":11,"columnNumber":15}]}}

removeBp(11, '/EFS/DEV-efsbastion/username/application/projects/service/module/index.js')
// {"id":1019,"result":{}}

Possible introduction of the issue

https://chromium.googlesource.com/chromium/src/+/2140ffeadf3e081e2e0821fc8e1ee040373b205d%5E%21/third_party/WebKit/Source/devtools/front_end/sdk/DebuggerModel.js

in combination with https://chromium.googlesource.com/chromium/src/+blame/master/third_party/blink/renderer/devtools/front_end/common/ParsedURL.js#103

[treffynnon]

Also referenced on https://bugs.chromium.org/p/chromium/issues/detail?id=841697

[alexkozy]

Thanks for great report!

Inside V8 on inspector backend side we try to match breakpoint url and script url. In this case we send incorrect url from frontend side and when we receive new parsed script on backend side we do not set breakpoint properly.

I will do following two points next week:

• check why we send incorrect url from frontend side,
• consider some kind of breakpoint url normalization on backend side to make our protocol a little more flexible.

I will update this issue as soon as I get some results.

(PhpStorm actually uses great regexp)

[mathieulj]

Just to corroborate that this is due to a change on the chrome side and provide a workaround. We experienced this recently running windows workstations with a linux development VM. We had 3 versions of chrome(64, 66, 68) connecting one at a time to a debug session and only 64 was working normally. This was true for node 6, 8 and 10.

As a workaround for those who need to debug, we found that using the id and manually visiting the link chrome-devtools://devtools/bundled/inspector.html?experiments=true&v8only=true&ws=${ip}:${port}/${id} works as expected even in the newer chrome versions. If, for example, you get the message Debugger listening on ws://0.0.0.0:3020/4dacc23a-f296-4b6d-bb47-18af21dfef67 during inspect start, the URL would be chrome-devtools://devtools/bundled/inspector.html?experiments=true&v8only=true&ws=${ip}:3020/4dacc23a-f296-4b6d-bb47-18af21dfef67 where ip is the ip of the remote machine. We also found that the plugin nodejs-v8-inspector-manag(https://chrome.google.com/webstore/detail/nodejs-v8-inspector-manag/gnhhdgbaldcilmgcpfddgdbkhjohddkj) seems to still work.

[brucejo75]

@ak239

Also see: https://bugs.chromium.org/p/chromium/issues/detail?id=844070
& meteor/meteor#9875

[tonymet]

I'm experiencing the lack-of-breakpoints with Windows 10 + WSL (aka Ubuntu Bash for Windows) + Node 8.11.3

@mathieulj 's workaround by manually building the chrome-devtools:// URL works for me.

Thanks for the help guys this was driving me bonkers for a while :D

Node v8.11.3
Windows 10 PRo 1803
Version 67.0.3396.87 (Official Build) (64-bit)

[tonymet]

Also mentioned in the chromium bug, Chrome Extensions "Inspector Manager" is a simpler workaround for now

https://chrome.google.com/webstore/detail/nodejs-v8-inspector-manag/gnhhdgbaldcilmgcpfddgdbkhjohddkj

[treffynnon]

@ak239 is there any progress being made with this issue?

Unfortunately, [Web/PHP]Storm has it's own issues with simply choosing the first file it sees with the name the breakpoint is set against. In the case of files named index.js this is an issue as so many modules use this filename that the breakpoint often ends up being set against the wrong file.

[alexkozy]

I landed following CL recently. It should fix provisional breakpoints in Node.js targets. At least it helps ndb.
I have another CL to land that fixes breakpoints in experimental Node.js modules: CL. I am going to land it soon.
It would be nice if someone could verify that issue is fixed in latest Canary.

[treffynnon]

I downloaded the canary with version: Version 70.0.3508.0 (Official Build) canary (64-bit) and it appears to still have the same problem if that helps.

[mathieulj]

It seems to have gotten worse with the release of chrome 69. The workarounds I posted above no longer work with chrome 69.0.3497.81(stable) and with chrome 71.0.3548.0 (canary). Workstations still running 68.0.3440.106 do work with both workarounds.

[mathieulj]

For those with a Webstorm licence, version 2018.2.3 seems to break correctly on breakpoints.

[treffynnon]

For those without a WebStorm licence you can use the Early Access Programme (EAP) builds for free with the following notice:

This is an early access version of the product. You expressly acknowledge that this version of the product may not be reliable, may not work as intended and may contain errors. Any use of the EAP product is at your own risk.

-- JetBrains

_{NOTE: I am not affiliated with JetBrains}

[alexkozy]

The root of this issue is using platform-specific file paths in our protocol as was mentioned before. I landed V8 patch and close to merging Node patch that will migrate Node starting Node 11 to file urls: nodejs/node#22251.
At the same time I am working on better solution for remote debugging as part of ndb project. I will update this issue as soon as it is ready for Node 8 and Node 10.