Disconnect button behaves differently across OSes with JLink GDB server

[AndriiLishchynskyi]

I have noticed that disconnect behaves differently on Win/Linux/Mac. On Windows after a disconnect(when request=attach) target gets halted regardless of the previous target state. On the other hand, on Linux/Mac detach resumes the target.

This is reproducible on PSoC6 devices.

I did some analysis and here are my observations:

1. It is not reproducible with OpenOCD(Cypress version), probably because 'Detach' is implemented there. If we are talking about JLink GDB server - it does not support detach command.

2. When JLink GDB shuts down "correctly" (exited, but not killed), it resumes the target. Based on the logs (attached) it looks like on Linux/Mac it shuts down "correctly", while on Windows it is just killed. The following message in JLink log indicate GDB was exited properly: "Restoring target state and closing J-Link connection..."

3. If I am launching "Cortex-Debug" debugging on Windows, I see that detach hangs at this line of code preventing GDB server from shutting down "correctly":

   cortex-debug/src/backend/mi2/mi2.ts

   Line 263 in e50947c

   await this.sendCommand('target-detach');

4. The issue is not something new, I have tried even very old Cortex-Debug versions.

Attached logs GDB traces and JLInk GDB server logs.
jlink_attach_detach_gdb_traces_ubuntu.txt
jlink_attach_detach_gdb_traces_win.txt
jlink_attach_detach_ubuntu.txt
jlink_attach_detach_win.txt

I am using:
VSCode-1.60.0
Cortex-Debug-v0.4.4
JLink_V754

So, the question: is it possible to unify behavior across OSes (I believe the proper one is to resume target upon detach)?

[haneefdm]

Thank you @AndriiLishchynskyi for the detailed issue description.

Unfortunately, I have not found even one gdb-server properly implement detach even if they claim support for extended-remote. No one cares, I guess. Yes, the proper behavior is if the session starts with an attach, then detach. Lets examine a few things.

OpenOCD

Yes, the publicly available one does not support detach properly. It should remove all breakpoints, put the processor in non-debug mode and let it continue. It did not do any of that other than detecting that the session is ending. I fixed this for Cypress almost 2 years ago. And, btw almost every vendor now has their own version of OpenOCD.

    await this.sendCommand('target-detach');

Yes, this is not supposed to hang, but it doesn't matter. We have that for debugging and telling us that we have yet another case that does not work properly. The command is supposed to be sent to gdb and in turn gdb requests the gdb-server (OpenOCD, JLink, etc.) to do the right thing. That doesn't happen with some servers. We use a backup plan that is unfortnate

cortex-debug/src/gdb.ts

Line 946 in e50947c

setTimeout(() => { // Give gdb a chance to disconnect and exit normally

We give the gdb and gdb-server 50 milli-seconds to do the right thing and then kill the gdb-server. As you noticed, there is nothing OS-specific in 99% of our code. But NodeJS process.kill() may work differently on different OSes. In the end though, we have to let know that we finished 'detaching' or your buttons in the GUI will be messed up.

Oh then there is another issue. gdb will not allow us to send commands (via gdb) if the program is still running. So, we interrupt first and then start the detaching process. This is the reason why you will find the target in halted state. OpenOCD (not sure about JLink) does allow talking to it but gdb wont. If we put gdb in async mode then will be able to but that is a whole another story.

I will look at all the log files and see if I can spot something. Btw, on jlink-ubuntu, it appears that you have another instance of OpenOCD running. Notice the TCP port numbers 50003/4/5, meaning another instance of OpenOCD is running on 50000/1/2

In the end though, there is nothing we can do to change the behaviour of the various gdb-servers. But, for a given server, we can see if we can have consistent behavior across OS platforms.

The issue is not something new, I have tried even very old Cortex-Debug versions.

Yup, we were hoping the respective gdb-servers will be fixed. From where we stand, we are not sure what we can do. Ideas?

[haneefdm]

Hmmm, in both cases I see the same thing happening as far as cortex-debug to gdb transactions

Windows:

43-target-detach
GDB -> App: {"token":43,"outOfBandRecord":[],"resultRecords":{"resultClass":"error","results":[["msg","Remote doesn't know how to detach"]]}}

Linux:

25-target-detach
GDB -> App: {"token":25,"outOfBandRecord":[],"resultRecords":{"resultClass":"error","results":[["msg","Remote doesn't know how to detach"]]}}

The await in that code is to get a 'done' or an 'error' response. Either way, the await will end. In this case, both responses are identical. So, it should not be hanging.

But it is JLink that is behaving differently. We did not do anything different between the two OSes. It appears that JLink gdb-server got killed (or dying) before it tried to respond?

Where did you capture the JLink traces? From Cortex-Debug or from the log file? Can you check the gdb-server section in the 'Terminal' tab to see if you see the same thing? It is possible that you are not seeing the tail end of the JLink output.

I don't have a Windows machine at my disposal right now.

[AndriiLishchynskyi]

Hello Haneef,

Thanks for looking into this issue.

Yup, we were hoping the respective gdb-servers will be fixed. From where we stand, we are not sure what we can do. Ideas?

I dont have much understanding how plugin works, the only thing that comes to mind its GDB-server specific implementation of detach/stop events, but not sure if that is efficient approach.

But it is JLink that is behaving differently. We did not do anything different between the two OSes. It appears that JLink gdb-server got killed (or dying) before it tried to respond?

I am not sure what is happening but here 2 more observations:

• I increased GDB shutdown timeout from 50 ms to 5 sec (inside of disconnectRequest) and that didn't help. Code execution still do not reach this line:

  cortex-debug/src/backend/mi2/mi2.ts

  Line 264 in e50947c

  this.stop();

• after removing 'await' keyword in await this.sendCommand('target-detach'); detach seems to work the way it should. Also, it is worth to say that after this hack 'Stop' also works differently - it resumes CPU on exit. I guess this case should be considered if we can ever fix it.

I have got JLink logs by passing '-log' command using serverArgs parameter, GDB logs - by setting "showDevDebugOutput": true

Where did you capture the JLink traces? From Cortex-Debug or from the log file? Can you check the gdb-server section in the 'Terminal' tab to see if you see the same thing? It is possible that you are not seeing the tail end of the JLink output.

I also found that weird. Unfortunately, I don't have access to the Linux machine right now, since I asked my colleague with Linux machine to collect those logs. Will try to check with him if that really matters.

[haneefdm]

Yes, the await is not necessary but also it should not matter if the gdb-server behaves properly.

Also, it is worth to say that after this hack 'Stop' also works differently

Not sure what you mean. You mean the 'Stop' button (red outlined square) after a launch? But that is unrelated ie, has nothing to do with detach. It has its own problems but it should work the same way as before.

Btw, the output of JLink or OpenOCD can be found in the Terminal tab of VSCode. Also in Output Tab (Adapter Output). This is being re-worked

[haneefdm]

Can you attach your launch.json. The one I have may not be the same as what you are using.

[haneefdm]

Can you try the following pre-release?

https://github.com/Marus/cortex-debug/releases

[AndriiLishchynskyi]

I have collected more info so we a full picture of Stop/Detach behavior. I have performed all possible debugging scenarios (with JLink and OpenOCD servers) with Cortex-Debug versions v0.4.5 vs v0.4.4: Possible outcome:

• case#1- Pressing the button does not affect target state (halted/running)
• case#2 - Target is being running after pressing the button
• case#3 - Target is being halted after pressing the button

1. Cortex-Debug version: v0.4.5-pre1

• OpenOCD Launch + Stop: case#1
• OpenOCD Launch + Disconnect: case#1
• OpenOCD Attach + Stop: case#2
• OpenOCD Attach + Disconnect: case#2
• JLink Launch + Stop: case#2
• JLink Launch + Disconnect: case#2
• JLink Attach + Stop: case#2
• JLink Attach + Disconnect: case#2

2. Cortex-Debug version v0.4.4:

• OpenOCD Launch + Stop: case#1
• OpenOCD Launch + Disconnect: case#1
• OpenOCD Attach + Stop: case#2
• OpenOCD Attach + Disconnect: case#2
• JLink Launch + Stop: case#1
• JLink Launch + Disconnect: case#1
• JLink Attach + Stop: case#3
• JLink Attach + Disconnect: case#3

Conclusion
As per my understanding, the ideal use cases are:

• Launch/Attach + Stop - case#1
• Launch/Attach + Disconnect - case#2

What we have in v0.4.5-pre1

• Disconnecting from JLink 'Attach' session now works properly
• Stopping JLink 'Launch' now behaves the same way JLink 'Attach'
• Buttons 'Stop' and 'Disconnect' behaves the same way in scope of single request type (Launch/Attach)

[haneefdm]

There are only two scenarios supported

1. Launch will always end with a Stop
2. Attach will always end with a Detach unless you do a reset/restart in which case it becomes Stop

No other scenarios are supported

Other debuggers (Python, JavaScript, Ruby, Java, etc.) who do not have to rely on gdb-servers may do more combinations but we cannot. We are limited by what gdb and the gdb-servers do as a combination and we simply accept whatever that is. JLink does behave differently than OpenOCD and that is the way it is.

Our original issue had something to do with OSes behaving differently. Is that better now?

[haneefdm]

There is something we could try to support the other two cases if you are willing to test with both JLink and OpenOCD. It has a low chance of working but we can try. I can test it with MacOS, I don't have a good test case though.

[haneefdm]

@AndriiLishchynskyi Can you try out this release?

https://github.com/Marus/cortex-debug/releases/tag/V0.4.5-pre2a

With IOpenOCD (Mac only)

• Launch + {Running | Stopped} + {Stop | Disconnect}
• Atttach + {Running | Stopped} + {Stop | Disconnect}

Works as expected. What I added was when you invoke a Stop or Disconnect, was the program in running or stopped state. So, that is quite a few combinations. In my testing, JLink always continues to run regardless of the combination above

Note that my version of OpenOCD may be slightly different from yours. Unfortunately, this release has a few other un-related changes

@bohdan-tymkiv may want to try this release too for #482

Here is some more info on JLink (and OpenOCD). They are actually expecting to be killed as part of the protocol. JLink does exit when gdb disconnects since we are using 'single run mode' but always lets the program continue.

J-Link GDB Server is normally terminated by a close or Ctrl-C event. When the single run mode is active it will also close when an error occurred during start or after all connections to GDB Server are closed.
On termination GDB Server will close all connections and disconnect from the target device, letting it run.

https://wiki.segger.com/J-Link_GDB_Server

OpenOCD has a similar language but the only way to get make it quit is to kill it. We can't (yet) issue a 'monitor shutdown' but you can do that in your own config file

Btw, I did find JLink hang around after it disconnected one time, cannot reproduce this.

[haneefdm]

Ooops, found a serious bug with stack traces. Deleting pre2

[haneefdm]

False alarm. But I revved the version number anyway. Here is the latest

https://github.com/Marus/cortex-debug/releases/tag/V0.4.5-pre3

All my comments above are still valid.

[AndriiLishchynskyi]

@haneefdm Please see my results based on v0.4.5-pre3 below:
OpenOCD Launch + Stop: target is halted
OpenOCD Launch + Disconnect: target is resumed
OpenOCD Attach + Stop: target is halted
OpenOCD Attach + Disconnect: target is resumed

JLink Launch + Stop: target is resumed
JLink Launch + Disconnect: target is resumed
JLink Attach + Stop: target is resumed
JLink Attach + Disconnect: target is resumed

[haneefdm]

Thanks. I didn't even know that MS added the Stop/Disconnect options to both Launch and Attach. They added it to the protocol and the GUI to tell us what the user is requesting, but don't use different terminology. FYI:

https://microsoft.github.io/debug-adapter-protocol/specification

You can find this if you study their protocol data structures but I don't see it mentioned in DisconnectArguments above.

https://microsoft.github.io/debug-adapter-protocol/overview

Scroll to the Debug session end section