Restarting the server on the fly ignores set levels and mixes everybody at 100%

[dingodoppelt]

Describe the bug
Restarting the server sets levels to 100%

To Reproduce
restart a running server with clients on it and notice how it ignores the set gain levels and mixes everybody at 100%

Expected behavior
the server should mix everybody at the individually set levels

Operating system
linux, windows, macos

Version of Jamulus
3.6.2 and before

[gilgongo]

3.6.2 and before

Just a to clarify - does that imply you've reproduced this in earlier versions, or just by inference?

[dingodoppelt]

@gilgongo: not everybody was on 3.6.2 when we noticed (around 17 clients on the server). and i didn't ask for every version because i think the code concerning the levels hasn't been touched in the last few releases (correct me, if i'm wrong)

[hoffie]

I don't think the server has any way to store this information (currently). So the two options to improve this would be:

• Store this state server-side and reload it upon reconnect. This may have the disadvantage that we could only match on IP and/or name. At least the name may well be not that unique ("No Name")
• Detect the server reconnection on the client side and make the client resend the mixing wishes to the server

[gilgongo]

Also, when you say "restarting" do you mean a kill -s SIGUSR1/2 as part of recording, or a kill -HUP (or stop/start)?

[dingodoppelt]

@gilgongo
by "restarting" i mean restarting the systemd service: "systemctl restart jamulus.service"
@hoffie
i'd go for the client side implementation since it can already send levels but doesn't seem to do it on a restart. would we need another protocol message signaling a reconnect? i think "restarting" means quitting and starting before the clients time out so a "restart" scenario isn't handled. could it be the client doesn't notice it is being reconnected (because it doesn't get disconnected) and therefore doesn't update the fader gains? if the server made a request to the client to send the gains on connect it should work even on a restart.

[pljones]

UDP is a connectionless protocol. Stopping the server cannot be directly detected by the client. One of two things needs to happen: either the server says "I'm disconnecting you" or the client says "It's been rather a long time since I heard from the server". The former only happens with --discononquit. The latter is a generous amount of time.

When the server restarts, the server has no way - currently - of knowing what clients were connected. It just waits for them to start talking to it again. Hence the generous amount of time they give it to recover.

What it sounds like we need here then is:

1. A protocol message from the server to the client to say "Hey, I've not seen you before, even if you think I have - send state!"
2. The client to understand that message and do so.

Now, there may be an existing message the server sends to the client when it first hears from it - can't remember... but I don't think the client's prepared to suddenly be told "You need to send state" when it thinks everything's okay.

[gene96817]

Regarding "a generous amount of time".... the server is always sending the audio mix... this can serve as a keep-alive... the server is up unless there is no audio mix output

I am assuming the audio mix output exists as long as there is at least one user connected...
If it is possible for all users to be quiet and there is no mix output, then the server has to a periodic quiet channel keep-alive message, saying the channel is quiet and all is well,

I agree that the server needs a "send state" request if it receives a message from an unexpected client.

[pljones]

Regarding "a generous amount of time".... the server is always sending the audio mix... this can serve as a keep-alive... the server is up unless there is no audio mix output

I'm pretty sure that's what's used -- audio is the only thing that's meant to run constantly. If it stops, you know the server's "not there"... but it could be one packet, two packets... twenty packets... buffer now seriously under running... Cloud servers sometimes do get swapped out by long running high priority tasks on the same metal - you'll see that "ping" suddenly go from 30ms to 500ms (or the maximum reported)... but the client hangs on in there, waiting for things to settle. And often after a few tens of seconds, they do. The server hasn't noticed anything (maybe a few dropped packets on its side, too, of course).

The problem only comes when the process terminates - i.e. the server stops existing. That doesn't normally happen. It's either a deliberate act (i.e. "stop the server", perhaps as part of "restart the server") or it's due to a crash. Some crash-type failures may result in an automatic restart.

But yes, simple solution: server tells client "No, no, believe me: I don't know you (any more)" and so client sends state.

[hoffie]

I'm wondering if we could (ab)use some existing side effect in the protocol to detect a server restart, e.g.:

• Using the Welcome message (a chat message with WELCOME_MESSAGE_PREFIX); disadvantage: would only work on servers with a welcome message
• Attempting to detect some packet counter wrap; not sure if reliable enough, haven't checked

[gene96817]

We should select a max time with no server transmissions to declare the server down and restart. It is almost arbitrary.

@hoffie When the server restarts, there will be no destination to send a restart message unless the server has some persistent data of connected clients just prior to the restart. What information persist across a restart?

[hoffie]

When the server restarts, there will be no destination to send a restart message unless the server has some persistent data of connected clients just prior to the restart.

Yes, that's true at the time of the restart (and it will be true for all other proposed detection mechanisms). The reason why it could work is that clients will "reconnect" quickly (technically: they will probably not notice that they had not been connected and will continue to behave as if nothing happened).

We would explicitly be making use of the fact that the server noticed knows that it was restarted. We would only have to make clients recognize this an act like it was a fresh connection.

Have you tried it? In a quick test, restarting a server will cause a new welcome message to be sent to existing clients.

[pljones]

I think there are some things the client "knows" to tell the server on a fresh connect and some things the server knows to tell the client, currently. What we have here is the server "knowing" it's a fresh connect -- but the client not having realised.

So we need a way to make it clear to the client "something happened you weren't expecting" and it then needs to act appropriately.

[hoffie]

Looks like there are already several opportunities in the server code which signal the client that it was reconnected (= is considered to be a new client by the server):

jamulus/src/server.cpp

Line 593 in 7d0801a

// inform the client about its own ID at the server (note that this

[pljones]

Interestingly, some of the messages explicitly allows for getting into an unexpected state
https://github.com/jamulussoftware/jamulus/blob/7d0801a1578afc41803ff30a33327ed5e38d8269/src/server.cpp#L605-L610,619-626

So the problem is reduced -- the server knows to try to poke at the client for the information it needs, it's just the client side. What does it decide to do on a ClientIDMes arriving? Seems to be this...

jamulus/src/client.cpp

Line 710 in 7d0801a

void CClient::OnClientIDReceived ( int iChanID )

which just triggers the UI to do something (probably not resend current values).

Potentially CClient::OnClientIDReceived could do the work but I'm wary as the style seems not to follow that idiom...