...
This happens about once every week on 1 or 2 on 1 of the 22 systems, so it is hard to diagnose.
...
Previously I tried to change ntpd so that it might catch the error and not hang, but no success.
Now I might know why the tee_tty is being sent the SIGINT signal. I turns out the serial port is opened in "cooked" mode, which means that if it somehow receives a ctrl-C on that port then the process is sent a SIGINT. I guess a ctrl-C could also be received on any pseudo-terminal that is opened for reading. A BREAK condition can also result in a SIGINT, but IGNBRK is set on the serial port, and I don't believe BREAKs can be sent over pseudo-terminals.
Today around 3 pm I logged into all the stations and updated the file /etc/gps.conf to change the "c" in GPS_TEE_OPTS to "r", so that the serial port is opened in "raw" mode:
...
Next I'm rebuilding tee_tty so that if the real serial port is opened in raw mode that , the pseudo-terminals are also opened in that wayraw mode. I don't think the ctrl-C could be coming from the dsm process. rserial traps the ctrl-C and terminates.
As of 4:33 pm, the tee_tty app has been updated on all DSMs, and their tee_tty and ntpd processes restarted.