Gordon Oct 25
Cockpit data was not coming in.
Traced this down a system clock problem on flux. Its clock gradually lost time, and then the dsm_server process started throwing away samples, with this message in /var/log/isfs/isfs.log:
Oct 25 14:54:54 flux dsm_server[30058]: WARNING|sample with timetag in future by 2.305084 secs. time: 2012 Oct 25 20:54:56.861 id=10,20 total future samples=39039001
The problem was in /etc/chrony.conf on flux and gully. The chrony processes were configured to use keys, which were different, and so flux was not able to get time information from gully. Commented out
# keyfile /etc/chrony.keys
statement in both, and restarted chronyd
systemctl restart chronyd.service
Note that chronyd on gully is configured to get time from the WIFI stations 192.168.0.120-122.
chronyd on flux is configured to get time from gully.