The DSM had been rebooting intermittently, so I visited the site around 4:30 pm 8/19 MT.

While there wondering if there was a way to unlock and open the battery box, I heard the TRH fan drop off and then start up again immediately. So the power outage was brief. I also noticed the ttstation ubiquiti wifi was up, which only happens for the first 15 minutes after boot. I connected to the ubiquiti and confirmed uptime of 10 minutes.

Wiggling the plug in the gfi outlet in the weatherproof box did nothing.

I verified poe passthrough is off on the radio, so I am able to connect to the DSM through the radio lan2 port. Just to remove the switch as a possible failure point, I disconnected the power from the switch and connected the ubiquiti directly to the Pi.

I had to restart the json data service after the system date updates, and then dashboard looked good.

systemctl --user restart json_data_stats

The power outages could be caused by a problem in the victron, so I looked for messages from the victron in the raw data which might indicate an error

aq@tt:~ $ lsu
USB1:
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        58G  1.8G   53G   4% /media/usbdisk
-r--r--r-- 1 daq eol  117670 Aug 19 19:25 tt_20100201_000129.dat
-r--r--r-- 1 daq eol  189288 Aug 19 19:40 tt_20100201_000130.dat
-r--r--r-- 1 daq eol  124325 Aug 19 19:52 tt_20100201_000131.dat
-r--r--r-- 1 daq eol  119383 Aug 19 20:04 tt_20100201_000132.dat
-r--r--r-- 1 daq eol  118837 Feb  1  2010 tt_20100201_000133.dat
-r--r--r-- 1 daq eol  121911 Feb  1  2010 tt_20100201_000134.dat
-r--r--r-- 1 daq eol  119202 Aug 19 22:28 tt_20100201_000135.dat
-r--r--r-- 1 daq eol  116753 Aug 19 22:40 tt_20100201_000136.dat
-r--r--r-- 1 daq eol   49322 Jun 24 23:01 tt_20100201_000851.dat
-r--r--r-- 1 daq eol 8806338 Aug 19 23:04 tt_20210819_224007.dat

In chrony/tracking.log, the reboot after 22:38 utc (16:38 mt) took 1 minute 12 seconds to sync to PPS. I think that means the ublox is keeping leap seconds, and I'd bet pps starts sooner than that, so maybe we need to change some chrony settings to sync to it faster. Or maybe that's how long it takes chrony to start up, in which case a battery-backed system clock really makes sense.

2021-08-19 22:38:32 PPS              1      6.860      0.009  1.660e-07 N  1  5.438e-08 -2.242e-08  0.000e+00  9.651e-06  2.594e-05
===================================================================================================================================
   Date (UTC) Time     IP Address   St   Freq ppm   Skew ppm     Offset L Co  Offset sd Rem. corr. Root delay Root disp. Max. error
===================================================================================================================================
2010-02-01 00:00:08 0.0.0.0          0      7.225      0.008  0.000e+00 ?  0  0.000e+00 -2.891e-16  1.000e+00  1.000e+00  1.500e+00
2010-02-01 00:01:12 PPS              1      7.225      0.008 -3.644e+08 N  1  1.209e-06 -2.989e-10  0.000e+00  4.525e+02  1.500e+00
2021-08-19 22:40:23 PPS              1      7.225      0.010 -9.001e-07 N  1  1.396e-06 -7.381e-11  0.000e+00  9.621e-06  3.644e+08
2021-08-19 22:40:39 PPS              1      7.224      0.017 -9.389e-07 N  1  1.322e-06  5.131e-07  0.000e+00  1.112e-05  2.937e-05
2021-08-19 22:40:55 PPS              1      7.223      0.030  5.480e-07 N  1  1.656e-06  3.884e-07  0.000e+00  1.168e-05  3.100e-05

There are no non-zero ERR reports in the latest data file:

data_dump -i 1,60 /media/usbdisk/projects/LOTOS2021/raw_data/tt_20210819_224007.dat |& egrep ERR | egrep -v 'ERR\\t0\\r' | less

I looked for a way to get an uptime from the victron, to see if the victron itself was power cycling, but I could not find that.

Looking at the wiki page for the victron (https://wiki.ucar.edu/pages/viewpage.action?pageId=398004758), the victron lowers the charging power by 50% every 10 minutes. Could that correlate with the outages?

daq@tt:~ $ data_dump -i -1,60 /media/usbdisk/projects/LOTOS2021/raw_data/tt_20210819_224007.dat | grep PPV | cut -c 50-70 | sed -e 's/\\t/ /g' -e 's/\\r\\n//' -e 's/PPV//' | sort -n | uniq 
2021-08-19,23:30:28|INFO|opening: /media/usbdisk/projects/LOTOS2021/raw_data/tt_20210819_224007.dat
2021-08-19,23:30:28|NOTICE|parsing: /home/daq//isfs/projects/LOTOS2021/ISFS/config/lotos2021.xml
Exception: EOFException: /media/usbdisk/projects/LOTOS2021/raw_data/tt_20210819_224007.dat: open: EOF
  21
 21
  22
  23
  24
 24
  25
 25
  26
 26
  27
 27
  28

The PPV is always from 21-28 V, so no sign that it is dropping off. However, if it dropped off suddenly, then that would kill everything before it could be logged.

I installed victronconnect app on my phone, got the bluetooth app and connection working, but I couldn't see any settings which could be causing output to cut. There is a setting to cut output according to battery temperature, but that is disabled. The output mode is "always on".

  • No labels