These are PC104 DSMs (I can't remember which are Titans and which Vipers) which have not been online even though their ubiquiti radios are online.  I visited each to see what I could figure out.

rsw02

My first realization was that I didn't have a console cable for these DSMs, so I didn't try to log into this one.  I did notice that the ethernet ribbon cable was disconnected from the interface panel, but I wasn't sure of the orientation of the plug.  I rebooted it anyway and left to check on the plugs in the other DSM.  At rsw05 I saw that the ribbon cable gets plugged in with ribbon exiting away from the RJ45 jack:

So that's how I plugged it back in at rsw02 on the way back.

Nagios on ustar reported that rsw02 then started responding to pings, for another few hours, until I tried to run an ansible playbook on it from ustar, and now it has not responded since.  I should have tried to log into it first to see whether anything else was working, but I didn't.

rsw05

Rebooted, just to start from scratch, but maybe not necessary.  The ethernet ribbon cable is plugged in and I see lights, so I can't see anything obviously wrong.

So I disconnected the ubiquiti at the bulgin port and connected my laptop to the internal RJ45 jack.  Ssh works, I can log into the DSM.  The IP address is correct: 192.168.1.173, GPS is locked and chrony in sync, usb is at 50%.

No data from the METEK sonic.

Next I disconnected the ethernet ribbon cable, plugged ubnt radio back in, with my laptop still plugged into the internal RJ45 jack.  I cannot ping or browse to rsw05u, but I know it's up because it has been pingable for the last week.

So should it be possible to connect to the ubiquiti through the internal RJ45 jack?  If so, does that mean there is a problem with the ethernet connections on the interface panel?

Next steps

I'm considering what to try next:

Log into the DSM with the console cable and investigate from there.  Ping the radio, look for log messages about errors in the ethernet device.

I have one of the RJ45 jacks which plug directly into the CPU card for either the Titan or Viper.  I could at least isolate the ribbon cable with a patch cable between the CPU jack and the RJ45 jack on the interface panel, but I don't think that works for both Titans and Vipers.

Work on a scheme to isolate the interface panel ethernet port.  My best guesses:

• Use a cable with a bulgin female plug and RJ45 plug to convert the Ubiquiti cable to RJ45 end, then plug that into a PoE injector which gets power from somewhere, either by tapping the power panel or the external bulgin ethernet port.  If the internal RJ45 works to connect to the CPU, then that can be used to patch the CPU into a switch, also powered off the power panel.
• Replace the bulgin cable entirely and run an ethernet cable up the tower same as the Pi towers.  Use the same PoE injector, but it might need a different power plug.  Connect that to the internal RJ45 jack same as above.
• If there is a problem between the internal RJ45 and CPU card, then I don't know what you do.  Can we attach the ribbon cable to a new RJ45 jack?  Use a USB ethernet interface instead of the onboard interface?
• Just replace the entire interface panel.  If we don't have spares here, take one from VERTEX.  Or replace the entire DSM with one from VERTEX, but keep the CPU stack.