These are PC104 DSMs (I can't remember which are Titans and which Vipers) which have not been online even though their ubiquiti radios are online. I visited each to see what I could figure out.
rsw02
My first realization was that I didn't have a console cable for these DSMs, so I didn't try to log into this one. I did notice that the ethernet ribbon cable was disconnected from the interface panel, but I wasn't sure of the orientation of the plug. I rebooted it anyway and left to check on the plugs in the other DSM. At rsw05 I saw that the ribbon cable gets plugged in with ribbon exiting away from the RJ45 jack:
So that's how I plugged it back in at rsw02 on the way back.
Nagios on ustar reported that rsw02 then started responding to pings, for another few hours, until I tried to run an ansible playbook on it from ustar, and now it has not responded since. I should have tried to log into it first to see whether anything else was working, but I didn't.
rsw05
Rebooted, just to start from scratch, but maybe not necessary. The ethernet ribbon cable is plugged in and I see lights, so I can't see anything obviously wrong.
So I disconnected the ubiquiti at the bulgin port and connected my laptop to the internal RJ45 jack. Ssh works, I can log into the DSM. The IP address is correct: 192.168.1.173, GPS is locked and chrony in sync, usb is at 50%.
No data from the METEK sonic.
Next I disconnected the ethernet ribbon cable, plugged ubnt radio back in, with my laptop still plugged into the internal RJ45 jack. I cannot ping or browse to rsw05u, but I know it's up because it has been pingable for the last week.
So should it be possible to connect to the ubiquiti through the internal RJ45 jack? If so, does that mean there is a problem with the ethernet connections on the interface panel?
Next steps
I'm considering what to try next:
Log into the DSM with the console cable and investigate from there. Ping the radio, look for log messages about errors in the ethernet device.
I have one of the RJ45 jacks which plug directly into the CPU card for either the Titan or Viper. I could at least isolate the ribbon cable with a patch cable between the CPU jack and the RJ45 jack on the interface panel, but I don't think that works for both Titans and Vipers.
Work on a scheme to isolate the interface panel ethernet port. My best guesses:
- Use a cable with a bulgin female plug and RJ45 plug to convert the Ubiquiti cable to RJ45 end, then plug that into a PoE injector which gets power from somewhere, either by tapping the power panel or the external bulgin ethernet port. If the internal RJ45 works to connect to the CPU, then that can be used to patch the CPU into a switch, also powered off the power panel.
- Replace the bulgin cable entirely and run an ethernet cable up the tower same as the Pi towers. Use the same PoE injector, but it might need a different power plug. Connect that to the internal RJ45 jack same as above.
- If there is a problem between the internal RJ45 and CPU card, then I don't know what you do. Can we attach the ribbon cable to a new RJ45 jack? Use a USB ethernet interface instead of the onboard interface?
- Just replace the entire interface panel. If we don't have spares here, take one from VERTEX. Or replace the entire DSM with one from VERTEX, but keep the CPU stack.
8 Comments
Gordon Maclean
I may be misunderstanding how you had things connected, but it won't work to have the ubiquiti connected to the external bulgin port, the laptop connected to the internal RJ45 port, and the ribbon to the PC104 CPU, since they can't share the same TX/RX lines - you need an ethernet switch to connect the 3 systems.
I think the second guess (replacing bulgin cable) has the best chance of succeeding. If it still shows problems (maybe intermittently) connect the ethernet cable from the POE injector directly into the Viper/Titan ethernet adaptor, not to the interface panel RJ45 (but it sounds like you don't have enough for all the systems). The adaptors work on both Vipers and Titans, with the "bottom" of the little circuit board (the side opposite the RJ45) nearest to the edge of the CPU card. Some adaptors needed to be shaved a bit because it was a close fit against the PC104 connector on a Titan.
Out in the FLAB lot there were times that I had to add an ethernet switch inside the DSM, because it didn't seem to be able to talk directly over a long cable to NETS ethernet switch box.
Gary Granger AUTHOR
Ok, thanks for the info on the CPU card ethernet adaptor. Since I only have one of them at the moment, hopefully the internal port works for the CPU connection and I won't need the adapter. I knew it would not work to connect three devices to the interface panel, and that's why I'm wondering how to work around the fact that we can't connect both CPU and Ubiquiti to a switch when neither has a standard RJ45 plug. If it should have worked to connect to the Ubiquiti through the internal jack with the ribbon cable disconnected, and that didn't work, then definitively there is a fault in the bulgin connection or cable.
So I guess we should find some extra PoE injectors and a way to connect them to power.
Gordon Maclean
Yea I agree it should have worked with the ubiquiti through the bulgin with a laptop on the internal RJ45 and the CPU disconnected. On old systems there is the cross-over issue, but any new laptop should be able to auto-sense. I assume those bulgin cables were tested, but maybe not sufficiently. On some systems the power to the ethernet port is controlled with vio/tio? I'm assuming you're seeing power on the ubiquiti (would be good to know what the voltage is at the ubiquiti and what it requires, but the voltage may be hard to probe.
A short ethernet cable between the internal RJ45 and the PoE should work, with the ribbon to the CPU. Sometimes I've suspected that the surge circuitry on the interface panel messed with things, but did not come to a firm conclusion.
Gary Granger AUTHOR
Yes, the Ubiquiti's have power. They are connected to the wireless network and can be browsed from the wireless end, they do not seem to be having any trouble with power.
Steve Oncley
We could have Semmer bring out another front panel and/or stack from those used in VERTEX, if it would help. The boxes assembled for Perdigao were put together with odd parts. If this were done, we'd have to revisit the power jumpering that I hacked to get the POE stuff working on these boxes.
Alternatively, we could replace these boxes with Pis, though then we'd have to come up with another option (Wisard board or off-the-shelf USB A/D) for the Cornell analog barometers.
Gary Granger AUTHOR
I think Gordon's suggestion is the preferred option, skip the bulgin cable and connector entirely using a PoE injector and an ethernet cable up the tower. Steve, do we have extra PoE injectors here, or should Semmer bring them? Meanwhile, if we have time we'll look at how to wire them into the PC104 box. I asked Kurt about that this morning to get his help with it. Or Steve, maybe you and Semmer want to come up with a plan that he can execute when he gets here, so he can bring whatever he needs? As long as we need to try this for two DSMs, I think we should be prepared to do it for the other two also.
Steve Oncley
Shouldn't you be sleeping, Gary??
Steve and I will talk when he's here on Tues.
Hopefully, the (5) spare DSMs have POE injector cables that you could steal. I can't recall if there are other extras – of course, feel free to look through the tubs (should be getting easy, now that most are getting empty!) I wouldn't hesitate to cut off the power barrel connector to attach it to the old DSM power.
Kurt should be able to figure out how the output of the DC-DC is routed to the Titan/Viper front-panel, though it was somewhat nasty. The solutions are different for the different PC104 DSM boxes.
Gordon Maclean