Posts tagged with “servers”

Spring Cleaning!

Spring has sprung here in Maine, so it's time to clean some computer equipment.

Today, we're going to start with Jennay, my HP DL380 G9 which acts as the primary server on my network for both production and lab uses. After shutting down the lab VMs and migrating the production VMs to a temporary host, the cleaning could commence! This box had 120 days uptime prior to being taken down, which is pretty good for me.

First, we have to clean the top of the server itself, which tends to take the role of a sort of shelf in my rack. Mostly a junk shelf. It also holds our cable modem, an external hard drive, two Pis with GPS clock duties, and the UPS for the network gear. The cables on this stuff are all long enough that they can be moved aside without being powered down.

Messy DL380G9

And now it's clean!

Clean DL380G9 Top

Don't forget to install your UPS-supporting cat food container.

UPS supporting cat food container

Then we can take it outside and get a closer look at what we have. Lots of buildup on these side vents by the power supplies.

Dusty Vents

Using the shopvac with the hose connected on the back to blow (it was moved after this photo) in conjunction with a firm paintbrush to get more stubborn dust out of there.

DL380G9 Outside 1

It's really not that bad inside, but there are a few corners where there's bad accumulation, and a fine surface layer over everything that will just get worse over time.

DL380 G9 Inside Pre

Again, it's hard to tell, but it's MUCH cleaner now. Especially everything around the fans, they were caked with very fine dust.

DL380 G9 Clean

The second goal with taking this server down was to add a small fan to the onboard HBA controller. It is regularly the hottest point in the system, usually between 52ºC and 55ºC. The larger part of the heatsink is almost perfectly 40mm on each side, and there is a 12v powerpoint nearby on the PCIe riser.

RAID Controller Heatsink

It's a good fit, and there were metal tabs on the ends of the wires that allowed them to snugly fit into the large 12v connector.

RAID Controller Heatsink Fan

I also used a very small amount of hot glue on the front and rear edges to tack it into place.

RAID Controller Fan Glued

Works great!

RAID Controller Fan Working

This modification brought the temperature of the onboard RAID chip down to 42ºC - it's still one of the warmest components, but a 10º drop is significant, and should lower the thermal stress on it significantly. The next time I take the server down I may remove the module and replace the thermal interface material between the controller and heatsink.

The important parts back in place, all the other junk is in a box on the floor. Here's to another 120 days!

DL380 G9 Returned

TimeHat ❤ Nokia FYGM

FYGM and TimeHat

For the past few monts GNSS-disciplined time servers have been taking up a large percentage of my hobby time. Building and distribuiting the TimeHat went very well, and it gave me a bit of extra capital to dig into a few things that I might not have been able to otherwise, like picking up some "real" GNSS timing modules! The unit I was most interested in was the Nokia FYGM, a GNSS timing receiver made to work with Nokia cell site equipment. The FYGM has a u-blox LEA-M8T, one of the better timing-focused u-blox modules available. Lots of people buy these (and other) timing modules from decomissioned cell sites through Chinese scrappers and recyclers just to harvest the receiver, but I wanted to keep them as "together" as possible.

Unlike typical hobbyist GNSS modules, the FYGM has a some extra hardware on it, namely an 8051 MCU. I have more information on this on the wiki, here, so I'll keep this short. To start, why not simply emancipate the M8T from this board? I have a few reasons:

  • I don't want to design a new carrier for the M8T
  • The original board has high quality power supplies for the M8T and active antenna
  • It also provides excellent antenna protection circuitry
  • The included aluminium case is really nice
  • The status LEDs (driven by the MCU) are useful
  • Once the board boots, it's not hard to get the MCU out of our way

That MCU, the 8051, started as being the biggest thorn in my side. Before I had one of these units in my hands all I had to go off of was ebay listing pictures and a couple of posts on the TimeNuts mailing list, but that was enough to get started. In addition to antenna input, the FYGM has one other connector - a 12 pin DIN amphenol-looking thing. Thankfully all of the FYGMs I have found for sale inclide a completed cable to mate with this (using an HDMI connector on the host device end, with CAT7 as the cable itself), otherwise this connector could be a real pain to deal with, becuase it's not an amphenol, or anything that seems to be generally available. Only 8 of the pins on this connector are used, Ground, DC in (taking anywhere from 12 to 35v), and then three RS422 differential pairs for Tx, Rx, and PPS. Two RS422 transceivers on the bottom of the module provide this signaling. My initial intention was to try and use these as-is, so I got some hardware to work with RS422 - more on that later.

Once I got an FYGM in my hands, I started tracing all of the connectors and headers. I was disappointed to learn that the RS422 transeivers' input does not come directly from the M8T, but rather it from the 8051. This meant that it probably wouldn't be spitting out UART signals directly from M8T, something that turned out to be true. On the bright side, the four pin connector adjacent to the M8T is, in fact, UART directly from the M8T, meaning that bypassing the 8051 seemed like a possibility, and it technically still is. The ten pin connector next ot the 8051 is a programming and debugging header for that device, so perhaps one day I can read its configuration out or even reprogram it to make it more useful to me. The more immediate use that header has is the exposed reset pin - dropping a jumper between it and ground causes the 8051 to not boot at all, so through the UART pins next to the M8T I can interface with it directly, and completely ignore the MCU. I didn't end up doing this though.

Why?

Well, to put it plainly, the engineers who developed this thing had some clue as to what they were doing - remember, this device is intended to be a highly precise GNSS time source for a clock in a cell site. If we can benefit from the work they put into developing this thing, why not? When the FYGM boots, the 8051 sends a number of strings to the M8T to put it into a configuration that makes it optimal for use in its intended purpose - which just happens to be the purpose I have for it! As far as I have been able to tell, once these strings are sent, the 8051 gets out of the way entirely, it simply takes UART from the M8T, process it, and spits it out over R422. Initially this was a disappointment, but... I don't need to use the RS422 - I can run the signals straight from that UART header to the host - and that's exactly what I ended up doing.

Early on I had started to design a Pi Hat with an HDMI port, a DC in jack for 12-35v, a 5v power supply for the Pi, and RS422 transeivers for the UART and PPS signals. As much as I like the aesthetic of this, it's just not reasonably feasible. Seeing as the UART from the M8T isn't connected directly to the RS422 transeiver on the FYGM, I would have to connect them directly with some bodge wires, but this means that the 8051 has to be taken out of the loop, and for the reasons described above, I don't want to do that anymore. What I ended up doing is pulling the pins out of the eight pin JST connector, everything but ground and DC in, and I am connecting to them directly to the UART signals out of the M8T. This also lets me do some sneaky stuff. It's kind of ugly, but it works well and allows me to continue to use the DIN connector which is solastic'd in place.

FYGM Open

Again, this is ugly - but it works. If you look closely you can see the sneakyness... As you can probably tell from the picture at the top of this post, what I have done is cut the included cable down to about a foot and run it straight onto one of the TimeHat boards, feeding in Tx, Rx, and PPS just as the MAX-M8Q modules do. Power comes in through that barrel jack zip tied to the cable. If you look closetly, though, you'll see something going to the 5v pins of the TimeHat, what's up with that? Well, I spent a while think about how best to handle powering this whole mess, and I really liked the idea of integrating the Pi and FYGM as closely as possible. The UART header on the FYGM has a pin for 3v3 input, so I could power the M8T directly. The 8051 and active antenna circuits have their power power regulators though, these are fed from 6.3v coming from the main power supply on the board.

The Pi has no easy way to source 6.3v, but you know what? Turns out it will run just fine on 6.3v! I haven't spent much time at all studying the power front end on the Pi, so I just threw 6.3v on the 5v rail and sure enough, it works just fine! This may very well impact something in a negative light, but I'm just going to let it roll for now - I'll update if it causes any problems, but I don't suspect that it will. Basically, what I'm doing is feeing 16v into the FYGM's main power supply, then pulling the 6.3v back out and into the 5v rail on the Pi along with the other signals mentioned earlier. How much power is safe to sync from this 6.3v rail? Probably not a lot - but - keep in mind that I have the Pis that I use in this role configured for very low power consumption - no WiFi, no BT, no GPU, no USB devices, and the CPU is clocked down about as low as it can go. Using a kill-a-watt, I measured the whole setup to pull a whopping 1.9 watts from the wall, including all the power supply overhead.

What does this mean as far as time server performance is concerned? Well, I don't know yet. I haven't done much of that measuring yet. I just recently got my GNSS antenna splitter, so I can finally start to compare receivers using the same signal source, which is a step in the right direction. What really needs to happen next, before I can start seriously start making this measurements, is getting a GNSS antenna mounted up above the roof line. I have my mast built and brackets in place, all that's left is to actually attach it to the house - I'm still working on figuring that out, more to come shortly, I hope.

That said, even in my "less than optimal" configuration, things are looking good. The Pi here is a Pi2, and that is on purpose. Can pairing a less-than-optimal Pi with a really good GNSS receiver help? Who knows! Here's what the chronyc sources output looks like at the moment:

FYGM chronyc sources

And here's what cgps looks like:

FYGM cgps

The 0.35 TDOP is what really stands out here. This receiver's antenna is in the basement, sitting on the ledge of a casement window. Using a high-mounted helical antenna sticking out of a 2nd story window I have seen TDOP below 0.2, so I expect this to improve well beyond the capabilities of the Pi once I get it mounted on the side of the house.

mSATA RAID?

Even though it's in packed away in the storage unit right now, I have been thinking a lot about the C220M3. Specifically about its own storage. Right now the only aspect of the machine that I have not seen to in one way or another is drives, specifically drive caddies - which, frankly, is not an uncommon issue with servers. Some machines, like the G5-7 HPs, and a few generations of Dells, share caddies (within their respective brands, granted). Because of this, their caddies are incredibly cheap. For this cisco, on the other hand, the caddies are relatively more expensive, often around $15 each. It's often more cost effective to just go ahead and buy caddies with drives in them already, provided you're okay with it being spinning media. The one drive I have for the machine now, the drive it came with, is an A03-D600GA2 - a Western Digital 600GB 10kRPM SAS hard drive. Aside from being a spinner, and having all of the downsides of spinners, there's nothing wrong with it.

Additional A03-D600GA2s, with caddy, are often about $50-60 shipped on ebay - which isn't bad, but I don't want them, for a few reasons typical of hard drives. I don't want to deal with the additional power consumption and the heat generation that goes with that. I also don't need the relatively cheaper dollar-per-gigabyte that spinners offer, since any large amount of data I need to to store can live on the NAS. Since SATA drives can be plugged into SAS backplanes, what I done prior to taking the C220M3 offline was have SSDs simply hanging out in the bays, two 480 and two 240 - the former in a RAID1, the latter in a RAID0. These were in bays 5, 6, 7, 8 whose SAS cable was connected to the Adaptec 2405 PCIe card I had added. This was necessary because ESXi has no support for the onboard intel soft raid - I had more posts about that in the past so I won't really go into it here.

Anywhoo, I was talking with some pals earlier tonight about storage and we started looking specifically for quad m.2 SATA PCIe cards. I know there are practical reasons for this not to be a thing, but they kind of are. Quad NVMe m.2 cards are actually fairly common. But that's not what this is about. This is about mSATA, which predated m.2 and was briefly popular. A card I found that I am particularly interested in is the Addonics AD4MSPX2-A - a $55 PCIe card with four mSATA slots and a Marvell RAID controller. It supports 1, 0, and 1+0 (which is what I would be using) and based on some cursory research on mSATA SSD prices, it seems rather competitive for what I am interested in - which is reliable SSDs that are "fast enough" - I don't need blazing.

256GB mSATA SSDs, name brand models from Samsung and the like, are about $50-60 on the second hand market. Four drives plus the card comes out at $260 or so, which is not bad at all for a bootable 512GB RAID1+0. Unfortunately 512GB mSATA SSDs are not very cost effective, still priced at around $100-115, putting a 1TB RAID1+0 closer to the price point of standard 2.5" SATA SSDs. Certainly one could take this in the other direction as well - 128GB mSATA drives can be found for as little as $20 shipped, meaning you could get a 512GB RAID0 (cough) for a mere $130 or so. Would I trust it? No, probably not, but I would sure as hell try it.

So why is this so interesting to me? Well, one of the PCIe slots in the C220M3 is already occupied by the the Adaptec 2405, so swapping that out with the Addonics card wouldn't cost me anything more. If I use the eight SAS bays on the front as individual drives in ESXi I can also pass them straight through to VMs, meaning I can use softraid in windows/linux, and at the very least have some dedicated IOPS for those VMs. Or I can just not use those bays at all, and only use the Addonics card. Now, I don't know yet if ESXi supports this Marvell chipset - I don't have high hopes, but I'm certain that windows and linux do, so there's always Hyper-V and Proxmox. (Hyper-V and Proxmox also support the Intel SoftRAID, so, you know... but... nevermind that).

Either way, look forward to a post with more information on this. I am incredibly interested in what a card like this can do, especially as mSATA prices continue to fall. More likely than not I would start with four 32 or 64GB for testing, then once I am satisfied with the performance, get another Addonics card and four 256GB drives. This gets me a 128/256GB fast scratch volume and a nice big protected working volume. It may be a while, but this will be interesting, I can promise that!

UPDATE 1 - Found a cheaper card here: Syba SI-PEX40109 - Currently $55 on prime. I also like the form factor a bit more than the Addonics model - it has an SFF adapter, and is a bit shorter because it has two mSATA on the front and two on the back. Not sure how the chipsets compare...

SpeedStep® Limiting Single-Threaded Tasks

This is something that will need to be further fleshed out by benchmarks and research, but I wanted to start writing about it now, before I forget. More is certainly to come.

Context

First, a little backstory for context. The company I work for, Esri, just released version 10.6.1 of their ArcGIS Enterprise product. Something I found very appealing about this release is that it includes a new interface and set of tools for processing imagery from UAVs into orthophotos, DSMs, and DEMs, called Ortho Maker. Esri has a desktop product that does this as well, Drone2Map, which is based on the Pix4D Mapper engine. Ortho Maker is unique in comparison because it is intended to run on server infrastructure rather than workstations or HEDTs, and anyone who knows me will know that I find that incredibly appealing.

So, when 10.6.1 was formally released, I wiped the TS140 'goonie' to a clean slate and configured it as follows: 2x8GB RAM (I moved the other 2 to TS140#1), 256GB Samsung 840 Pro boot/primary data drive, 2TB WD Green as a backup/scratch drive - this simply because it was laying around and I figured I could use some extra storage that didn't need performance or redundancy. Finally, a fresh copy of Server 2016 - which took forever to update, then a base enterprise deployment of ArcGIS Enterprise 10.6.1. Now, this also required an Image Server license because raster processing tools are used in order to generate the final products. In a production environment the server with the image server license would not be the same as the server running the portal and hosting server (you can read more about ArcGIS Enterprise architecture here), but since I am the only individual that will be accessing this server, and I know that nothing will be putting load on the hosting server or portal while the image server components are working.

I have a few different sample UAV imagery collections that I use to test this software, both my own and others that are available publicly on the internet, so I began with processing a few to get a better feel for the capabilities of the software. As a solution engineer, part of my job is to thoroughly test all the components of our software that I could find myself recommending to a customer, and this is no exception.

Where SpeedStep comes in...

After watching Process Explorer and the Windows Task Manager while the various stages of the ortho maker's processes were run, it became clear that these are single threaded tools. I imagine this will change in future releases, and there may be an option to make it multi threaded in the future, but as it stands each of the tasks only runs one thread at a time - each process never taking more than 25% of the total CPU time (the E3-1225v3 is 4c/4t).

Windows Task Manager shows the current clock speed of the CPU, which is very useful in cases like these, and what I was seeing was clock speeds much lower than expected. The E3-1225v3's base is 3.2GHz, with boost up to 3.6GHz, however Task Manager was showing that the system was rarely, if ever, hitting higher than 1.2GHz. Now, we all know what SpeedStep is - it's that fabulous technology that changes CPU speed on the fly, keeping power consumption and temperatures down. The dark underbelly to SpeedStep, apparently, is that it uses total CPU usage to gauge whether it needs to crank up the speed rather than per-core usage.

In this case, that's a problem. If the server is running two Ortho Maker tasks, and is otherwise idle, the CPU will only ever be at 50% + a small amount of system overhead, maybe 15%. What that means is that the system will never determine that it's necessary to step up, and the performance of those single threads will be far less than they could be. I suppose if I was concerned about temperatures and fan speeds then maybe that would apply, but this is a server we're talking about here.

How to resolve it

Thankfully this is an easy problem to solve. Disable SpeedStep. This can even be done from within windows, without rebooting. Launch powercfg.cpl and change the power plan from "Balanced" to "High Performance". If you want to see what this does, click "Change plan settings" then "Change advanced power settings" - under processor power management both the minimum processor state and maximum processor state will be set to 100% - meaning that in my case the CPU will always be at 3.2GHz, and will boost when possible.

SpeedStep can also (usually) be disabled from the BIOS, and this would be my recommendation. I'm currently working on this server remotely so that's not an option for me, but I will be addressing it soon!

What other people are saying

Not surprisingly, quite a few people have discovered this quirk, and Microsoft even has some KBs that address this when it comes to using Hyper-V. Basically what it comes down to is "know your workloads, and know your hardware". In many cases, leaving speedstep enabled will be best.

What I need to do next

More research, and some single threaded benchmarks of my own. I also want to test this theory across multiple machines - especially the C220M3 (currently in storage...).

Resources

ADS-B and PiAware

I have run some form of an RTL-SDR based ADS-B receiver every since I first learned it was possible back in college. The idea of using a cheap ($10-20) USB radio to pick up aircraft transponder signals, and then use some open source software to decode and visualize it was beyond cool. A few years ago flightware came out with a program called "PiAware" where if you used their software (a combination of raspbian and dump1090 plus some of their own customizations) you could feed your data into their system to help augment their data, and in the process you got a free premium account.

I never used this because 1) I wasn't interested in a premium account, and 2) I ran my own customized version of dump1090 and didn't want to lose out on my sweet added features. Additionally, even though I know that the software can be run on any linux system, I had always felt that the Raspberry Pi was under-powered and not stable enough to run like this for an extended period of time. About a week ago, I decided to revisit the matter, mostly because of the premium account.

I've been doing a lot of testing with my company's real time & big data spatial processing tool, GeoEvent Server and I was under the impression that the premium account would give me an API key to access their feed. Turns out I was wrong, that has to be paid for separately regardless, but I decided to follow through with setting up PiAware anyway. I have an Raspberry Pi B 3 floating around doing nothing in particular, and I figured it should be beefy enough to handle the task at hand.

(picture of setup here)

My setup is simple but works well, it has been running for just a few days now. My number one question, as it always is with deployed RPis, is how well the SD card will hold up. I'm using a 32GB SanDisk Class 1 card, but you never know. My antenna is the official FlightAware branded ADS-B antenna, a dark green plastic tube about 2ft long with an N-connector on the bottom. I have it mounted on top of a tripod standing up in the attic. 20ft of low-loss coax connect it to the receiver - first to a band pass filter that kicks out FM and all the other RF noise, then to the FlightAware Pro radio itself - this is a standard RTL-SDR with a tuner and amplifier somehow "tuned" to 1090MHz. There is a "Pro Plus" version of this receiver now that has the band pass filter integrated, and I would very much like to see how it compares. The last piece is the Pi itself, the only other thing connected to it is a CanaKit 2.5amp PSU - it's hanging off the network via WiFi, also a test.

Performance has been good, the software barely taxes the resources of the machine, and it hasn't had any issues holding up to the network. Their updated version of dump1090 has a number of features that I had added to my own, so I have no qualms with using it now. It also still has the always-updated JSON endpoint with aircraft information which can be readily ingested into GeoEvent Server - more on that later!

Despite being mounted in an attic, surrounded by trees, the coverage is quite good. I find that 7am-1pm tends to be the busiest time.