Posts tagged with “infrastructure”

mSATA RAID?

Even though it's in packed away in the storage unit right now, I have been thinking a lot about the C220M3. Specifically about its own storage. Right now the only aspect of the machine that I have not seen to in one way or another is drives, specifically drive caddies - which, frankly, is not an uncommon issue with servers. Some machines, like the G5-7 HPs, and a few generations of Dells, share caddies (within their respective brands, granted). Because of this, their caddies are incredibly cheap. For this cisco, on the other hand, the caddies are relatively more expensive, often around $15 each. It's often more cost effective to just go ahead and buy caddies with drives in them already, provided you're okay with it being spinning media. The one drive I have for the machine now, the drive it came with, is an A03-D600GA2 - a Western Digital 600GB 10kRPM SAS hard drive. Aside from being a spinner, and having all of the downsides of spinners, there's nothing wrong with it.

Additional A03-D600GA2s, with caddy, are often about $50-60 shipped on ebay - which isn't bad, but I don't want them, for a few reasons typical of hard drives. I don't want to deal with the additional power consumption and the heat generation that goes with that. I also don't need the relatively cheaper dollar-per-gigabyte that spinners offer, since any large amount of data I need to to store can live on the NAS. Since SATA drives can be plugged into SAS backplanes, what I done prior to taking the C220M3 offline was have SSDs simply hanging out in the bays, two 480 and two 240 - the former in a RAID1, the latter in a RAID0. These were in bays 5, 6, 7, 8 whose SAS cable was connected to the Adaptec 2405 PCIe card I had added. This was necessary because ESXi has no support for the onboard intel soft raid - I had more posts about that in the past so I won't really go into it here.

Anywhoo, I was talking with some pals earlier tonight about storage and we started looking specifically for quad m.2 SATA PCIe cards. I know there are practical reasons for this not to be a thing, but they kind of are. Quad NVMe m.2 cards are actually fairly common. But that's not what this is about. This is about mSATA, which predated m.2 and was briefly popular. A card I found that I am particularly interested in is the Addonics AD4MSPX2-A - a $55 PCIe card with four mSATA slots and a Marvell RAID controller. It supports 1, 0, and 1+0 (which is what I would be using) and based on some cursory research on mSATA SSD prices, it seems rather competitive for what I am interested in - which is reliable SSDs that are "fast enough" - I don't need blazing.

256GB mSATA SSDs, name brand models from Samsung and the like, are about $50-60 on the second hand market. Four drives plus the card comes out at $260 or so, which is not bad at all for a bootable 512GB RAID1+0. Unfortunately 512GB mSATA SSDs are not very cost effective, still priced at around $100-115, putting a 1TB RAID1+0 closer to the price point of standard 2.5" SATA SSDs. Certainly one could take this in the other direction as well - 128GB mSATA drives can be found for as little as $20 shipped, meaning you could get a 512GB RAID0 (cough) for a mere $130 or so. Would I trust it? No, probably not, but I would sure as hell try it.

So why is this so interesting to me? Well, one of the PCIe slots in the C220M3 is already occupied by the the Adaptec 2405, so swapping that out with the Addonics card wouldn't cost me anything more. If I use the eight SAS bays on the front as individual drives in ESXi I can also pass them straight through to VMs, meaning I can use softraid in windows/linux, and at the very least have some dedicated IOPS for those VMs. Or I can just not use those bays at all, and only use the Addonics card. Now, I don't know yet if ESXi supports this Marvell chipset - I don't have high hopes, but I'm certain that windows and linux do, so there's always Hyper-V and Proxmox. (Hyper-V and Proxmox also support the Intel SoftRAID, so, you know... but... nevermind that).

Either way, look forward to a post with more information on this. I am incredibly interested in what a card like this can do, especially as mSATA prices continue to fall. More likely than not I would start with four 32 or 64GB for testing, then once I am satisfied with the performance, get another Addonics card and four 256GB drives. This gets me a 128/256GB fast scratch volume and a nice big protected working volume. It may be a while, but this will be interesting, I can promise that!

UPDATE 1 - Found a cheaper card here: Syba SI-PEX40109 - Currently $55 on prime. I also like the form factor a bit more than the Addonics model - it has an SFF adapter, and is a bit shorter because it has two mSATA on the front and two on the back. Not sure how the chipsets compare...

NVMe Upgrade

I found a good deal for a 480GB Kingston NVMe drive (true NVMe, not the "SATA on m.2 rubbish) as part of prime day, so I decided to go ahead and bite the bullet and slap one in the Inspiron. The main advantage this would give me is the ability to configure the second drive, the 2.5" SATA drive, as a secondary boot drive for linux, or additional storage if necessary. A day layer the drive showed up in a tiny envelope and the upgrade could commence. Prior to wiping the 2.5" drive, I ran Veeam to get a full system image over to the NAS. I wasn't planning to restore any data from it, but just in case something wonky happened I wanted to have an image I could quickly restore from. Under normal circumstances I would have simply mirrored the 2.5 over to the NVMe drive, but I wanted to use this as an opportunity to upgrade from home to pro, and I wanted to do that as a fresh install rather than an upgrade from within windows.

The physical installation was a breeze - pop off the bottom panel, remove the m.2 retention screw, drop in the drive, replace the screw. Could hardly be easier. The software installation, however, was somewhat more involved. The reason being that I didn't want to use the Windows 10 Home license key that is embedded in the system firmware. It took some trial and error, but what eventually worked was creating a PID.txt file in D:\sources\ on the installer. The key I wanted to use was embedded within the PID.txt file, and the installer automatically applied it, then activated it after the system started.

I wont bother listing any benchmark results - I ran CrystalDiskBench and it scored exactly as well as it should have, what you would expect for a mid-range NVMe drive. This wasn't installed for speed, however, rather it was installed to gain a second drive in this system. Once it was done and configured as I wanted, I popped in Ubuntu 18.04 install media and installed that onto the the 2.5" drive. Initially I had intended to remove the original 2.5" drive I was using and replace it with one of my older 128GB drives, but apparently this machine has some very strict thickness requirements and only 7mm drives will work. As it stands, it has a 480GB NVMe and 480GB 2.5" - more storage than I need, but until I can get a slim 120 or 240 that's not in use elsewhere, it's what will stay. Alternatively, I could split it in half and make a scratch disk with what's left over, but that's just not fairly low down on the importance ladder.

SpeedStep® Limiting Single-Threaded Tasks

This is something that will need to be further fleshed out by benchmarks and research, but I wanted to start writing about it now, before I forget. More is certainly to come.

Context

First, a little backstory for context. The company I work for, Esri, just released version 10.6.1 of their ArcGIS Enterprise product. Something I found very appealing about this release is that it includes a new interface and set of tools for processing imagery from UAVs into orthophotos, DSMs, and DEMs, called Ortho Maker. Esri has a desktop product that does this as well, Drone2Map, which is based on the Pix4D Mapper engine. Ortho Maker is unique in comparison because it is intended to run on server infrastructure rather than workstations or HEDTs, and anyone who knows me will know that I find that incredibly appealing.

So, when 10.6.1 was formally released, I wiped the TS140 'goonie' to a clean slate and configured it as follows: 2x8GB RAM (I moved the other 2 to TS140#1), 256GB Samsung 840 Pro boot/primary data drive, 2TB WD Green as a backup/scratch drive - this simply because it was laying around and I figured I could use some extra storage that didn't need performance or redundancy. Finally, a fresh copy of Server 2016 - which took forever to update, then a base enterprise deployment of ArcGIS Enterprise 10.6.1. Now, this also required an Image Server license because raster processing tools are used in order to generate the final products. In a production environment the server with the image server license would not be the same as the server running the portal and hosting server (you can read more about ArcGIS Enterprise architecture here), but since I am the only individual that will be accessing this server, and I know that nothing will be putting load on the hosting server or portal while the image server components are working.

I have a few different sample UAV imagery collections that I use to test this software, both my own and others that are available publicly on the internet, so I began with processing a few to get a better feel for the capabilities of the software. As a solution engineer, part of my job is to thoroughly test all the components of our software that I could find myself recommending to a customer, and this is no exception.

Where SpeedStep comes in...

After watching Process Explorer and the Windows Task Manager while the various stages of the ortho maker's processes were run, it became clear that these are single threaded tools. I imagine this will change in future releases, and there may be an option to make it multi threaded in the future, but as it stands each of the tasks only runs one thread at a time - each process never taking more than 25% of the total CPU time (the E3-1225v3 is 4c/4t).

Windows Task Manager shows the current clock speed of the CPU, which is very useful in cases like these, and what I was seeing was clock speeds much lower than expected. The E3-1225v3's base is 3.2GHz, with boost up to 3.6GHz, however Task Manager was showing that the system was rarely, if ever, hitting higher than 1.2GHz. Now, we all know what SpeedStep is - it's that fabulous technology that changes CPU speed on the fly, keeping power consumption and temperatures down. The dark underbelly to SpeedStep, apparently, is that it uses total CPU usage to gauge whether it needs to crank up the speed rather than per-core usage.

In this case, that's a problem. If the server is running two Ortho Maker tasks, and is otherwise idle, the CPU will only ever be at 50% + a small amount of system overhead, maybe 15%. What that means is that the system will never determine that it's necessary to step up, and the performance of those single threads will be far less than they could be. I suppose if I was concerned about temperatures and fan speeds then maybe that would apply, but this is a server we're talking about here.

How to resolve it

Thankfully this is an easy problem to solve. Disable SpeedStep. This can even be done from within windows, without rebooting. Launch powercfg.cpl and change the power plan from "Balanced" to "High Performance". If you want to see what this does, click "Change plan settings" then "Change advanced power settings" - under processor power management both the minimum processor state and maximum processor state will be set to 100% - meaning that in my case the CPU will always be at 3.2GHz, and will boost when possible.

SpeedStep can also (usually) be disabled from the BIOS, and this would be my recommendation. I'm currently working on this server remotely so that's not an option for me, but I will be addressing it soon!

What other people are saying

Not surprisingly, quite a few people have discovered this quirk, and Microsoft even has some KBs that address this when it comes to using Hyper-V. Basically what it comes down to is "know your workloads, and know your hardware". In many cases, leaving speedstep enabled will be best.

What I need to do next

More research, and some single threaded benchmarks of my own. I also want to test this theory across multiple machines - especially the C220M3 (currently in storage...).

Resources

GPU Weirdness

Earlier this week I encountered a very strange problem with the 1050ti in my desktop TS140 - the same setup I had posted about here earlier. This is what happened. The machine had just completed a windows update and prompted to reboot. I was working on the other TS140, goonie, at the time and needed local video on it, so while the desktop-TS140 rebooted I pulled it's displayport cable and moved it to goonie. I finished doing what I needed to do and moved the cable back, yet, no video. So I reset the machine - the BIOS splash displayed as normal, but when the loading screen normally transitions to the windows login screen, the display went dark. And not just black - no video signal was being received.

I figured a windows update nuked something, so I booted off my veeam recovery disk and restored the previous day's backup (yay for taking regular backups), then rebooted, but encountered the same behavior. As soon as windows started, the display went dark. So I grabbed my windows 10 install drive and did a fresh install - the install process went just fine, but as soon as it loaded the desktop and windows started automatically downloading and enabling divers - boom, display goes dark again. At this point I don't know what to think. I tested the HDMI and DVI as well, but only saw the same behavior.

Wanting to test on more than windows, I grabbed my ubuntu 18.04 drive and did a live boot off the install media, and while it did display full resolution, I didn't think that the video was being accelerated. At this point it was getting late, so I pulled the card and switched back to the onboard video, then once again restored the previous day's backup. Windows booted up just fine, and after updating the driver for the iGPU (which most certainly made me nervous) it was just fine. At this point it was about 1AM so I packed up and went to bed.

On the way home from work the next day I had to swing by the storage unit to drop off some more stuff, and while I was there I decided to pull out the T3500 - my only other full-size PC with PCIe. I do have the other TS140 at the house but I wanted to test on something totally different. Once I got it all setup and the card installed I did a fresh install of ubuntu 18.04. When it booted up the display was again at full resolution, but after running some GPU tests it was most certainly not accelerated. I installed the latest stable nvidia drivers for ubuntu and rebooted again, fingers crossed. As soon as the machine came back up and the display manager started... you guessed it... the video signal dropped out. Every time the GPU was initialized with drivers it was crashing. I ran the windows 10 install on this machine too and again saw the same behavior - no problem during the install and first boot, but as soon as windows grabbed and started the GPU driver the display dropped. Frustrated and certain that the card was not only dead, but not covered under warranty, I went to bed.

The following evening (yesterday evening, if we're counting), I decided to pop off the card's cooler and take a look at the thermal situation. I didn't think that this was a heat related issue, but still - might as well take a look at the card and the die while I can. The thermal paste was plentiful and very dry, so I grabbed some alcohol and q-tips and went to town cleaning everything up. I re-applied the paste and put the card back together, and then wondered if it was worth testing again. I figured it was, but wanted to take a slightly different approach.

I powered down and grabbed the other TS140, goonie, and put a spare SSD in it. I made a windows 7 install drive, and after mounting the card in the machine, began the process of installing and updating 7 on this box. Once it was all up and running, I made the fateful trip to the nvidia download page and grabbed the latest installer. Unlike in windows 10, the GPU driver wasn't initialized right away, so I rebooted, expecting the worst. It came right back up and displayed a full resolution login screen, so after I logged in I checked device manager right away and the 1050 ti was listed, with no errors.

So far I have run benchmarks with Geekbench, Cinebench, and FurMark - including a 10 minute stress test with FurMark. The card gets rather toasty, about 60ºC, but the fans never ramped up, so I guess it was doing just fine. In any case, the system never crashed - it just kept chugging along. I don't know what to think. I really doubt that my replacing the thermal paste did anything, and I would be surprised if both the T3500 and other TS140 were messed up somehow. I have another 1050 Ti (a short card this time...) on order, should get here tomorrow, and I'm really hoping that it doesn't exhibit any weirdness. I am going to test this card in at least one more system, a friend's Optiplex 790, and until then I'm going to essentially expect it to die at any time. Right now the only test I have left that I'd really like to do is Ubuntu 18.04 on goonie and see how that compares. If I have to use this card on this machine in windows 7 that's hardly the worst thing in the world, but it certainly is confusing....

New Laptop!

The last "contemporary" laptop I purchased was a Samsung Chromebook, for about $150, three years ago. It worked well enough, but as a then-low end machine with a small 1366x768 screen and not much in the way of RAM, CPU power, or storage, I didn't end up doing much with it. Prior to that was my Lenovo T410, a thinkpad purchased as a refurb unit in late 2010 while I was in college. The T410 was my daily driver all through college, and until a couple of years after, when I decided that I no longer really needed a laptop as my main machine, and would rather use a desktop. The T410 never went anywhere, but despite having 8GB of RAM and an SSD did start to feel "old" fairly quickly. Granted, this was a first generation i5 from 2010, and while in 2014 it was still capable, it had it's limitations. The battery was long gone, and the CPU seemed to always run hot, no matter how often I would tear it down and blow out dust and replace the thermal paste. Every key on the keyboard is worn shiny, as is the once bumpy trackpad and pebbled wrist rests, and the screen has just enough annoying bits of dust trapped within it to be frustrating. I should note that I do still have it, though, and I have no intentions of getting rid of it... it's just... not really a laptop anymore, it's a small computer that runs linux like a dream and sits on a desk.

Now, I've been in my current professional position for a bit more than two and a half years now, and for about 18 months of that, I have had a very nice Precision 5510. Prior to that I had a Precision M4800, which when docked was the perfect computer, but at a heftyy 13 pounds (with charger, granted) was less than pleasant to travel with - and I travel a lot. The M4800 developed some problems and I was able to get upgraded, thus, 5510. With a 4c/8t i7, 32GB RAM, a 1TB NVMe SSD and 2GB Quadro GPU it's nothing to sneeze at - though it's nothing I would ever purchase for myself. It's basically the business equivalent of the XPS 15, just with a Quadro instead of a GeForce Mobile, or whatever.

The problem is, really, that this machine had more or less developed into my standard laptop for travel. Vacations to Maine or weekend trips to Richmond, Charleston, or Savannah I would always bring it along. It was only ever used to checking email, streaming Netflix or Plex, or perhaps offloading pictures from a full SD card, but was nice to have along. Until... well, outlook was right there, so was lync and all my other projects and material for work, and just like that I'd be working. This never developed into a problem, per se, but I recognized that it could lead to a more dangerous habit, and that it would be more healthy to just leave well enough alone, and just leave my laptop at work unless I needed to take it home for some legitimate reason.

While this did free me from the shackles of my employment (at least when I wasn't supposed to be working), it did shove in my face the fact that a modern laptop might actually be a good thing to have again. That being said, we did recently purchase a laptop for my wife, a very handsome 13" Inspiron 2-in-1 - which for most trips is fine, often we just need one machine. However there are other cases (like sitting at home on the couch, or when I travel personally alone, or whatever) that it would be convienient for me to have my own laptop. I tried using the chromebook for a while, but that didn't really work that well. I also picked up a Latitude E4310 cheap from The Grid a while back, initially with the intention of flipping it, but I ended up getting attached to it. I tried using it for a while, and while it is smaller than the T410, it was no faster (same CPU, even) nor any cooler - though it did have a working battery, and the backlit keyboard was fairly nice.

Ultimately my biggest problem was the lack of a 1080 screen. I know people use higher resolution screens with higher DPIs these days, but I like 1080. I use it at home on my desk, I use it at work on my desk, and it's what my work laptop runs at. I am very comfortable in 1920x1080. My two requirements came down to this: a 1920x1080 screen, and a screen size of less than or equal to 15", prefferably 14". I started with looking at various secondhand Latitudes, especially the E5540 and E5440 - and while I liked what I saw for the most part, I had mixed feelings about purchasing another secondhand laptop, especially as this is something I wanted to last at least as long as the T410 had. While out to dinner not long ago, and discussing this with Liz, I couldn't help but wonder if I would just be better off getting a low end machine with a 1080 screen, then dropping in an SSD of my own. She suggested that we just go ahead and go to Best Buy after dinner and see what they have, so we did.

Once we got there I was surprised at how disappointed I ended up being - the selection was just awful. The cheapest machine with a 1080 screen was an ASUS, and if I recall correctly it was around $950. There may have been a cheaper option in a 17" desktop replacement, but I didn't want anything massive. I was about to shrug off the adventure and just go home to look on Amazon, but while looking around we happened to notice that the open box "cage", as it was, was absolutely full to the brim with generic brown boxes with either returned or display model machines. It took a moment to track down someone who was willing to unlock it and help us out, but once we did it was a fairly straightforward experience. The guy we worked with had no idea what was in any of the boxes - the labels affixed to them simply had a price and a manufacturer, so one by one we pulled out machines that were in my price range and scanned them into the system, looking up their components.

Eventually, we got lucky - a machine that exactly matched what I was looking for, an Inspiron 15 7573 2-in-1. Granted I didn't really want a touch screen machine, but it had other things I found appealing - the i7-8550U say, or the fact that it was configured with 12GB RAM. Really the only "problem" was that the 2TB hard drive was just that, a hard drive - but no problem, I have spare SSDs at home. Of course it has a 1080 screen, and a price tag of less than $600 was more than reasonable. So far I've had it about two weeks, and I'm very happy with it. Who knows if it will outlast the T410, but right now I'm not really that worried. I'm just glad I found what I was looking for.

My model in particular is a 7537-7012, which seems to be a Best Buy exclusive SKU. Since the C220M3 is currently out of operation, at least until we move, I pulled one of its 480GB SSDs for use in the inspiron and have since listed the original hard drive for sale, as I have no use for a 2TB 2.5" drive. Getting into the machine is very easy, just 10 screws - four of which, along the front edge, are fully removable, where the other six are captive. Why they chose to make any of them captive at all is beyond me. Once they're removed (or loosened) the whole bottom panel lifts off, as many newer laptops with metal frames do these days, revealing the entire motherboard, drives, and battery. Swapping the drive was just a few more screws. In addition to the 2.5" SATA bay, this machine does support full size NVMe drives as well - something I plan on getting in the near future. I'll probably aim for a ~500GB drive and return this 2.5" SSD to the C220M3 once it's back up and running.

So now that I've had this thing a while, what are my impressions? Well, I really like it - it has been able to do exactly what I want, and it's plenty comfortable to use. A common complaint I see about this series is that it's very heavy, but I don't really notice it any more than any other similarly sized laptop. The metal frame feels very sturdy and solid, and doesn't flex at all as far as I can tell. The keyboard feels great and is backlit, which I'm fond of - though I know that is very typical these days. Overall it feels very similar to the 5510, though I would still consider the 5510 to be better, though I'm a bit stumped at explaining exactly how. The trackpad does the job, but I don't love it - again, the 5510 wins here with it's silky smooth surface. The inspiron is a bit rougher, but it does track very well.

As far as IO goes it leaves me wanting nothing - three USB3, including one that can be "always on" for power, HDMI, a headset jack, USB-C (though I don't know what protocols it carries - it does not support the 5510's USB-C to Ethernet adapter), and a full size SD card slot that doesn't leave any of the card hanging out, which the 5510 does. The SD card slot is only USB2 attached though, topping out at around 25MB/s, which is a bummer. The 5510's SD slot, while not fully recessed, is much faster.

The screen is very attractive, but I agree with others in that it could be brighter. I don't have any issues with the color reproduction or viewing angle, however. I love the windows hello feature and have been using it extensively. Some Inspiron 15 models have a fingerprint reader built into the power button, which I would prefer, but this model doesn't have that.

Right now, I think, my only issue is with windows licensing. There is a windows license embedded within the system, and right now I don't know how to disable it. I'd rather use an existing windows 10 pro key I have, because with just 10 home there are a few features I find I'm missing - like WSFL, Hyper-V, etc. When I get the NVMe SSD and do a reinstall I'll dig into resolving this problem more seriously.