The Dome

A couple of weeks ago I found a very reasonably priced iMac G4 being sold locally, and having always been somewhat interested in them (and having never had one before) I decided to get in touch with the seller and go pick it up. What I ended up getting was a fantastic condition first generation iMac G4/800. It has a few minor scuffs and marks on it here and there, but nothing a couple of alcohol wipes couldn't take care of.

What has kept me away from iMac G4s in the past has been the fact that, by and large, they are OS X only machines. I have very little interest in PPC OS X as it largely still feels too new to be "retro" and yet is too slow and incompatible to be used for much. I tinker with it from time to time, but System 7 and Mac OS 8 and 9 are where most of my interest lies. Only the first revision of the first generation of iMac G4 could run 9, and even then only it's special version of 9.2.2 (which is not hard to find), but that is good enough for me. I decided to take a gamble on this particular machine because 1) it was very cheap, 2) it looked to be in good condition, and 3) it was a 15", and all of the first gen iMac G4s were 15", so there was a chance that this one was a first gen. Reading the service manual I also learned that not only is the first gen the only generation to be able to boot natively into 9, but only machines with serial numbers less than xx305xxxxxx will as well. This machine is in the xx200 range, so it should boot 9 just fine. It's also the highest end first generation model sold - with the 800MHz CPU, 60GB HDD, and Superdrive. It was also factory upgraded to 512MB with a single PC133 DIMM, leaving the SO-DIMM slot vacant. The airport slot is also surprisingly empty, but that's not really a problem for me as I have little interest in WEP encrypted 802.11b.

Now, the bad news. Unfortunately this particular machine has screen issues. Particularly, the "white screen of death" that seems to be not entirely uncommon when researching these machines. This one in particular boots as one would normally expect when the power button is pressed - it chmes, the fan spins up, if an HDD or CD with OS loaded is present you can hear it begin to boot - but the screen is forever white. When it first kicks on the screen is more of a gray, in fact it's the same gray that you would expect for a machine of this era's first image (all of my G4s do it too) but instead of showing a happy mac, or the "OS not found" symbols, within a few seconds it transitions to a bright white. One way I know that the machine is booting is that I am able to enter open firmware and use commands to shut down, reboot, and eject the optical drive tray. This is good news, as it tells me that the motherboard works properly, and when I hear it booting from the HDD or optical drive I know that the IDE bus works as well.

I don't know which part is causing the problem, but I have been able to narrow it down to one of three things: the video connector on the motherboard (very unlikely), the video cable that runs from the board, up the neck, into the screen (somewhat likely, but I would expect flickering when the screen is articulated and wiggled), or it's the LCD panel itself - most likely, I believe, and frankly also probably the easiest to replace. The second easiest would be the logic board, but the video cable is contained within the neck itself, and getting a new neck with wiring harness so far seems rather difficult. I would also like to avoid replacing the logic board because there is little certainty that I would get another 800MHz first-gen board, which is what I very specifically want.

The Neck

Now, this machine does actually have video output. It's one of the few macs (along with a handful of iBooks, PowerBooks, and other iMac G4s and G5s) from the early 2000s to use "Mini VGA" which, aside from having a weird connector, is exactly what it sounds like. On this machine it doesn't allow for adding a second extended display - it can only mirror the primary display - but in this case that happens to be exactly what I want to do. Enough macs used Mini VGA that there are a plethora of the adapters available, and I was able to pick one up for about $4 shipped. So you may be able to see where I'm going here.

What I have chosen to do is remove the neck and screen assembly entirely, and what remains is what I have chosen to name "The Dome". The Dome is, obviously, the bottom half of an iMac G4 - a white plastic hemisphere driving an external monitor. It looks a bit strange, sure, like a mac mini about to pop, but it is a more compact G4 running OS9 than either the Quicksilver or Digital Audio, so that's a plus. And at 800MHz with 512MB RAM, it's no slouch either. Granted the GPU is fixed (GeForce2 with 32MB) but seeing as it's only ever driving 1024x768, that's not really an issue. No potential for a SATA card either, but I plan to at least get it upgraded with a 7.2k hard drive - the original superdrive will stay.

The process for removing the neck and screen is fairly trivial. Remove the bottom case, disconnect all the wires, and set it aside. Remove the drive carrier, power supply, and fan from the upper housing. The hardest part is removing the extraneous wires from the blind mate connector. This connector carriers a number of signals - power/sleep LED, microphone, display inverter voltages, fan, and speaker. Everything but the fan and speaker is in the display, so I removed those pins from the connector. The fan could be powered from the drives' molex connectors, and you could forgo a speaker, but I didn't want to do either. I also took this opportunity to clean out the fan and top of the case, as well as the peculiar two-part power supply. The neck is secured to the base with five screws, and once they are removed it pops straight off, and the various wires are fed through the hole left behind. I may 3D print something to cap this hole, but for the time being I am going to leave it as is.

As mentioned earlier, I do plan to upgrade the hard drive to a 7200RPM unit. The costs of PC133 SODIMMs are also so low right now that, at $8 shipped, there's really no good reason to not upgrade it to 1GB RAM. The fan is a standard 92mm PWM unit, so I plan to upgrade that to a Noctua as well. There's no real need for this - the included fan works just fine - but I would like something with greater airflow at lower speed. I would also like to find something that allows me to monitor the temperature and fan speed in OS9, though I don't know that such a utility exists. I have also noticed, in a couple of days of using it, that it is far more stable than my Digital Audio. This will require some further followup and investigation, and perhaps a future post.

mSATA RAID?

Even though it's in packed away in the storage unit right now, I have been thinking a lot about the C220M3. Specifically about its own storage. Right now the only aspect of the machine that I have not seen to in one way or another is drives, specifically drive caddies - which, frankly, is not an uncommon issue with servers. Some machines, like the G5-7 HPs, and a few generations of Dells, share caddies (within their respective brands, granted). Because of this, their caddies are incredibly cheap. For this cisco, on the other hand, the caddies are relatively more expensive, often around $15 each. It's often more cost effective to just go ahead and buy caddies with drives in them already, provided you're okay with it being spinning media. The one drive I have for the machine now, the drive it came with, is an A03-D600GA2 - a Western Digital 600GB 10kRPM SAS hard drive. Aside from being a spinner, and having all of the downsides of spinners, there's nothing wrong with it.

Additional A03-D600GA2s, with caddy, are often about $50-60 shipped on ebay - which isn't bad, but I don't want them, for a few reasons typical of hard drives. I don't want to deal with the additional power consumption and the heat generation that goes with that. I also don't need the relatively cheaper dollar-per-gigabyte that spinners offer, since any large amount of data I need to to store can live on the NAS. Since SATA drives can be plugged into SAS backplanes, what I done prior to taking the C220M3 offline was have SSDs simply hanging out in the bays, two 480 and two 240 - the former in a RAID1, the latter in a RAID0. These were in bays 5, 6, 7, 8 whose SAS cable was connected to the Adaptec 2405 PCIe card I had added. This was necessary because ESXi has no support for the onboard intel soft raid - I had more posts about that in the past so I won't really go into it here.

Anywhoo, I was talking with some pals earlier tonight about storage and we started looking specifically for quad m.2 SATA PCIe cards. I know there are practical reasons for this not to be a thing, but they kind of are. Quad NVMe m.2 cards are actually fairly common. But that's not what this is about. This is about mSATA, which predated m.2 and was briefly popular. A card I found that I am particularly interested in is the Addonics AD4MSPX2-A - a $55 PCIe card with four mSATA slots and a Marvell RAID controller. It supports 1, 0, and 1+0 (which is what I would be using) and based on some cursory research on mSATA SSD prices, it seems rather competitive for what I am interested in - which is reliable SSDs that are "fast enough" - I don't need blazing.

256GB mSATA SSDs, name brand models from Samsung and the like, are about $50-60 on the second hand market. Four drives plus the card comes out at $260 or so, which is not bad at all for a bootable 512GB RAID1+0. Unfortunately 512GB mSATA SSDs are not very cost effective, still priced at around $100-115, putting a 1TB RAID1+0 closer to the price point of standard 2.5" SATA SSDs. Certainly one could take this in the other direction as well - 128GB mSATA drives can be found for as little as $20 shipped, meaning you could get a 512GB RAID0 (cough) for a mere $130 or so. Would I trust it? No, probably not, but I would sure as hell try it.

So why is this so interesting to me? Well, one of the PCIe slots in the C220M3 is already occupied by the the Adaptec 2405, so swapping that out with the Addonics card wouldn't cost me anything more. If I use the eight SAS bays on the front as individual drives in ESXi I can also pass them straight through to VMs, meaning I can use softraid in windows/linux, and at the very least have some dedicated IOPS for those VMs. Or I can just not use those bays at all, and only use the Addonics card. Now, I don't know yet if ESXi supports this Marvell chipset - I don't have high hopes, but I'm certain that windows and linux do, so there's always Hyper-V and Proxmox. (Hyper-V and Proxmox also support the Intel SoftRAID, so, you know... but... nevermind that).

Either way, look forward to a post with more information on this. I am incredibly interested in what a card like this can do, especially as mSATA prices continue to fall. More likely than not I would start with four 32 or 64GB for testing, then once I am satisfied with the performance, get another Addonics card and four 256GB drives. This gets me a 128/256GB fast scratch volume and a nice big protected working volume. It may be a while, but this will be interesting, I can promise that!

UPDATE 1 - Found a cheaper card here: Syba SI-PEX40109 - Currently $55 on prime. I also like the form factor a bit more than the Addonics model - it has an SFF adapter, and is a bit shorter because it has two mSATA on the front and two on the back. Not sure how the chipsets compare...

NVMe Upgrade

I found a good deal for a 480GB Kingston NVMe drive (true NVMe, not the "SATA on m.2 rubbish) as part of prime day, so I decided to go ahead and bite the bullet and slap one in the Inspiron. The main advantage this would give me is the ability to configure the second drive, the 2.5" SATA drive, as a secondary boot drive for linux, or additional storage if necessary. A day layer the drive showed up in a tiny envelope and the upgrade could commence. Prior to wiping the 2.5" drive, I ran Veeam to get a full system image over to the NAS. I wasn't planning to restore any data from it, but just in case something wonky happened I wanted to have an image I could quickly restore from. Under normal circumstances I would have simply mirrored the 2.5 over to the NVMe drive, but I wanted to use this as an opportunity to upgrade from home to pro, and I wanted to do that as a fresh install rather than an upgrade from within windows.

The physical installation was a breeze - pop off the bottom panel, remove the m.2 retention screw, drop in the drive, replace the screw. Could hardly be easier. The software installation, however, was somewhat more involved. The reason being that I didn't want to use the Windows 10 Home license key that is embedded in the system firmware. It took some trial and error, but what eventually worked was creating a PID.txt file in D:\sources\ on the installer. The key I wanted to use was embedded within the PID.txt file, and the installer automatically applied it, then activated it after the system started.

I wont bother listing any benchmark results - I ran CrystalDiskBench and it scored exactly as well as it should have, what you would expect for a mid-range NVMe drive. This wasn't installed for speed, however, rather it was installed to gain a second drive in this system. Once it was done and configured as I wanted, I popped in Ubuntu 18.04 install media and installed that onto the the 2.5" drive. Initially I had intended to remove the original 2.5" drive I was using and replace it with one of my older 128GB drives, but apparently this machine has some very strict thickness requirements and only 7mm drives will work. As it stands, it has a 480GB NVMe and 480GB 2.5" - more storage than I need, but until I can get a slim 120 or 240 that's not in use elsewhere, it's what will stay. Alternatively, I could split it in half and make a scratch disk with what's left over, but that's just not fairly low down on the importance ladder.

SpeedStep® Limiting Single-Threaded Tasks

This is something that will need to be further fleshed out by benchmarks and research, but I wanted to start writing about it now, before I forget. More is certainly to come.

Context

First, a little backstory for context. The company I work for, Esri, just released version 10.6.1 of their ArcGIS Enterprise product. Something I found very appealing about this release is that it includes a new interface and set of tools for processing imagery from UAVs into orthophotos, DSMs, and DEMs, called Ortho Maker. Esri has a desktop product that does this as well, Drone2Map, which is based on the Pix4D Mapper engine. Ortho Maker is unique in comparison because it is intended to run on server infrastructure rather than workstations or HEDTs, and anyone who knows me will know that I find that incredibly appealing.

So, when 10.6.1 was formally released, I wiped the TS140 'goonie' to a clean slate and configured it as follows: 2x8GB RAM (I moved the other 2 to TS140#1), 256GB Samsung 840 Pro boot/primary data drive, 2TB WD Green as a backup/scratch drive - this simply because it was laying around and I figured I could use some extra storage that didn't need performance or redundancy. Finally, a fresh copy of Server 2016 - which took forever to update, then a base enterprise deployment of ArcGIS Enterprise 10.6.1. Now, this also required an Image Server license because raster processing tools are used in order to generate the final products. In a production environment the server with the image server license would not be the same as the server running the portal and hosting server (you can read more about ArcGIS Enterprise architecture here), but since I am the only individual that will be accessing this server, and I know that nothing will be putting load on the hosting server or portal while the image server components are working.

I have a few different sample UAV imagery collections that I use to test this software, both my own and others that are available publicly on the internet, so I began with processing a few to get a better feel for the capabilities of the software. As a solution engineer, part of my job is to thoroughly test all the components of our software that I could find myself recommending to a customer, and this is no exception.

Where SpeedStep comes in...

After watching Process Explorer and the Windows Task Manager while the various stages of the ortho maker's processes were run, it became clear that these are single threaded tools. I imagine this will change in future releases, and there may be an option to make it multi threaded in the future, but as it stands each of the tasks only runs one thread at a time - each process never taking more than 25% of the total CPU time (the E3-1225v3 is 4c/4t).

Windows Task Manager shows the current clock speed of the CPU, which is very useful in cases like these, and what I was seeing was clock speeds much lower than expected. The E3-1225v3's base is 3.2GHz, with boost up to 3.6GHz, however Task Manager was showing that the system was rarely, if ever, hitting higher than 1.2GHz. Now, we all know what SpeedStep is - it's that fabulous technology that changes CPU speed on the fly, keeping power consumption and temperatures down. The dark underbelly to SpeedStep, apparently, is that it uses total CPU usage to gauge whether it needs to crank up the speed rather than per-core usage.

In this case, that's a problem. If the server is running two Ortho Maker tasks, and is otherwise idle, the CPU will only ever be at 50% + a small amount of system overhead, maybe 15%. What that means is that the system will never determine that it's necessary to step up, and the performance of those single threads will be far less than they could be. I suppose if I was concerned about temperatures and fan speeds then maybe that would apply, but this is a server we're talking about here.

How to resolve it

Thankfully this is an easy problem to solve. Disable SpeedStep. This can even be done from within windows, without rebooting. Launch powercfg.cpl and change the power plan from "Balanced" to "High Performance". If you want to see what this does, click "Change plan settings" then "Change advanced power settings" - under processor power management both the minimum processor state and maximum processor state will be set to 100% - meaning that in my case the CPU will always be at 3.2GHz, and will boost when possible.

SpeedStep can also (usually) be disabled from the BIOS, and this would be my recommendation. I'm currently working on this server remotely so that's not an option for me, but I will be addressing it soon!

What other people are saying

Not surprisingly, quite a few people have discovered this quirk, and Microsoft even has some KBs that address this when it comes to using Hyper-V. Basically what it comes down to is "know your workloads, and know your hardware". In many cases, leaving speedstep enabled will be best.

What I need to do next

More research, and some single threaded benchmarks of my own. I also want to test this theory across multiple machines - especially the C220M3 (currently in storage...).

Resources

GPU Weirdness

Earlier this week I encountered a very strange problem with the 1050ti in my desktop TS140 - the same setup I had posted about here earlier. This is what happened. The machine had just completed a windows update and prompted to reboot. I was working on the other TS140, goonie, at the time and needed local video on it, so while the desktop-TS140 rebooted I pulled it's displayport cable and moved it to goonie. I finished doing what I needed to do and moved the cable back, yet, no video. So I reset the machine - the BIOS splash displayed as normal, but when the loading screen normally transitions to the windows login screen, the display went dark. And not just black - no video signal was being received.

I figured a windows update nuked something, so I booted off my veeam recovery disk and restored the previous day's backup (yay for taking regular backups), then rebooted, but encountered the same behavior. As soon as windows started, the display went dark. So I grabbed my windows 10 install drive and did a fresh install - the install process went just fine, but as soon as it loaded the desktop and windows started automatically downloading and enabling divers - boom, display goes dark again. At this point I don't know what to think. I tested the HDMI and DVI as well, but only saw the same behavior.

Wanting to test on more than windows, I grabbed my ubuntu 18.04 drive and did a live boot off the install media, and while it did display full resolution, I didn't think that the video was being accelerated. At this point it was getting late, so I pulled the card and switched back to the onboard video, then once again restored the previous day's backup. Windows booted up just fine, and after updating the driver for the iGPU (which most certainly made me nervous) it was just fine. At this point it was about 1AM so I packed up and went to bed.

On the way home from work the next day I had to swing by the storage unit to drop off some more stuff, and while I was there I decided to pull out the T3500 - my only other full-size PC with PCIe. I do have the other TS140 at the house but I wanted to test on something totally different. Once I got it all setup and the card installed I did a fresh install of ubuntu 18.04. When it booted up the display was again at full resolution, but after running some GPU tests it was most certainly not accelerated. I installed the latest stable nvidia drivers for ubuntu and rebooted again, fingers crossed. As soon as the machine came back up and the display manager started... you guessed it... the video signal dropped out. Every time the GPU was initialized with drivers it was crashing. I ran the windows 10 install on this machine too and again saw the same behavior - no problem during the install and first boot, but as soon as windows grabbed and started the GPU driver the display dropped. Frustrated and certain that the card was not only dead, but not covered under warranty, I went to bed.

The following evening (yesterday evening, if we're counting), I decided to pop off the card's cooler and take a look at the thermal situation. I didn't think that this was a heat related issue, but still - might as well take a look at the card and the die while I can. The thermal paste was plentiful and very dry, so I grabbed some alcohol and q-tips and went to town cleaning everything up. I re-applied the paste and put the card back together, and then wondered if it was worth testing again. I figured it was, but wanted to take a slightly different approach.

I powered down and grabbed the other TS140, goonie, and put a spare SSD in it. I made a windows 7 install drive, and after mounting the card in the machine, began the process of installing and updating 7 on this box. Once it was all up and running, I made the fateful trip to the nvidia download page and grabbed the latest installer. Unlike in windows 10, the GPU driver wasn't initialized right away, so I rebooted, expecting the worst. It came right back up and displayed a full resolution login screen, so after I logged in I checked device manager right away and the 1050 ti was listed, with no errors.

So far I have run benchmarks with Geekbench, Cinebench, and FurMark - including a 10 minute stress test with FurMark. The card gets rather toasty, about 60ºC, but the fans never ramped up, so I guess it was doing just fine. In any case, the system never crashed - it just kept chugging along. I don't know what to think. I really doubt that my replacing the thermal paste did anything, and I would be surprised if both the T3500 and other TS140 were messed up somehow. I have another 1050 Ti (a short card this time...) on order, should get here tomorrow, and I'm really hoping that it doesn't exhibit any weirdness. I am going to test this card in at least one more system, a friend's Optiplex 790, and until then I'm going to essentially expect it to die at any time. Right now the only test I have left that I'd really like to do is Ubuntu 18.04 on goonie and see how that compares. If I have to use this card on this machine in windows 7 that's hardly the worst thing in the world, but it certainly is confusing....