28 May 2020

Rather Closer to the End

Well, the beginning was a good long time ago. And one thing I can always take away from the Drake Equation is that a primary reason that we’re alone is because civilizations just don’t last long enough to get past the deeply stupid stage that follows the enlightenment. Whether we do it to ourselves, or Giant Asteroid ’20 does the job for us, the bell is tolling.

I know, I’m a cheery sort of bloke, eh? You look at the news. I’ll go back to the small shit that I have control over. Like …

The garden is alive. Almost two weeks and nothing’s dead yet. On Saturday I’ll take off the bird netting, weed out the beds, and set the tomato plant cages. Yay!

SSH Agent Persistence vs CSH/TCSH

Y’all can skip this part if you want. It’s here as much for me to find it again if I ever need it as for anyone in Greater OutThereLandia.

The problem is “simple”. I have a group of systems I’m responsible for. The developers have a whole suite of scripts and processes based on the fact that they use the modern incarnation of the C shell, tcsh. I’m migrating them to newer platforms, and newer code repositories, remote code repositories. Access to these repositories requires SSH access. SSH key pairs are the answer to the problem, but the private key requires a passphrase to meet our security requirements.

Persistence across multiple login sessions is the key for the developers. Once they have an active login session on the platform, they want to leverage their SSH authentication without re-entering the passphrase each time it’s needed, or even each time they login (simulateous sessions) on the system.

All the tools I have laying about for managing persistent SSH keys across multiple sessions are pretty much Bourne shell based, and mostly I use a tool called keychain, written a while back by Daniel Robbins of Gentoo fame. I needed to find out how other people are solving this problem for csh/tcsh users.

What I learned is that not very many people are using csh/tcsh anymore. BUT. But, I found Mark A. Hershberger, who many moons ago wrote a page about managing SSH Agent via scripts, with a link to a sub-page with solutions for alternate shells (like tcsh/csh) – http://mah.everybody.org/docs/ssh-agent-startup … yes, that’s a plain HTTP link, no SSL, and your browser will hate it. Don’t worry, you’re not logging in or doing any banking there.

This was an awesome find for someone like me that spends little or no time in tcsh, but knows that it’s a bit of a janky environment for scripting things. I read, understood, and implemented the script as written. It didn’t work. Sigh. After a couple of hours complete with gnashing of teeth and pulling of hair, it turns out I was getting clobbered by a default feature of the environment called noclobber, which effectively prevented me from overwriting a file that already exists via redirection. So, if noclobber is set (which is part of the environment for these devs), then this code won’t work, if the target file, /tmp/blue.txt, already exists:

echo "I\'ve got the blues!" > /tmp/blue.txt

Once I learned about noclobber, and determined that I could not unset it and leave it unset because of user expectations, I found that I could force the overwrite with the judicious application of a ‘!’ character to decorate the redirection. This works, whether or not /tmp/blue.txt exists:

echo "I\'ve got the blues!" >! /tmp/blue.txt

With that problem out of the way, I was able to get the code to run at login. Then I started piecing together the logic I wanted to actually apply for these development users. For login, look for a file that defines an existing SSH Agent session. If that file doesn’t exist, or if it contains information about a defunct session, it will start a new SSH Agent session. IF there’s a running SSH Agent session, it’ll check to ensure the key is loaded, and prompt to load it if needed. So, this code goes into the users .login file in their home directory:

set sshAgent=/usr/bin/ssh-agent
set sshAgentArgs="-c"
set tmpFile=~/.ssh/ssh-agent-info
#
Check for existing ssh-agent process
#
if ( -s $tmpFile ) source $tmpFile
  echo $SSH_AGENT_PID
  if (! $?SSH_AGENT_PID ) then
    # echo "No $tmpFile, starting new agent…"
    $sshAgent $sshAgentArgs | head -2 >! $tmpFile
    source $tmpFile
    echo "ssh agent started [${SSH_AGENT_PID}]"
    ssh-add
else
  # the tmpfile was present with data, check it…
  # echo "Found $tmpFile, check data"
  set this=`ps -elf | grep ${SSH_AGENT_PID} | grep ssh-agent`
  # start ssh-agent if status is nonzero
  if (( $? != 0 ) && ( -x "$sshAgent" )) then
    # tmpFile exists, but stale data
    $sshAgent $sshAgentArgs | head -2 >! $tmpFile
    source $tmpFile
    echo "ssh agent started [${SSH_AGENT_PID}]"
    ssh-add
  else
    # Agent running, ensure a key is present
    set sa_data=`ssh-add -l`
    if ( $? != 0 ) then
      # need to add key
      ssh-add
    endif
  endif
endif

The original script for exiting the session would kill the SSH Agent outright. This is not so useful if you still have other login sessions running. So I wrote a few lines to attempt to ensure that only when the last running login was being exited, would the SSH Agent be reaped. This code goes in the user’s .logout file in their home directory:

set tmpFile=~/.ssh/ssh-agent-info
set sessCount=`w | grep $user | wc -l`
if ( $sessCount == 1 ) then
  # last user, clear out the ssh-agent
  eval `ssh-agent -c -k`
  /bin/rm $tmpFile
endif

And the solution works. The developers are minimally unhappy about the increased security wrapped around access to the code base, because they know I worked to make it as painless as possible while meeting policy requirements. It could always be more robust, but I tried to get all the common failure cases, and mostly the resolution to something I didn’t catch is for the user to log out of all of their sessions, then log back in again to reset.

Winding Down

Our condolences to the family and friends of 1st Lt. Trevarius Ravon Bowman, 25, from Spartanburg, South Carolina, who died on May 19, 2020, in Bagram Air Force Base, Afghanistan, from a non-combat-related incident.

Our hearts go out the families who have lost loved ones to COVID-19. Y’all, socially distance, wear a mask when you need to (indoors around other people for sure), and WASH YOUR DAMNED HANDS way more often.

Please don’t be one of the people who want to go into retail (or any other) businesses without a mask. Don’t be one of the people trying to up your chances of being DEAD by the time the General Election rolls around. Even if you don’t care for yourself, do you want to get ill, pass it on to an elderly parent (or a young child) and have them DIE because you’re behaving like a petulant child yourself? No, no, I understand that your role model in this case does indeed behave much like a petulant child. Don’t follow that particular lemming off the cliff. Stop. Think. Listen to medical professionals and keep safe.

I love you ALL.

20 April 2020

Fun with WordPress

Note – this is a discussion and solution for a technical problem for a WordPress instance that uses an SSL certificate signed by a non-public CA. If you don’t care about this sort of thing, please move your eyes down to the next section.

The error text that I saw in the new-to-me Site Health page following upgrading to WordPress 5.4:

cURL error 60: SSL certificate problem: unable to get local issuer certificate

The error above was generated because WordPress/PHP couldn’t verify the site certificate. When this is broken, the impact can be significant on a WordPress instance. Some features just don’t work quite right. Auto updating can fail, and so on.

The context here is that for a variety of internal and external sites, I use site-specific SSL certificates that are signed by our internal CA. That’s a good thing, because prior to Let’s Encrypt, it was easy to spend a bunch of money on SSL certificates from a reputable source. We won’t discuss the non-reputable sources. Since I’m using an external resource for caching and web app firewalling, I am able to use the internally signed certificate for several external sites as well.

With the most recent update adding Site Health as a core feature, this error surfaced for me on a couple of sites. It took a couple of hours and some false starts before I found this solution.

In the WordPress file tree, there’s a file at wp-includes/certificates/ca-bundle.crt (using UNIX-style slashes). This is the file of CA certificates that WordPress and the PHP functions use to verify a certificate is valid. Tryijg to get WordPress and PHP to use the system CA certs file (which has my Root Certificate added as a trust source) was a non-starter, although I tried. So I copied the text of my Internal Root Certificate into thewp-includes/certificates/ca-bundle.crt file. Boom! Problem solved … for now.

The downside of this solution is that any given WordPress update in the future may (will?) overwrite that file with newer info, and will once again exclude my Internal Root Certificate. So I created a text file that contained an identifying header string and the Internal Root Certificate. I then wrote a shell script to check thewp-includes/certificates/ca-bundle.crt for that header string, and if not found, adds the content of the text file to the ca-bundle.crt file. That shell script runs once a day in the wee hours of the morning.

Now, anytime there’s a WordPress update that overwrites ca-bundle.crt, by the next morning, the Internal Root CA certificate will be back in place, and things will continue humming along nicely.

Staying at Home

We continue to stay at home, which is a good thing.

I’ll ask you to determine for yourself if it’s a good thing that some people who, for reasons of politics, mistrust etc., continue to gather in groups, putting themselves and their loved ones at heightened risk of severe illness and death. I personally would rather that people be sane and safe. But bailing any water at all from the deeply stupid side of the gene pool can only be for the good of humanity, in the long term.

I didn’t do any yardwork this weekend. We did a number of other inside chores, including re-loading shelves and such after dealing with a multi-phased ant invasion.

Additionally, on the yardwork front, I will point out that planting veggies HAS brought the usual effects on to our region: We had two overnight frosts in the last week, and we’re due for one more on Tuesday night. I’ve been tarping the veggie beds for those events, and so far haven’t lost plants to them.

Happy Dog

While I was dealing with a training event late last week, I ran across the first picture we took of Lexi on her gotcha date in March 27, 2010:

Our first picture of Lexi the chipuggle mutt, taken on March 27, 2010.
Lexi’s First Photo Op

Winding Down

Nothing particular to report here. Be well, okay?

6 October 2019

Continuation

Yep, it’s been two weeks since last time. A week ago Saturday we went out to Centennial Lake and spent the morning fishing. I caught one small bass, 13-14″ long:

A small bass (fish), caught on Sept. 28, 2019 at Centennial Lake in Maryland
A small bass, caught on Sept. 28, 2019 at Centennial Lake

I’ve also been doing a lot of reading: Clarkesworld, Forever Magazine, Alan Dean Foster’s Icerigger Trilogy, Neil Gaiman’s Don’t Panic, and not least: everything I can reasonably lay hands on by Martha Wells and Robert J. Sawyer in advance of this year’s Capclave (where Wells and Sawyer are GoH!!!).

This weekend I managed a fair bit of yardwork out front – cleaning out the beds and preparing to winterize the whole joint. Fall is here, overnight temps are going to be regularly in the low 50’s and high 40’s now.

Computing

I’ve been spending a fair bit of time diving into the depths of Red Hat’s latest Enterprise Linux offering, and wrapping my head around more effective automation using Ansible. Good fun, but after a fairly static run of years (in terms of system management), a lot of things have changed under the hood, and there’s a bunch to learn. So I’m reading a bunch of manual pages to supplement the formal documentation.

All the while, I’m still running FreeBSD, Ubuntu (on PC and ARM hardware), CentOS, Windows 10, and MacOS. So I keep all the mad skillz sharp by changing up platforms several times a day. Assume nothing!

Winding Down

No new casualties were reported by DoD in the last two weeks.

22 September 2019

Last of Summer

According to the calendar, tomorrow is Fall. According to the forecast, Summer looks to be in an extended run. We did have two nights last week drop into the high 40’s overnight – but highs for the next 10 days are supposed to be in the mid- to high 80’s. So, summer, still. Unless, y’know, the forecasters are wrong. That would be so unusual!

Mysterious Absence

So, yeah … no post last week. I’ve got no explanation but sheer laziness. This weekend was a bit more of the same. We did make it out fishing at Cash Lake yesterday morning for a few hours. No fish, of course. But relaxing right up until I cast while twisting my torso. That gave me a muscular tweak under the lower left scapula that continues to vex me.

Reading

My read of the last week is the biographical Don’t Panic: Douglas Adams & the Hitchhiker’s Guide to the Galaxy, by some bloke named Neil Gaiman. At the moment, it’s free for $AMZN Prime members. And it’s a jolly good read, full of stuff I knew about Adams and H2G2, and MUCH other stuff that I didn’t. Fun.

On dead trees, I’m most of the way through a long-delayed re-reading of Cliff Stoll’s The Cuckoo’s Egg. Good stuff if you like 80’s computers and tracking down a cracker with a penchant for military computers that have crappy security policies applied to them.

Winding down

Our condolences to the family and friends of Sgt. 1st Class Jeremy W. Griffin, 40, from Greenbrier, Tennessee, who was killed in action on Sept. 16, 2019, by small arms fire when his unit was engaged in combat operations in Wardak Province, Afghanistan.

11 August 2019

Better Than The Alternative

Still busy, that is, and that’s better than the alternative. Yesterday was given over mostly to food work – a new big batch of salsa, with only tomatoes out of the garden. Everything else came from the store, sadly. I roasted a pountd of coffee. I also made a batch of my potato/spicy sausage concoction, 10 meals worth. That’ll see me through the week.

Today: shopping, clean the shower, patching day for some systems at work, lunch, and lawn. And I’m a bit whacked. Time to get back to the office tomorrow, and spend the week recharging for the next weekend’s worth of chores.

OS News

On the computing front, I’m starting to move on to Red Hat Enterprise Linux version 8 (RHEL8). There don’t seem to be the same underlying massive changes that version 7 brought – no amount of sheer disruption similar to that brought by systemd. Note about that – I can (and must) live with it. There are even features that I shudder to say that I like. But it breaks the UNIX “small tools doing things well” paradigm so very, very much. Anyway, back to RHEL8 – it has a better built-in system for keeping optional subsystems — from the Apache webserver, to PHP, to Python, etc., etc. — more current and easier to work with. I have work to do…

Reading

I just finished up Charlie Jane Anders’ All The Birds In The Sky – it’s a complex story that includes some comfortable old ideas, just to suck you into the story, but then it gets weird (in a good way). Magic vs. technology, and not in a good way. Some of the sub-plots resolve precisely as telegraphed, but I didn’t see that ending coming, so … good? Recommended.

In other venues, I’m waiting to see how my voting aligned with the Hugo Awards. Tick, tock.

Winding Down

No new news from DoD – good. News from most of the rest of the world: floating between weird and sucks. But Al Yankovic is touring, so something is going right.

7 August 2019

Busy days

I’m feeling a bit broken, with the shootings of recent days, and the inflammatory rhetoric of the GOP and their Beloved Leader. Sorry.

The garden is two thirds dead. No peppers. No zucchini. But the tomatoes, they are in full production. We had salsa this weekend, and I’m going to make more tomorrow night, since Marcia pulled out a bushel of red fruit today.

I did manage to find time to roast some coffee the other night. Here I’m pre-heating the roaster before doing the actual deed…

Pictured: In the background, my @behmor 1600+ coffee roaster, pre-heating (the elements are glowing red). In the foreground, a bag containing the remaining pound of Colombian green coffee beans from @sweetmarias, about to be roasted.
Pre-heating before roasting the coffee…

We’re drinking that coffee, and it is super tasty.

I’ve got a new home firewall I’m ready to install and test out, except that brilliantly, I no longer have any VGA cables here at home. I disposed of the last one a couple of years ago, because … why not, I’m not going to have any more gear that doesn’t have HDMI or DisplayPort or … this firewall appliance that only has a VGA output. Sigh.

To be honest, it also has a COM port, and with some little effort I could manage to get a system with a COM port running around here. But the main home server these days is an Intel NUC, with no COM ports. The Macs … no COM ports. I have gear with COM ports, but they’re running Windows, and I just can’t be bothered. I’ll snag a spare VGA cable from work, use it to do what’s needed, then store it with a BIG note saying KEEP ME, YOU REMEMBER WHY!

Winding Down

Our condolences to the friends and families of these fallen warriors:

  • Aviation Electronics Technician 2nd Class Slayton Saldana, 24, had been listed Duty Status Whereabouts Unknown (DUSTWUN) following a July 17 non-combat, man overboard incident while underway onboard USS Abraham Lincoln (CVN 72) in the Arabian Sea.
  • Pfc. Brandon Jay Kreischer, 20, of Stryker, Ohio, died on July 29, 2019, in Tarin Kowt, Uruzgan Province, Afghanistan, as a result of wounds sustained in a combat related incident.
  • Spc. Michael Isaiah Nance, 24, of Chicago, Illinois, died on July 29, 2019, in Tarin Kowt, Uruzgan Province, Afghanistan, as a result of wounds sustained in a combat related incident.

9 June 2019

Hullo

Yes, I know. But me IRL has a lot going on. I’ve been doing a lot of home maintenance work lately, and not a lot of energy for things Internet-ish. Last weekend, along with the never-ending yardwork of Spring through Autumn, I managed to get nearly all of the bi-annual pressure washing done. This weekend, I did some sanding and started painting front trim. I got perhaps 60% of the way through. I might have finished today, but exterior painting and rain rarely mix well (or rather, mix far too well!).

So that’s waiting for another day. I have similar work to do on the back of the house, but that will be done over the next few weeks.

RHEL 8

In what little spare time I do have, I’ve started exploring the recently released next version of Red Hat Enterprise Linux. They’ve done some interesting things to manage the fact that we want the operating system to be stable yet secure, for a long, long time… and we also want the most secure, feature-full, and latest tools available to us to support modern workloads. Just for example, until recently, it was fairly difficult to get a supported recent version of PHP that would run on Red Hat. So, I’ve more to learn, which is always a good thing.

Entertainment

We’ve been watching Good Omens on Amazon Prime. Highly Recommended.

Winding down

DoD announced no new casualties in the last couple of weeks. Now, to rest…

24 March 2019

Sad News

My friend Mark Camack died this last week, after a battle with throat cancer. We’ve been in the exchange holiday cards / biannual phone call place for the last couple of decades, but I’d still have jumped up and headed out if he needed help. I didn’t hear about this until his wife Bonnie got in touch, the other day. My heart goes out to Bonnie and their extended families. Rest in peace, my friend.

New Beginnings, Old Endings

The big news is that I migrated all of the personal sites I manage from an old server to a new server. Not super-exciting from an external perspective, but I did manage to separate the WordPress instances from being embedded in the old sites far too deeply. We *should* have had them be something like blog.orbdesigns.com, but then the site was already a blog, just pre-dating WordPress. So, anyway, www.mumble is the wordpress site for this place, and for Marcia’s two sites. The older, more static sites are more easily accessible via legacy URLs, for example legacy.orbdesigns.com.

One of the links on the legacy site that I clicked in testing was from December 28, 2009. And that was the day we said goodbye to Lucy, our cocker spaniel. So, “Old Endings.” There’s plenty of stuff there in legacy land, from . I’m probably going to fix just a couple of top-level internal links that will make the site work better.

I’m also changing horses on the two-factor authentication tools I’m using, both on the device and on the sites. So, lots of behind-the-scenes technology updates. Drop me an email, or comment here if you find something that’s so deeply broken that I absolutely must fix it. Of course, I may choose to leave something broken, but that’s another story.

Winding down

I’ve got a busy week in front of me, so pardon if I’m even less loquacious than usual.

Our condolences to the families and friends of these fallen warriors:

  • Spc. Joseph P. Collette, 29, of Lancaster, Ohio, died on March 22, 2019, in Kunduz Province, Afghanistan, as a result of wounds sustained while engaged in combat operations.
  • Sgt. 1st Class Will D. Lindsay, 33, of Cortez, Colorado, died on March 22, 2019, in Kunduz Province, Afghanistan, as a result of wounds sustained while engaged in combat operations.

24 February 2019

Farewell

Our family said their goodbyes to Dennis yesterday. He’s resting by his grandparents now. Sadly, I didn’t know him as well as I’d have liked, and we didn’t make the trip for his memorial, because it was a small affair, and nobody needed out-of-towners to deal with on top of everything else. He’ll be missed, rest in peace.

Hardware Woes

Work was a week like so many others, not worth remarking on. However, on the home front, the computing environment continues to have drastic changes.

Yesterday, I pulled the final backup from my FreeBSD 11 server, Serenity. After that, I shut it down. Later in the evening, I went to spin it up in prep for putting VMware ESXi 6.7 on the hardware. But it stayed dark. Hmmm. Power supply bad? Dunno. Left that for today.

Today: Same thing. Swapped power supply to my backup Antec. Still no joy. Plopped in the emergency Corsair PS. Still no joy. Okay. the hardware is actually dead. This chassis has seen 3 or 4 motherboards, several different Linuxen, FreeBSD 10 and 11. It’s been the home server and primary home workstation for a long time. But lately it’s ONLY been the home server, which is asking not very much for a server motherboard with a Xeon and ECC RAM. And I was burning 100W continuously keeping that system online. So it was time for a re-purpose.

The little NUC box is now running FreeBSD 12, disk mirrored on a couple of 1TB drives. I’ve got too much RAM in it, 32G, because I planned on something different. Best part: 11W draw in normal operations. So, a good home server.

But it isn’t a good VMware host for a couple of reasons. So I wanted to use the Xeon box for that. Nope. I’ve got another NUC8 Performance on order, and that’ll be the VMware box.

Winding Down

Not much to promote – we’ve not got a show at Annapolis Shakespeare until next weekend. And I’ve not had much time for reading. Pretty boring, outside of my normal tech routine.

DoD announced no new casualties in the last week.

Ciao!

17 February 2019

Sheepishly

Yet utterly unlike a sheep. Last Sunday, we were out at Cabaret Night at Annapolis Shakespeare. It was song and dance and showtunes from the 50’s, and the joint was rockin’! So much fun I forgot to post. And as I remembered, each ensuing day, well, I was at work, not relaxing at home, so I kicked the can down the road and here we are. The cabaret was wonderful, as always. Up next at the theater, another Cabaret Night, 60’s this time, and then it’s time for Pride and Predjudice to open (runs for 6 weeks starting on March 15). So good, so looking forward to the work!

Computationally

I’ve been building out new infrastructure for this joint, and for the home server. FreeBSD 12 has been out for a while, and I wanted to upgrade at a reasonable pace, rather than waiting for the 11 branch to go out of support. Additionally, unlike the transition from 10 to 11, I’m not upgrading these systems, but building out fresh, and doing clean configurations of the services I really need (and leaving every prior experiment behind).

Additionally, for home backups, I’m migrating to using the Free edition of the Veeam Agent for Windows. I’m a big fan of Veeam, and there are features in the free agent that are perfect for my needs.

First, the backups can be encrypted. Do that. Keep the credentials in a password manager software, use a good passphrase, and secure your backups. Why? Because you plug in the disk, run the backup, unplug it and take it offsite (like to work, or keep it in the glove box of your car). Because the backup might be in a less-than-secure environment, encrypt that backup.

Secondly, by default Veeam runs backups based upon changed blocks in the volume, rather than changed files. So if you edit a few bytes of a huge document, you may only have to back up a few tens of kilobytes instead of the whole file.

Finally (for me, there are lots of features), while Veeam defaults to configuring a job with a calendar schedule, that doesn’t make much sense when you’re backing up to media (say, a 1TB USB3 mobile disk) that only gets plugged in when you remember to bring it home. The answer is to change the job schedule to simply run automatically when the media is plugged in, then eject the media when the backup is done. That’s about the best you’ll be able to do. Why is this important? If you click on a malicious link in an email, and your AV solution lets you down, some jerk’s ransomware will encrypt or wipe every file on your system. Your backup won’t help you IF it’s also attached to the system, encrypted or not. Only attach your backup disks for as long as you need to run the backup.

One last point – every once in a while, plug up the backup disk, and restore a file or two and check them. Backups are wonderful, but you don’t know if they’re good unless you test them, regularly. You don’t want an emergency restore to be the time you find out that the backup wasn’t really working because you misconfigured the job, or the disk has errors, or whatever.

Bottom line: Make backups. Encrypt the backups. Test the backups. Only leave them connected for as long as needed for backups and testing. Store your backups offsite – a disaster that takes your house shouldn’t take your precious data, too!

Winding Down

It’s been a slightly busy weekend – we celebrated Marcia’s birthday on Friday, so I took off work that day. We hung out, did stuff together, and went out to supper in the evening. Saturday was a total write-off. I relaxed. Today was shopping and remote work and haircut and coffee roasting, oh my! Tomorrow’s a holiday, so it’s a four day weekend, but I have an office that needs cleaning, yet.

DoD announced no new casualties in the last two weeks.