Author Archives: bilborg

LISA 15 Report

The LISA 2015 conference was held this year at the Washington Marriott Wardman Park, off Connecticut Avenue in north east DC. It’s 15 miles from home, but the best driving time I had was Wednesday (Veteran’s Day) morning, which took half an hour, and the worst was a bit over 1.5 hours, coming home in weeknight traffic, in the rain. It’s a nice venue, though I’ve never stayed there, only attended events.

Saturday, 11/7

Saturday night was badge pickup and opening reception. I attended that mostly to do a handoff of the give-away items for the LOPSA general business meeting. Because I’m local, I volunteered to be a drop ship site for stuff that arrived over the course of the month leading up to LISA. That evening, I made contact with LOPSA’s President, Chris Kacoroski (‘Ski’), and we grabbed a couple of other willing bodies and emptied out my trunk, which was chock-full of Lego kits, books, booth collateral, etc. An hour or two of chatting with early-arriving attendees, then I headed back home to get an early bedtime – I was facing a long week.

Sunday, 11/8

Sunday was the first of three consecutive days of tutorials. In the morning, I attended a half-day session presented by Chris McEniry on the topic of Go for Sysadmins. Go was developed at Google, and released under an open source license in 2009. To my eye, it combines some of the best features of C, Python, and Java (but the FAQ says that Pascal has a strong influence – it’s been a long, long time). With larger data sets to work with each passing year, a faster and better language seems to be a useful tool for the continuously learning system administrator, and Go provides that sort of tool. Chris was an excellent presenter, and his examples and supporting code were pertinent and useful. Effective? Yep, I want to learn more about Go … in my copious spare time.

Sunday afternoon was all about Software Testing for Sysadmin Programs, presented by someone I’ve known for a few years now, Adam Moskowitz. Adam is a pleasant bloke, and like everyone at LISA, smart as all get out. He makes the valid point that all of the tools that we encourage our programmers to use, from version control to testing and deployment automation, belong in our toolbox as well. And for UNIX-ish sysadmins, lots of stuff is written in shell. Adam developed a suite of tools based on Maven, Groovy, and Spock, and gave us a working configuration to test code with. Impressive and useful. Now all I have to do is do it!

In the evening, I hung out for a bit for what’s called the “Hallway Track”, which is all of the non-programmed activities from games to BoF (Birds of a Feather) sessions, to conversations about employers, recruiting, tools, and users. Always fulfilling, the hallway track.

Monday 11/9

On Monday, I over-committed myself. Caskey L. Dickson was putting on a full-day tutorial on Operating System Internals for Administrators (a shortened version of the actual title). I attended the morning session of that, which was awesome. One would suspect that hardware is so fast that it just doesn’t matter so much anymore. But it turns out that such things as memory affinity in multi-socket, multi-core systems can have significant performance impacts if the load isn’t planned well. And while storage is getting faster, so are busses and networks. The bottlenecks keep moving around and we can’t count on knowing what to fix without proper metrics. Caskey presents an excellent tutorial, it’s actually in some senses a pre-requisite for  the Linux Performance Tuning tutorial that Ted Ts’o does (I’ve attended that in years past). I would have stuck around for the second half day of Internals, but…

Instead, I attended a half-day tutorial  called systemd, the Next-Generation Linux System Manager. Presented by Alison Chaiken, I learned a lot about the latest generation of system manager software that’s taken over from the System V init scripts model that’s ruled for the last few decades. While change is always a PITA, and there are definitely people who vehemently dislike systemd, I find that (A) I have to use it in my work, so I should learn more; and (B) there are features that I really quite like. Alison knows a lot about the software and the subject, and helped me understand where I needed to fill in the gaps in my systemd education.

Tuesday 11/10

For me, Tuesday was all about Docker. Until not that long ago, I’d have been managing one service (or suite of services) on a given piece of hardware. Programs ran on the Operating System, which ran on the hardware, which sat in the rack in the data center, mostly idle but with bursts of activity. Always burning electricity, and needing cooling, a growing workload meant adding new racks, more cooling, more electric capacity. In the last decade, virtualization has taken the data center by storm. Where once a rack full of 2U servers (2U stands for the vertical space that the server takes up in the rack – most racks have 42 U {units} of space, and servers most commonly are 1, 2 or 4 U) sat mostly idling, we now have a single more powerful 2U or 4U server that runs software like VMware’s ESXi hypervisor, Microsoft’s Hyper-V, or Xen/KVM running on a Linux host. On “top” of those hypervisors, multiple Operating System installs are running, each providing their service(s) and at much higher density. Today’s high-end 2U server can provision as much compute capacity as a couple of racks worth of servers from 5-10 years ago. It’s awesome.

But that’s so … yesterday. Today, the new hotness is containers, and Docker is the big player in containers right now. The premise is that running a whole copy of the OS just to run a service seems silly. Why not have a “container” that just has the software  and configurations needed to provide the service, and have multiple containers running on a single OS instance, physical or virtualized. The density of services provided can go up by a factor of 10 or more, using containers. It’s the new awesome!

I don’t have to use Docker or containers in my current situation, but that day may come, and for once I’d like to be ahead of the curve. So in the morning, I attended Introduction to Docker and Containers, presented by Jerome Petazzoni, of Docker. Dude seriously knows his stuff. But I’ve never attended a half-day tutorial that had more than 250 slides before, and he got through more than 220 of them in the time at hand, while ALSO showing some quick demos. Amazingly, I wasn’t lost at the time. And I’ve got a copy so that I can go back through at my leisure. Containers launch quickly, just like Jerome’s tutorial. I think I learned a lot. But it’s still due for unpacking in my brain.

In the afternoon, Jerome continued with Advanced Docker Concepts and Container Orchestration. Tools now regarded as stable (such as Swarm, which reached the 1.0 milestone a couple of weeks before the presentation) (grin) and Docker Compose were discussed and demonstrated to show how to manage scaling up and out. Another immense info dump, but I’m grateful I attended these tutorials. I think I learned a lot.

In the evening, I hit up the Storage BoF put on by Cambridge Computers, and dropped into the Red Hat vendor BoF on the topic of Open Storage. A long day.

Wednesday, 11/11

Veteran’s Day dawned bright and sunny. Like each day of this week, I left the house at 0630. I was surprised, rolling into the parking garage at 0700 … until I remembered the holiday, and that no Feds were working (and clogging my drive) as a result. Win!

The morning keynote was given by Mikey Dickerson, head of the USDS. He spoke on the challenges of healthcare.gov (his first Federal engagement), and being called back to head up the new US Digital Service. Mikey is a neat, genuine guy who has assembled a team of technologists who are making a difference in government services. Excellent keynote, fun guy.

I took a hallway track break for the next hour and a half – catching up with folks I hadn’t seen in a couple of years.

After lunch, I attended first a talk by George Wilson on current state of the art for OpenZFS. ZFS is an awesome filesystem that was built by Sun (Yay!), then closed by Oracle (Boo!). OpenZFS took off as a fork of the last OpenSolaris release, some years ago. Since then it’s been at the core of IllumOS and other OpenSolaris-derived operating systems, as well as FreeBSD and other projects. I’m a huge fan of ZFS, and it’s always good to learn more about successes, progress, and pitfalls.

Then I sat in on Nicole Forsgren’s talk: My First Year at Chef: Measuring All the Things. Nicole is a smart, smart person, and left a tenure-track position to join Chef last year. She brought her observational super-powers and statistics-fu to bear on all the previously unmeasured things at Chef, and learned lots. Chef let her tell us (most of) what she learned, which is also awesome. The key take-away: Learn how to measure things, set goals, and measure progress. Excellent!

After dinner up the street at Zoo Bar and Grill with Chas and Peter, I attended the annual LOPSA business meeting. I didn’t stay for the LOPSA BoF in the bar upstairs, since my steam was running out and I was driving, not staying at the hotel.

Thursday, 11/12

Christopher Soghoian provided the frankly depressing Thursday morning keynote: Sysadmins and Their Role in Cyberwar: Why Several Governments Want to Spy on and Hack You, Even If You Have Nothing to Hide. Seriously. Chris is the Chief Technologist for the ACLU, and his “war” stories are hair-raising. We’re all targets, because we run systems that might let the (good|bad|huh?) guys get to other people. All admins are targets, not of opportunity, but of collateral access. Sigh. Sigh. Good talk, wish it wasn’t needed.

The morning talk I attended was about Sysdig, using it to monitor cloud and container environments. Presented by Gianluca Borello, I found that sysdig is a tool I really should learn more about.

In the afternoon, I spent some time in the Vendor Expo area, catching up with people and learning about the products that they think are important to my demographic. I was going to attend a mini-tutorial later in the afternoon called Git, Got, Gotten on using git for sysadmin version control … but by the time I got to the room it was SRO. So I bailed out way early (skipping the in-hotel conference evening reception – I expected a disappointment following last year’s wonderful event at the EMP Museum), unwound, and got a good night’s sleep.

Friday, 11/13

I started the day with Jez Humble of Chef, who talked to the big room about Lean Configuration Management. An excellent talk on, among other things, what tools from the Dev side of the aisle we can use on the Ops side. Jez is an excellent speaker, and he brings up a good point about how the data points to high-performing IT groups as being a driver of innovation AND profit.

My second morning session was Lightweight Change Control Using Git, by George Beech of Stack Overflow. A big hunk of time was given to what’s wrong, before progressing into the organization of managing configs and processes with version control, explicitly git. Good talk.

After lunch, I spent a couple of hours on the hallway track, since there was nothing that really called out my name in the formal program. And for the closing keynote … well, I decided to beat the Friday traffic out of the district instead. But the presentation has been made available already – it’s here: It Was Never Going to Work, So Let’s Have Some Tea, by James Mickens of Harvard. You can watch it with me.

Thanksgiving and stuff

It was a good week, though I did work on Friday. Thanksgiving Day was a nice quiet day at home. Pancakes and espresso in the morning. Turkey, mashed potatoes, gravy, cranberry sauce, apple pie, … other stuff, I think … through the late afternoon and evening. Food coma #FTW, with lots of leftovers. We called and talked to family in lots of places, and that was fun, too. The weekend has been catching up on chores, putting up the Christmas crap, and roasting coffee.

Fallen Warriors

DoD reported no new casualties in the last week.