(Upgrade) Times are changing…..

Stop me if you’ve heard this one before. It’s Monday morning, you open up your newsreader and there are 25 different articles about an exploit that has been found that is sweeping the net. It affects nearly 90% of systems out there. You know it’s only a matter of time until this news goes from being on the tech sites only to the Wall Street Journal and The New York Times. Once that happens, the alert level hits red. Now all of your C-Level execs are aware of the problem and someone is going to be calling you asking for a status update. If you’re Peter Gibbons, you may even have 8 different people calling you. Where do you go from here? In the old days, this would mean, any plans you had for that weekend were scrapped. You’d now have to coordinate outages with your application teams, IT staff, sometimes you’d even have to get your building’s security team involved. You’d also have to break the news to your wife, husband, girlfriend, boyfriend, kids, or whoever that you may not see them again until Tuesday (assuming that all goes well). Then you get to go through this scenario:
  • Planning and executing the downing all of your affected DEV/TEST systems.
  • Preemptively opening cases with your vendors in case you run into an issue (you would hate to get stuck in the queue without a case number while your systems are down)
  • Downloading and applying the patches to fix the vulnerability.
  • Bringing all of said systems back up and running.
  • Contacting all of your applications owners once the systems are back up and having them test all of the applications.
  • Squeeze in a phone call to your loved ones asking about how life is on the outside.
  • Notifying all of your users that the systems are back up and running and that now regular weekend work can commence.
  • Once all of this is done, and you’ve verified that everything is OK and there are no issues, you can now plan to do the same thing to your Production systems. YAY! That usually means another weekend down the toilet.
Many times, some of the pain involved with this type of maintenance can be lessened through mechanisms like vMotion, Exchange DAGs, and clustered systems in general. Typically, you patch each of the secondary nodes in the cluster, then you patch the primary node and you’re good to go. This process of upgrading different cluster nodes can take hours depending on the size of your environment and requires total concentration and focus. If you run into an issue during a failover, you’ll be happy you opened that support case.
Why do I bring all of this up? Traditionally, the one system that usually has the biggest issues during this kind of upgrade/update scenario is your storage environment. Especially if you are on legacy storage for one reason or another. In most cases that I have seen, storage code upgrades are completely ignored unless absolutely necessary. I can see why people make that argument. If your storage goes down, especially in a small to medium sized shop, EVERYTHING goes down. This scares the pants off of a lot people, with good reason. They would rather take the “If it ain’t broke, don’t fix it.” approach of yesteryear. Nobody wants to run into those kinds of problems and lose their weekends because of storage issues. This kind of thinking leads to rolling the dice and hoping that the storage environment will just keep on chugging along and that no one will exploit the vulnerabilities that are out there. I think this model is changing in storage though, along the same lines that the break/fix mentality was replaced with a proactive approach. IT departments are getting more sophisticated and are looking to get everything patched and protected BEFORE someone tries to exploit the vulnerabilities. 
What if you, the IT engineer, could avoid those sleeping in the office kind of issues and get your weekends back? Who would say no to that? As I’ve written about in the past, I’ve been a customer of Pure Storage for about two and a half years now. I started out on an FA-320 array, I’m currently using their FA-400 series and I’m getting ready to start playing with the FlashArray//m as soon as it arrives. One of the things that sold me on Pure Storage was the Non-Disruptive Upgrade (NDU) capabilities for both the software and the hardware of the array (you can see a demo of their NDU here). I’ve gone through almost every iteration imaginable. I’ve done code upgrades (both minor and major revisions), I’ve added additional shelves of disk, I’ve gone from 300 series to 400 series controllers, you name it and I probably done it. The one similarity in every upgrade was that it happened like they said it would happen. No downtime, no performance degradation, no idea that it was happening from a user perspective. They were all quick, seamless, and pain free. They also happened during the week (we played it safe and did them on Friday evenings for our Production units) but on Saturday morning I was home playing with my little boy which is what I care about most.
As I said earlier, this approach appears to be the new status quo. Many other vendors besides Pure Storage are trying to follow suit. EMC has stated that they now support NDU’s (although I’m not sure that is the case for different hardware versions). Other vendors such as Solid Fire and Nimble also support NDU’s. This is a direction that I think everyone in IT welcomes. Being able to provide services quickly to the end user without disturbing their workflow is the goal of nearly every IT staff. This new model greatly increases the success rate of achieving that goal. Pure Storage has gone one step further and changed the typical storage lifecycle model around this principle when they launched Evergreen Storage. The belief is that forklift upgrades will go the way of the dodo bird and you can just replace individual components when needed. Your maintenance never increases (unless you add capacity). Your storage system can stay the same for as long as you need it too saving you tons of money in the long run while also providing you with a solid foundation to house your infrastructure on.
If other systems start following suit and rethink how we look at system lifecycles, the end result can be great for IT Admins. What if it was as easy to upgrade the code on your core switches and routers as it is to upgrade an app on your iPhone? What if said code could be upgraded FROM your iPhone while you’re sipping margaritas on a beach somewhere (just don’t drink too many until the upgrade is done)? What if upgrading your email servers wasn’t a 6 month project? Whether it’s PC refreshes, server upgrades, or application upgrades, a pain-free process is something everyone would welcome and what we currently strive for as IT pros. It’s nice to see that not only can we make end users’ lives easier, I think it’s time that we make our own lives easier as well. Don’t we as IT admins deserve the same level of happiness and time away from the office as our users do? I sure think so. I think you all would agree with me. It’s nice to see that vendors like Pure Storage share that same vision and are doing something to achieve it.

How To Be An Awesome IT Professional

I bet a lot of you are reading this and saying to yourself, “I’m already an awesome IT Professional!” You know what? You’re right! The fact that you would read something to try and better yourself even though you’re already awesome is one of the many things that makes you awesome. For some of the newbies to the field or the ones that may find themselves in a rut, or even those that just want to get to the next level of their careers, this guide is for you.
What makes an awesome IT Professional? How can you spot one? How can you become one? How can you tell if you’re going in the other direction? Here are some of my tips for being an awesome IT Professional.
I’m not saying you need to be a nag or that annoying telemarketer from Sirius XM who keeps calling you to renew the free subscription you got when you bought your car. Being relentless is about looking at your career as a lifelong quest to improve yourself. Keep on learning. Read blogs. Build home labs. Take and pass certification exams. Join user groups. Do whatever you can to learn everything about your field. I look at it this way, whenever someone on my team has a question or can’t figure out how to do something, I want them coming to me first for an answer. Better yet, I want to have an answer to give them. No one has all the answers but that doesn’t mean you shouldn’t try to learn as many of them as you possibly can. Try to be the best you can be both professionally and personally. Never stop pursuing your goals. It’s never too late to get there.
I can’t emphasize this one enough. You don’t need to be super anal but you need to be close to it. Document everything that you can. If you’re rolling out a new application, have all of the IP’s and spreadsheets done before hand so you’re just reading off a prepared list. When an issue comes up you don’t want to be fumbling for information. You want it to be easy to find and intuitive. Uniformity in your environment goes a long way in this regard. If you have multiple sites and your gateway has a similar IP at each location it takes some of the guesswork out of troubleshooting. It also makes things easier for new employees to get up to speed if your environment is set up logically. Use tags, use Organizational Units, put descriptions on your router interfaces. It may be more work up front but in the long run, it will help you exponentially. This goes for your day to day work too. Organize your email into folders, use contacts, do all of the things that you know you SHOULD do but probably aren’t. Keep notes when issues arrive so you have something to refer back to if it pops up again. The best IT Professionals do the majority if not all of these things.
“Be like water making its way through cracks. Do not be assertive, but adjust to the object, and you shall find a way around or through it. If nothing within you stays rigid, outward things will disclose themselves.
Empty your mind, be formless. Shapeless, like water. If you put water into a cup, it becomes the cup. You put water into a bottle and it becomes the bottle. You put it in a teapot, it becomes the teapot. Now, water can flow or it can crash. Be water, my friend.”
― Bruce Lee
Those of you who get the reference probably know where I’m going with this. This has been a common theme in my life. You should be able to adapt to any situation that arises quickly and easily. A similar motto is used in the Marine Corps. Adapt (improvise), and overcome. Most IT Professionals will deal with an end user or customer in one way, shape or form. Every user is different. Every user has a different personality and you’ll have to adjust to the personality that you are faced with. Some are pleasant to deal with, some can be complete and total nightmares. You’re not going to know at first until you are faced with the situation. The same goes for selling to a customer, you’re not going to know how to approach a customer until you hear what their situation is currently and where they are trying to get to. Once you have a clear understanding of what they are dealing with, you’ll better know how to assist. Be water, my friend.
A lot of IT troubleshooting comes down to identifying a problem and resolving it as quickly as you can. A particular issue may come up but have more than one reason for why it is occurring. Take for example a user not having internet access. This could be DNS related, it could be a bad cable, it could be a bad network device, or it could be a problem with the ISP. You’ll need to find a way to go through all of the possible causes and come up with a resolution while keeping an open mind that every situation is different and may require different troubleshooting methodologies. You may not know the answer at first and you may need to do a little detective work to find the solution. How do you know where to look? This comes with experience, every IT Professional has their own way of doing things but one things mostly everyone share is the use of Google. As an IT Professional you will spend countless hours searching through KB articles, blogs and obscure tech websites searching for the one page that someone created after going through the same mess you’re going through. Once you have a good way of troubleshooting any problem can be resolved. Knowing the answer is sometimes less important than being able to find the answer. No one has all the answers.
This one is often overlooked. How many of you know of a super smart IT Professional who always knows what the issue is but they are a complete pain in the ass to deal with? Almost as if you’re being a horrible person for asking them to their job. I’m not going to lie. This was me for a few years there. I was a nightmare. I just wanted to sit in the air conditioned datacenter and pound away at all the project work I could find but I didn’t want to interact with anyone. It may have been all of the years of support that chipped away at me, I don’t know. At the end of the day though, it was counter-productive. You need to build relationships to be successful. Whether it’s with your end users, management, your vendors, or your peers. At some point, you’re going to need help in procuring equipment, a reference, or even a new job. This also goes for managers. You want your employees to be able to come with you with issues, ideas or general concerns while having the feeling that they can have a constructive conversation. Building relationships with those around you will help you find answers in situations where looking for them yourself is not working. It will also grow your network for years and years to come.
Unless you are in the smallest of companies, you work with a team. You all have similar (if not the same goals) so why not pool your resources? If you are strong in a particular area and you know that one of your comrades is not, bring them up to speed. Sit down with them and try to explain as much as you can. Cross train with each other. If you know virtualization and the person next to you knows networking, you can both be two-trick ponies. It will also help give you different perspectives on technologies that you may not have had before. In the end you’re only as strong as your weakest link. If your team is strong, you all look good. If one person isn’t, the same holds true. You ALL look bad. I’ve found them some of the best relationships I have to this day are with people that I worked side by side with for months and months collaborating, sharing ideas, and discussing the task at hand with. Some people out there take the opposite approach and like to horde knowledge and info like a pack rat. What happens when they are not there? If you’re the only one who knows something and you go on vacation, guess who’s getting a phone call on the 4th tee. Yep. No one likes getting work calls on vacation. If everyone on your team is in the loop though, you won’t be getting a phone call because the issue will be resolved before it gets to that. Everybody wins.
As I mentioned earlier, I hit a point a few years back where I wasn’t the greatest person around. I can look back on that time now and realize that I was burned out. I was working crazy hours, I was working on multiple projects and I was re-certifying a bunch of my certifications all at the same time. That translates to not a lot of sleep and a level of irritability seen only in garbage cans on Sesame Street. One of the things that really helped me was when I realized that there are only so many hours in a day. You’re not going to finish everything you set out to do every single day. You need to manage your time wisely and learn how to prioritize your tasks. This will go a long way to ensuring that you are productive while not hitting the wall because you tried to do too much all at once.
These are just a few of the guidelines that I’ve used in my career. If you already do some or all of these things you’re probably already awesome. No matter how great you already are you can always improve. Not just in IT but in life as well. The journey is a marathon, not a sprint. BE AWESOME TODAY!!
I’m sure you all have even more great ideas that I’m forgetting or I just didn’t think of BE COLLABORATIVE and share them with the group. 🙂 As always, comments are welcomed and encouraged. Thanks for reading.