By day, I am a mild mannered (sort of) DBA/DBD (Database Developer). By night, I get to be a volunteer fire fighter for my community. Being a fire fighter is COMPLETELY different from working on databases all day. At the same time, it CAN be almost identical. Let me explain.
Fire fighters are usually in one of two different modes of operation at any given point in time. They are either fighting fire or doing fire prevention for the most part. For this post, I am going to focus on fire fighting and save fire prevention for a follow up post.
I’m sure that you’ve all experienced this. A call comes in, the customer is frantic and demands that you fix a problem (pick one) that you didn’t create. Depending on the issue and your environment, this could be utter chaos. You have been called in to help control a chaotic event. Yet, this is very similar to a true fire emergency, where the fire fighters are called to help control a chaotic event, say a structure fire. Granted, a life probably isn’t in the balance, but someone’s job might be.
So, hotshot, what do you do?
You would probably start to look around the server and ask some questions. Is there a backup? Does my database mirroring help solve the issue? A snap shot? Log shipping maybe? Could a solution be found in my source control? You start to ask questions to determine just how bad of a problem it might be. You do a “scene size-up”. Fire fighters do the same thing. When arriving on scene, they start to investigate and try to determine exactly what the problems are and the best way to tackle them. Usually that’s done by a chief officer who arrives on scene before any engines (depending on how the department operates). They usually do other things, but for the purpose of this post, that’s how they operate.
After the scene size up has been completed, we now know a couple of things:
- How critical the problem is.
- Is it a production issue?
- How large the problem is.
- How many users are affected?
- How many resources we need to address the problem.
- Do we need to contact Iron Mountain to get the tape that holds the backup?
Odds are by this time, we know these answers and probably get to work on getting the
fire out. Now, the issue becomes, what tools do you use to tackle the problem?
select db_name(st.dbid) as 'Database Name' , object_name(st.objectid) as 'Object' , st.text as 'Query' , cp.size_in_bytes as 'Size' FROM sys.dm_exec_cached_plans as cp cross apply sys.dm_exec_sql_text(cp.plan_handle) as st GO
Of course, the tools used by fire fighters are “slightly” different from what a DBA uses, but a tool is a tool regards of where it’s a halligan bar or a TSQL script. The script above is just one of millions of scripts that you can use as a tool. I use this one just to give me a glance as to what SQL queries is in the cache and how big the cache plan is.
The goal of putting out any fire, whether literal or figuratively, is knowing how to use your tools to their maximum potential. What good is using the script above if you don’t understand what your looking at? If you don’t know how to handle a halligan bar or a fire hose, then I doubt that you would have much luck in putting out the fire, especially the latter of the two!
Assuming that you know how to your tools and you get that fire put out, I think that it’s important to reflect on the problem and learn from it. I think just about every issue you run across can help you to become a better DBA (or fire fighter in my case). Review your tools and how you can use them better. Check out the scene and see if there was something that might have prevented the issue in the first place, say a backup perhaps? I will blog later about “fire prevention”.
All in all, I think that it’s interesting how two completely seperate worlds can be similar in so many ways. The next time your on the street and see a fire engine go by running lights and sirens, take a moment and reflect on your last “fire” you had to fight. What tools did you use? What would you do differently?
I leave this post with this. Years ago, my fire chief gave me the best advice during my first year as a fire fighter. “We are all rookies. No fire is the same. Every call is different”. These are words that I took to heart both as a fire fighter and as a DBA. I try my best as a DBA to expect the unexpected and think about 10 steps ahead. Sometimes that works and sometimes it bites me.
Happy fire fighting!
© 2011, John Morehouse. All rights reserved.