The hosting provider, being really helpful, goes in as root and first deletes ib_logfile* then ib* in /var/lib/mysql. He later says “I am sorry if I deleted it. I thought I deleted the log only. Sorry again.”  Now this may appear nice, but people who know what they’re doing with MySQL will realise that deleting the iblogfiles actually destroys data also. MySQL of course screams loudly that while it has FRM files it can’t find the tables. No kidding!

Then, while he’s been told to not touch anything any more, and I’m trying to see if I can recover the deleted files on ext3 filesystem (yes there are tools for that), he goes in again and puts an ibdata1 file back. No, not the logfiles – but he had those somewhere else too. The files get restored and turn out to be two months old (no info on how they were made in the first place but that’s minor detail in this grand scheme). All the extra write activity on the partition would’ve also made potential deleted file recovery more difficult or impossible.

This story will still get a “happy” ending, using a recent mysqldump to load a new server at a different hosting provider. Really – some helpfulness is not what you want. Secondary lesson: pick your hosting provider with care. Feel free to ask us from recommendations, we know some good/excellent and have encountered plenty bad.

CFEngine is both the oldest and the newest of the popular tools for automating site administration. Mark Burgess invented it as a free software project in 1993, and years later, as deployments in the field outgrew its original design he gave it a complete rethink and developed the powerful concept of promise theory to make it modular and maintainable. In this guise as version 3, CFEngine stands along with two other pieces of free software, Puppet and Chef, as key parts of enterprise computing. Along the way, Burgess also started a commercial venture, CFEngine AS, that maintains both the open source and proprietary versions of CFEngine.

Diego Zamboni has recently taken the position of Senior Security Advisor at CFEngine AS and is writing a book for O'Reilly on CFEngine 3. I talked to him this week about the recent new release of the open source version (3.2.4) in tandem with a new commercial release of CFEngine 3 Nova (version 2.1.3). Here's are excerpts of what he has written to introduce CFEngine 3.

CFEngine 3 is fine-tuned to the features and design that make it possible to automate very large numbers of systems in a scalable and manageable way. CFEngine 3 is also very lightweight--its binaries normally use less than 30MB of disk space, it requires a single TCP port to communicate among servers and clients, and it has been designed to be very resource-efficient. CFEngine 3 can run on everything from smartphones to supercomputers.

CFEngine 3 is different from many other automation mechanisms in that you do not need to tell it what to do. Instead, you specify the state in which you wish the system to be, and CFEngine 3 will automatically and iteratively decide the actions to take to reach the desired state, or as close to it as possible. Underlying this ability is a powerful theoretical model known as Promise Theory, which was initially developed for CFEngine 3, but which has also found other applications in Computer Science and in other fields such as Economics and Organization.

This allows you to develop building blocks for complex promises that remain readable and manageable because the lower-level components are encapsulated. Each promise represents the desired state of certain parts of the system. At the lowest level, these are some of the things that you can express to CFEngine 3 as desired states:

• "Make sure file /foo/bar contains line xyz"

• "Make sure user foobar exists/does not exist"

• "Make sure process foo is/is not running"

At a higher level of abstraction, you can encapsulate CFEngine 3 operations and express high-level desired states:

• "Make sure all web servers have Apache installed"

• "Make sure all root accounts have the same, centrally-designated password"

• "Make sure parameters EnableDNS and AllowRoot are disabled on all sshd configurations"

And at an even higher level, you can express top-level desired states like these:

• "Configure host xyz as a database server"

• "Create a new cluster of VMs to use as web servers"

So what's in the new versions? CFEngine 3 Nova includes:

• System monitoring extensions, which extend the monitoring capabilities of CFEngine 3 Community (to monitor system state such as CPU load, number of processes and network connections, disk utilization, etc.) to allow for defining custom monitors for any type of information.

• Support for manipulating virtual machines on Xen, VMware ESX, and KVM.

• Native Windows support.

• Flexible searching of reports in a brand new scalable interface that supports thousands of hosts on a single hub.

• Improved machine learning and anomaly monitoring for diagnostics and capacity planning. Additional sensors have been added to detect operating system performance and behavioral trends, especially on Linux kernels.

• The NoSQL document-oriented database MongoDB, used instead of MySQL for all storage on Nova's Mission Portal.

• Generic JSON return values so that users can customize the interface and JQuery framework of the Mission Portal. This allows direct access to data in a way that makes higher levels of scripting more effective.

CFEngine 3 Community also includes a large number of improvements, all of which are in Nova too:

• A vastly improved bootstrapping process, which makes it easy to get new CFEngine 3 servers and clients up and running with very little manual configuration.

• Support for environments, which are a way of grouping hosts according to arbitrary definitions. This makes it very easy to define, for example, "development," "testing," and "production" environments for CFEngine 3 policies.

• The new cf-report command, available in both Community and Nova, which allows extraction of data and generation of reports from the command line. It can produce reports both about the behavior of the current CFEngine 3 environment (policies, hosts, etc.) and about internal information, such as a CFEngine 3 syntax summary.

• Many performance and concurrency improvements and bug fixes.

• Several new functions and parsing improvements, including and(), not(), and or() functions, to ease writing of complex class expressions.

• A new and improved Emacs mode for editing CFEngine 3 policy files.

Velocity Europe, being held Nov. 8-9 in Berlin, will bring together the web operations and performance communities for two days of critical training, best practices, and case studies.

Save 20% on registration with the code RADAR20

I learn therefore I am!

I’ve just wrote few bits about learning a new technology and after skimming through my Google Reader, I noticed a great post by Chen Shapira — Deliberate Practice. That’s reminded me about another aspect of learning that I didn’t mention — learning is a continuous process.

There are two aspects…

• No matter how good I am and how much I know, my knowledge and expertize become outdated relatively quickly these days unless I keep up with the new stuff. Unfortunately, there is so much new technologies these days that I have to be very selective on what I want to follow which is a big challenge in itself. On the other hand, I’d rather be challenged than bored. As DBA’s we are luckier than Developers — their world changes much faster than ours.
• I forget things I don’t use/do/read about/think about regularly. Refreshing my “old” memory is a must do. This is why I think Chen’s idea of deliberate practice is so great.

Some people are better on keeping the existing tools sharp, while others succeed on the lookout for new ways and techniques. To truly excel, we need to master both.

This photo (it was taken 5 years ago in Australia in one of Sydney’s “wild” parks) reminds me of this continuous learning process — perhaps, it will do the job for you as well.

But Poderosa is not the tool I wanted to mention….Another user group member mentioned PuTTY Connection Manager. It wraps around PuTTY and gets the existing saved connections, makes a nicely tabbed browsing window where you can open sessions by double-clicking the connections, which are now listed on the right-hand side.

See screenshot below:

I have not played with other features such as sending a command to multiple windows, but even just having this is a HUGE win.

Example:
- System admins were being put in charge of scheduling rooms in the building
- They wrote a PHP web application to help them automate the task
- They refined the app, documented how to use it, and handed it off to a secretary
- They have to maintain the app, but it’s far less work.

The Community Pattern:

- Prior to launch of a new service, create user documentation for it.
- Point a few early adopters at the documentation and see if they can use the service with minimal support
- Use feedback to improve documentation, and the service
- Upon launch, create a mailing list, forum, IRC channel, or Jabber chat room and ask early adopters to help test it out.
- Upon launch, your early adopters are the community, and they’ll tell new users to use the tools you’ve provided instead of calling you.

Example:
- A beowulf cluster for an academic department
- Documented like crazy, early adopters were given early access to the cluster (demand was high)
- Crated a mailing list, early adopters were added to it with their consent, functionality was tested with them.
- Email announcing launch mentioned the early adopters in a ‘thank you’ secion, and linked them to their mailing list.

The DRY pattern
DRY = Don’t repeat yourself
Identify duplicate code in your automation scripts
Put subroutines that exist in an include file, and include them in your scripts.

Example:
- “sysadmin library”
- /var/lib/adm/.*pl
- Elapsed time and # of lines to script a task for which the library was useful plunged dramatically
– new tasks were thought up that were not considered before but were obvious now (ie, users that want to change their username)
– migrating to new services became much easier

The Chameleon Pattern
- Identify commonalities among your services
- Leverage those to create “Chameleon” servers that can be re-purposed on the fly
- Abstract as much of this away from the physical hardware
- Doesn’t need to involve virtualization, though it’s awfully handy if you can do it that way.
[this one is a bit harder to do with MySQL config files]

Example:
[puppet/cfengine were mentioned...]
ldapconfig.py – more than a script: a methodology

- But isn’t installing packages you don’t need bad? Depends on the package….ie, gcc is bad for enterprise

“Junior annoynances”

Terminal issues

Junior:
open terminal, login to machine1
think issue is with machine2, talks to machine1.
log out of machine1
log into machine2

Senior:
opens 2 terminals each of machine1 and machine2 to start

Junior:
networking issue ticket arrives
logs into server
runs tcpdump

Senior:
networking issue ticket arrives
logs into server
looks at logs

“Fix” vs. “Solution” ie “taking orders”
Junior will try fix a problem, senior will try to figure out what the problem is. ie, “I need a samba directory mounted under an NFS mount” a junior admin will try to do exactly that, a senior admin will ask “what are you trying to do with that?” because maybe all they need is a symlink.

Fanboyism
Signs you might be a fanboy:
- Disparaging users of latest stable release of $THING for not using the nightly (unstable) build which fixes more issues
- Creating false/invalid comparisons based on popular opinion instead of experience/facts
- Going against internal standards, breaking environmental consistency, to use $THING instead of $STANDARD (but this is also how disruptive technology works)
- Being in complete denial that most technology at some point or another stinks.
- Evaluating solutions based on “I like” instead of “we need” and “this does”.

Example:
- System admins were being put in charge of scheduling rooms in the building
- They wrote a PHP web application to help them automate the task
- They refined the app, documented how to use it, and handed it off to a secretary
- They have to maintain the app, but it’s far less work.

The Community Pattern:

- Prior to launch of a new service, create user documentation for it.
- Point a few early adopters at the documentation and see if they can use the service with minimal support
- Use feedback to improve documentation, and the service
- Upon launch, create a mailing list, forum, IRC channel, or Jabber chat room and ask early adopters to help test it out.
- Upon launch, your early adopters are the community, and they’ll tell new users to use the tools you’ve provided instead of calling you.

Example:
- A beowulf cluster for an academic department
- Documented like crazy, early adopters were given early access to the cluster (demand was high)
- Crated a mailing list, early adopters were added to it with their consent, functionality was tested with them.
- Email announcing launch mentioned the early adopters in a ‘thank you’ secion, and linked them to their mailing list.

The DRY pattern
DRY = Don’t repeat yourself
Identify duplicate code in your automation scripts
Put subroutines that exist in an include file, and include them in your scripts.

Example:
- “sysadmin library”
- /var/lib/adm/.*pl
- Elapsed time and # of lines to script a task for which the library was useful plunged dramatically
– new tasks were thought up that were not considered before but were obvious now (ie, users that want to change their username)
– migrating to new services became much easier

The Chameleon Pattern
- Identify commonalities among your services
- Leverage those to create “Chameleon” servers that can be re-purposed on the fly
- Abstract as much of this away from the physical hardware
- Doesn’t need to involve virtualization, though it’s awfully handy if you can do it that way.
[this one is a bit harder to do with MySQL config files]

Example:
[puppet/cfengine were mentioned...]
ldapconfig.py – more than a script: a methodology

- But isn’t installing packages you don’t need bad? Depends on the package….ie, gcc is bad for enterprise

“Junior annoynances”

Terminal issues

Junior:
open terminal, login to machine1
think issue is with machine2, talks to machine1.
log out of machine1
log into machine2

Senior:
opens 2 terminals each of machine1 and machine2 to start

Junior:
networking issue ticket arrives
logs into server
runs tcpdump

Senior:
networking issue ticket arrives
logs into server
looks at logs

“Fix” vs. “Solution” ie “taking orders”
Junior will try fix a problem, senior will try to figure out what the problem is. ie, “I need a samba directory mounted under an NFS mount” a junior admin will try to do exactly that, a senior admin will ask “what are you trying to do with that?” because maybe all they need is a symlink.

Fanboyism
Signs you might be a fanboy:
- Disparaging users of latest stable release of $THING for not using the nightly (unstable) build which fixes more issues
- Creating false/invalid comparisons based on popular opinion instead of experience/facts
- Going against internal standards, breaking environmental consistency, to use $THING instead of $STANDARD (but this is also how disruptive technology works)
- Being in complete denial that most technology at some point or another stinks.
- Evaluating solutions based on “I like” instead of “we need” and “this does”.

So I am in Seeking Senior and Beyond: The Tech Skills That Get You Promoted. The first part talks about the definition of what it means to be senior, and it completely relates to DBA work:
works and plays well with other
understands “ability”
leads by example
lives to share knowledge
understands “Service”
thoughtful of the consequences of their actions
understands projects
cool under pressure

Good Qualities:
confident
empathetic
humane
personal
forthright
respectful
thorough

Bad Qualities:
disrespective
insensitive
incompetent
[my own addition - no follow through, lack of attention to detail]

The Dice/Monster Factor – what do job sites see as important for a senior position?

They back up the SAGE 5-year experience requirement
Ability to code in newer languages (Ruby/Python) is more prevalent (perhaps cloud-induced?)

The cloud allows sysadmin tasks to be done by anyone…..so developers can do sysadmin work, and you end up seeing schizophrenic job descriptions such as

About the 5-year requirement:
- Senior after 5 years? What happens after 10 years?
- Most electricians, by comparison, haven’t even completed an *apprenticeship* in 5 years.

Senior Administrators Code
- not just 20-line shell scripts
- coding skills are part of a sysadmin skill
- ability to code competently *is* a factor that separates juniors from seniors
- hiring managers expect senior admins to be competent coders.

If you are not a coder
- pick a language, any language
- do not listen to fans, find one that fits how you think, they all work…..
- …that being said, some languages are more practical than others (ie, .NET probably is not the best language to learn if you are a Unix sysadmin).

Popular admin languages:
- Perl: classic admin scripting language. Learn at least the basics, because you will see it in any environment that has been around for more than 5 years.

- Ruby: object-oriented language for people who mostly like Perl (except for its OO implementation)

- Python: object-oriented language for people who mostly hate Perl, objects or no objects. For example, you don’t have to create a String object to send an output.

But what if you do not have time to learn how to program?

- senior admins are better at managing their time than junior admins, so perhaps managing time
- time management means you’ll have more time to do things, it doesn’t mean all work work work.
- Read Time Management for System Administrators – there is Google Video of a presentation by the author, Tom Limoncelli.

Consider “The Cloud”
- starting to use developer APIs to perform sysadmin tasks, so learning programming is good.
- still growing, could supplant large portions of datacenter real estate
- a coder with sysadmin knowledge: Good
- a sysadmin with coding knowledge: Good
- a coder without sysadmin knowledge: OK
- a sysadmin with no coding interest/experience: Tough place to be in

Senior Admins Have Problems Too
Many don’t document or share knowledge
Maany don’t do a good job keeping up with their craft
Cannot always be highlighted as an example of how to deal with clients
Often reinvent the wheel – also usually there is no repository
Often don’t progress beyond the “senior admin” role

….on the other hand…..
cynicism can be good…..

Advice:
learn from the good traits
observe how others respond to their bad traits
think about how you might improve upon that
strive to work and play well with others, even if you don’t have a mentor for good/bad examples.

Now he’s going into talking about Patterns in System Administration….

So I am in Seeking Senior and Beyond: The Tech Skills That Get You Promoted. The first part talks about the definition of what it means to be senior, and it completely relates to DBA work:
works and plays well with other
understands “ability”
leads by example
lives to share knowledge
understands “Service”
thoughtful of the consequences of their actions
understands projects
cool under pressure

Good Qualities:
confident
empathetic
humane
personal
forthright
respectful
thorough

Bad Qualities:
disrespective
insensitive
incompetent
[my own addition - no follow through, lack of attention to detail]

The Dice/Monster Factor – what do job sites see as important for a senior position?

They back up the SAGE 5-year experience requirement
Ability to code in newer languages (Ruby/Python) is more prevalent (perhaps cloud-induced?)

The cloud allows sysadmin tasks to be done by anyone…..so developers can do sysadmin work, and you end up seeing schizophrenic job descriptions such as

About the 5-year requirement:
- Senior after 5 years? What happens after 10 years?
- Most electricians, by comparison, haven’t even completed an *apprenticeship* in 5 years.

Senior Administrators Code
- not just 20-line shell scripts
- coding skills are part of a sysadmin skill
- ability to code competently *is* a factor that separates juniors from seniors
- hiring managers expect senior admins to be competent coders.

If you are not a coder
- pick a language, any language
- do not listen to fans, find one that fits how you think, they all work…..
- …that being said, some languages are more practical than others (ie, .NET probably is not the best language to learn if you are a Unix sysadmin).

Popular admin languages:
- Perl: classic admin scripting language. Learn at least the basics, because you will see it in any environment that has been around for more than 5 years.

- Ruby: object-oriented language for people who mostly like Perl (except for its OO implementation)

- Python: object-oriented language for people who mostly hate Perl, objects or no objects. For example, you don’t have to create a String object to send an output.

But what if you do not have time to learn how to program?

- senior admins are better at managing their time than junior admins, so perhaps managing time
- time management means you’ll have more time to do things, it doesn’t mean all work work work.
- Read Time Management for System Administrators – there is Google Video of a presentation by the author, Tom Limoncelli.

Consider “The Cloud”
- starting to use developer APIs to perform sysadmin tasks, so learning programming is good.
- still growing, could supplant large portions of datacenter real estate
- a coder with sysadmin knowledge: Good
- a sysadmin with coding knowledge: Good
- a coder without sysadmin knowledge: OK
- a sysadmin with no coding interest/experience: Tough place to be in

Senior Admins Have Problems Too
Many don’t document or share knowledge
Maany don’t do a good job keeping up with their craft
Cannot always be highlighted as an example of how to deal with clients
Often reinvent the wheel – also usually there is no repository
Often don’t progress beyond the “senior admin” role

….on the other hand…..
cynicism can be good…..

Advice:
learn from the good traits
observe how others respond to their bad traits
think about how you might improve upon that
strive to work and play well with others, even if you don’t have a mentor for good/bad examples.

Now he’s going into talking about Patterns in System Administration….

You're installing MySQL using the Generic Linux Binary tar ball? If you do, you'll need to create a system user and group using useradd and groupadd. However, you would like to use the standard system IDs. No problem, Red Hat defines their standard system users and groups in their manual respectively as 27 and 27.

Here are the commands to create the system user mysql and its group mysql:

 shell> groupadd -r -g 27 mysql
 shell> useradd -u 27 -g 27 -r -d /var/lib/mysql -s /bin/bash mysql

Years ago, but for some now still, one objective is to see long uptime for a server, sometimes years. It means the sysadmin is doing everything right, and thus some serious pride is attached to this number. As described only last week in Modern Uptime on the Standalone Sysadmin blog, security patches are a serious issue these days, and so (except if you’re using ksplice sysadmin quality has become more a question of when you last did security updates (which might have involved a reboot), rather than the uptime number.

But I think the aforementioned booboo illustrates an additional aspect, I think it might be quite sensible to reboot a system every so many weeks (we can debate the interval and it may differ per system and situation) since in the end it will be rebooted some time, and that may show trouble at an inconvenient time. Better to test and fine out when you’re there.

Of course this also has consequences for either your external uptime (scheduled maintenance slots with outages), or thinking about your architecture differently. Can you take out any individual system in your infrastructure without some service getting interrupted? It’s doable, but not necessarily with some traditional approaches or equipment that carries the “enterprise” label.

Food for thought! As always, comments welcome.