Over the last few weeks I had been quite silent, but that's about to change: Over the next few weeks I'll give a few presentations. Feel free to join any of those.
Cloud technologies and services increase every day. Global players like Amazon put continuously new features to their cloud and products to provide sufficiently functionality of actually web development activities and requirements.
In fact, Amazon starts with the S3 (Simple Storages Service) to provide cloud storage space since 2006 and only a few months later EC2 (Elastic Compute Cloud) was launched a service to provide computing power to manage capacity as many as we need over time. Now, RDS (Relational Database Service) is available to get elastic database power as much as we need to cover all components that we require to implement the whole stack of current web applications.
Use that services to transform the development and of course every application in a liquid and full dynamic status. Every service increase productivity and can be used to cover architecture requirements in sufficient way. That means, use every service as long as we need and allocate capacity as many as we need in the finest way up to 100% not more, not less to reduce resources and costs.
Ok let us start to introduce each important service below to get more details and show examples what we have to do to use these services. Amazon provides a SDK especially for PHP developers to connect each service. You can download the SDK under the following URL:
http://aws.amazon.com/sdkforphp/
To start, download and install the SDK to your destination directory in your application root folder. Before we can start edit the configuration file and change the privacy settings with your AWS (Amazon Web Services) credentials (see Listing 1). To get credentials please create an AWS account on:
http://aws.amazon.com/de/ and follow the instructions.
S3 is the storages service from Amazon where we can store (Create), retrieve (Read), overwrite (Update) and remove (Delete) data. The logical structure based on buckets that contains items like files e.g. images, videos or documents or any data that we want to store. Buckets must have a unique developer-assigned key across the account each item can be contain 1 byte up to 5 terabytes.
First, to store our objects we create a bucket with the method create_bucket. All what we have to do is specify a name of the bucket testbucket and the region where we want to create it. In our case the region is EU_E1 (European 1- Region). Additionally, a simple if check covers the return value whether the call was correct.
In the next step, we put files to the bucket and store some data to the cloud. Simply, call the create_object method to handle our request.
Listing 3 shows the PHP snippet to create a test file. Call the method with the bucket name testbucket, name of the file testfile.txt and some additional parameters in our case body, acl and contentType to specify the characteristic of our data.
Now we want to get back our stored data. To finalize that request we have to use the method get_object.
Listing 5 covers our example and echoed the string that we have inserted a few minutes before.
To see it is very simple to communicate to our cloud storage with the SDK and execute the proper methods. If you do not know the right method take a look in the library service file where every single method is described well that enclose every required parameter and excepted return values.
EC2 is the next create cloud services that we can use to allocate computing resources for our web environment. Through defined methods from the SDK and the proper service file you can easily start an AMI (Amazon Machine Image) with an operating system that match our requirements. There are images from the market leaders like Linux and Windows systems and you can upload an own image as well whether you need a self defined system e.g. pre-installed web, db or chat server. Please go to the Amazon website:
http://aws.amazon.com/ec2/
to see which kind of images are available that fits your requirements. After we have chosen a suitable images we can start it with a method call from the SDK similar create a bucket in S3.
Listing 6 shows that method call run_instances with the parameter ami-48aa4921 with specifies the machine images that will be loaded and some optional parameters in that case the type of the underlying hardware configuration which is specified by m1.micro instance - currently the smallest type with 2 processor kernels (CPUs) - and round about 613MB RAM. Check the response value with the method isOk. If everything is fine we can connect our recently started machine through a secure shell.
Listing 7 visualize a connection to the virtual machine that we have started. We need only our private key that you can download from the AWS account and the hostname of the machine that you can find on the EC2 overview simply go to the AWS website:
http://aws.amazon.com/ec2/.
After the login (see Listing 8) we see our secure shell of the machine where we can input commands like on our self administrated local systems.
Similar S3, EC2 is also pretty easy to use with some SDK commands like run_instances that helps us to integrate this service into our web application. You need only a valid AWS account (key credentials) to access the service, a machine image like ami-48aa4921 and a proper instance type e.g. m1.micro with sufficient processor and memory power.
The final service that we want to cover is the RDS service the third part to complete our web environment stack. It is also pretty easy to use it with the SDK. Call only a single method to start a new RDS instance and connect to the machine.
See Listing 9 to show an example how to create a database instance. Execute simple the method create_db_instance with the parameter testinstance the name of the instance, 5 specifies the size of the db storages in GB, db.m1.small clarifies the size of the underlying instance type (varies in processor and memory power), MySQL the type of the database system, testuser a user name and testpassword the password to connect the database. Check also with the isOk method the response value to guarantee the availability of the system.
Now we can connect to our new instantiated database through a secure shell. Input only the following known MySQL client command to connect. Specify the user name, password and the database host (see Listing 10).
After we have connected to the database server we will see the start screen where we can input commands (see Listing 11).
Check out which tables are available and execute the show databases command. Additionally, show all tables in the MySQL database with the show tables command (see Listing 12).
In this blog post we have cover common cloud services from the Amazon service. S3, EC2 and the RDS are services to extend and of course to replace current development architectures where normally dedicated machines and systems work. With the SDK it is very simple to connect all the services, request commands and response values e.g. S3 items. These proposed services cover a small overview about available services. There are much more excellent functionalities we can use to integrate.
SimpleDB is an additional database service that based on a key value store without any relation between items unlike relational database systems. Handle data in domains similar buckets in S3 and retrieve values with proper SQL statements.
Last but not least, SQS (Simple Queue Service) integrate a SOA service. Loosely coupled items communicate through defined interfaces. Add values to a queue at one application point and retrieve items on another. Works asynchronous and can be used to handle jobs that converts large data like image or video rendering. Thank you for your attention to read until the end ;-) if you have any comments please add your notes and ideas after that post.
Today's topic is deployment. It's called one-click deployment for a reason: Developers are lazy.
It's hard to do less than clicking on one button, so that's our goal.
With the growing need for lower time-to-market and faster response to user feedback it is inevitable to not be limited by technical factors (there are enough other obstacles already). The focus lies on reproducible results.
So, what do we need? Actually, not much. Disregarding the tools and practices that build the foundation of agile software development, you only need a central build server. But you've already got that one covered, right?
If you don't, you should get one. It's a huge help to discover errors quickly and be alerted instantly. This usually leads to a shorter time frame until a fix is done. Tests are run continuously and new parts are integrated into the whole code base.
Everything written here is based on Apache ant, so the easy choices here are Hudson and Jenkins. We're focusing on Jenkins here, but we've successfully used some variation of this continuous integration/deployment script on both.
We've also used CruiseControl/phpUnderControl and Atlassian Bamboo in the past for continuous integration, so the deployment parts should work as well.
Back to the hard facts. We're using the Template for Jenkins Jobs for PHP Projects here for our basic checks, so let's assume you got it installed and your build server is working.
The typical workflow for a developer now consists of these steps:
There's not much deployment in this, so a few more steps:
There are a few preconditions in our environment that haven't been named yet. Not all of them are hard requirements, but they make some things a lot easier:
The frontend nodes use a setup including varnish, nginx, PHP 5.3+APC, ZF 1.11, Dojo 1.6 and Doctrine 2.1.
The backend nodes use MySQL(Master+Slave), Apache Solr, RabbitMQ, memcached and ejabberd.
It basically executes these steps:
And that's it.
The manually executed task is called deploy-to-staging and consists of these steps:
Here's one caveat: We've set up our staging (and production) nodes to include a directory (like /opt/repository) in /etc/apt/sources.list and puppet periodically checks if there are new packages in this directory, and installs it. Of course you could just call dpkg -i through ssh as well.
So far, so good. After waiting 5mins we've got our "release 1.32c" deployed to all 4 staging nodes and the testers can do their work.
We're not quite done yet though. The testers really liked release 1.32c and we want to deploy to live now!
So this means:
In greater detail:
If everything went well, you just deployed a new release!
The first thing we noticed is the build time. The build used to take 15 minutes because of massive disk I/O. Mostly all the analysis/metrics tools are to blame, if you have a few of those, every single one has to read and check all of your hundreds of files even if you exclude the external dependencies and libraries (which is a good idea, anyway). We put the files on a ram disk and brought the build from 15 minutes down to 4. Good enough.
The setup is quite complex (as you've seen if you're still reading). It takes a bit to get used to and there are some moving parts. But so far it's working nicely.
Jenkins is a single point of failure. If your build server goes the way of the dodo, it's very hard to get a release going. There's an easy fix, however: Have a second machine ready that's running an identical copy of your Jenkins setup. If your main build server is down you can still use that one then. As your developer VMs are using the same ant build.xml for daily work, even using that as a last resort is possible.
There are also some goodies that aren't crucial, but nice:
Having talked with developers from other companies there seem to be quite a few similar setups including chef, rake, phing or even make - so you're surely not forced to use ant or a debian-based distribution.
Good luck simplifying your own deployment!
Object-relational mapping (ORM) frameworks have been around for several years now and for some people, ORM is already outdated by now. As we have seen with other technologies and concepts before, PHP is not exactly what we call an early adopter among the programming languages. Thus it took some time for ORM to grow up in the PHP context.
There have been some frameworks before Doctrine 2 that implement ORM (remember e.g. Propel) specific tasks but most of them lack the required maturity to be used in large projects. With Doctrine 2, PHP takes a huge step into the right direction Doctrine 2 is fast, extensible and easy to use.
This article will take you on a tour through the main concepts of Doctrine 2 in the first part and then explain how to use it in a real world application in the second part. Since at the time of writing Zend Framework 1.11.xx (ZF) is very popular, we will integrate Doctrine 2 into a ZF project.
To understand Doctrine 2, we have to take a look at some relevant terms (or in this case objects), study their behavior and practice their usage. We start with some introductory phrases on ORM systems and then go on to the concepts underlying Doctrine 2: Entity Objects, the Entity Manager, Repositories and Proxies.
Since the beginning of Object-Orientation, people had to manage the persistence of their application's state resp.
their objects. In the context of Web Application Development, this usually involves a Database server which is being
consulted using a Query Language. One example for this pattern is a PHP application that uses some kind of SQL server
by sending SQL queries to it. Another one is an application using a CouchDB server by querying it via its REST API.
Due to the author's laziness, we will talk in terms of relational databases from now on. Keep in mind, that you can
accomplish almost everything mentioned here with NoSQL databases, too.
ORM relates value objects that exist in an application's business logic to database records.
Thus every object that should be persistent is saved in one row of a database table. The most common approach is to
map classes to tables and the classes' objects to rows in the these tables.
Besides writing objects to a database, ORM systems are also intended to ease the process of finding data stored in the
database. When talking in terms of ORM, finding data always means making the framework fetch one or many objects
that meet a certain criteria.
The objects that are being managed by an ORM system are called Entity Objects. Every entity object relates to one entry in a table. In Doctrine 2, the classes that represent entities do not have to fulfill special requirements like inheriting from a certain super class (as you might have seen in other database abstraction frameworks like Zend_Db). When creating a new entity class with Doctrine 2, all you have to do is to write down a regular PHP class with properties. Besides this, you have to provide some hints on how these attributes should be persisted. The information how entity attributes relate to columns in the DB is called Metadata. Metadata can be described in different ways: By default there are metadata drivers for descriptions in XML, YAML and PHP. The fourth and most popular driver is based on DocBlock annotations (since in PHP, annotations aren't a language feature as in Java (see Wikipedia), they are contained by the classes' and attributes' DocBlocks). We will use annotations to describe our entities metadata. To get an impression on how easy this is, take a look at the following example.
This example contains all it needs to tell Doctrine 2 about the new entity User. With this class, you can create,
find, delete and modify user objects and persist their state to the underlying database. But keep in mind: as long
as you don't need any persistence features, you can use your user objects just like any other objects!
The next two objects resp. object types we will describe are responsible for doing the ORM functionality: persisting and finding.
To use ORM functionality, the Entity Manager (Doctrine\ORM\EntityManager) is the main access point to Doctrine 2. The entity manager is
responsible as you might have guessed for managing entities and for building a facade for the whole framework.
To accomplish its tasks, the entity manager uses some helpers. The Unit of Work object for example collects entities
that should be written back to the database and is capable of doing this in batches. This way, database operation
can be executed with almost no overhead and therefore are really fast.
Another dependency of the entity manager is the Event Manager. To be as extensible as possible, Doctrine 2 comes with an event system that publishes all important state changes to the outside as events. You can register for such events and extend the life cycle of your entity objects at one single point.
The entity manager's API combines methods for managing entities (find, persist, contains, copy, detatch, merge,
remove and refresh), methods that control the use of transations (beginTransaction, commit, flush, rollback and
transactional) and some helper methods for creating custom queries and accessing some of the entity manager's dependencies.
The following example shows how to query an object from the entity manager, modify it and write the changes back into the database.
Creating a new persistent object is almost as easy as modifying it:
For finding entities, Repositories are used. Every entity class has its own repository which is responsible for finding entities of that type. By default, repositories have some handy methods for fetching entities that match certain criteria:
find: Finds an entity by its primary key / identifierfindAll: Finds all entities of the repository's entity typefindBy / findOneBy: Finds all resp. one entity that matches the passed criteria:findBy<attribute> / findOneBy<attribute>: Magic methods that ease the filtering by a single attribute:
To access a repository, all you have to do is ask the entity manager for one. If you have implemented your own
repository, it will be returned by Doctrine\ORM\EntityManager::getRepository(). Otherwise, Doctrine 2 will provide
a generic repository. The main reason to implement custom repository classes is to group custom queries for an entity
type to make them reusable. For custom query logic, there are several mechanisms you can use: You can either use
Doctrine's query builder that implements an API similar to Zend_Db_Select or queries written in the Doctrine Query
Language (DQL) or you can even execute plain SQL queries. With these options, it is also possible to migrate old
applications which use complex queries by just wrapping these queries into the methods of custom repositories.
When traversing a graph of entity objects (which is required when entities are having relations to other entities),
it would be very expensive (in the sense of requiring many database queries) to fetch every depending entity with
an additional query. Therefore Doctrine 2 uses the concept of Proxy objects that represent regular entity objects
which have not been populated with data from the database. Take a look at the following example where the entity
Group aggregates a list of User objects in its member property. When accessing the members list, Doctrine 2 provides a
collection of proxy objects instead of complete User objects. When an object of this collection is being asked for
one of its properties, Doctrine loads the object's data from the database. This way, the users' data is not loaded
until it is really needed.
This section describes some advanced concepts that are required when mapping entity classes that have relationships to
other entity classes. Possible relationship types are association and inheritance. Inheritance is the mechanism used for
representing subtypes in object-oriented programming languages. An example would be a class User that implements
methods every user of a software should have and a class Administrator inheriting from User that adds methods for
determining the administrator's access rights.
Association is a weaker relation type. It means that an entity object can be related to other entity objects of other types. In terms of relational databases, there are three types of association which differ in the number of entities an object is related to: 1:1, 1:n and n:m relationships. n and m are placeholders and mean multiple.
To put objects of an entity type into relation, you just have to mention this relation in the entity class' mapping
information. The simplest case is a unidirectional 1:1 relationship. In the following example we describe a User entity
which has its access information (user credentials) encapsulated into another entity class called UserCredential. Since
every user has at most one credential object and every credential object may only be associated to one user object,
this is a 1:1 relationship.
If the relationship should be bidirectional, include the OneToOne attribute in the other class, too, and add an
attribute which denotes the attribute of the other entity that mapps the related object:
This way, you can access the user object from the credentials object, too.
Most of the times, developers have to deal with relationships which include many objects on at least one side.
These relationships are called 1:n or n:m relationships. This means that either one or multiple entities are standing
in relationship with an arbitrary number of entities of another type. To accomplish this, you have to use the mapping
keywords OneToMany or ManyToMany when describing your entities. Besides that, the mapping works the exact same way as
with 1:1 relationships.
There are however some tricks you should know when dealing with collections of associated entity objects. Consider
the following relationship between the entity classes User and Group:
When a group has at least one member, the group object will have a collection of the type
Doctrine\Common\Collections\ArrayCollection set as its members property. This collection contains
all user objects (or proxy objects as we have seen before) and can be modified intuitively with the methods add and
removeElement. To honor object-orientation, you might want to introduce custom methods for these tasks. If you do so,
you get into trouble when the group object does not have any users associated. In this case, the collection will simply be
set to null. To avoid checks whether the collection has already be initialized, you should to this by yourself
in the entity class' constructor:
It is also important to notice that one entity has to update the other entity's state as well when a relationship between to objects is created or removed. Take care to do this only in one class to avoid endless recursion loops! This class is called the Owning Side of the relationship. When implementing a bidirectional relationship, the other class is called the Inverse Side. It is important to determine owning and inverse side and implement the the classes accordingly to avoid greater trouble during debugging.
There are some more features implemented by Doctrine 2 enabling developers to specify their entities' relationships including sorting, pre-fetching and indexing. These topics are not covered in this article but are explained very understandable in the Doctrine 2 documentation.
Subtyping can be implemented in different ways using Doctrine 2. The main difference between these implementations is how the inheritance is mapped to the database. The options are to have one table for every class (Class Table Inheritance), to have one table for all classes in a hierarchy (Single Table Inheritance) and to have a table for every specialized sub-class of a given super-class (Mapped Super Class).We will give a short overview on all three alternatives, you have to pick the right one yourself. This decision should be made based on how many common attributes there are in your sub-classes.
Introducing a mapped superclass is probably the easiest way for specifying inheritance but might lead to many duplicate columns in your database schema. The superclass of your entities is not being declared as an entity itself (and might also be declared abstract) but provides attributes and optionally methods that will be available in all subclasses. When creating the database schema, Doctrine 2 merges all attributes and relationships of the superclass into the definitions of the subclasses and processes them as regular entities.
After creating the database from this mapping information, your tables will look like this:
When having entities that are very similar besides some few attributes, you might want to store them together in one database table. This approach is called Single Table Inheritance. To distinguish between the different types, there is always a column marked as discriminator column and a discriminator map that tells Doctrine 2 which values in the discriminator indicate what entity types.
These definitions cause the existing of one single table called User with all the attributes declared inside the classes User and Administrator plus a column type the discriminator column. When working with entities of these types, Doctrine will manage the type flag automatically for you.
The resulting database schema looks as illustrated by the following diagram:
Having each entity type stored in its own table is always good for keeping your schema extensible. When you have to create a new subtype, Doctrine 2 will just create a new table for this type and it can inherit the logic and common attributes of a superclass. The only overhead you have with this approach is that all tables that correspond to subtypes have to maintain a relationship to their supertype's table. Using class table inheritance, the example with the entities User and Administrator looks like this:
Besides the inheritance type, there is no difference to the example using single table inheritance. The outcome on the resulting database scheme is huge. Now you have to separate tables which store users and administrators. Every record in the table Administrator has a corresponding record in the User table.
This was the first part of this article. Stay tuned for part II which will be published tomorrow (on 6th of December 2011)! In the second part, we will integrate Doctrine 2 into a Zend Framework application and include a generic sandbox (ZF-)project with Doctrine 2!
I was giving a presentation of the MySQL’s Enterprise Monitor* application to a client recently. I was demonstrating the “graphs” section of MEM, where you can monitor MySQL sessions, connections, replication latency and more with 60+ graphs. Usually, you view the graphs from within the MEM Enterprise Dashboard (via a web browser). But the client asked if there was a way to automatically download graphs. I wasn’t sure why he wanted to download the graphs (I didn’t ask), but I knew it wasn’t possible by using MEM alone. However, in the past I have written Perl scripts to automatically download files from web sites, so I thought I would see if it was possible with MEM.
|
*The MySQL Enterprise Monitor (MEM) continuously monitors your MySQL servers and alerts you to potential problems before they impact your system. Its like having a “Virtual DBA Assistant” at your side to recommend best practices to eliminate security vulnerabilities, improve replication, optimize performance and more. As a result, the productivity of your developers, DBAs and System Administrators is improved significantly. (from: http://www.mysql.com/products/enterprise/monitor.html) |
Of course, you have to install MEM and at least one agent. Let’s assume that you have already accomplished this task, and that MEM is running properly. Open MEM in your browser, login, click on the graphs tab, and then you will see a list of all of the available graphs.
For this example, we are going to automatically download the Agent Reporting Delay and the Disk IO Usage graphs. We will download the first graph for all of the servers in a particular group, and the second graph for an individual server. First, click on a server group in your server list on the left side of MEM.
Next, we will need to change the Time Range settings to “From/To”, so that we can enter the a timeline for the graph in our script. Don’t worry about the time settings that are in MEM, as we will change these settings later, but we need them so that they will be included in the URL that we will use (more on this later). After you have changed the Time Range settings, click on the “Filter” button.
Next, click on the plus sign for the graph that you want to use so that MEM will draw the graph. For this example, we will click on the “Agent Reporting Delay” graph:
You will notice two icons to the right of the graph name. The first icon (on the left) allows you click on the icon to download the graph as a .csv file. The second icon (on the right) allows you to click on the icon and download the graph as a PNG image file.
We need some information from the actual link that is used when you click on the PNG icon. So, we will need to right-click on the icon to get the URL link location information for the Agent Reporting Delay graph:
The URL for this graph is then copied to your clipboard. This is the URL location (which is for all servers in the group that I selected):
http://192.168.1.2:18080/Graph.action?dims_height=300&dims_width=800&graph=f924cb42-fed5-11df-923c-a6466b4620ce&locale=en_US&noDefaults=false&servers_group=0&style=NORMAL&time_fromDate=2011-11-16&time_fromTime=11%3A24&time_toDate=2011-11-16&time_toTime=11%3A54&time_type=FROMTO&tzName=America%2FNew_York
|
As you can see in the URL above, there are several variable values that we will include in our script to produce our graphs (in blue text above). In this example, we will only be working with the following variables:
- dims_height
- dims_width
- time_fromDate
- time_fromTime
- time_toDate
- time_toTime
- servers_group and servers_server
(the servers_server variable and value are not shown in the above example, but will be in the next example below)
We will be using a text file named files.txt to store some of the graph variable values that will be used by the script. Now that you know how to copy the URL for a graph, you will need to extract the value for the graph variable and the value for the servers variable and place the values into your files.txt file. The graph value for the above URL (shown again below) is in blue text, and the value for the server variable is in red text: (notice that all values are separated on the left by an equal sign “=” and on the right by an ampersand “&”)
http://192.168.1.2:18080/Graph.action?dims_height=300&dims_width=800&graph=f924cb42-fed5-11df-923c-a6466b4620ce&locale=en_US&noDefaults=false&servers_group=0&style=NORMAL&time_fromDate=2011-11-16&time_fromTime=11%3A24&time_toDate=2011-11-16&time_toTime=11%3A54&time_type=FROMTO&tzName=America%2FNew_York
|
In the above example, we had selected a group of servers in our server list (on the left side of MEM), and therefore the URL will not have a value for the individual server (variable named servers_server). The graph that we will extract will be for this group of servers (in this case servers_group has a value of zero, which is still a value). This is what we had chosen under our Servers list:
Now, we want to select an individual server. In this case, we will click on “iMac-Tony”:
Now that we have chosen an individual server, in the URL for that graph, you will have a value for the variable named “servers_server”, as well as a value for servers_group – and you will need both values together. So, if you want a graph for an individual server, you will need to click on that individual server in your servers list, reselect the “Time Range” value of “From/To”, click “Filter”, and re-copy the PNG graph URL. Once we have copied the URL for this graph for an individual server, you will see a different value for the graph variable (in red) and a value for servers_group and servers_server (in blue) like this:
http://192.168.1.2:18080/Graph.action?dims_height=300&dims_width=800&graph=6d9c8ac0-7a3b-11df-9df0-f30c5eb77a3c&locale=en_US&noDefaults=false&servers_group=0&servers_server=111&style=NORMAL&time_fromDate=2011-11-16&time_fromTime=15%3A27&time_toDate=2011-11-16&time_toTime=15%3A57&time_type=FROMTO&tzName=America%2FNew_York
|
We will use the above URL information for our second graph – the Disk IO Usage graph. You will need to copy all of the graph and server values for the graphs that you want to download. For the above URL, we will grab these values, to be placed in our files.txt file:
graph = 6d9c8ac0-7a3b-11df-9df0-f30c5eb77a3c
server group and server name = servers_group=0&servers_server=111
This is a tedious process at first, but you should only have to do this once for each graph. (Caveat – I am not sure if the graph values change over time or when you upgrade the MEM software.)
In the files.txt file, we also want a name for the graph (which will also be used for the PNG image file name), the graph value, the servers value and the server or server group values from the above URLs, as well as the name of the server group or individual server. You should separate the values with a delimiter of three tildes “~~~”.
So, for the two example graphs above, your files.txt file should contain the following values – Graph Name~~~graph value~~~server information~~~server or group name: (please note that the graph values that I have here may not be the same values that you would have for the same graph)
Agent Reporting Delay~~~f924cb42-fed5-11df-923c-a6466b4620ce~~~servers_group=0~~~All Servers |
The first line above will produce an “Agent Report Delay” graph for “All Servers”. The second line will produce a “Disk IO Usage” graph for only the server named “iMac-Tony”.
Now that we have our files.txt file in place (it should be placed in the same folder as the Perl script – or you may modify the Perl script for a different file location), we will use this Perl script to download our graphs as PNG image files. In case you want to place this script in a cron job to run every X number of minutes, we will include a variable to allow you to select the previous number of minutes to include in your graph. For example, in the Perl script, if you set the value of the variable $time_interval to 60 (minutes) and run the job at 30 minutes past the hour, the script will retrieve a graph for the past 60 minutes from the time of script execution.
For this example, we will name the Perl script “get_graphs.pl”. There will be some variables in the script that you will have to change once, to match your system’s information. The variables that you need to change are highlighted in blue text in the script:
#!/usr/bin/perl
|
When we executed the script, two files were created and downloaded – All_Servers_Agent_Reporting_Delay_2011_11_16_16_17.png and iMac_Tony_Disk_IO_Usage_2011_11_16_16_17.png:
All_Servers_Agent_Reporting_Delay_2011_11_16_16_17.png
iMac_Tony_Disk_IO_Usage_2011_11_16_16_17.png
You could also create a similar script to download the information as a .csv file, but the syntax is very different (maybe I will do that in a future post). But for now, I have a possible solution for the client – and I hope that he likes it.
—————————————–
|
Tony Darnell is a Principal Sales Consultant for MySQL, a division of Oracle, Inc. MySQL is the world’s most popular open-source database program. Tony may be reached at info [at] ScriptingMySQL.com and on LinkedIn. |
One of the great things about the HTTP protocol, besides status code 418, is that it's stateless. A web server therefore is not required to store any information on the user or allocate resources for a user after the individual request is done. By that a single web server can handle many many many different users easily, and well if it can't anymore one can add a new server, put a simple load balancer in front and scale out. Each of those web servers then handles its requests without the need for communication which leads to linear scaling (assuming network provides enough bandwidth etc.).
Now the Web isn't used for serving static documents only anymore but we have all these fancy web apps. And those applications often have the need for a state. The most trivial information they need is the current user. HTTP is a great protocol and provides a way to do authentication which works well with its stateless nature - unfortunately this authentication is implemented badly in current clients. Ugly popups, no logout button, ... I don't have to tell more I think. For having nicer login systems people want web forms. Now the stateless nature of HTTP is a problem: The user may login and then browse around. On later requests it should still be known who that user is - with a custom HTML form based login alone this is not possible. A solution might be cookies. At least one might think so for a second. But setting a cookie "this is an authorized user" alone doesn't make sense as it could easily be faked. Better is to simply store a random identifier in a cookie and then keep a state information on the server. Then all session data is protected and only the user who knows this random identifier is authenticated. If this identifier is wisely chosen and hard to guess this works quite well. Luckily this is a mostly PHP- and MySQL-focused blog and as PHP is a system for building web applications this functionality is part of the core language: The PHP session module.
The session module, which was introduced in PHP 4, partly based on work on the famous phplib library, is quite a fascinating piece of code. It is open and extendable in so many directions but still so simple to use that everybody uses it, often newcomers learn about it on their first day in PHP land. Of course you can not only store the information whether the user is logged in but cache some user-specific data or keep the state on some transactions by the user, like multi-page forms or such.
In its default configuration session state will be stored on the web server's file system. Each session's data in its own file in serialized form. If the filesystem does some caching or one uses a ramdisk or something this can be quite efficient. But as we suddenly have a state on the web server we can't scale as easily as before anymore: If we add a new server and then route a user with an existing session to the new server all the session data won't be there. That is bad. This is often solved by a configuration of the load balancer to route all requests from the same user to the same web server. In some cases this works quite ok, but it is often seen that this might cause problems. Let's assume you want to take a machine down for maintenance. All sessions there will die. Or imagine there's a bunch of users who do complex and expensive tasks - then one of your servers will have a hard time, giving these users bad response times which feels like bad service, even though your other systems are mostly idle.
A nice solution for this would be to store the sessions in a central repository which can be accessed from all web servers.