Scribd has an agile, startup culture and an unusually close working relationship between engineering and ops. You’ll regularly find cross-over work at Scribd, with ops people writing application-layer code and engineers figuring out operations-level problems. We think we’re able to make that work because of the uniquely talented people we have on the team.

To allow us to keep scaling, we’re now looking to add a strong, experienced operations guru to the team. As a member of Scribd operations, you’ll have tremendous ownership and responsibility for one of the web’s most popular applications. Because Scribd is a startup, you will wear many hats and have broader responsibility than you would at a larger company.

If you read this blog, you should already have a good sense of the kind of work you’ll be doing on this position.

The Ideal Profile

You are an experienced operations professional and have run ops at at least one large-scale website. You have comprehensive knowledge of a broad variety of system tools, from MySQL and Nginx to Squid and Memcached. You should also have strong software development skills and be well-versed in major programming languages. You should be strongly motivated, a creative solution finder, and ready to jump into the thorniest technical problems whenever necessary.

Responsibilities

• Develop and maintain all aspects of Scribd’s operations infrastructure, including system monitoring, backups, server configuration, databases, and caching systems
• Collaborate with engineering to create next generation infrastructure to support changing requirements
• Predict scaling problems before they occur and work with engineering to prevent them
• Write and debug application level ruby code
• Participate in an on-call rotation
• Quickly diagnose server problems and employ preventive measures to maintain high availability servers

Qualifications

• Bachelors degree in CS or equivalent experience
• 3-5 years of professional experience in site operations
• Strong software engineering skills, including knowledge of major programming languages
• Strong database skills, preferably with MySQL, and overall linux knowledge
• Experience with most of the following technologies: MySQL, Nginx, Ruby, Memcached, Squid, git, Solr, HBase, Postfix
• Proven ability to quickly learn and implement unfamiliar technologies
• Strong desire to work hard at a rapidly growing company

Location: You are preferably located near San Francisco, CA. Relocation assistance is designed on a per-case basis. In short, we’ll be creative to get you here.

Contact: Please send your email cover letter and resume with the subject “Your name – Senior Site Operations Engineer – via Kovyrin.net” to jobs@scribd.com or contact me directly using any of my contacts. All communication and correspondence is held in the strictest confidence to ensure that you can connect and learn more without exposure.

Many are speculating that this is simply an effort to further and more effectively monetize Java, a storied program language that has move more toward openness and survived several supposed death sentences as newer languages arrived. Still, with all of the open source parts — GlassFish application server, MySQL database, OpenOffice.org suite — is Java the most significant to Oracle? It may be, but regardless of what Oracle is doing, its legal moves here may certainly have an impact on the many other open source projects from Sun that are now under Oracle’s umbrella.

Oracle may also simply be initiating an IP licensing effort around Java, but as Microsoft has found, this can be a delicate endeavor to say the least. Another possibility is that Oracle, not typically mentioned or meaningful when we discuss the hot market of smartphones, wants to make sure the world knows its Java code is in many of that Android technology. Still, there are more constructive ways to go about that, I would think.

We have questioned Oracle’s full appreciation for open source software before, but its latest action simply brings more questions to mind.

The smartphone market is seeing incredible opportunity, competition and innovation right now? In addition, with waves of iPhone and more recently Android popularity, the smartphone market might even be poised for a slow in growth (even though it is by many accounts the fastest growing technology market). Still, if there is some slowing that was part of the natural market cycle, will Oracle take some or even all of the blame?

Given that Google is adept at software development and using open source, we also have to wonder about the impact of any and all major workarounds. Plans may already be well underway to circumvent the use of Java in Android and any range of other devices or markets where it has managed to stay relevant despite its age. This could finally make Java less relevant, or at the least have a negative effect on Java development going forward.

One thing seems clear, Oracle’s move makes all that software patent discussion and debate more relevant and more real. We have sensed a coming storm over software patents, but we did not anticipate a first shot from Oracle, frankly. One of the biggest questions now is what kind of reaction will this trigger from the likes of the Open Invention Network, Linux Foundation (of which Oracle is a Platinum member and Google is a Gold member) or others with resources and interest in legally defending Linux and open source software?

As I’ll show, DROP operations are significantly simpler than creation operations. Interestingly, it the same as in life.

DROP COLUMN

A column turns to be redundant, unused. Before it is dropped from the database, we must ensure no one is using it anymore. The steps are:

1. App: V1 -> V2. Remove all references to column; make sure no queries use said column.
2. DB: V1 -> V2 (possibly failover from M1 to M2), change is DROP COLUMN.

DROP INDEX

A possibly simpler case here. Why would you drop an index? Is it because you found out you never use it anymore? Then all you have to do is just drop it.

Or perhaps you don’t need the functionality the index supports anymore? Then first drop the functionality:

1. (optional) App: V1 -> V2. Discard using functionality which relies on index.
2. DB: V1 -> V2 (possibly failover from M1 to M2), change is DROP INDEX. Check out InnoDB Plugin here.

DROP UNIQUE INDEX

When using Master-Slave failover for table refactoring, we’re now removing a constraint from the slave. Since the master is more constrained than the slave, there is no problem here. It’s mostly the same as with a normal DROP INDEX, with a minor addition:

1. (optional) App: V1 -> V2. Discard using functionality which relies on index.
2. DB: V1 -> V2 (possibly failover from M1 to M2), change is DROP INDEX.
3. (optional) App: V2 -> V3. Enable functionality that inserts duplicates.

DROP FOREIGN KEY

Again, we are removing a constraint.

1. DB: V1 -> V2 (possibly failover from M1 to M2), change is DROP INDEX.
2. (optional) App: V2 -> V3. Enable functionality that conflicts with removed constraint. I mean, if you really know what you are doing.

DROP TABLE

The very simple steps are:

1. App: V1 -> V2. Make sure no reference to table is made.
2. DB: V1 -> V2. Issue a DROP TABLE.

With ext3 dropping a large table is no less than a nightmare. Not only does the action take long time, it also locks down the table cache, which very quickly leads to having dozens of queries hang. xfs is a good alternative.

Conclusion

We looked at single table operations, coupled with application upgrades. By carefully looking at the process breakdown, multiple changes can be addressed with ease and safety. Not all operations are completely safe when used with replication failover. But they are mostly safe if you have some trust in your code.

I will break down the discussion into types of database refactoring operations, and I will limit to single table refactoring. The discussion will try to understand the need for refactoring and will dictate the steps towards a successful upgrade.

Reader prerequisites

I will assume MySQL to be the underlying database. To take a major component out of the equation: we may need to deal with very large tables, for which an ALTER command may take long hours. I will assume familiarity with Master-Master (Active-Passive) replication, with possible use of MMM for MySQL. When I describe “Failover from M1 to M2“, I mean “Make the ALTER changes on M2 (passive), then switch your application from M1 to M2 (change of IPs, VIP, etc.), promoting M2 to active position, then apply same changes on M1 (now passive) or completely rebuild it”.

Phew, a one sentence description of M-M usage…

I also assume the reader’s understanding that a table’s schema can be different on master & slave, which is the basis for the “use replication for refactoring” trick. But it cannot be too different, or, to be precise, the two schemata must both support the ongoing queries for the table.

A full discussion of the above is beyond the scope of this post.

Types of refactoring needs

As I limit this discussion to single table refactoring,we can look at major refactoring operations and their impact on application & upgrades. We will discuss ADD/DROP COLUMN, ADD/DROP INDEX, ADD/DROP UNIQUE INDEX, ADD/DROP FOREIGN KEY, ADD/DROP TABLE.

We will assume the database and application are both in Version #1 (V1), and need to be upgraded to V2 or greater.

ADD INDEX

Starting with the easier actions. Why would you add an index? Either:

1. There is some existing query which can be optimized by the new query
2. Or there is some new functionality which issues a query for which the new index is required.

Adding an index is an easy action in that the table’s data does not really change.

In case #1, all you need to do is to add the new index (if the table is large, fail over from M1 to M2). There is no application upgrade, so all that happens is that the database upgrades V1 -> V2.

In case #2, the database must be prepared with new schema before the new functionality/query is introduced (since it depends on the existence of the index). The steps, therefore, are:

1. DB: V1 -> V2 (possibly failover from M1 to M2)
2. (Sometime later) App: V1 -> V2. Application will issue queries which utilize the new index.

The application does not have to be upgraded at the same instant the DB gets upgraded. In fact, we’ll see that this is a typical scenario: we can separate upgrades into smaller steps, which allow for time lapse. One could work out steps 1 & 2 together, but that would take an extra effort.

ADD COLUMN

This must be one of the most common table schema upgrades: a new property is needed on the application side. It must be supported by the database. Perhaps a new field in some Java Object, with Hibernate mapping that field onto a new column. Or maybe the new column is there for purpose of de-normalization.

This is also a more complicated task. Let’s look at the required steps:

1. DB: V1 -> V2 (possibly failover from M1 to M2), change is ADD COLUMN.
2. App: V1 -> V2. Change is: provide column value for newly INSERTed rows.
3. If needed, retroactively update column values for all pre-existing rows.
4. App: V2 -> V3. Application begins to use (read, SELECT) new column.

The above procedure assumes that the new column must have some calculated value. A 10-million rows table must now be updated, to have the correct values filled in. So we ask of the application to start filling in data for new rows, which makes the invalid row set static. We can just take a “from row” and a “to row” and fill in the missing column’s value for those rows. Only when all rows contain valid values can we let the application start using that row. This makes for two application upgrades.

If you’re content with just a static DEFAULT value, then step 3 can be skipped, and step 4 can be merged with step 2.

ADD UNIQUE INDEX

This is an altogether different case than the normal ADD INDEX, even though they may seem similar. And the case is particularly different when using Master-Slave failover for rebuilding the table.

Consider the case where we add a UNIQUE INDEX on a slave. Some INSERT query executes on the master, successfully, and is logged to the binary log. The slave picks it up, tries to execute it, to find that it fails on a DUPLICATE KEY error.

The UNIQUE INDEX is a constraint, and it makes the slave more constrained than the master. This is a delicate situation. Here how to (mostly) work it out:

1. App: V1 -> V2. Change INSERT queries on relevant table to INSERT IGNORE or REPLACE queries, whichever is more appropriate.
2. DB: V1 -> V2 (possibly failover from M1 to M2), change is ADD UNIQUE KEY (and while at it, a tip: are you aware of ALTER IGNORE TABLE?)

The change of query ensures that the query will succeed on the slave (either by silently doing nothing or by actually replacing content). It also means that the slave can now have different data than the master. Of course, it you trust your application to never INSERT duplicates, you can sleep better.

We do not handle UPDATE statements here.

ADD CONSTRAINT FOREIGN KEY

As with ADD UNIQUE INDEX, there is a new constraint here. A slave becomes more constrained than the master. But we now have to make sure INSERT, UPDATE and DELETE statements all go peacefully (well, it also depends on the type of ON DELETE and ON UPDATE property of the FK).

The steps would be:

1. DB: V1 -> V2 (possibly failover from M1 to M2), change is ADD CONSTRAINT FOREIGN KEY.

And then cross your fingers or have trust in your application. If the table is small enough, one does not have to use replication to do the refactoring, and life is simpler. Just execute the ALTER on the active master, and continue with your life.

CREATE TABLE

This is a simple case, since the table is new. The steps are:

1. DB: V1 -> V2 (no need to use slaves here)
2. App: V1 -> V2. Application will start using new table.

Conslusion

Having such steps formalized help with development management and database management. It makes clear what is expected of the application, and what is expected of the database. The breaking down of these operations into sequential steps allows us to work more slowly; make preparation work; work within our own working hours; get a chance to see the family.

In this post we took a look at “creation” refactoring changes. New columns, new keys, new constraints. In the next part of this article, we’ll discuss DROP operations.

We had speculated on the fate of Sun open source software under Oracle and while we acknowledged Oracle’s participation in, contribution and commitment to and opportunity from open source software, we questioned its appreciation of open source software communities beyond code and customers. It appears the OpenSolaris community and thus the OS itself, which we believe is key to advancing development of the more popular, proprietary cousin Solaris — are not a priority for Oracle.

The same cannot be said for all open source from Sun, and there’s a lot of it, now at Oracle. Amid the struggles of the OpenSolaris community, one of the other open source keystones from Sun, MySQL, seems to be doing well, despite persisting claims Oracle purchased Sun and MySQL simply to keep it from competing with Oracle database products. According to a Jaspersoft survey of customers/developers, there is a lack of awareness or concern of Oracle’s involvement in MySQL (59 percent were not aware Oracle reorganized and established a separate MySQL business unit apart from Oracle’s traditional RDBMS business …). Another 43% of Jaspersoft’s respondents said MySQL development and innovation would improve under Oracle.

The Jaspersoft survey found even more love for Java under Oracle, with 80 percent of respondents indicating they believe the Java process will improve or stay the same under Oracle. The related GlassFish application server also appears to be healthy with both community and commercial versions recently released.

The OpenOffice community appears also to be continuing forward supported and unfettered by Oracle (perhaps because it was typically fettered by Sun?), but it may also me failing to fully seize the opportunity.

It has also been interesting to see how Sun’s cloud computing technology has helped give Oracle new love for the term and the market.

There are a number of key open source projects and pieces from Sun, those listed above as well as many others, that may be on the line right now (or may have already been branded ’stay’ or ’stop’). We will be watching to see how Sun’s open source continues to shine or to set at Oracle.

I list four major places where SQL comments are helpful. I’ll use the sakila database. It is originally scarcely commented; I’ll present it now enhanced with comments, to illustrate.

Table definitions

The CREATE TABLE statement allows for a comment, intended to describe the nature of the table:

CREATE TABLE `film_text` (
 `film_id` smallint(6) NOT NULL,
 `title` varchar(255) NOT NULL,
 `description` text,
 PRIMARY KEY (`film_id`),
 FULLTEXT KEY `idx_title_description` (`title`,`description`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8 COMMENT='Reflection of `film`, used for FULLTEXT search.'

It’s too bad the comment’s max length is 60 characters, though. However, it’s a very powerful field.

Column definitions

One may comment particular columns:

CREATE TABLE `film` (
 `film_id` smallint(5) unsigned NOT NULL AUTO_INCREMENT,
 `title` varchar(255) NOT NULL,
 `description` text,
 `release_year` year(4) DEFAULT NULL,
 `language_id` tinyint(3) unsigned NOT NULL COMMENT 'Soundtrack spoken language',
 `original_language_id` tinyint(3) unsigned DEFAULT NULL COMMENT 'Filmed spoken language',
 `rental_duration` tinyint(3) unsigned NOT NULL DEFAULT '3',
 `rental_rate` decimal(4,2) NOT NULL DEFAULT '4.99',
 `length` smallint(5) unsigned DEFAULT NULL,
 `replacement_cost` decimal(5,2) NOT NULL DEFAULT '19.99',
  ...
) ENGINE=InnoDB AUTO_INCREMENT=1001 DEFAULT CHARSET=utf8

Stored routines definitions

Here’s an original sakila procedure, untouched. It is already commented:

CREATE DEFINER=`root`@`localhost` PROCEDURE `rewards_report`(
 IN min_monthly_purchases TINYINT UNSIGNED
 , IN min_dollar_amount_purchased DECIMAL(10,2) UNSIGNED
 , OUT count_rewardees INT
)
 READS SQL DATA
 COMMENT 'Provides a customizable report on best customers'
BEGIN

 DECLARE last_month_start DATE;
 DECLARE last_month_end DATE;
 ...

SQL queries

Last but not least, while not part of the schema, SQL queries define the use of the schema. That is, the schema exists for the sole reason of being able to query it.

Where did that query come from? Which piece of code issued it? Why? What’s its purpose?

Looking at the PROCESSLIST, the slow log, etc., it is easier when the queries are commented:

SELECT
 /* List film details along with participating actors */
 /* Issued by analytics module */
 film.*,
 COUNT(*) AS count_actors,
 GROUP_CONCAT(CONCAT(actor.first_name, ' ', actor.last_name))
FROM
 film
 JOIN film_actor USING(film_id)
 JOIN actor USING(actor_id)
GROUP BY film.film_id;

Conclusion

Source code commenting is an important practice, and usually watched out for. SQL & table definitions commenting are often scarce or non-existent. I urge DBAs to adopt a comments coding convention for SQL, and apply it whenever they can.

Before we get started here, let me state that I am using Ruby 1.9.1 (I refuse to look back!), and that I have not tested this solution on Ruby 1.8.6, but it should work there as well, though I may have some 1.9-isms in my code. Should be easy enough to spot.

I am working on writing an application in Ruby that can talk to an Windows application that has an ActiveX COM Automation object exposed. Ruby is basically the wrapper so that I can access the application from the Linux side of the world. So, I am using Ruby’s DRb to bridge those worlds because, after all, I am the Linux Bloke!

Well, as you may have guessed, I ran into problems with this approach. I simply could not call the COM objects from a call initiated with DRb, though I could call them directly just fine. After scratching my head a bit, I figured it out.

The win32ole module that runs on the Windows side of the world in Ruby only wants to run in the same thread that it was started in. win32ole is simply not thread-safe, and this has to do in large part to how ActiveX works under Windows. No need to delve into the gory details as we want code that works already!

DRb is very much all about threads. The DRb Server runs in a separate thread, and threads are launched each time a DRb request comes in. Threads abound like crazy! After all, it is very clear that the implementation of DRb was based, in part, on the Java threading model and Java’s RMI. But we knew that. We know that Ruby Threads parrot Java Threads. And I’ve done a lot of work with Java Threads in the past and almost feel a bit of “déjà vu” in working with them in Ruby. Oh the days…

But I digress.

We have a major problem here. How do we get around it, without having to throw out DRb and doing something funky like writing some custom RPC bit just to make Windows happy?

Well, as you may have guess, the Linux Bloke created the very solution you need!! Funnel!

Funnel works by wrapping a given object with a “meta” object that can then be called from any thread. All the calls are actually queued up and processed by the thread the target object wants to run in. The calling threads block until the target object returns the call, and the result objects are stuffed somewhere so that the calling thread can find them.

It’s all very transparent and you need not do anything special — much. You will need to call process_funnel_messages() in the funneled thread. And you may do this once in which case process_funnel_messages() will loop forever and never return, or you can call it at regular intervals if you need to do other processing in that same thread.

You, of course, can use Funnel anywhere you need to funnel calls from multiple threads to a single thread to access something that is not inherently thread-safe or thread-aware.

The downloadable code is posted here:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73

=begin rdoc
Funnel created by Fred Mitchell (LinuxBloke.com) on 2010-06-05                         
 
=Funnel -- funnel calls to an object to a specific thread that created said object.    
 
With some systems, like win32ole, the system basically wants to run on the same thread
the system was started on. To facilitate that need in a multi-threaded environment,
we create the Funnel.                                                                  
 
The Funnel wrapper on an object will basically intercept all method calls and
funnel those calls to the wrapped object in the thread it was created in. The
caller thread will basically block until the Funnel calls the target object's method
and will be given, as a return, the result object of that call.                        
 
The Funnel thread will basically sit in a loop waiting for something to come in,
and wake up to process the entries, then go back to sleep until the next ones come
in.                                                                                    
 
Any exceptions (or errors) that occur in the Funnel shall be
thrown to the caller thread, as though the exception took place in that thread.        
 
This code is released under the GPLv3.                                                 
 
=end                         
 
module Funnel
  class Wrapper
    def initialize(target)
      @targetOb = target
      @targetThr = Thread.current
      @targetThr[:methQueue] = [] if @targetThr[:methQueue].nil?
    end                                                                                
 
    def method_missing(meth, *parms)
      Thread.current[:methResult] = :nothing_yet
      @targetThr[:methQueue] << [@targetOb, meth, Thread.current, parms]               
 
      # Thing is, we may have gotten a response already!
      while Thread.current[:methResult] == :nothing_yet
        if @targetThr.stop?
          @targetThr.wakeup
          # Thread.stop
        end
        Thread.pass
      end
      Thread.current[:methResult]
    end
  end                                                                                  
 
  # Called by the orginal thread to process object messages.
  # This function never returns.
  def process_funnel_messages(loop_forever = true)
    begin
      meth = nil
      (ob, meth, thr, parms) = Thread.current[:methQueue].shift unless Thread.current[\
:methQueue].nil?
      unless meth.nil?
        begin
          thr[:methResult] = ob.send(meth, *parms)
          thr.run
        rescue
          thr.raise($!)
        end
      else
        Thread.stop if loop_forever
      end
    end while loop_forever
  end                                                                                  
 
  def wrap(target)
    Wrapper.new(target)
  end
end

And here is an example of its use:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

require 'funnel'                                                                       
include Funnel                                                                         
 
class StupidThreadUnsafeThing                                                          
  def callme                                                                           
    puts "*** I've been called. My thread is"                                          
    p Thread.current                                                                   
    puts                                                                               
  end                                                                                  
end                                                                                    
 
stut = StupidThreadUnsafeThing.new                                                     
 
# This is the easy to use wrapper                                                      
fstut = wrap stut                                                                      
 
stut.callme                                                                            
 
Thread.new do                                                                          
  10.times do |i|                                                                      
    sleep 1                                                                            
    Thread.new {                                                                       
      puts "XXX #{i} calling stut from thread"                                         
      p Thread.current                                                                 
      fstut.callme                                                                     
    }                                                                                  
  end                                                                                  
  exit                                                                                 
end                                                                                    
 
# Here we loop forever processing messages.                                            
# Optionally, we could call this repeateady                                            
# to process messages by using a parameter of                                          
# "false".                                                                             
process_funnel_messages

This code is fairly straightforward, as you can see. If there is enough interest, I’ll consider turning this into a gem.

First of all, here is what you may need this feature. Let’s imagine you have a file storage on Amazon S3 where you store tons of content. And you have an application where you have some content downloading functionality that you want to be available for logged-in/paying/premium users and/or you want to keep track of downloads your users perform on your site. If your content was on your web server, you could have used simple controlled downloads functionality built-in to nginx out of the box. But the problem is that your content is remote.

Here is what we do to solve this problem.

First, we create a special location on our nginx server. This location will be used as a proxy for all our accelerated file downloads:

After adding this location to our nginx config we could start sending responses with headers like the following:

Here is an example code you could use in a Rails application to use our internal redirect location:

As you can see, nginx is really powerful tool and when you turn your creativity on you can make it even more powerful. Stay tuned for more Nginx-Fu posts.

It is a complex problem that usually consists of two smaller ones: individual cache elements invalidation (you need to keep an eye on your data changes and invalidate cached pages when related data changes) and full cache purges (sometimes your site layout or page templates change and you need to purge all the cached pages to make sure users will get new visual elements of layout changes). In this post I’d like to look at a few techniques we use at Scribd to solve cache invalidation problems.

Since we use haproxy to balance our traffic in the cluster it is hard to predict where should we send a purge request. So we fan those out to all cache servers.

To make sure cache purging won’t slow the site down, especially considering we need to do more that just a simple cache purge (submit documents to search indexes, etc, etc), we just spool a “document changed” request to a queue and then have a set of asynchronous processes that do all the work in background.

Next, The Hard Problem – handling full cache purges w/o killing our backend servers with 5x-10x traffic (our normal hit ratio is ~90-95%).

We’ve spent a lot of time thinking about this problem and the first idea we came up with was to have a loop process somewhere that would iterate all documents we have cached and purge them one by one… but that does not seem to be a practical solution when you have tens of millions documents (and few page versions per document) and obviously the solution would not scale with constantly growing documents corpus.

So we kept brainstorming and finally got one idea that works just perfectly for us: what if we’d be able to take our traffic and define a function f(t) that would return a percentage of the traffic that should be purged at any moment in time. So we did it – we’ve implemented a nginx module that would version our cache by assigning every cached page a revision (using a custom HTTP-headers + Vary-caching) and would be able to slowly migrate the cache from one revision to another over a pre-defined period of time.

Having that module we are able to do so called “slow” cache purges that could take any time from a few minutes (that still helps to reduce the load spike generated by the hottest content) up to many hours (this is what we normally use) or days (never used this option, but it is definitely possible).

Here is an example 100% cache purge over an 8 hour interval:

1. Daily hit ratio graph:
   
2. Weekly hit ratio graph:
   

As you can see, during those slow purges our cached pages would be slowly updated without putting too much pressure on the backend. Cache hit ratio would slowly degrade and then slowly get back to its normal levels, but with our normal (6-8 hours) purges hit ratio never gets lower that 65-70% which makes it possible for us to save huge amounts of money on not having 90% spare capacity just for the cache purge load surges (we used to have lots of spare application cluster capacity before introducing this approach).