Recent Tips and News on Java, Java EE 6, GlassFish & more :
PlanetMySQL Voting: Vote UP / Vote DOWN
Archive for the ‘json’ Category
Tab Sweep — JSON, OSGi, GlassFish+NetBeans experience, JAXB and JodaTime, file uploads & more
Январь 7th, 2012PlanetMySQL Voting: Vote UP / Vote DOWN
Liveblogging at Confoo: Blending NoSQL and SQL
Март 11th, 2010Persistence Smoothie: Blending NoSQL and SQL – see user feedback and comments at http://joind.in/talk/view/1332.
Michael Bleigh from Intridea, high-end Ruby and Ruby on Rails consultants, build apps from start to finish, making it scalable. He’s written a lot of stuff, available at http://github.com/intridea. @mbleigh on twitter
NoSQL is a new way to think about persistence. Most NoSQL systems are not ACID compliant (Atomicity, Consistency, Isolation, Durability).
Generally, most NoSQL systems have:
- Denormalization
- Eventual Consistency
- Schema-Free
- Horizontal Scale
NoSQL tries to scale (more) simply, it is starting to go mainstream – NY Times, BBC, SourceForge, Digg, Sony, ShopWiki, Meebo, and more. But it’s not *entirely* mainstream, it’s still hard to sell due to compliance and other reasons.
NoSQL has gotten very popular, lots of blog posts about them, but they reach this hype peak and obviously it can’t do everything.
“NoSQL is a (growing) collection of tools, not a new way of life.”
What is NoSQL? Can be several things:
- Key-Value Stores
- Document Databases
- Column-oriented data stores
- Graph Databases
Key-Value Stores
memcached is a “big hash in the sky” – it is a key value store. Similarly, NoSQL key-value stores “add to that big hash in the sky” and store to disk.
Speaker’s favorite is Redis because it’s similar to memcached.
- key-value store + datatypes (list, sets, scored sets, soon hashes will be there)
- cache-like functions (like expiration)
- (Mostly) in-memory
Another interesting key-value store is Riak
- Combination of key-value store and document database
- heavy into HTTP REST
- You can create links between documents, and do “link walking” that you don’t normally get out of a key-value store
- built-in Map Reduce
Map Reduce:
- Massively parallel way to process large datasets
- First you scour data and “map” a new set of dataM
- Then you “reduce” the data down to a salient result — for example, map reduce function to make a tag cloud: map function makes an array with a tag name and a count of 1 for each instance of that tag, and the reduce tag goes through that array and counts them…
- http://en.wikipedia.org/wiki/MapReduce
Other key-value stores:
Document Databases
Some say that it’s the “closest” thing to real SQL.
- MongoDB – Document store that speaks BSON (Binary JSON, which is compact). This is the speaker’s favorite because it has a rich query syntax that makes it close to SQL. Can’t do joins, but can embed objects in other objects, so it’s a tradeoff
- Also has GridFS that can store large files efficiently, can scale to petabytes of data
- does have MapReduce but it’s deliberate and you run it every so often.
- CouchDB
- Pure JSON Document Store – can query directly with nearly pure javascript (there are auth issues) but it’s an interesting paradigm to be able to run your app almost entirely through javascript.
- HTTP REST interface
- MapReduce only to see items in CouchDB. Incremental MapReduce, every time you add or modify a document, it dynamically changes the functions you’ve written. You can do really powerful queries as easy as you can do simple queries. However, some things are really complex, ie, pagination is almost impossible to do.
- Intelligent Replication – CouchDB is designed to work with offline integration. Could be used instead of SQLite as the HTML5 data store, but you need CouchDB running locally to be doing offline stuff w/CouchDB
- Pure JSON Document Store – can query directly with nearly pure javascript (there are auth issues) but it’s an interesting paradigm to be able to run your app almost entirely through javascript.
Column-oriented store
Columns are stored together (ie, names) instead of rows. Lets you be schema-less because you don’t care about a row’s consistency, you can just add a column to a table very easily.
- Cassandra – Built by Facebook, also used by Twitter
- BigTable
- Hypertable
- HBase
Graph Databases
speaker’s opinion – there aren’t enough of these.
Neo4J – can handle modeling complex relationships – “friends of friends of cousins” but it requires a license.
When should I use this stuff?
| If you have: | Use |
|---|---|
| Complex, slow joins for an “activity stream” | Denormalize, use a key-value store. |
| Variable schema, vertical interaction | Document database or column store |
| Modeling multi-step relationships (linkedin, friends of friends, etc) | Graph |
Don’t look for a single tool that does every job. Use more than one if it’s appropriate, weigh the tradeoffs (ie, don’t have 7 different data stores either!)
NoSQL solves real scalability and data design issues. But financial transactions HAVE to be atomic, so don’t use NoSQL for those.
A good presentation is http://www.slideshare.net/bscofield/the-state-of-nosql.
Using SQL and NoSQL together
Why? Well, your data is already in an SQL database (most likely).
You can blend by hand, but the easy way is DataMapper:
Generic, relational ORM (adapters for many SQL dbs and many NoSQL stores)
Implements Identity Map
Module-based inclusion (instead of extending from a class, you just include into a class).
You can set up multiple data targets (default is MySQL, example sets up MongoDB too).
DataMapper is:
- Ultimate Polyglot ORM
- simple r’ships btween persistence engines are easy
- jack of all, master none
- Sometimes perpetuates false assumptions –
- If you’re in Ruby, your legacy stuff is in ActiveRecord, so you’re going to have to rewrite your code anyway.
Speaker’s idea to be less generic and better use of features of each data store – Gloo – “Gloo glues together different ORMs by providing relationship proxies.” this software is ALPHA ALPHA ALPHA.
The goal is to be able to define relationships on the terms of any ORM from any class, ORM or not
Right now – partially working activeRecord relationships
Is he doing it wrong? Is it a crazy/stupid idea? Maybe.
Example:
| Need | Use |
|---|---|
| Assume you already have an auth system | it’s already in SQL, so leave it there. |
| Need users to be able to purchase items from the storefront – Can’t lose transactions, need full ACID compliance | use MySQL. |
| Social Graph – want to have activity streams and 1-way and 2-way relationships. Need speed, but not consistency | use Redis |
| Product Listings — selling moves and books, both have different properties, products are pretty much non-relational | use MongoDB |
He wrote the example in about 3 hours, so integration of multiple data stores can be done quickly and work.
PlanetMySQL Voting: Vote UP / Vote DOWN
RESTful PHP Web Services – reviewed
Январь 21st, 2010I’ve been using a lot of RESTful services these days and have been waiting for a good book that is dedicated to the topic. I recently received a copy of ‘RESTful PHP Web Services’, which does a successful job of outlining proven concepts in current web technology. If you want to learn the methods for creating and consuming RESTful services then you will find many examples in this book. From the architectural plans to well thought out code samples, the book covers a lot of ground in a relatively quick read.
The first chapter gives the reader a quick introduction to RESTful services and the most common PHP frameworks in use at the time of writing. I particularly enjoyed the section on the Zend framework due to the explanation of benefits over the other frameworks. The chapter also covers the very basics which include a detailed look at exactly what RESTful services means and what technologies are required to use and benefit from a RESTful architecture. The second chapter gives a quick run down of the various methods in use for consumption of data; these being Curl, several HTTP methods, processing data with XML, DOM, and SimpleXML. After those are covered there is a simple example of consuming services like Flickr using the previous methods. This transitions into many more examples of consuming real world services that any developer would find interesting and exciting for data mashups.
The real meat of the book starts in chapter four where we get into designing the resource utilization systems and then the resource clients in chapter five. Those topics basically go over the nuts and bolts of gathering data, manipulating it, updating it, as well as creating fresh data. We get more instruction and usage examples on the Zend framework in chapter seven where the author gives us information on the controllers, models, and view (MVC model). This would not be too useful without knowing how to debug the code that we’re using so there is, thankfully, a chapter dedicated to debugging XML building and parsing errors. A couple of short appendixes cover the author’s own WSO2 web service framework as well as REST Client Classes which should prove useful for writing your own reusable classes.
Overall this book covers the majority of topics that a new developer needs to understand in order to start developing and deploying RESTful code and web services in PHP. From frameworks to consumable service samples, and everything in between, RESTful PHP Web Services comes through in a concise and enjoyable style that will not disappoint. I highly recommend this book for developers that are new to this topic or experienced developers that need a quick refresher course.
PlanetMySQL Voting: Vote UP / Vote DOWN
Drizzle query monitoring
Август 28th, 2009Disclaimer:
This blog post is about things I did on my own free time, not endorsed by my employer.
Meanwhile, a few weeks ago, I started looking at the REST API that comes with the MySQL Enterprise Monitor.
The result is that we can now see most of the information returned by the plug-in, on the Dashboard.
How?
A colleague at work, wrote a little Perl script that interacts with the REST API, and I took his work as the foundation for my agent.pl script.
The next problem was to find a way to call this script as soon as there was a new entry on the log. After a little Google search, I went ahead and decided to ask my friend Adriano Di Paulo (who among other things, introduced me to MySQL).
A few minutes later, he showed me a working example of the Tail Perl module.
That was exactly what I needed, as soon as there is a new entry, I call the function assemble_queries() and I pass the new log entry as the parameter.
The assemble_queries() function is mostly based on what MC blogged about some time ago. On his blog post, he shows how to collect query related data using Dtrace and Perl.
Then, every n number of queries, I use send_json_data() to send the query information to the Dashboard, delete the sent data and it is ready to process more queries.
sub tail_log {
my $file=File::Tail->new(name=>$log_file, maxinterval=>1, reset_tail=>0);
while( defined (my $line=$file->read ) ) {
print "\n" . $line . "\n" if $DEBUG > 3;
assemble_queries( $line );
}
}
The assemble_queries() function is mostly based on what MC blogged about some time ago. On his blog post, he shows how to collect query related data using Dtrace and Perl.
Then, every n number of queries, I use send_json_data() to send the query information to the Dashboard, delete the sent data and it is ready to process more queries.
Now that I'm writing this, I realized that if send_json_data() fails, the information related to the queries are lost :|. (Note to self, fix it).
There are other functions in there, but they are mostly for housekeeping.
How do I use it?
Very simple, get the agent.pl script from the MySQL Forge website, edit the credentials, hosts, and ports to fit your needs (Future versions would include some kind of config file).
And then you call the script like this:
As soon as the scripts starts, it will add the drizzle server to the service manager, and once you start sending queries to drizzle, those queries will end up on the UI.
Next?
Next is already done :). I modified the agent.pl script to use the gearman logging plugin. I'll write a blog about it very soon.
Thanks for reading and enjoy!
There are other functions in there, but they are mostly for housekeeping.
How do I use it?
Very simple, get the agent.pl script from the MySQL Forge website, edit the credentials, hosts, and ports to fit your needs (Future versions would include some kind of config file).
And then you call the script like this:
$ DEBUG=1 perl agent.pl --serveruuid="22222222-5555-5555-5555-222222222211" \
--serverhostuuid="ssh:{11:11:11:11:11:11:11:11:11:11:11:11:11:11:11:21}"\
--serverdisplayname="Main-Drizzle-web2" \
--log-file=/path/to/log/file
As soon as the scripts starts, it will add the drizzle server to the service manager, and once you start sending queries to drizzle, those queries will end up on the UI.
Next?
Next is already done :). I modified the agent.pl script to use the gearman logging plugin. I'll write a blog about it very soon.
Thanks for reading and enjoy!
PlanetMySQL Voting: Vote UP / Vote DOWN