Topics for this podcast:
*Hadoop v1.0 and year ahead
*Oracle-Cloudera deal for more Hadoop
*Oracle’s ‘Sun spot’ with Solaris
*Open Source M&A outlook for 2012
*Our new MySQL/NoSQL/NewSQL survey
iTunes or direct download (28:49, 4.9MB)
Previously, I’ve called out years for non-desktop Linux in 2008, Linux in both the low and high-ends of the market in 2009, ‘hidden’ Linux in 2010 and last year, cloud computing in 2011. For 2012, I see continued growth, prevalence, innovation and impact from Linux, thus leading to a 2012 that is dominated by Linux.
I expect to see nothing but continued strength for Linux and open source in cloud computing in 2012. The cloud continues to be the biggest disruptor and opportunity for Linux providers. 2012 got off to an interesting start with Microsoft’s efforts to support for Linux on Azure, which highlights just how pervasive Linux has become in cloud computing. As detail in our special report on The Changing Linux Landscape, we also expect Linux to continue to be the basis for most offerings in IaaS and particularly PaaS, which is burgeoning across open source languages and frameworks as well as verticals and enterprise customers. Its popularity among enterprise and other developers will also bolster Linux and open source software in 2012.
We can certainly expect to see Linux continue its domination in supercomputing and the Top 500 Supercomputer List, where Linux continues to grow its share above 90% while others, such as Microsoft, Apple and BSD, fall off of the list.
I also expect Linux will grow its presence and impact on the wider, more mainstream server market, where Red Hat and SUSE continue to benefit from Unix migration, particularly from Solaris. Our analysis with survey data from 451 Research division TheInfoPro shows server spending for databases and data warehousing favoring Red Hat with Linux over Oracle with either Linux or Solaris. Out of more than 165 server professionals interviewed by TIP, 67% are planning to spend more with Red Hat on database/data-warehousing, and only 6% plan to spend less. The positive figures for Red Hat mirror negative spending intentions for Oracle, with 55% planning to spend less and only 9% planning to spend more. Spending continues to decline strongly for all of the primary Unix providers in the study, which in addition to Oracle includes IBM and Hewlett-Packard.
We may also see further expansion for Red Hat, which may be eyeing key acquisitions, and other Linux and open source vendors as they continue building their channels and wade more into midmarket and SMB customers.
In smartphones and mobile software, I also expect Linux will do quite well in 2012 with continued Android strength, diminished FUD and possibly an open source boost from a newly-open sourced WebOS. We also see Ubuntu arriving on the mobile and converged device scene, including ‘concept’ appearance at CES.
We’re also likely to see Linux in automobiles, health care and other electronics even more in 2012, though you may never hear Linux or open source. Don’t be fooled though, Linux is expanding its already impressive, wide presence and 2012 looks to be another year of significant gains.
Oracle released many MySQL 5.6 and MySQL Cluster 7.2 DMRs accompanied by new versions of MySQL Enterprise Monitor, MySQL Enterprise Backup, MySQL Workbench (and utilities), MySQL Proxy, MySQL Cluster Manager and Connectors.
The reason we are confident that the comparative decline in the use of the GNU GPL family of licenses and the increasing significance of complementary vendors in relation to funding for open source software-related vendors will continue is due to the analysis of our database of more than 400 open source software-related vendors, past and present.
We previously used the database to analyze the engagement of vendors with open source projects for our Control and Community report, plotting the strategies used by the vendors against the year in which they first began to engage with open source projects to get an approximate view of open source-related strategy changes over time.
For example, we found that the engagement of vendors with projects that used strong copyleft licenses peaked in 2006, while the engagement of vendors with projects using non-copyleft licenses had been rising steadily since 2002.
Analysis of our updated database shows that the the number of new vendors engaging with open source projects in each year has risen steadily in recent years, from 26 in 2008 to 44 in 2011. However, as noted last week, we have also seen a shift towards ‘complementary vendors’ – those that are dependent on open source software to build their products and services, even though those products and services may not themselves be open source.
2010 was the first year in which we saw more complementary vendors engage with open source projects than open source specialist, and that trend accelerated in 2011.
As previously explained, complementary vendors were responsible for over 30% of open source software-related funding raised in 2011, and we should expect that proportion to remain high given that over 57% of the vendors engaging with open source in 2011 were complementary vendors.
We have also seen that complementary vendors are more likely to engage with projects with non-copyleft licenses (38% of complementary vendors have engaged with projects with non-copyleft licenses, compared to 24% that have engaged with projects with strong copyleft licenses).
If we look at all 400+ vendors in our database in terms of open source software license preference, the trend towards new vendors engaging with non-copyleft licenses is clear.
There has been a strong shift from vendors towards non-copyleft licenses in recent years, accelerated in 2011 by the likes of Apache Hadoop and OpenStack in particular. This does not mean that the number of projects using strong copyleft licensing has decreased (although as we previously saw the proportion of projects using the GPL family of licenses has declined).
It is indicative, we believe, of the shift away from specialist open source vendors using vendor-led projects and strong copyleft licenses towards multi-vendor collaborative projects and proprietary implementations of open source code, however.
This trend should not really surprise anyone. For some time we have seen open source becoming part of the fabric of modern software development and licensing strategies, rather than a competitive differentiator. Back in 2009 we predicted the increased importance of business strategies that relied on vendor-led development communities, rather than projects dominated by a single vendor.
We called this “open source 4.0″ and later suggested that it might be considered the golden age of open source, based on our belief that vendors had learned that they stand to gain more from collaborating on open source projects and differentiating at another stage in the software stack than they do from attempting to control open source projects.
Updating the results of our analysis to the end of 2011 and 400+ vendors indicates that, from the perspective of the commercial adoption of open source business strategies at least, we were not far off.
Some might not consider the proliferation of multi-vendor open source communities and proprietary distributions of open source software as the peak of achievement for open source. Each is of course entitled to come to their own conclusions about the implications.
Our perspective, as always, is that open source methodologies present a potentially disruptive, and also valuable, asset that complements the way both vendors and enterprise IT organizations conduct their businesses.
Our analysis indicates, however, that open source methodologies are increasingly being employed by ‘complementary vendors’ with a leaning towards more permissive licensing.
One of the favourite blog topics on CAOS Theory blog over the years has been our quarterly and annual updates on venture capital funding for open source-related businesses, based on our database of over 600 funding deals since January 1997 involving nearly 250 companies, and over $4.8bn.
There are still a few days left for funding deals to be announced in 2011 but it is already clear that 2011 will be a record year. $672.8m has been invested in open source-related vendors in 2011, according to our preliminary figures, an increase of over 48% on 2010, and the highest total amount invested in any year, beating the previous best of $623.6m, raised in 2006.
Following the largest single quarter for funding for open source-related vendors ever in Q3, Q4 was the second largest single quarter for funding for open source-related vendors ever, as $230.4m was invested in companies including Cloudera, Hortonworks, and Rapid7.
As with Q3, however, the list of vendors presents us with something of an existential dilemma, as we see an increasing amount of activity by what we have referred to as ‘complementary vendors’ – those that are dependent on open source software to build their products and services, even though those products and services may not themselves be open source – as opposed to open source specialists.
The list of complementary vendors has grown rapidly in 2011, particularly around projects such as OpenStack and Apache Hadoop. If we examine the figures in more detail we find that over 30% of the funding raised in 2011 was raised by complementary vendors, compared to just 4% in 2006.
In fact, as the chart below indicates, VC funding for specialist open source vendors in 2011 was actually less than that in 2006 and 2008, and only marginally up on 2010, when again just 4% of funding went to complementary vendors.
The low amount of funding for complementary vendors in 2010 shows that the significance of complementary vendors is not growing at a constant rate, although for reasons that will become clear when we publish a follow-up post on the latest trends regarding the engagement of vendors with open source projects, we do expect that the proportion of funding related to complementary vendors is more likely to increase in the future, rather than decline.
This has implications for the ongoing trends related to open source software licensing, as covered yesterday. Examining our database of over 400 open source-related vendors – funded and unfunded, complementary and specialist – indicates that specialist vendors are much more likely to engage with projects using strong copyleft licenses than complementary vendors.
Specifically, our data indicates that 55% of open source specialists have engaged with projects that use strong copyleft licenses, while just 20% have engaged with projects with non-copyleft licenses. In comparison, 38% of complementary vendors have engaged with projects with non-copyleft licenses, compared to 24% that have engaged with projects with strong copyleft licenses.
Will will take a more detailed look at the trends related to the engagement of vendors with open source projects in the concluding part of this series of posts.
There has been no shortage of reaction to HP’s move to make the Linux-based WebOS open source software. Below, I offer some of my thoughts on the meaning for the different players affected.
*What’s it mean for WebOS?
Moving WebOS to open source was best option for HP. It retains some value in the software depending on its involvement. It is also the best fate for the code, rather then being sold or simmered to its IP and patent value or even used as another weapon in the ongoing mobile software patent wars. Still, the move comes amid huge developer and consumer uncertainty for WebOS. Nevertheless, at least WebOS was already in the market with a compelling products, the Palm the Pre, in the modern smartphone market. WebOS will hopefully have a faster path to open source than Symbian since the former is based on Linux. I still think the greatest opportunity for WebOS may be in serving as an open alternative in the market, particularly after Android has proven to handset makers, wireless carriers, OEMs and others that a Linux-based, open source mobile OS can succeed in the market and provide profit for multiple parties. Furthering this opportunity, WebOS may be even more attractive to these key vendors, channel players and other stakeholders who are tired of the IP and patent stress and expense around Android. Of course, Android was not under patent or IP attack until it was successful in the market and the same may be the case for WebOS, though we think its IP roots and history in touch and smartphone technology are less complex in terms of origin and ownership.
*What’s it mean for competitors?
For Apple, an open source WebOS means more market pressure and open pressure, more competition for developers and a real danger WebOS hooks into the Android ecosystem. WebOS may also be harder to attack from a patent and IP standpoint since it is older and more singular in ownership (Palm and now HP). Other factors include HP’s own formidable patent portfolio and the perception of Apple as a patent aggressor, which would be reinforced if it attacked WebOS the way it has gone after Android.
For Android, it may finally get a dose of its own open medicine, feeling the pressure of another Linux-based, open source mobile OS that is familiar to many developers, compatible with newer smartphone technologies and appealing to handset makers and other key OEMs. However, WebOS is also a validation of Android, which paved the path for mobile Linux and open source to finally break through beyond geeks to reach a mass consumer audience.
As for other proprietary players such as Microsoft and RIM, another open source rival is bad news. It presents another open source option and potentially serious competition on developers, applications, devices, carriers and consumers. An open source WebOS may also make Android, in effect, more open with faster, easier access to code for both Android and WebOS compete. This could make it even harder for these older, proprietary players to get developer or consumer mind share that is already slipping.
*What’s it mean for open source? Really, there is no downside for open source except that it will be viewed as a form of software cemetery if WebOS is not developed or delivered to market. HP’s WebOS move does give open source greater prominence in mobile software. Again, it is a validation of Android, which is Linux-based and open source, and shows that we haven’t seen the last of mobile Linux and open source software in Android.
Funding for BlazeMeter and Digital Reasoning. Red Hat goes unstructured. And more.
# BlazeMeter announced $1.2m in Series A funding and launched the a cloud service for load and performance testing.
# Digital Reasoning announced a second round of funding to help develop its Hadoop-based analytics offering.
# Red Hat announced the availability of Red Hat Storage Software Appliance, based on its recent acquisition of Gluster.
# Red Hat also announced the general availability of Red Hat Enterprise Linux 6.2.
# Jaspersoft released Jaspersoft 4.5, delivering drag-and-drop analytics and reporting on Apache Hadoop, NoSQL and analytic databases.
# Jaspersoft also delivered a second-generation native connector to MongoDB.
# CloudBees announced the availability of Jenkins Enterprise by CloudBees providing support and enhanced capabilities for the Jenkins Continuous Integration platform.
# Diaspora* is back in action, and outlined its plans.
# Talend announced that Bi3 Solutions has embedded Talend Integration Suite inside its Software-as-a-Service platform.
# DataStax announced new versions of Apache Cassandra, DataStax Community, and DataStax Enterprise.
# The H reported that Microsoft’s Windows Store agreement has open source exception.
# Black Duck Software announced the release of Export 6.0.
# Antelink launched SourceSquare, a free open source scanning engine.
Object-relational mapping (ORM) frameworks have been around for several years now and for some people, ORM is already outdated by now. As we have seen with other technologies and concepts before, PHP is not exactly what we call an early adopter among the programming languages. Thus it took some time for ORM to grow up in the PHP context.
There have been some frameworks before Doctrine 2 that implement ORM (remember e.g. Propel) specific tasks but most of them lack the required maturity to be used in large projects. With Doctrine 2, PHP takes a huge step into the right direction Doctrine 2 is fast, extensible and easy to use.
This article will take you on a tour through the main concepts of Doctrine 2 in the first part and then explain how to use it in a real world application in the second part. Since at the time of writing Zend Framework 1.11.xx (ZF) is very popular, we will integrate Doctrine 2 into a ZF project.
To understand Doctrine 2, we have to take a look at some relevant terms (or in this case objects), study their behavior and practice their usage. We start with some introductory phrases on ORM systems and then go on to the concepts underlying Doctrine 2: Entity Objects, the Entity Manager, Repositories and Proxies.
Since the beginning of Object-Orientation, people had to manage the persistence of their application's state resp.
their objects. In the context of Web Application Development, this usually involves a Database server which is being
consulted using a Query Language. One example for this pattern is a PHP application that uses some kind of SQL server
by sending SQL queries to it. Another one is an application using a CouchDB server by querying it via its REST API.
Due to the author's laziness, we will talk in terms of relational databases from now on. Keep in mind, that you can
accomplish almost everything mentioned here with NoSQL databases, too.
ORM relates value objects that exist in an application's business logic to database records.
Thus every object that should be persistent is saved in one row of a database table. The most common approach is to
map classes to tables and the classes' objects to rows in the these tables.
Besides writing objects to a database, ORM systems are also intended to ease the process of finding data stored in the
database. When talking in terms of ORM, finding data always means making the framework fetch one or many objects
that meet a certain criteria.
The objects that are being managed by an ORM system are called Entity Objects. Every entity object relates to one entry in a table. In Doctrine 2, the classes that represent entities do not have to fulfill special requirements like inheriting from a certain super class (as you might have seen in other database abstraction frameworks like Zend_Db). When creating a new entity class with Doctrine 2, all you have to do is to write down a regular PHP class with properties. Besides this, you have to provide some hints on how these attributes should be persisted. The information how entity attributes relate to columns in the DB is called Metadata. Metadata can be described in different ways: By default there are metadata drivers for descriptions in XML, YAML and PHP. The fourth and most popular driver is based on DocBlock annotations (since in PHP, annotations aren't a language feature as in Java (see Wikipedia), they are contained by the classes' and attributes' DocBlocks). We will use annotations to describe our entities metadata. To get an impression on how easy this is, take a look at the following example.
This example contains all it needs to tell Doctrine 2 about the new entity User. With this class, you can create,
find, delete and modify user objects and persist their state to the underlying database. But keep in mind: as long
as you don't need any persistence features, you can use your user objects just like any other objects!
The next two objects resp. object types we will describe are responsible for doing the ORM functionality: persisting and finding.
To use ORM functionality, the Entity Manager (Doctrine\ORM\EntityManager) is the main access point to Doctrine 2. The entity manager is
responsible as you might have guessed for managing entities and for building a facade for the whole framework.
To accomplish its tasks, the entity manager uses some helpers. The Unit of Work object for example collects entities
that should be written back to the database and is capable of doing this in batches. This way, database operation
can be executed with almost no overhead and therefore are really fast.
Another dependency of the entity manager is the Event Manager. To be as extensible as possible, Doctrine 2 comes with an event system that publishes all important state changes to the outside as events. You can register for such events and extend the life cycle of your entity objects at one single point.
The entity manager's API combines methods for managing entities (find, persist, contains, copy, detatch, merge,
remove and refresh), methods that control the use of transations (beginTransaction, commit, flush, rollback and
transactional) and some helper methods for creating custom queries and accessing some of the entity manager's dependencies.
The following example shows how to query an object from the entity manager, modify it and write the changes back into the database.
Creating a new persistent object is almost as easy as modifying it:
For finding entities, Repositories are used. Every entity class has its own repository which is responsible for finding entities of that type. By default, repositories have some handy methods for fetching entities that match certain criteria:
find: Finds an entity by its primary key / identifierfindAll: Finds all entities of the repository's entity typefindBy / findOneBy: Finds all resp. one entity that matches the passed criteria:findBy<attribute> / findOneBy<attribute>: Magic methods that ease the filtering by a single attribute:
To access a repository, all you have to do is ask the entity manager for one. If you have implemented your own
repository, it will be returned by Doctrine\ORM\EntityManager::getRepository(). Otherwise, Doctrine 2 will provide
a generic repository. The main reason to implement custom repository classes is to group custom queries for an entity
type to make them reusable. For custom query logic, there are several mechanisms you can use: You can either use
Doctrine's query builder that implements an API similar to Zend_Db_Select or queries written in the Doctrine Query
Language (DQL) or you can even execute plain SQL queries. With these options, it is also possible to migrate old
applications which use complex queries by just wrapping these queries into the methods of custom repositories.
When traversing a graph of entity objects (which is required when entities are having relations to other entities),
it would be very expensive (in the sense of requiring many database queries) to fetch every depending entity with
an additional query. Therefore Doctrine 2 uses the concept of Proxy objects that represent regular entity objects
which have not been populated with data from the database. Take a look at the following example where the entity
Group aggregates a list of User objects in its member property. When accessing the members list, Doctrine 2 provides a
collection of proxy objects instead of complete User objects. When an object of this collection is being asked for
one of its properties, Doctrine loads the object's data from the database. This way, the users' data is not loaded
until it is really needed.
This section describes some advanced concepts that are required when mapping entity classes that have relationships to
other entity classes. Possible relationship types are association and inheritance. Inheritance is the mechanism used for
representing subtypes in object-oriented programming languages. An example would be a class User that implements
methods every user of a software should have and a class Administrator inheriting from User that adds methods for
determining the administrator's access rights.
Association is a weaker relation type. It means that an entity object can be related to other entity objects of other types. In terms of relational databases, there are three types of association which differ in the number of entities an object is related to: 1:1, 1:n and n:m relationships. n and m are placeholders and mean multiple.
To put objects of an entity type into relation, you just have to mention this relation in the entity class' mapping
information. The simplest case is a unidirectional 1:1 relationship. In the following example we describe a User entity
which has its access information (user credentials) encapsulated into another entity class called UserCredential. Since
every user has at most one credential object and every credential object may only be associated to one user object,
this is a 1:1 relationship.
If the relationship should be bidirectional, include the OneToOne attribute in the other class, too, and add an
attribute which denotes the attribute of the other entity that mapps the related object:
This way, you can access the user object from the credentials object, too.
Most of the times, developers have to deal with relationships which include many objects on at least one side.
These relationships are called 1:n or n:m relationships. This means that either one or multiple entities are standing
in relationship with an arbitrary number of entities of another type. To accomplish this, you have to use the mapping
keywords OneToMany or ManyToMany when describing your entities. Besides that, the mapping works the exact same way as
with 1:1 relationships.
There are however some tricks you should know when dealing with collections of associated entity objects. Consider
the following relationship between the entity classes User and Group:
When a group has at least one member, the group object will have a collection of the type
Doctrine\Common\Collections\ArrayCollection set as its members property. This collection contains
all user objects (or proxy objects as we have seen before) and can be modified intuitively with the methods add and
removeElement. To honor object-orientation, you might want to introduce custom methods for these tasks. If you do so,
you get into trouble when the group object does not have any users associated. In this case, the collection will simply be
set to null. To avoid checks whether the collection has already be initialized, you should to this by yourself
in the entity class' constructor:
It is also important to notice that one entity has to update the other entity's state as well when a relationship between to objects is created or removed. Take care to do this only in one class to avoid endless recursion loops! This class is called the Owning Side of the relationship. When implementing a bidirectional relationship, the other class is called the Inverse Side. It is important to determine owning and inverse side and implement the the classes accordingly to avoid greater trouble during debugging.
There are some more features implemented by Doctrine 2 enabling developers to specify their entities' relationships including sorting, pre-fetching and indexing. These topics are not covered in this article but are explained very understandable in the Doctrine 2 documentation.
Subtyping can be implemented in different ways using Doctrine 2. The main difference between these implementations is how the inheritance is mapped to the database. The options are to have one table for every class (Class Table Inheritance), to have one table for all classes in a hierarchy (Single Table Inheritance) and to have a table for every specialized sub-class of a given super-class (Mapped Super Class).We will give a short overview on all three alternatives, you have to pick the right one yourself. This decision should be made based on how many common attributes there are in your sub-classes.
Introducing a mapped superclass is probably the easiest way for specifying inheritance but might lead to many duplicate columns in your database schema. The superclass of your entities is not being declared as an entity itself (and might also be declared abstract) but provides attributes and optionally methods that will be available in all subclasses. When creating the database schema, Doctrine 2 merges all attributes and relationships of the superclass into the definitions of the subclasses and processes them as regular entities.
After creating the database from this mapping information, your tables will look like this:
When having entities that are very similar besides some few attributes, you might want to store them together in one database table. This approach is called Single Table Inheritance. To distinguish between the different types, there is always a column marked as discriminator column and a discriminator map that tells Doctrine 2 which values in the discriminator indicate what entity types.
These definitions cause the existing of one single table called User with all the attributes declared inside the classes User and Administrator plus a column type the discriminator column. When working with entities of these types, Doctrine will manage the type flag automatically for you.
The resulting database schema looks as illustrated by the following diagram:
Having each entity type stored in its own table is always good for keeping your schema extensible. When you have to create a new subtype, Doctrine 2 will just create a new table for this type and it can inherit the logic and common attributes of a superclass. The only overhead you have with this approach is that all tables that correspond to subtypes have to maintain a relationship to their supertype's table. Using class table inheritance, the example with the entities User and Administrator looks like this:
Besides the inheritance type, there is no difference to the example using single table inheritance. The outcome on the resulting database scheme is huge. Now you have to separate tables which store users and administrators. Every record in the table Administrator has a corresponding record in the User table.
This was the first part of this article. Stay tuned for part II which will be published tomorrow (on 6th of December 2011)! In the second part, we will integrate Doctrine 2 into a Zend Framework application and include a generic sandbox (ZF-)project with Doctrine 2!
Talend delivers v5. Zentyal raises series A. The TCO of OSS. And more.
# Talend announced version 5 of its data integration suite, adding business process management capabilities via an OEM relationship with BonitaSoft. Yves De Montcheuil explained the name changes in version 5.
# Zentyal closed a series A venture capital funding of over $1m by Open Ocean Capital.
# The London School of Economics released a report on the total cost of ownership of open source software.
# Couchbase announced the availability of the Couchbase Hadoop Connector, developed in conjunction with Cloudera.
# Rackspace announced the private beta of Rackspace MySQL Cloud Database.
# The debate over the role of open source foundations in the Git era continued, including a follow-up by the instigator, Mikael Rogers, a rallying cry for autonomy from Ceki Gülcü, and Simon Phipps warning about throwing the baby out with the bathwater.
# Marco Abis is stepping down as CEO of Sourcesense.
# NGINX usage has grown almost 300% over the last year, according to Netcraft figures discussed by Royal Pingdom.
# The Wireless Innovation Forum announced the formation of the Open Source Framework for Commercial Baseband Software project.