Big Data Refinery Fuels Next-Generation Data Architecture

Since joining Hortonworks at the beginning of the year, a question I’ve heard over and over again is “What is Apache Hadoop and what is it used for?”

There’s clearly a lot of hype [and confusion] in this emerging Big Data market, and it feels as if each new technology, as well as existing technologies, are pushing the meme of “all your data are belong to us”. It is great to see the wave of innovation occurring across the landscape of SQL, NoSQL, NewSQL, EDW, MPP DBMS, Data Marts, and Apache Hadoop (to name just a few), but enterprises and the market in general can use a healthy dose of clarity on just how to use and interconnect these various technologies in ways that benefit the business.

In my post entitled “7 Key Drivers for the Big Data Market”, I asserted that the Big Data movement is not only about the classic world of transactions, but it factors in the new(er) worlds of interactions and observations. And this new world brings with it a wide range of multi-structured data sources that are forcing a new way of looking at things.

In order to make sense of this emerging space, I’ve created two graphics designed to walk through a vision of a next-generation data architecture. At the highest level, I describe three broad areas of data processing and outline how these areas interconnect.

The three areas are:

• Business Transactions & Interactions
• Business Intelligence & Analytics
• Big Data Refinery

The graphic below illustrates a vision for how these three types of systems can interconnect in ways aimed at deriving maximum value from all forms of data.   

Enterprise IT has been connecting systems via classic ETL processing, as illustrated in Step 1 above, for many years in order to deliver structured and repeatable analysis. In this step, the business determines the questions to ask and IT collects and structures the data needed to answer those questions.

The “Big Data Refinery”, as highlighted in Step 2, is a new system capable of storing, aggregating, and transforming a wide range of multi-structured raw data sources into usable formats that help fuel new insights for the business. The Big Data Refinery provides a cost-effective platform for unlocking the potential value within data and discovering the business questions worth answering with this data. A popular example of big data refining is processing Web logs, clickstreams, social interactions, social feeds, and other user generated data sources into more accurate assessments of customer churn or more effective creation of personalized offers.

More interestingly, there are businesses deriving value from processing large video, audio, and image files. Retail stores, for example, are leveraging in-store video feeds to help them better understand how customers navigate the aisles as they find and purchase products. Retailers that provide optimized shopping paths and intelligent product placement within their stores are able to drive more revenue for the business. In this case, while the video files may be big in size, the refined output of the analysis is typically small in size but potentially big in value.

The Big Data Refinery platform provides fertile ground for new types of tools and data processing workloads to emerge in support of rich multi-level data refinement solutions.

With that as backdrop, Step 3 takes the model further by showing how the Big Data Refinery interacts with the systems powering Business Transactions & Interactions and Business Intelligence & Analytics. Interacting in this way opens up the ability for businesses to get a richer and more informed 360 degree view of customers, for example.

By directly integrating the Big Data Refinery with existing Business Intelligence & Analytics solutions that contain much of the transactional information for the business, companies can enhance their ability to more accurately understand the customer behaviors that lead to the transactions.

Moreover, systems focused on Business Transactions & Interactions can also benefit from connecting with the Big Data Refinery. Complex analytics and calculations of key parameters can be performed in the refinery and flow downstream to fuel runtime models powering business applications with the goal of more accurately targeting customers with the best and most relevant offers, for example.

Since the Big Data Refinery is great at retaining large volumes of data for long periods of time, the model is completed with the feedback loops illustrated in Steps 4 and 5. Retaining the past 10 years of historical “Black Friday” retail data, for example, can benefit the business, especially if it’s blended with other data sources such as 10 years of weather data accessed from a 3^rd-party data provider. The point here is that the opportunities for creating value from multi-structured data sources available inside and outside the enterprise are virtually endless if you have a platform that can do it cost effectively and at scale.

Let me conclude by describing how the various data processing technologies fit within this next-generation data architecture.

In the graphic above, Apache Hadoop acts as the Big Data Refinery. It’s great at storing, aggregating, and transforming multi-structured data into more useful and valuable formats.

Apache Hive is a Hadoop-related component that fits within the Business Intelligence & Analytics category since it is commonly used for querying and analyzing data within Hadoop in a SQL-like manner. Apache Hadoop can also be integrated with other EDW, MPP, and NewSQL components such as Teradata, Aster Data, HP Vertica, IBM Netezza, EMC Greenplum, SAP Hana, Microsoft SQL Server PDW and many others.

Apache HBase is a Hadoop-related NoSQL Key/Value store that is commonly used for building highly responsive next-generation applications. Apache Hadoop can also be integrated with other SQL, NoSQL, and NewSQL technologies such as Oracle, MySQL, PostgreSQL, Microsoft SQL Server, IBM DB2, MongoDB, DynamoDB, MarkLogic, Riak, Redis, Neo4J, Terracotta, GemFire, SQLFire, VoltDB and many others.

Finally, data movement and integration technologies help ensure data flows seamlessly between the systems in the above diagrams; the lines in the graphic are powered by technologies such as WebHDFS, Apache HCatalog, Apache Sqoop, Talend Open Studio for Big Data, Informatica, Pentaho, SnapLogic, Splunk, Attunity and many others.

Key Takeaway

A next-generation data architecture is emerging that connects the classic systems powering Business Transactions & Interactions and Business Intelligence & Analytics with Apache Hadoop, a “Big Data Refinery” capable of storing, aggregating, and transforming multi-structured raw data sources into usable formats that help fuel new insights for the business.

Enterprises that get good at maximizing the value from all of their data (i.e. transactions, interactions, and observations) will put themselves in a position to drive more business, enhance productivity, or discover new and lucrative business opportunities.

7 Key Drivers for the Big Data Market

I attended the Goldman Sachs Cloud Conference and participated on a panel focused on “Data: The New Competitive Advantage”. The panel covered a wide range of questions, but kicked off covering two basic questions:

“What is Big Data?” and “What are the drivers behind the Big Data market?”

While most definitions of Big Data focus on the new forms of unstructured data flowing through businesses with new levels of “volume, velocity, variety, and complexity”, I tend to answer the question using a simple equation:

Big Data = Transactions + Interactions + Observations

The following graphic illustrates what I mean:

ERP, SCM, CRM, and transactional Web applications are classic examples of systems processing Transactions. And the highly structured data in these systems is typically stored in SQL databases.

Interactions are about how people and things interact with each other or with your business. Web Logs, User Click Streams, Social Interactions & Feeds, and User-Generated Content are classic places to find Interaction data.

Observational data tends to come from the “Internet of Things”. Sensors for heat, motion, pressure and RFID and GPS chips within such things as mobile devices, ATM machines, and even aircraft engines provide just some examples of “things” that output Observation data.

With that basic definition of Big Data as background, let’s answer the question:

What are the 7 Key Drivers Behind the Big Data Market?

Business

1.     Opportunity to enable innovative new business models

2.     Potential for new insights that drive competitive advantage

Technical

3.     Data collected and stored continues to grow exponentially

4.     Data is increasingly everywhere and in many formats

5.     Traditional solutions are failing under new requirements 

Financial

6.     Cost of data systems, as % of IT spend, continues to grow

7.     Cost advantages of commodity hardware and open source software

There’s a new generation of data management technologies, such as Apache Hadoop, that are providing an innovative and cost effective foundation for the emerging landscape of Big Data processing and analytics solutions. Needless to say, I’m excited to see how this market will mature and grow over the coming years.

Key Takeaway

Being able to dovetail the classic world of Transactions with the new(er) worlds of Interactions and Observations in ways that drives more business, enhances productivity, or discovers new and lucrative business opportunities is why Big Data is important.

One promise of Big Data is that companies who get good at collecting, aggregating, refining, analyzing, and maximizing the value derived from Transactions, Interactions, and Observations will put themselves in a position to answer such questions as:

What are the behaviors that lead to the transaction?

And even more interestingly:

How can I better encourage those behaviors and grow my business?

So ask yourself, what’s your Big Data strategy?

Solving the Data Problem in a Big Way

I recently joined Hortonworks as VP of Corporate Strategy, and I wanted to share my thoughts as to what attracted me to Hortonworks.

For me, it’s important to 1) work with a top-notch team and 2) focus on unique market-changing business opportunities.

Hortonworks has a strong team of technical founders (Eric14, Alan, Arun, Deveraj, Mahadev, Owen, Sanjay, and Suresh) doing impressive work within the Apache Hadoop community. Hortonworks also has an impressive Board of Directors that includes folks like Peter Fenton, Mike Volpi, Jay Rossiter, Rob Bearden, as well as our most recent board member Paul Cormier (Red Hat’s President of Products and Technology).

So “top-notch team”? Check!

Regarding “unique market-changing business opportunities”, the top 3 technology areas right now are arguably: Mobility, Cloud, and Big Data. Apache Hadoop is clearly a technology in the Big Data category that is enabling a new approach to data processing (both from a capabilities perspective and an economics perspective).

I’ve spent the last few years in the Cloud space (at SpringSource and VMware), and I met with many customers who loved VMware’s Cloud Application Platform vision. One of the common questions that came up, however, was:

“What are you going to do about the data problem?”

Traditional application architectures focus on moving structured data from backend datastores to the applications that need the data. Elastic Caching Platforms such as VMware’s vFabric Gemfire help with scalability and latency issues for these types of applications.

Rather than move data to applications, Hadoop provides a platform that cost effectively stores petabytes of data and enables application logic to execute directly on that data in a massively parallel manner.

I believe Hadoop provides a very compelling solution to the “data problem” since it’s explicitly designed to deal with the volume, velocity, variety and exponential scale of unstructured and semi-structured data that businesses increasingly need to deal with. Moreover, Hadoop does this within an economic model (a la commodity servers and storage) that makes the platform useful for a wide range of problems.

While 2011 was the year where a critical mass of enterprise customers and vendors began to realize the size and scope of the opportunity and value behind this Apache Hadoop phenomenon, the wave is just getting started, and I’m excited to be a part of the fun!

Food Zombies - A Tasty Top-Down Shooter Game

Over the past few years, I've written various blog posts covering my son's interest in programming.

Billy is a high school junior now, and his efforts over the past 4 years have spanned from creating interactive virtual playworlds and sharing LUA scripting code in ROBLOX, to cool programs for the TI-84 calculator, to Block Dude Evolved for iOS devices (a port of the classic TI-84 Block Dude game), to Prom Checklist for high school girls to keep track of all their prom-related details on their iOS devices, as well as Prom Checklist West, a branded version for Cherry Hill High School West where the proceeds will go to the school's "project graduation".

His most recent effort brings him back to his area of passion...that is game development for iOS devices (iPod Touch, iPhone, etc.).

Food Zombies is his craziest, most entertaining game to date.

While Food Zombies is a classic top-down zombie shooter, it's also good clean (and unique) fun since the zombies are fast food (in the form of pizza, fries, donuts, pies, etc.):



The power-ups are, naturally, good-for-you fruits:



And since this a shooter game, it offers many different weapons to satisfy even the pickiest food-slaying cravings:



I highly recommend the flame thrower. :-)

Anyhow, if you have an Apple iPod Touch or iPhone and you can't get enough of top-down Zombie shooter games, then I recommend you satisfy your cravings by giving Food Zombies a try.

Oh yeah, one final note for those Block Dude Evolved fans out there. Billy has just finished a major 2.0 version of the game that adds in a bunch of new features. The game is working its way through Apple approvals, so stay tuned!

Block Dude Evolved

My son's first foray into programming started a few years ago playing ROBLOX; a virtual playworld where kids can create and customize the look and behavior of their own online worlds. I wrote a couple of posts including "Professor ROBLOX: Class In Session" covering how ROBLOX is actually shaping the lives of future programmers since kids use the Lua scripting language to customize the behavior of their worlds.

My son moved on past Lua and taught himself TI Assembly Programming and Visual C++. His goal: create games for the TI-84 graphing calculator so he and his middle-school friends could play games rather than pay attention in math class. :-)

With the rise of the Apple iPod Touch and iPhone, he has launched headfirst into Objective-C and delivered "Block Dude Evolved", a recreation of the all-time classic TI calculator game called "Block Dude".


Block Dude Evolved is a puzzle game. The goal is to move your little man across obstacles and out the exit door on the level. The challenge is that you need to pick up and move blocks to help you climb over obstacles that are between you and the door. You can only step up one block at a time, so if you are facing a wall two blocks high, then you need to grab a movable block and plop it down so you can climb up. The first level is pretty simple, but the levels increase in difficulty after that.

The controls of Block Dude Evolved are pretty simple. To move the little man left or right, you just tap those sides of the screen. To climb up a block, just tap the upper portion of the screen. If you are standing next to a block that can be picked up, just tap the block and you will lift it above your head. Then you simply move to where you want to be and tap the spot where you want to drop the block. If you want to exit out of the game, just tap two fingers at the same time.

Block Dude Evolved has a Settings dialog that enables you to customize the look:


For example, you can choose the Future look:


Or the Revamped look:


Anyhow, if you have an Apple iPod Touch or iPhone and yearn for the days of classic brain-puzzle games, then I recommend you give Block Dude Evolved a try.

Job Trends: Spring, WebSphere, WebLogic - what a difference a year makes!

Last year I wrote "Job Trends: Tomcat, Spring, Weblogic, JBoss, EJB" where I discussed the trend towards "Lean Software" and the role that Spring plays in this important movement.

A lot has happened over the past year. CIO's have identified Virtualization and Cloud computing as their top two strategic technologies for 2010. Lean Software has become even more of a Business Technology Imperative than it was a year ago. And, the job market over the past year has been challenging at best.

With that as a backdrop, let's see what the job market looks like for Spring Java developer skills versus the other industry heavyweights.

The chart nicely illustrates that Spring Java developer skills (green line) have been on an inexorable path upwards for the past 5 years. WebSphere Java developer skills (blue line) are next and have been on a downward path for the past year and a half. WebLogic Java developer skills (orange line) round out the chart and have been relatively flat over the past few years.

Conclusions?
Companies continue to value lightweight application infrastructure skills (i.e. Spring) since this provides them a way to create applications more quickly and therefore be more competitive. More speculatively, I believe that Virtualization and Cloud computing initiatives are accelerating this trend since these initiatives are forcing enterprises to take a hard look at how they are building and deploying applications...and to take measures (and hire talent) that dramatically simplify the process.

Since I work at the SpringSource division of VMware, I have a keen interest in the health and vibrancy of the Spring community. I'm happy to see that even in a tough job market, the demand for Spring Java developer skills continues to grow.


Credits: I used Indeed.com to generate the chart above. Indeed.com searches millions of jobs from thousands of job sites and provides a neat service that lets you see job trends for whatever search criteria you may have. My criteria was Java Developers that have Spring, WebSphere, or WebLogic skills. Click here to go to Indeed.com to see the latest view of my chart above.

I LIKE PI(E)

Things that make you go hmmmmm....I never realized that when looked at it in the mirror, 3.14 can be both mathematical and delicious:



MMMMM.....PUMPKIN.....PIE.....

SpringSource + VMware: My VMworld 2009 Interview

I had the chance to sit down with John Troyer to discuss SpringSource, VMware, cloud computing, and breaking down the barriers between Development and Operations...live (at the time) on the show floor at VMworld 2009.

VMware Fanboy

After arriving at SFO airport earlier this week, I just had to stop and snap the photo below:


The placement and timing of the ad was special since 1) I'm excited to be joining VMware as part of the SpringSource acquisition, and 2) I flew in to SFO in order to attend VMworld 2009 (the event was ginormously huge with lots of great sessions, excited attendees, and impressive partners).

I've been a fanboy of VMware for years: I've been an avid and happy user of VMware's products, and earlier this year I declared VMware the "Virtualization Big Dog" as part of my "Big Dog" series of posts.

In that post I compared VMware with Microsoft, Citrix, and Red Hat, and I concluded by saying "As long as VMware stays focused on creating great solutions to customer problems and honestly assesses the threats posed by the other market players, they should be able to retain their market leadership status for many years to come."

I love working for companies that provide REAL VALUE to customers, and I look forward to joining VMware as soon as the deal closes and extending its market leadership from virtualization to the cloud and beyond.

Multiple Languages, Multiple Platforms: Choice Is A Good Thing

Over the past decade or so, the Microsoft vs. Java landscape has been summed up as follows:

• Microsoft: Multiple languages, single platform.
• Java: Single language, multiple platforms.

In "Java yields to other languages on the Java Virtual Machine" Paul Krill from InfoWorld covers the fact that Java is no longer the only language in town when it comes to creating applications that run on the Java Virtual Machine.

Languages such as Groovy, JRuby, and Scala are just a handful of languages beyond Java available for the JVM.

The fact is that over the past few years the landscape has changed to be:

• Microsoft: Multiple languages, single platform.
• Java: Multiple languages, multiple platforms.

Why is offering a choice of languages important? Neil McAllister provides some good reasons in "We need more polyglot programmers". The emergence of other languages targeting the proven and scalable Java platform is a good thing for the Java market. It will help keep the Java platform vibrant and expand its market reach.

Neil makes the point that Groovy "offers a Java-like syntax but is actually a dynamic language, similar to Perl, Python, and Ruby. It gives developers the safety and stability of the Java runtime but frees them from the often-restrictive Java syntax.".

All of these points factored into why SpringSource added Groovy and Grails into our portfolio of product offerings.

Bottom-line: Choice of language and platform is a good thing...for developers, customers, software vendors, and the market in general.

Harry Kalas - Outta Here!

Philadelphia paid tribute today to Harry Kalas, the voice of the Philadelphia Phillies. Kalas passed away earlier this week and the fans filled Citizens Bank Park to pay their respects and to hear Harry Kalas' close friends and colleagues share their thoughts and feelings.

I think Mike Schmidt summed it up the best in his eulogy when he said that Harry Kalas' life was "bountiful".

I think we should all strive to lead "bountiful" lives.

While I've never met Harry Kalas, he has been a part of my life for many years. Thinking of him reminds me of my father who taught me to love baseball and the Phillies. My dad was my little league coach, and he came to watch me play baseball through high school, college, and even into my 30's when I played over-30 baseball with some of my old high school buddies.

I visited my dad's grave today to spend some time reminiscing with him about the great calls that Harry Kalas made over his broadcasting career, including:

"Long drive! It is....outta here!
Home run...Michael Jack Schmidt!"

We definitely heard that call many, many times.

Unfortunately my dad passed away before hearing one of Harry's best:

"The 0 - 2 pitch...swing and a miss...struck eem out!
The Philadelphia Phillies are 2008 World Champions of Baseball!!"

So long Harry. We'll miss you.
And if you get a chance, please say hi to my dad for me.

For those who weren't Phillies fans or for those who want to reminisce along with me, here's a handful of Harry's best calls:


And here's a nice tribute to Harry Kalas by Comcast:

Google App Engine Is Groovy Baby!

The SpringSource Groovy team and the Google App Engine Java team worked together to make sure that Groovy works on Google App Engine now that App Engine supports Java: "Seriously this time, the new language on App Engine: Java™"

Guillaume Laforge, of SpringSource, has a great post about how to "Write your Google App Engine applications in Groovy".

Check out some initial Groovy sample apps running on App Engine:
http://myowngroovy.appspot.com

Here you'll find the standard "Hello World" app, a Google Maps, Google Geocoding service, Flickr mashup application, and more.

10 Rules For Building A Great Organization

I'm reading "Enough. True Measures of Money, Business, and Life" by John C. Bogle, the founder and former CEO of Vanguard Mutual Fund Company. In the book, Bogle offers 10 Rules for Building a Great Organization:

1. Make caring the soul of the organization.


2. Forget about employees. (Vanguard instead uses the term "crew member")


3. Set high standards and values - and stick to them.


4. Talk the talk. Repeat the values endlessly.


5. Walk the walk. Actions speak louder than words.


6. Don't overmanage.


7. Recognize individual achievement.


8. A reminder - loyalty is a two-way street.


9. Lead and manage for the long term.


10. Press on, regardless.

A pretty solid set of rules if you ask me.

Job Trends: Tomcat, Spring, Weblogic, JBoss, EJB

Forrester recently described a trend that they refer to as "lean software" in their paper entitled Lean Software Is Agile, Fit-To-Purpose, And Efficient. They state that "lean software is emerging as the antidote to bloatware" and that "the trend toward lean software has been building for years, but the worldwide recession is accelerating it".

Forrester mentions SpringSource as one of four companies at the forefront of the lean software movement. This is due to our leadership within the Spring, Apache, Groovy and Grails communities, as well as our active encouragement, via SpringSource dm Server, of enterprise OSGi as the basis for next-generation application infrastructure.

Since Spring and Apache Tomcat play an important role in lean application infrastructure strategies, I've researched what the job market looks like for developers with Spring and Tomcat skills as compared to EJB, Weblogic, and JBoss skills.


I used Indeed.com to generate the chart above. Indeed.com searches millions of jobs from thousands of job sites and provides a neat service that lets you see job trends for whatever search criteria you may have.

My criteria was Java Developers that have Tomcat, Spring, Weblogic, JBoss, or EJB skills. Click here to go to Indeed.com to see the latest view of my chart above.

The chart nicely illustrates that Spring skills (green line) are in highest demand, well ahead of the others, and has been on a steep incline for the past year. Weblogic skills (blue line) are next and have remained fairly flat over the years. EJB skills (red line) and Tomcat skills (orange line) are neck and neck behind that, with JBoss skills (black line) tracking behind Tomcat but on a similar path.

NOTE: I've been asked why I did not include Glassfish, Geronimo, or WebSphere CE in the above chart. The reason is simple: they were effectively zero on the graph and therefore statistically irrelevant for my analysis.

UPDATED ON MARCH 30 2009:
Let me reiterate that the point of this post is to show that Spring and Tomcat Java developer skills are as easy to find in today's job market as other popular enterprise Java developer skills. With that said, I've been asked why I did not include WebSphere in the chart above. One could argue that WebSphere as a brand is broader than the other items in the search criteria, but I've modified the criteria to include it; the chart now covers Java Developers that have Tomcat, Spring, Weblogic, JBoss, EJB, or WebSphere skills.

Click here to go to Indeed.com to see the latest view of the chart above.

Conclusions?
More and more companies are looking at lean application infrastructure to help them remain competitive, and the difficult economy is only accelerating this trend. These businesses not only need to feel comfortable with the technologies they embrace, but they also need to make sure they can easily find people experienced with the technologies.

Since I work at SpringSource, I have a keen interest in the health and vibrancy of both Spring and Tomcat. I'm happy to see that Spring and Tomcat are doing so well in the job market for Java developers. This fact will help my customers feel more comfortable choosing SpringSource Enterprise and SpringSource tc Server to power their applications.

Credits: I'd like to thank Indeed.com for providing this valuable job trends service on their website. It provides a great way to understand the past and current directions of whatever type of job you may be interested in.

JVM as a Service: Mashup Azul and Amazon

This may be a crazy stupid idea, but I was having a conversation today about the kinds of things that make sense to be "as a service". Amazon EC2 and S3 are essentially compute as a service and storage as a service.

Since I'm a Java guy, I wonder:
Does it make sense to have a JVM as a service?

Why? Because while running Java workloads on Amazon may be cool for scale out scenarios, it does nothing to help with scale up scenarios for Java workloads. The classic JVM is a limiting factor in scale up scenarios because the more memory it has to deal with the more garbage collection time becomes a real performance issue.

Azul Systems offers a JVM compute appliance that illustrates the logical model of what I have in mind. Basically the Azul appliance centralizes JVM processing onto a machine designed to provide low response times, pauseless garbage collection, and efficient huge memory management. Basically each server has a JVM proxy that hands off processing to the compute appliance.

So, what if we mashup Azul with Amazon?

That is, what if we take the Azul JVM appliance concept and make it a cloud service? That could enable any EC2 instance running a Java workload to leverage a model where a JVM proxy on a given EC2 instance hands off its processing to a JVM as a Service. That JVM as a Service could then centrally handle dynamic scale up/scale down of JVM resources as well as eliminate the nasty garbage collection performance issues once and for all.

Now all we need is the JVM gurus at Azul or Oracle/BEA JRockit to get to work on it. I want my JVM as a Service guys, and I want it now!!! :-)

If you are a JVM guru, feel free to chime in as to why this idea is stupid and could never possibly work. Or if you have an equally outrageous "as a service" idea, please share.

Top 20 Most Commonly Used Open Source Licenses

My recent post "Tim Yeaton Snares Black Duck" gave me the encouragement to check out what Black Duck is up to these days.

While surfing the Black Duck Software website, I came across a very useful page worth bookmarking:
Top 20 Most Commonly Used Licenses in Open Source Projects

Black Duck's site says that they update the data in the table below on a daily basis, so please click over to their website for the latest data:
There is nothing unexpected or overly shocking in the data. For example, I have known that GPL 2.0 is the dominant open source license. I always thought the Apache 2.0 license was a little more prevalent, but the data is not far off from what I expected. GPL 3.0 appears to be doing really well since it is already above the Apache 2.0 and Mozilla Public License.

Anyhow, I figured I'd share the link. I'll certainly visit it periodically to stay abreast of the open source license landscape.

App Server SmackDown Panel 2009

I'm participating in a panel discussion focused on New Application Server Frontiers at SD West on Thursday March 12 2009 from 10:15 am - 11:45 am PT in room 204 at the Santa Clara Convention Center in Santa Clara, Calif.

The Panel is moderated by Chris Haddad, Burton Group vice president and service director, and the panelists include:

• Shaun Connolly, SpringSource
• Adam Gross, SalesForce.com
• Rich Sharples, JBoss Division of Red Hat
• Larry Cable, Oracle (BEA)
  
• Jerry Waldorf, Sun Microsystems
• Savio Rodrigues, IBM

Chris Haddad from Burton Group describes the session as follows:
"Cloud, server componentization, declarative programming models, and innovative application frameworks are forcing structural IT change and opening new server frontiers. Application servers, development frameworks, and tooling are adapting at either an evolutionary and revolutionary pace. ‘New Server Frontier’ panelists will describe how application stacks will either encourage exploration or strand teams and applications on a legacy island."

I simply describe the session as: App Server SmackDown Panel 2009!

Come see the fun!

For more information on SD West 2009, please visit: www.sdexpo.com.

One other note, SpringSource's Colin Sampaleanu will also be presenting a session titled: "Spring Framework 3.0 - New and Notable," on Friday, March 13, 8:30 am P.T. — 10:00 am P.T. in Room E at SD West. So if you want to hear the latest on Spring, then please go to Colin's session.

Virtualization Big Dog: VMware, Microsoft, Citrix, or Red Hat?

I was catching up on industry news today when I saw the following eWeek post: Red Hat Is Getting Ready to Take on VMware

It has been a while since I've written a "Big Dog" article, but this topic is worthy; so here goes.

Who is the virtualization big dog?

VMware is the incumbent big dog, of course. They have a compelling portfolio of offerings that are proven, feature-rich, and getting better all the time. Moreover, while I was at Princeton Softech, I was an avid and happy user of VMware's products.

Virtualization is big business these days and will only get bigger since it plays a fundamental and foundational role in cloud computing. VMware is faced with competitors interested in chipping away at their market dominance:

• Microsoft and its Hyper-V is clearly targeting VMware


• Citrix and its XenServer are focused on undercutting VMware


• Red Hat continues its bottom-up commoditization strategy

Just for fun, I've compared/contrasted the players in the Virtualization market with the players in the Enterprise Middleware market back in 2003/2004:

• VMware and BEA: Early/dominant market leaders driving innovative technology
  
• Microsoft and IBM: Huge players who think/execute over the long term
  
• Citrix and Sun: Xen ignited these market dynamics; Java made the middleware market
• Red Hat and JBoss/Tomcat: The power of bottom-up market groundswell
  

The middleware market leader was BEA Systems. IBM was the much bigger player applying top-down market pressure and working its usual long-term plan to gain market dominance. Sun was the vendor responsible for the technology that made the market but never became the market leader. And JBoss and Tomcat were generating significant and unyielding bottom-up market groundswell.

How did it play out? BEA became so focused on fighting IBM at the high end that they ignored/denied the bottom-up groundswell: BEA Chief Downplays Open-Source Alternatives. Alfred Chuang never gave credence to the JBoss or Tomcat threat and ultimately BEA got swept up by Oracle.

Bob Pasker (founder of Weblogic) had an interesting post after BEA got acquired by Oracle: JBoss (and possibly Tomcat) should never have happened. "JBoss launched an innovators dilemma attack against BEA, not with a revolutionary product, but with a revolutionary business model, one that BEA couldn’t hope to copy without cannibalizing its existing revenue stream. BEA fell right into the trap."

That Was Then But This Is Now
While there are some interesting similarities between the middleware and virtualization markets, the market dynamics today are clearly different. The open source model has clearly made its mark across many software markets. Neither Microsoft nor VMware are denying the power of open source or the threat that companies such as Red Hat pose to their business.

Add in the fact that VMware has Paul Maritz (ex-Microsoft executive) as their CEO and that changes things as well. While I don't like the article's title, I do agree with much of Chris Mellor's points in: VMware's one-trick pony: Destined to be a platform?. Chris highlights Maritz as an inspirational leader who has a great feel for where the market is heading.

So with all of that said, my answer to the question:

Who is the virtualization big dog?

VMware

Bottom-line: As long as VMware stays focused on creating great solutions to customer problems and honestly assesses the threats posed by the other market players, they should be able to retain their market leadership status for many years to come.

Programming Language Popularity

My 13 year old son has been programming in Lua, TI Basic and Assembler, ActionScript, JavaScript, and he's into C++/Dark GDK these days. While I've mostly focused on Java for the past few years, I've programmed in a wide range of statically typed and dynamically typed languages.

Since my son is just starting his programming journey, I naturally wondered:

What are the most popular programming languages these days?

I used tiobe.com and langpop.com to answer this question. Both sites provide programming language popularity statistics and rankings, and I was happy to see that Java, C, and C++ rank in the top three on both sites.

What I like about both of these sites is that they gather information across a wide range of search engines and websites in order to generate a popularity score. Neither site is focused on declaring the "best" programming language or the language in which the most lines of code have been written. They simply provide information that, as TIOBE states, "can be used to check whether your programming skills are still up to date or to make a strategic decision about what programming language should be adopted when starting to build a new software system."

TIOBE Programming Index for Feb 2009
TIOBE gathers information from Google, MSN, Yahoo!, and YouTube to calculate the ratings, and they have a great web page that defines how the TIOBE index is assembled. Below is a summary of the top 20 programming languages for February 2009. I especially like how they compare against last year's ranking so we can gauge how the language is doing year over year.

Below you will see that Java has been #1 the past two years. C++ and C# are on the rise. Visual Basic, PHP and Perl have declined however. I also find it interesting that while Ruby and Python generate a lot of buzz, neither has risen in rank over the past year.



LangPop Normalized Comparison on Feb 2, 2009
Below is LangPop's Normalized Comparison Chart that combines the data gathered across Yahoo, Craigslist, Amazon, Freshmeat, Google Code, and Delicious for 29 different programming languages. Click on the chart to see it more clearly.

Java is #2 behind C. Visual Basic is much lower in ranking than in TIOBE's ranking. Python, Perl, and Ruby are slightly higher than in TIOBE's results.


Conclusions?
Since I work at SpringSource and focus on enterprise Java (a la Spring) and dynamic languages that run on Java (a la Groovy/Grails), I'm pretty happy to see that Java not only maintains its relevance but continues to dominate as a top programming language.

With that said, the interest in dynamic languages such as PHP, Python, Ruby, and Groovy clearly tells me that developers crave more productivity and less complexity. Another reason I'm happy to be at SpringSource, since simplifying enterprise Java is our area of focus and passion.

Credits: I'd like to thank TIOBE and LangPop for maintaining and sharing the information on their website. They provide a valuable service and I will continue to visit their sites to stay abreast of how the programming language landscape evolves over the coming months and years.

Tim Yeaton Snares Black Duck

I agree with the sentiment behind Dana Blankenhorn's post asserting that it is "Happy days for Black Duck". As Dana states:

• Bookings were up 42% year over year, and service revenues were up 65%.
• Black Duck has secured $9.5 million in new financing, including $4.5 million in loans.
• Black Duck recruited Tim Yeaton, once vice president for worldwide marketing at Red Hat, as its new CEO. He was given credit there for driving the acquisition of JBoss, still one of the biggest open source merger deals ever.

Having had the pleasure of working closely with Tim Yeaton before, during, and after Red Hat's acquisition of JBoss, I'm happy to see him join Black Duck as CEO. I'm quite familiar with some of Black Duck's offerings from prior due diligence efforts that used Black Duck to scan source code for the open source licenses contained within the code. Definitely useful for understanding any potential legal exposure you may have when acquiring medium to large code bases.

Anyhow, as Dana Blankenhorn points out, Tim Yeaton drove Red Hat's acquisition of JBoss in June 2006. Tim truly understood the value and strategic importance of expanding Red Hat's footprint to include middleware. And while the integration of the two companies was challenging (as are most integration efforts), Red Hat clearly benefited from the move.

Tim Yeaton did a great job of making all of the JBoss team feel comfortable and welcome at Red Hat. And when I left Red Hat in March 2008 for some "Purposeful Risk-Taking", I publicly thanked Tim and others at Red Hat / JBoss for making the experience a great one.

With Tim at the helm, I do indeed hope that there are "happy days" ahead for Black Duck.