This article is inspired by the many questions I see in forums about what is the best language, NoSQL database, or cloud service to use today.
A frequent reply to these questions is “What is your exact goal and use case?”, which is something people often find difficult to answer. For writing a use case is a tricky exercise indeed. In which depth of details to go? What is meaningful and what is not? How to make it fit inside a forum post? And how to keep your startup's end goal private?
Another recurring and objective reply is “Experiment to find what works best for you”. Still, one who is new to the Cloud and NoSQL space cannot experiment with everything. Aiming to reduce one's experimentation space is legitimate, which brings back to the question about use case.
I therefore thought that I would share my personal use case, and how I decided to experiment with Clojure, CouchDB, Heroku and PostgreSQL before anything else. My hope is that it will help you formulate your own use case and put you on the track of finding your first tool set, in particular by showing you in which way a use case and a first technological bet connect together.
A word of warning: objectivity is not the point of this article. Everything that follows is personal and subjective. It is based on my very own experience (or lack thereof), understanding (or lack thereof) and intuition (or lack thereof). This is intentional. Making your first bet should involve a huge amount of your own subjectivity and intuition. Experimentation will bring objective answers.
In the end, writing about this is a risky exercise. At best I will change my mind. At worst I got it all wrong... but I don't think so.
Interestingly, I found that my needs could be expressed in pretty generic terms, and without revealing the details of what I am up to; so if you want to ask for help on a forum, there is a lot you can say without selling your soul.
My use case
I have a few projects in the pipe now, and here is my reality in a nutshell.
- 75% of my projects are data harvesting, aggregation and
- The shape of the data is data source driven / supplier driven. Data are structured, but heterogeneous in the sense that a same data item may come in different forms from different sources (e.g. same dataset for different national government offices)
- Data can be stored as text.
- I value rapidity of development, robustness, data and process redundancy, ease of data harvesting.
- Data harvesting and pre-processing should happen in the cloud, research and prototyping on an in-house box, and production crunching and deployment back in the the cloud.
- Research data on a in-house box should ideally mirror production data in the cloud.
- 25% of my projects are social web applications
- The shape of the data is application driven / consumer driven.
- I value rapidity of development, ease of deployment, robustness
- The user base will not be gigantic.
- The ideal software development environment should work for data exploration, prototyping and large scale deployment (think Matlab and .Net in one).
- Some projects may reach “big data” scale, so the platform should allow replicating and sharding data.
- Administration should be minimal, i.e. doable by one or two developers alongside development work.
- Managing several environments should be easy (dev, research, test, prod).
- Cost should be minimal on day 1, and I choose linear costs over logarithmic cost.
- The solution should be be part of a recognised and standard ecosystem (like Java, .Net, AWS, Azure, Hadoop).
- I want to have a good understanding of every bit of the platform on day 1. My IT experience is in desktop, server, and web development with Java, .Net, MS SQL Server and a bit of Oracle, and data analysis with SQL Server and Matlab.
If you are an online shop, a news publisher, an online RPG, or a S&P500 company, then bits of your equation should be different from this. You might have a supply of developers and administrators, you might be working with in-house data only, you might soon be snowed under petabytes of data, you might be streaming digital data.
A McKinsey analysis would be more structured and polished, but this use case is really good enough for the purpose of narrowing which technologies to start experimenting with.
What about the technologies I know most?
Technically, these are things that Azure, .Net, SQL server and Matlab would do for me. But I think they would benefit projects that are a bit more mature than mine, if only for the higher upfront costs that they entail. Also:
- I would prefer a single environment for research, development and deployment.
- My intuition is that deploying and administrating data and processes redundantly across cloud and a local environment will require more skill and time than I would like.
- I think my data will need a fair bit of work to normalise and store in SQL server; some NoSQL document stores might allow me to cut on data harvesting development.
- I won't use some of Microsoft's (and Matlab's) killer features: rich desktop/office applications, highly accessed tiered systems and transactional databases, corporate standards, corporate support.
What follows is my first shot at an overall platform that could work for my projects, and reasons why. The choice is very much an overall one, and not just a choice of individual parts, and the process to arrive there was very iterative.
I will also describe technologies that I did put on the side for now. I may well end up using them as plan B if plan A doesn't work as well as I hope.
Software: Open Source
- reduced upfront cost
- freedom to experiment
Cloud infrastructure: Amazon Web Services
- hosts and connects a competitive ecosystem of services, many of which operate on a freemium model.
- vision of modular and elastic services.
- Runs on Amazon Web Services
- Clear deployment and scaling model; web & workers
- Hub for a wealth of add-on products that fit my needs, like databases, caching, messaging...
- Github based development and deployment workflow
- Minimal administration
- Supports Clojure applications
- Freemium model (same for many add-ons too)
- Heroku's choice of hosted database
- I feel at home there coming from Microsoft SQL Server: features, stored procedure (although in beta at Heroku at the time of writing)
- Candidate for main data warehouse.
- I understand all of it, and see the benefit of all of it:
- Document database, JSON
- MVCC and Replication
- Map/Reduce and View indexing
- Web integration (REST, CouchApps)
- CouchDB's limitations are honestly and clearly explained; best in class in the NoSQL space in this respect.
- CouchDB's functionalities complement relational databases for
what I need to do.
- Focus on large and atomic reads and non competitive writes.
- Simple out-of-the box mirroring; subject to how well replication works over a slow connection.
- If I got it right, a JSON document could contain the code to map itself, which would be a killer denormalisation feature for my heterogeneous data (will write about this when I make it work)
- CouchDB feels like a mini-hadoop over JSON docs with Master-Master replication and Map-reduce.
- CouchDB is scalable enough for my projects
- Administration seems low and tuning straightforward
- Creating a new test or development database from live data is trivial.
- Creating data harvesting processes that are redundant and partition-tolerant should be trivial.
- Couchapps are a clever deployment model for simple web applications.
- Hosted by Iris Couch and Cloudant (Heroku add-on), which both have a freemium model.
- CouchDB is an Apache project, and a recognised actor of the big data world
Functional programming: Clojure
- Best language I found to use all the way from research and prototyping to industrial deployment
- Compact, expressive and a delight to use
- Runs in interactive and compiled modes
- Wonderful devices for multi-threaded programming
- Java interoperability.
- Straightforward and elegant web frameworks.
- Also: Incanter for stats, Clutch and Clojurescript for CouchDB, Cascalog for Hadoop
- Compact ecosystem (for now)
- Hosted on Heroku
Object programming: Java
- Multi-platform OO language
- Costless development environment
- Libraries relevant to my projects (weka, gephi...)
- The language in which Hadoop is written.
- Huge hosting ecosystem. Hosted on Heroku.
In a nushell, Plan A is very much:
a bet on the trio Heroku, Clojure, CouchDB
with PostgreSQL as safe SQL bet
and Hadoop over AWS as a long term cloud scaling environment.
What I need to assess now is whether Heroku and hosted CouchDB deliver good enough performance, and whether CouchDB meets all my high expectations (too high?).
To summarize how tools match the use case:
|Data analysis projects|
|Heterogeneous and source driven data||CouchDB|
|Data can be stored as text||CouchDB|
|Rapidity of development||CouchDB, Clojure|
|robustness, data redundancy||CouchDB|
|Process redundancy||Heroku // in-house|
|Easy data harvesting||CouchDB, Clojure|
|Harvesting in the cloud, research in-house, production in the cloud||CouchDB, Clojure, Heroku|
|In-house research data mirror production data in the cloud||CouchDB|
|Web application projects|
|Application driven data||CouchDB, PostgreSQL|
|Rapidity of development, ease of deployment||Clojure, Heroku|
|Single environment from data exploration to deployment||Clojure, CouchDB|
|Big data||CouchDB, AWS|
|Minimal administration||Heroku, CouchDB|
|Managing several environments||CouchDB|
|Minimal upfront cost, linear costs||Open source and freemium|
|Recognised ecosystem||AWS, Java|
|Good personal understanding||Java, Clojure, CouchDB, PostgreSQL, Heroku|
I hope the above will help you to formulate your own use case and to find technologies that fulfil it. In particular, I hope this gave you an idea of how advanced an analysis you can perform based on the information that can be found on the web, in forums and in books (and a little bit of experimentation of the side). I will love to hear your own stories, and to advertise use cases and solutions that are different from the above.
In the end, the one piece of advice I would give when approaching the space is: understand what you plan to use, and feel certain that you and your team can become intimate with it. For example: I don't think I can become intimate enough with EC2 in the short-term, so I am happily giving its flavour of elasticity up for now. I see myself becoming intimate with PostgreSQL more easily than with MySQL. I felt intimate with CouchDB soon after I started reading about it, and less so with other NoSQL solutions.
Now back to experimenting with all this. And I am looking forward (am I?) to telling you whether I change my mind or not!