Thursday, December 15, 2005

Shared Nothing Architecture and Short Lived Processes

WARNNG - This post may offend fans of megalithic application servers

After having worked in the Java environment for the past several years and watching Enterprise Java Beans destroy whatever simplicity existed in the Java world, I was ready to explore new development environs with Collective Intellect. Further spurring this curiousity were the thoughts penned by Paul Graham in his enlightening book, Hackers and Painters. Paul is a big fan of Lisp and spends a few chapters talking about the elegance of the language and economy of expression required to implement powerful concepts. While Lisp is all of those things, it does seem to lack a large ecosystem of open source libraries ready made to bake into your current project. I ended up settling on Ruby and Ruby on Rails to implement our burgeoning ideas and business requirements. I will speak to the long and short comings of Ruby/Rails in a future post.

What I wanted to discuss on this post is Shared Nothing Architecture (SNA), coupled with Short Lived Processes (SLP). Shared Nothing Architecture is a concept made popular by the folks at google and is defined in Wikipedia as:

A shared nothing architecture is a distributed database architecture without a single point of failure. The term "shared nothing" is by imitation of other terms such as "shared disk", "shared network" and so on.

A typical shared nothing system would have duplicated disks, processors, power supplies, and networks. To make the system truly resilient, it should also have these systems split between different physical sites, and have multiple sources of electrical power and network connections to the outside world through physically diverse paths.

The advantage of SNA is that it greatly simplifies application construction as well as increasing the scalability and reliability of applications. I would argue that you don't even need to go so far as creating a distributed database and in some cases don't even require a database at all. Certainly having a distributed database creates a more resilient infrastructure, but if you don't have the time or resource pool to put a distributed database in place, you can still take advantage of SNA with a bit less resiliency at lower cost but still have a warm failover capability.

The implementation that we have at Collective Intellect uses a single mySQL database running on an industrial strength box with RAID 5. We have several other boxes that process information. They all point to this database box and file server to recover the context for the processing that they perform. One of these boxes also contains a hot nightly backup of all of the information in the database, so if the database box dies and the RAID file system completely fails, we can warm failover to the next box. We can do this because we also have an information architecture that will allow us to roll data forward (with a little pain) from the point at which the last backup was run.

This architecture allows us to write independent, specialized applications that each process a small chunk of our processing operation. This is a vast change from using a monolithic application server based architecture. Application server architectures require all applications to run inside of the architecture and manage state for all of those applications as part of that framework. One typical example of this is Enterprise Java Beans. These frameworks tend to carry a lot of baggage which results in two rather severe implications to your app.

  1. Bringing up an entire application server infrastructure requires very large memory and load time overhead vs. a non application server based app.
  2. All applications reside inside the application server. This means they are long lived and have a fairly large chance of becoming more and more intertwined with either legacy code or code that is meant to support other apps (i.e. spaghetti)
In our shared nothing architecture our apps can be short lived and only carry the code required by the app. Freeing applications from the application server means that these apps can be short lived. We have a number of small specialized Ruby applications that automatically get launched when work exists for them to do. This allows all components of the system to do what they were designed to do.

  • The database stores data state used by the applications
  • The operating system controls the sharing of hardware resources (network, disk, cpu, memory) as opposed to the application server controlling the sharing of these resources as a proxy or replacement of the OS.
  • The application processes implement the specific features required by the business model


Anonymous said...

Interesting post. A great topic to consider. I do have a question about the database component.

"The implementation that we have ... uses a single mySQL database ... with RAID 5."

How is this shared nothing? The database box is potentially full of shared components ( processors, RAM, etc.. ). RAID 5 is not a perfect solution. SCSI drives can go rogue and corrupt an entire shelf.

Is RAID 5 a good idea? Sure and this post spells out some sound concepts, but it's pushing the envelope to say RAID 5 with log shipping is "shared nothing" or even "takes advantage of shared nothing architecture".


Travis Reeder said...

You're architecture is hardly shared nothing, it sounds just like a normal multi-tiered application. A single db to stuff all your information is more of a shared-everything-architecture.

Tim Wolters said...

The Shared Nothing Architecture name is a play on older architectures that were referred to as Shared Memory or Shared Disk. What SNA is really about is creating applications that don't store state locally. Sharing local state across a distributed environment is a bit of a nightmare. BTW, multi-tier architectures put a persistent data layer in between external apps and the database. SNA goes in the opposite direction, with smaller apps directly accessing the database.

Tim Wolters said...


Your point is a good one. My belief is that you can phase into shared nothing. We started on the processing side, with each app being independent of the others and can run on their own box. The next phase would be to move to a distributed database on the back end. The other point I was trying to make in the post was regarding the advantages of short lived processes. These are only possible using a stateless app architecture.

Anonymous said...

The livejournal guys have a paper / preso on this topic that details horizontal partitioning to address this problem. At least when one guy slides off the face of K2, the rest of the party doesn't go with him.


Darpan Dinker said...

Where did you pick the definition of shared-nothing?

The current Wikipedia ( states the following:
A shared nothing architecture (SN) is a distributed computing architecture where each node is independent and self-sufficient, and there is no single point of contention across the system. People typically contrast SN with systems that keep a large amount of centrally-stored state information, whether in a database, an application server, or any other similar single point of contention. While SN is best known in the context of web development, the concept predates the web: Michael Stonebraker at UC Berkeley used the term in a 1986 database paper, and it is possible that earlier references also exist.

Tim Wolters said...

The primary goal of SNA (at least my interpretation of it) is to not share state between processes. This means you do not have the immense overhead of mirroring state among all of your processes. Any request can be handled by one of many identical processes. Will each process likely need to access a central database? Yes! Is that OK? Yes! Does that make the application less stable or less scalable? No! The data in the database does not represent a process centric state. The returned data represents the process centric state.

Text Messages said...

Now it is very difficult to find a really good blog, which is written the truth, not rumors. I really liked your blog and I completely share your thoughts. Latest Mobile

richie007 said...

nice topic to consider, we still use php 4 but are considering upgrading

james vick
why company

mikroenjeksiyon said...

Was a beautiful page. Thanks to the designers and managers.

turbo fire said...

What I want to know is why you didnt think to include the other side of this issue? There are so many things that youre missing here that I dont see how you could actually form an intelligent opinion on the subject. Its like you didnt even consider that there me be another side here. Im kind of disappointed.

kepçe kulak ameliyatı said...

Site's character and a great color match ..Göğüs küçültme Ameliyatı I will recommend your site to the other platforms.

Meme Estetiği said...

I enjoyed this site. Çene Cerrahisi I will recommend your site to other places. Saç Ekim Everyone does not show care. Estetik But you were preparing a site so rich in content. Estetik Burun Ameliyatı Everyone who helped, thank you.

Anonymous said...

paper towel coupons.
janitorial services.
toilet parts.
basement finishing.
soap dispenser.
paper towel holder.
activated carbon.
wooden boxes.
cleaning products.
carbon filter.
wet dry vac.
dual flush toilet.
5 gallon bucket.
recycle bins.
rubbermaid containers.
hand dryers.
recycling bin.
restroom signs.
swiffer wet jet coupon.
swiffer sweeper.
sloan flushmate.
traffic cones.
mansfield toilets.
finishing a basement.
floor scrubber.
feminine odor.

Anonymous said...

Great info.I like all your post.I will keep visiting this blog very often.It is good to see you verbalise from the heart and your clarity on this important subject can be easily observed. beachbody coach | insanity workout | P90X workout | revabs | insanity asylum | brazil butt lift | turbo fire | P90X2 Workout | shakeology Thanks again!

Fue Saç Ekimi Nedir said...

The livejournal guys have a paper / preso on this topic that details horizontal partitioning to address this problem. At least when one guy slides off the face of K2, the rest of the party doesn't go with him.

Algevis said...

A very nice page. I think the effort has passed, we have to thank you:))
Estetik diş

Unknown said...

a very nice site. Very useful. I will recommend your site to any environment.Estetik

Website Designers Bangalore said...

this touched my heart… your words and her music. i am honored to have found you both

pull your banner ads until google does a better job