2009-06-27

On the road to Camel 2.0 - OnCompletion

Its time to write a new blog entry on another new feature in Apache Camel 2.0 - on completion.

On completion allows you to do additional routing when an exchange has been completed.

Imagine you need to register a special log entry when a particular route is complete. What you could do at first sight was just to add a processor at the end of the route that would log the entry.

But now the processor is part of the original route and it could potential affect the outcome. For instance lets say the processor could not log the entry and fails, or it accidently changed or altered the payload, or it was time consuming. All affecting the original route.

So how can we avoid this? What we want to do is to execute our processor after the original route is complete. With the new onCompletion DSL you can do this.

Lets look at a little sample to help understand how its setup and works.

First we define a route as below:
from("file://inbox")
.to("bean:validateFile")
.to("activemq:queue:inbox");

And then we want to do run our custom processor FileCompletionProcessor when the route is complete. So what we do is to register an onCompletion on this route.

This is done by adding our onCompletion
.onCompletion()
.process(new FileCompletionProcessor())
.end()

... to the route as shown below:

from("file://inbox")
.onCompletion()
.process(new FileCompletionProcessor())
.end()
.to("bean:validateFile")
.to("activemq:queue:inbox");

Notice: We need to declare an end() on the onCompletion route to indicate the end of this one. What it means is that the onCompletion is a first class route on its own. So you can do all the DSL in this route as you like. You can do content based routing, filtering, invoking additional processors or beans etc. For Camel its just another route.

What happens is that when the original route is complete, e.g. the file content have been send to the activemq queue and the route is complete. Then the onCompletion kicks in. What Camel does internally is that it starts a new route in another thread that has a copy of the original exchange, that is routed in the onCompletion route. So the original route continues to run in its own thread to complete immediately. This ensures that the onCompletion route will not affect the original route.

The sample above was a basic example. What if you only want to do special work in case the original route failed? Well Camel supports DSL to specify:
- onFailureOnly
- onCompleOnly

For instance we want to invoke a bean registered with the id failureService in case the route failed. We can do this as:
.onCompletion().onFailureOnly()
.beanRef("failureService", "fileFailed")
.end()

And just like other functions in Camel onCompletion also supports scopes:
- global
- route

And the onWhen predicate that you might have seen as well.

Take a look at the documentation for further details and sample. It provides XML samples as well.

2009-06-16

Apache Camel 2.0m2 released

Yeah finally we got the 2nd and last milestone for Apache Camel 2 released.

And this time even the front page at Apache Camel has the news item in time :)

I had used 30 minutes to write a nice summary of the important new features in this release. But somehow Blogger managed to shrew things up so it got lost. Damn why does it not auto save and let you look back revisions of edits. What I could get back was just the first 2 lines. So I am not in the mood to rewrite it again, so I keep this blog post short.

You can check out from the release notes here.

This is the last planned milestone release. The plan is to use close in on the final release, fixing bugs and adding a few new features. And maybe taper a little with the API as we can seize the moment and do it before 2.0 is out. We hope to get 2.0 out within 1 month.

Please give 2.0m2 a test drive and we are especially interested in hearing about your experience if you upgrade from existing Camel 1.x applications.

2009-06-03

Need tooling for Camel? We got FUSE tooling

At FUSE we do have tooling for Apache Camel. In fact the new FUSE Integration Designer 1.2 release is out of the door.

This tool have visual route editing for Apache Camel. My fellow colleague Oisin Hurley, whom is the leader on FUSE tooling announced it on his blog.

What is especially interesting for Camel in this release is that they have added all the EIP patterns we have in Camel. And its really cool to finally be able to see the routes in great looking diagrams. Its like being able to see the Matrix code as motion picture instead of green scrolling text.




Its a two way editor so you can import existing Camel routes, continue to edit them and export it back. And yes you can also export the diagrams as image files so you can put them in word documents or the likes.

You can read more about the FUSE Integration Designer and downloads at this link.

And head over to the FUSE Tooling forum where you can ask questions about the tooling and even meet Oisin himself on the forum.

2009-05-23

On the road to Camel 2.0 - Concurrency with the async DSL

Its time to continue the road to Camel 2.0 and I thought that I should detail a bit more on the async DSL we have in Camel 2.0. It replaces the old thread DSL from Camel 1.x.

The async DSL is like the thread DSL used for concurrency. Under the async DSL lies the excellent Java Concurrency API where we leverage the ExecutorService for submitting concurrent tasks.  By default a thread pool with 5 threads is created. But you can pass in the core pool size if you like. For instance: from(x).async(20).to(y)

Suppose we have this simple route where we poll a folder for new files, process the files and afterwards move the files to a backup folder when complete.

from("file://inbox?move=../backup-${date:now:yyyyMMdd}")
    .to("bean:calculateBean");

The route is synchronous and there is only a single consumer running at any given time. This scenario is well known and it doesn't affect thread safety as we only have one active thread involved at any given time.

Now imagine that the inbox folder is filled with filers quicker than we can process. So we want to speed up this process. How can we do this?

Well we could try adding a 2nd route with the same route path. Well that doesn't work so well as we have competing consumers for the same files. That requires however that we use file locking so we wont have two consumers compete for the same file. By default Camel support this with its file locking option on the file component. But what if the component doesn't support this, or its not possible to add a 2nd consumer for the same endpoint? And yes its a bit of a hack and the route logic code is duplicated. And what if we need more, then we need to add a 3rd, a 4th and so on.

What if the processing of the file itself is the bottleneck? That is the calculateBean is slow. So how can we process messages with this bean concurrently?

Yeah we can use the async DSL, so if we insert it in the route we get:

from("file://inbox?move=../backup-${date:now:yyyyMMdd}")
    .async(10)
    .to("bean:calculateBean");

So by inserting async(10) we have instructed Camel that from this point forward in the route it should use a thread pool with up till 10 concurrent threads. So when the file consumer delivers a message to the async, then the async take it from there and the file consumer can return and continue to poll the next file. By leveraging this fact we can still use a single file consumer to poll new files. And polling a directory to just grab the file handle is very fast. And we wont have problem with file locking, sorting, filtering and whatnot. And at the same time we can leverage the fact that we can process the file messages concurrently by the calculate bean.

Here at the end lets take a closer look what happens with the synchronous thread and the asynchronous thread. The synchronous thread hands over the exchange to the new async thread and as such the synchronous thread completes. The asynchronous thread is then routing and processing the message. And when this thread finishes it will take care of the file completion strategy to move the file into the backup folder. This is an important note, that the on completion is done by the async thread. This ensures the file is not moved before the file is processed successfully. Suppose the calculate bean could not process one of the files. If it was the sync thread that should do the on completion strategy then the file would have been moved to early into the backup folder. By handing over this to the async thread we do it after we have processed the message completely.

For more information about the new Async API in Camel check out my previous blog entry and the Camel documentation.

Those where the days where the internet led did not go green

Monday night a storm hit southern Sweden and as the blog title indicates the storm did bring in other issues than just rain and thunders. Next morning the internet connection led did not go green and thus not much you can do. The enjoyment of having the first cup of coffee isn't quite the same as you cant take it easy and read the news on the net and turn to your mail box to see how many millions I have won in all the lotteries or what's the spam of today is.
 
As I get up before most of other people I had to wait until 8 am before my ISP provider opens the support center. Well you still get a voice machine.

In the old days you used to navigate these call centers by pressing the numbers on the phone. But not today they use voice recognition. So as a Danish citizen living and working in Sweden its a challenge to understandable swedish for a voice machine to understand that I just want to get a status when the internet will be back. After several tries and having a lot of "I don't understand you" from the voice lady, I won, it gave up and presented me the good old fashioned button menu navigation. With this I could press 4 and then 3 and get the status. And guess what, the next time I call in its much faster with the button navigation as I could just press 4 and 3 again. But no the damn phone company want to pesky me with that voice lady again. 

As the phone company is the largest in Scandinavia I guess they reaction to remedy the internet for a local town in the southern Sweden isn't on their top of the list. So today the internet is still not green and the latest estimate is 3pm this afternoon. Well guess what they said that yesterday as well. They just add 6 hours to the clock each time. 

Well one service they offered that I thought would be cool was that you could type in your mobile phone number and will receive a text message when the internet is green again. But yet again they would not accept my mobile number as it was from another vendor. Aint it sweet when companies just go to big and don't bother with the little people.

And on top of that I missed the latest episode of Heros last night, couldn't see digital TV either as its also requires the internet led to be green.

So appreciate when you internet led is green. You newer know when a storm is coming and having a ISP provider that is slower than a turtle to remedy infrastructure that people start to take for granted. 

Maybe wireless broadband is the future, haven't heard of any major phone company having 2 days downtime on the mobile phone net.

Just heard that in Denmark the record was topped for the most used active hour last day. More than 2.600.000 minutes was registered on the mobile net during a hour last day. That is 50% of the population was talking for 1 minute in that hour!

Yeah since you can read this blog entry I guess the internet led is green again.
The downtime was 3 days, on the 4th day the internet led did go green again :)

2009-05-22

Apache Camel 1.6.1 Released

We the Camel riders is happy to announce that Apache Camel 1.6.1 has been released.

This is a maintenance release with approx 93 issues resolved. As we want the 1.6.x series to be production stable the next 1.6.2 release should only contain important bug fixes. Its important for the Camel community that uses of 1.6.x in production can rely on that we wont break or anyhow compromise the release.

All future development and features go into the 2.0 release.

2009-05-16

Apache Camel online training

I take my good employee hat on and posts about this new offerings we have.

On-line training for Apache ServiceMix with Camel
Interactive classes without having to travel

Not getting the most out of your open source integration projects? Register today for one of our travel-free Apache ServiceMix with Camel classes. This is an intensive two-day, instructor-led, live training class that is delivered in a virtual classroom via the Web.
The course includes lectures and hands-on, code-level exercises targeted for developers and architects. Throughout the course the instructor, a teaching assistant and a member of the Progress FUSE Technical Services team are available to answer questions and assist with any technical issues. Click here to view the full agenda.