You have many servers that doing a lot of jobs. You have created many applications to make your system work well. You have defined different procedures to achieving your desired goal. As a result you get some systematic complexity, because of these distributed applications doing some messy work in different servers and you are away from your first intention to create these applications. Everything still works fine, you only lost the connection between applications because they became complicated.
Sometimes you have no time to research and you prefer to go wild way. But if you would have enough time to make some research for making a convenient decision, how would you change the architecture above?
Here, “convenient” word is the key of your research. You shouldn’t go too far, or shouldn’t choose a provider that lesser than your current solution. In first you get irrelevant complexity, in second you still can’t solve any problem.
While ago I was researching for a solution, I wrote down some necessities and draw schema to build our architecture. They are really helpful for seeing the bigger picture. Here are some of important considerations for me:
- I must see network topology easily, so I can understand everything smoothly.
- Applications must have different units of work (means different boundaries).
- Applications must be (or can be) distributed.
- Time consuming is important, so I can distribute them by consumed time.
- Application must only apply given work.
- If an application on a server cannot do a work, then it must be iterated to another application.
- If a given work cannot be applied on any application, then admin must be able to modify it.
- Applications on any servers can be able to connect each other in any necessity.
- Given work must be traceable.
- And Something (a control bus) must manage everything in one control panel.
Until here this looked like to me that I need a broker which is the responsible of communication between servers. Then I can send some messages between servers via that broker. So most probably it’s going to be a messaging broker. Now, I can make some research to what fits better to my problem.
Here are some options:
+ Redis PUBSUB RPOPLPUSH eg
+ Custom Designed SQL Queue Mechanism
I picked RabbitMQ in these options. Also I didn’t add some other options like ZeroMQ because it needs more abstract work to make it work IMO.
RabbitMQ is proven mature tool, usable in Windows, has a well explained documentation, it is designed for messaging, and written in Erlang is also a plus.
Here, I am sharing some basic rules with you that I want to follow when I create a message broker, especially with RabbitMQ.
Your producer shouldn’t be connected to queue directly, connect it to exchange directly.
Exchange is the head of your message broker. So don’t pass the exchange and connect queue directly. Always connect to exchange, it’s why we use it as a broker.
If your consumer crashes, then don’t lost the processing message (Ack Rule)
When your consumer application crashes for some reason, if it has a work to do at that moment, that work going to be a loss. It is better to not lose an unaccomplished work. For this, you simply send Ack messages back to your exchanges from your consumer applications after you finish your work for each message.
Disable too many message sending at that moment (QoS Rule)
When you created many consumers, some of them might wait in idle mode. Some work may take longer than others. But RabbitMQ doesn’t care which one is idle or busy. It just sends incoming message from queue down to the next consumer. This is fairly acceptable for a message broker. It just does its job. But we can tell RabbitMQ to wait for sending new message from queue until current message is processed by consumer.
Make queues persistence at least until they consumed (durable)
In Ack rule we guaranteed that if our consumer dies, we can still process same message in another consumer. But if RabbitMQ server halts or stops for some reason, we shouldn’t lost messages sent to RabbitMQ. For this purpose, we need to define our queues as durable from the beginning. Also messages should be flagged with persistent option. That is quite nice but it is still not guarantees the message losses in two ways:
1- Message could be sent and RabbitMQ stops
2- RabbitMQ server could not be reachable, TCP connection problems occurs.
For first option, we need a transaction on the channel. For second option, you may need more customized producers and consumers. So durable is not actually persistence.
Don’t use fanout exchange type until you need it
Fanout exchanges are replicates all incoming messages to connected queues. This is sometimes good, when you broadcast some messages to all connected queues. But if traffic and throughput are high, then your servers will get into bottlenecks.
Fanout exchanges mindlessly replicates messages and sends it to each queues. It creates high latency and it is not manageable. Use direct or topic based exchanges instead. There are simple rules for exchange types:
fanout : broadcast everywhere
direct: when you created a message with routing key same with binding key of queue then it becomes a direct message. This is what Direct exchange do.
topic : great flexibility, both fanout and direct messages can be simulated by topic exchange. When you use a # character as binding key for a queue, that queue gets all the messages comes to exchange. This is what happens in fanout exchange for queue’s perspective. And when you don’t use any special character in binding key, then it act like direct messaging for that queue.
Create your topology wisely and use Topic exchanges mostly
Here you can talk alot. You can decide so many things, even further than your necessities but don’t forget project time management. You want to achieve what you want, but you have not time. No problem! Just scratch some paper and use flow diagrams for better visualizing your server topology and try to make it flexible. So you can find more space for further queues next steps. You can use RabbitMQSimulator or start directly scratching something on tryrabbitmq.com website.
Add a Control Bus for dynamically control of producers and consumers
For obvious reasons you may want to control your applications availability, consistency. Also you may want to gather info from your applications like ping rate, heartbeat, logging, etc…
This can be done in different ways of course, but when you consider high number of applications, different locations, you start thinking to control everything from a center. So sending and receiving this extra info and work should be done in another channel for not effecting the actual work channel. This is what control bus planned for. You create an extra channel to sending and receiving that center’s messages.
If you need to reply message, then correlate it with request message
RabbitMQ lets you to define a correlation Id for each message. This gives you the ability of creating a relation between produced message and reply of it. You can understand which response created for which request.
Use some good extension libraries
I recommend you to take a look of each one and see if it fits with anything you want to build.
Before coding take a look at System Management patterns for your custom requirement
I strongly recommend you to check some messaging patterns, System management (also this) patterns. Those will guide you how to design your structure. There are very popular books also, which I think super informative and mind blowing. One of them of course Enterprise Integration Patterns and RabbitMQ in Action.
I also recommend you to watch Alvaro Videla’s this video:
Messaging systems are used by enterprise applications in different approaches. Being professional for each use case needs many years. Message based programming and distributed systems are way to go. I tried to touch some messaging design decisions in this article. Instead of focusing to one messaging application, I prefer to learn essentials of message based programming. And what I do in this article, I tried to collect all my decisions related to RabbitMQ and messaging systems. Have a nice one and keep programming…