Since the good weather is coming again, I started doing some sport, I got tanned, and trying to quit drugs that I used to deal around acting as a broker. Unfortunately higher level complain about no well distributed, local crews used to fight each other aggregate and …
No, not that movie. That’s just work on Software Architecture, but it is about solving a long standing problem on a software platform I worked for (and still I support http://starsellersworld.com ). I hope you enjoy, anyway.
The architecture involves message broker, orchestrator, and job execution log. The previous arrangement was like this
Well, that was sensible. Since the adopted message broker (RabbitMQ, but really any other message broker) can not support all kind of queue policy that comply with external API throttled based on usage, I developed a Saga Orchestrator based on command / dispatch concepts: a command can be “add a job”, the dispatcher is based on the job-type and dispatch the message through the message broker. This is a very “MessageBroker-Centric” architecture, and it fit the job.
At the time of development, I did not know about the strategyService, that was implicit, in fact it just take files from the Storage to craft strategy actions. The existence was implicit, and specification unknown, but it happens that it retrieved as much data as possible. The worst part was the enqueue logic, someway overcomplicated.
there are 2 two problems to be faced here:
- additional API call that is billed from the external service
- Saga Orchestrator handling storage and message broker traffic
The messages travel over two channel:
- From DataRetrieveService To Message Broker
- From MessageBroker To SagaOrchestrator
- From SagaOrchestrator To MessageBroker
- From MessageBroker To externalPlatformAPI
Then back:
- From externalPlatformAPI to …
… the same path other way around.
All these message passing are not very expensive in term of resource, but really cause a pile of crufty log stored in the Saga Orchestrator for handling something that happens to be already available on storage, on behalf of strategyService.
Behind this arrangement, with overlaying and duplicated messages, there are historical reasons and wrong evaluation. The former are not on place anymore, and the latter … well, it’s the point of the rework.
Working on another project, https://github.com/danielecr/hot-fixture-tool , gave me inspiration.
The (easy) idea is to rely on data already present on storage (and skip the message broker).
Implementation is inspired by unix find command philosophy: instead of getting all files from a folder, traverse the folder one file at time, and get the right based on filter (see https://github.com/danielecr/hfit-doc/blob/1671de76632f9f6f9f8948d616e95fc61d558612/docs/client/commands.md#files files interface***)
Well, after this the newer architecture is:
On this schema DataRetrieverService rely directly on storage to get the data, Orchestrator does not keep logs, but still the data is taken effectively.
This solved other feature for dealing with customer complain on usability.
Note
The adopted message broker was RabbitMQ, the saga orchestrator for dealing with throttled channel with different policies is the most effective way, and it does not depends on message broker architecture or limitation.
I am proud of my personal project, https://github.com/danielecr/hot-fixture-tool, as it solves an everyday problem when you work with legacy code. But still I got no star. Please, give me your star!