Foreign (non-PHP) Workers for Laravel Queue

[g41797]

Foreign Worker:

• Is application written on non-PHP language
• Listens on Queue and processes Jobs
• Supports the same set of drivers (SQS, Redis, ...)
• Extendable: possibility to add
  • new type of driver
  • job handlers
• Simple for development and maintenance
• Supports php-nonPhp e2e testing

I'd like to implement this software on Go, but don't know whether it's valuable use case: some months ago I raised the same question on Yii Queue repo without success.

Or possible it may be done straightforward and does not require any additional framework/library/package

[bulletproof-coding]

Hi. This is the same as calling from php an api to process something in that target project queue. In this case you don't have to define the job in php just to send the data as it would for your proposal (not to mention that you would need the same job FQN in the Go processor).
The job dispatcher should be also the job processor for simplicity.

[wizzymore]

I don't see why this will ever be a need, the solution already exists.
You can already use something like Redis or SQS ( anything not laravel specific like the db driver ) and push messages that don't get consumed by Laravel but by another consumer and have a generic serialization like json or whatever you want.

[wizzymore]

Well, I am talking from experience here.
When you build a microservice type of arhitecture you want to KISS, that means i don't want to have vendor code where I don't need to, I don't want breaking changes because of changes in the Job dispatching from Laravel and I also don't want to base my other language code on PHP's serialization/deserialization.

I don't see this as a solution because there is no need for one, you want your jobs to be as simple as possible and as fast as possible, making a go worker to process Laravel issued jobs is not that.

If you have a proof of concept maybe I'd agree with you, but i tried it and it doesn't feel "Laravely" anyway you'd do it. It was better to just use the SQS Client and send messages to the queue and having a Service Class abstract this away.

[g41797]

I expected JSON marshall/unmarshall
I think job dispatching still should be responsibility of
Laravel Scheduler and worker should only work
As far as I understand you tried to implement similar solution
Do you have it in public repo?

[wizzymore]

No, unfortunately the work done was for a paying client.

But i can say what we used instead:

• Faster processing that needed to be done in sync: HTTP requests to a GO API server that stayed in a private network that had no public access
• Processing that needed to be done in background that we did not know what language will process it we just sent a message on a SQS queue using a proprietary encoding for the data sent so it always stayed private and you can safely deserialize it.

To be honest, if I would have made the app from the ground app, I would have chosen to do everything in PHP. In jobs we can use concurrency and threads so we can parallelize the processing if we really had to but i would prefer to split a job into a batch when a lot of processing needs to be done at the same time, this is also more monitorable. ( Something that Github covered here: https://github.blog/2024-06-11-how-we-improved-push-processing-on-github/ )

And also, a lot of stuff needs to be processed in the background because you can't handle concurrency in PHP-FPM, but now i use Octane (with Swoole) for all the apps I deploy and I can easily use Coroutines, they are amazing and speed up your application by a lot. A lot of stuff that i had to put in jobs now can be done in parallel, imagine doing 10 api calls that take 10 seconds in total but the max wait time for individual requests is 1s, add them in a coroutine and now it takes only 1s and you can skip the job.

[wizzymore]

You can indeed get more speed by using better languages, but what I realised is that at the end of the day, the reason we chose PHP and not C++ is because we want stuff to be simple and as performant as possible, but a big emphasis on simple.

Adding one more language on your stack only adds complexity and you really need to think about that, do you really need that second off a job but maintain 2 code bases or 3 or 4?

Currently at the company I work at alongside PHP we use Java and Python for data processing and believe me, it only adds complexity, life of ALL of the devs in the company would have been a lot easier and we would have had a lot less bugs if we just stuck to what Laravel provided, but people hate monoliths for the bad reasons and create bad solutions because of their ego or because they don't want to put '$' in front of a variable name.

[g41797]

I liked note regarding '$' in front of a variable name
My constant slip during php development

[menpablo]

Maybe it would be useful, at least it would have been for some of my projects, to make the worker able to receive messages on plain json. This would allow third party services to send messages to be processed by laravel.

It could be done this way:

The message payload should have a field MessageType that could be match with some configuration on the queue.php file to map it to a job. I.e:

{
"Message": {
"MessageType": "process_order",
"data": {
"resource_id": "8888888",
"user_id": "191583287"3870419",
"attempts": 1
}
},
"Timestamp": "2024-07-01T21:12:58+00:00"
}

on queue.php it could be mapped with something like this:

'worker_map' => [
'process_order' => App\Jobs\ProductOrderJob::class
]

What do you think?

[g41797]

i am talking about opposite direction:

• laravel app sends plain json to q
• go worker process job

in order to send job from foreign service to laravel worker plain json is not enough
we also need to implement the same transport protocol

it is not a problem at least on go

do you have real use case?

[menpablo]

I have a central inventory systems that receive inventory movements and other data from integrations.
We just have a contract where the message have to have the format previously mentioned in the previous comment.

For what you need I guess you could just send the messages to the queue and have a go app to consume those messages I wouldn't think it is necessary to have the go app inside laravel.

Cheers!