Preface: I work as a professional software dev, and part of my job involves scaling applications to pretty high demands.

There's a statement Chris made in his post that stuck out to me, and I really wanna point it out as a big deal, cause its easy for folks to miss:

Chris Wilson: I want to emphasize that these changes have been load-tested before deployment, so we have no explanation for why they are failing under the load of real users.

^{Source: https://www.reddit.com/r/pathofexile/comments/msbiuv/extremely_slow_queue_processing/}

Now I wanna say something here... that situation is basically the absolute nightmare scenario for any Dev

This scenario is the "We did the load testing, we QAd and QCd it, we simulated this situation, we were confident this wasn't going to happen. This wasn't laziness, we genuinely specifically were prepping for this to be an issue and pre-emptively tested to make sure it wasn't

And then, after all that effort... it still happened anyways and we have no idea why

That is absolutely the "Oh no" moment for devs. I can 100% call right now their are devs, engineers, testers, Chris, and many others who are having to accept the fact they probably arent making it home for dinner tonight at this rate.

I have personally been in that situation myself and I want to say, It sucks. Really bad.

Right now there's likely an exhausted team of devs trying to figure out wtf is happening, they're running tonnes of tests trying to isolate the source.

And I 100% guarantee Chris Wilson has probably been on hold for a few hours now trying to get ahold of his database/cloud providers that host PoE on a Friday night, escalating shit up the tech chain from lv 1, lv 2, and lv 3 tech support to find out why the hell his servers are on fire and wtf is going on, and probably keeps getting put on hold.

Right now, GGG needs some support. This is not a "Fuckin GGG how dare they fuck us over" day

This is a "Fuck that sucks GGG, that's basically the worst case scenario, Take our energy!"

To kind of make a metaphor...

This isn't like an anti-masker going out and getting COVID and you gloating "haha sucks to be you"

This is someone who did everything right, did the steps, wore their mask, social distanced... and somehow still got COVID anyway (prolly cause someone else fucked em over)

So, let me go ahead and say it:

༼ つ ◕_◕ ༽つ GGG DEVS TAKE MY ENERGY ༼ つ ◕_◕ ༽つ

Edit: Addressing some common misconceptions

1. "Just shut it down, fix it, then turn it back on

Shutting it down wont make things go faster, and wont help anything. Also, the devs are likely using the live data from the servers breaking as important information to help isolate the problem, its pretty likely right now they have logging and data collection happening everytime things break to continue trying to isolate the problem.

In other words, if GGG shut things down right now, they'd stop getting that useful data they can use to isolate the problem and solve it

2. "GGG had 10/12/whatever years to fix this"

Based on Chris's post, this is a totally new problem they havent encountered before. This isn't something that crept up.

Awhile back last league IIRC, Chris also made a post discussing how they were working on migrating to a more scalable solution to prevent previous issues.

It's pretty likely that in the process of fixing the stuff that happened in Heist, they encountered new issues.

Fundamentally, scaling large scale many many user applications is simply just super fucking hard and extremely prone to breaking

It just happens and shit breaking league start is probably always gonna be a thing that happens for what is effectively the #1 most popular (and thus most load tested) ARPG on the market

If you think this is purely a GGG problem, even big triple A (much much bigger) corporations encounter this exact same issue.

Anyone who has played FFXI, WoW, or FFXIV can attest that Day one released of new content that produce huge influxes of players often results in a lot of problems.

If companies 20x bigger than GGG still have this issue, its kind of silly to expect GGG to be any less capable of errors.

Feel free to google "Raubahn Ex" for example memes of when Square Enix, a WAAAAAAY bigger company fell to the exact same sorts of issues on FFXIV.

3. Why didnt they test it on live servers before big patch?

It is distinctly possible this issue has been present for who knows how long on live servers, and it only just shows up under stressed loads.

For all we know this was a thing for the last 2 months but we just weren't stress testing the game at that level and only now did it show up today.

4: Giving this post Awards

Hey I love the enthusiasm and appreciate it.

But instead of giving awards to me, go show Chris some love and give him some "Take My Energy" awards on his post over here:

https://www.reddit.com/r/pathofexile/comments/msbiuv/extremely_slow_queue_processing/

5: Make a beta test / stress test temp league before real league!

As nice as this idea is, it also breaks a really core part of Path of Exile's identity as a game, a big part of what makes it special, and would kind of destroy pretty much all of GGG's marketing strategy.

Such a huge part of the league is the spoiler season, the teasers, the build up, and the hidden surprises set up for us ahead of time.

Creating any form of, even short and temporary, "beta test" system would absolutely destroy that entire concept and ruin the hype train.

If you make it limited access, now its not a stress test. If you make it a stress test, then all you get is just a bunch of people playing then and then peacing out and not being invested in the actual league.

And anyone who avoids it and wants to wait for the league risks getting spoilers from the beta testers too.

So altogether its kind of a non-option, unless of course you are okay with giving up the Bex Teaser Season fun we all like to have here.

6: This shit happens every league!

Well... No. No. Actually. It doesnt and hasnt

Every league has had its issues. Absolutely. But it has been a distinct and different issue every time

Delve league was client side issues causing crashes due to missing models, and that one crashed you to desktop.

Bestiary and Synth were distinct UX problems.

Heist was a localized scaling issue with hardware.

Betrayal was engine performance issues causing FPS spiking.

Blight league was the Trade API itself choking, and ritual it was a specific app and specific couple of users basically DDoSing the Trade API*

The list goes on and on, sure every league has been rough but every time it was a different kind of issue

And thats simply because Path of Exile is a big ass game and has a lot of moving parts, so stuff is just gonna break sometimes. Thats just how it is and will always be for a game of this size.