Page 1 of 31

Server maintenance & server instability

PostPosted: Fri Sep 25, 2015 6:05 pm
by Daemon
Dear community,

As you are most likely aware, a lot of changes have been applied recently regarding server performance and content release (the upcoming patch notes will explain the changes in more detail). However, the recent updates seriously affected the servers overall stability. Nostalrius rarely crashes in usual circumstances, but right now we are having multiple server crashes per day and this issue is a top priority for our team.
After a few days of analysis, we have several reasons to believe that these crashes are indirectly related to BWL raid.

To confirm this hypothesis, BWL will be closed after the next server crash (if any!) for a short time. All the characters that were inside the raid will get ported to the entrance. Raid IDs won't be reset.
BWL will be available again tomorrow around 16:00 (server time / GMT + 2).

Best regards,
Nostalrius Begins Dev Team.

EDIT (2015-09-26 10:10): The server still crashes without BWL and with a few performance systems disabled.
The live server now runs with advanced monitoring tools to track crash origins. This may affect (even more) server stability, and performance.
More information on that topic will be available as soon as our investigation goes.
Since yesterday, 5 "hidden" crash sources have been fixed, but not the main one sadly.

EDIT (2015-09-26 10:36): The server may be unavailable for at least one hour. We are fixing a lot of crashes at the moment thanks to the monitoring tools.

EDIT (2015-09-26 11:18): The monitoring tools have been removed from Nostalrius. It should be more stable now. Analysis is still ongoing on the datas we gathered during the last crashes.

EDIT (2015-09-26 16:00): BWL is now accessible again.
The monitoring done on the server this morning allowed us fix 10 confirmed crash sources after analysis.

Tomorrow morning between 9AM & 10AM server time, we will take the tracking to the next level. During this period of time, the monitoring tools will crash the server on errors to identify actual crash sources in live conditions. Huge changes are being done towards the server stability since last week in preparation for the future.

Thank you again for the support!


EDIT (2015-09-27 10:10):
The server maintenance is over.
This second analysis period helped us identifying (at least) 3 other crash sources and a few bugs that we did not manage to explain.
To give an idea about the number of crashes fixed, we had yesterday less than 1 minute uptime with our monitoring tools, and in the end of this session we had more than 10 minutes uptime (with more players!).
A full feedback will be available shortly.

Re: BWL Temporarily Unavailable

PostPosted: Fri Sep 25, 2015 6:07 pm
by Robotron
I'm so glad my guild doesn't raid tonight.

Thanks for the update on the crashes, Daemon.

Re: BWL Temporarily Unavailable

PostPosted: Fri Sep 25, 2015 6:14 pm
by Pokeypointy
Hopefully you'll get the data you need. This will only delay everything down the line and nothing more.

Thanks for the update.

Re: BWL Temporarily Unavailable

PostPosted: Fri Sep 25, 2015 6:28 pm
by notgunnahappen
Very few guilds raid on friday, so good call. Glad you guys are looking into this.

Re: BWL Temporarily Unavailable

PostPosted: Fri Sep 25, 2015 6:54 pm
by Imbaslap
notgunnahappen wrote:Very few guilds raid on friday, so good call. Glad you guys are looking into this.


except entitled does..
poor fellas.. :(

Re: BWL Temporarily Unavailable

PostPosted: Fri Sep 25, 2015 8:09 pm
by Maladocious
Thanks for the update, Daemon.

Re: BWL Temporarily Unavailable

PostPosted: Fri Sep 25, 2015 8:14 pm
by Niko
Could we get an official information on what's causing the crashes?
If there is something necessary to be done towards upgraded ddos protection, I'm pretty sure more than just me are willing to give a donation towards better soft/hardware.

Regards,
Niko

Re: BWL Temporarily Unavailable

PostPosted: Fri Sep 25, 2015 8:29 pm
by rekc
Niko wrote:Could we get an official information on what's causing the crashes?
If there is something necessary to be done towards upgraded ddos protection, I'm pretty sure more than just me are willing to give a donation towards better soft/hardware.

Regards,
Niko


They just said that they believe that BWL is causing the crashes and they are closing it down to trouble shoot the issue.

It seems that the BWL patch always ruins every private server, ever.

Re: BWL Temporarily Unavailable

PostPosted: Fri Sep 25, 2015 8:44 pm
by Niko
Yeah, I was suspecting so, but I know Nostalrius is under a high threat from other projects who failed to listen to their community, thus getting fukt in de ass by this one))

Re: BWL Temporarily Unavailable

PostPosted: Fri Sep 25, 2015 10:00 pm
by whitekidney
Don't spam