5 months already!

Nostalrius official announcements

5 months already!

by Viper » Fri Jul 24, 2015 7:38 pm

Dear community,

Nostalrius is now the biggest worldwide private server among all the expansions available: Congratulations!

The server has been online for nearly 5 months now, so let's give an overview of how things have gone and how things will go in the future. Since the release of the server, a lot of unexpected events that we had to tackle occurred which caused a delay in content release. Here is a list of the issues we encountered, with the solution found, and the extra time needed to reach the current status.

Glossary:
FTE : Full-Time Equivalent
ISS : Information Systems Security
ALM : Application Lifecycle Management
IsVV : Independant Software Verification & Validation
PTR : Public Test Realm

A * Amount of players

Solutions:
+ Core optimization to handle a great deal of players,
+ New hardware and a way to support the new upkeep,
+ Complete internal reorganization.

Upon release, our server was designed, for the first time in private server history, to support ~5.000 players online simultaneously without lagging. We reached much more players than expected (up to 9.100 players online, still growing). Unique core re-optimizations were done by the team after release to make the server be able to support +7.500 players without lag (+50% performance increase). Some additional server performance plans are in the backlog, but none of them are planned for now as we may have found another possible solution to allow a lot more players.
Extra work unscheduled : 1 FTE for 3 weeks.

B * Unexpected amount of attempts to use custom hacks in-game


Solutions:
+ Enhance tracking security
+ Create an automatic bot detector system

Security protection differs from a basic development task as once the system is established, you always have to update it to intercept new kinds of threats, much like an antivirus. Our Dev Team discovered a lot of new ways to hack since the release and fixed all of them so far, but we do not plan on stopping there. An exclusive automatic anti-bot system was also created to track botting activities. Extra time has been planned to mitigate new threats encountered later on.
Extra work unscheduled : 1 FTE for 4 weeks, work still needed.

C * Unexpected way to hack the infrastructure

Solutions:
+ Enhance internal ISS
+ Enhance provider ISS

Besides the game, hackers can also try to take down a system by using different kinds of attacks. During the last 2 weeks, a lot of new attacks tried to take down the system. We had to react fast to improve the ISS from our internal server and provider by suggesting new kinds of mitigation systems. The biggest attack spotted since server release was mitigated successfully without any impact on the server performance last week.
Extra work unscheduled : 2 FTE for 1 week.

D * Gold sellers

Solutions:
+ Enhance tracking system
+ Create an automatic anti-advertising system

This has been something very new for us since we have never reached this level of popularity before. We had to take actions against them by enhancing our tracking system and creating an exclusive automatic anti-advertising system to spot and ban the goldsellers.
Extra work unscheduled : 2 FTE for 2 weeks, work still needed.

E * Itemization

Solutions:
+ Gathering all of the items evolutions for each patch since the release of retail,
+ Need to create tools to deal with big data information (up to 15 gigabits of data to handle),
+ False alarm detection regarding wrong itemization for a dedicated patch

Itemization was something very important for Nostalrius when the server was rebuilt. To be honest, it was (and still is) a very difficult task to handle. Creation of tools to deal with the big amount of data was kind of new for the team too. Issues regarding itemization are always difficult to check because the server is lacking resources to analyze/check that everything is on point before releasing the content.
Extra work unscheduled : 3 FTE for 2 weeks, work still needed.

F * Transparency / Communication

Solutions:
+ Bugtracker
+ PTR Realm with new hardware and solutions to handle the upkeep
+ Streaming
+ Recruitment

A lot of people asked for more transparency from our side regarding how the development team was handling new tasks. After a state of the art analysis, an ALM was installed including the Bugtracker to help players report issues they find. Nearly 1.500 players subscribed on the Bugtracker to report all of the issues existing on the server.
Dealing with thousands of players also meant that we had to rework a lot the existing content because even the smallest issues (NPC with the wrong weapon, an object to be moved by one meter, etc.) are actually reported. As a consequence, IsVV Team couldn't handle all of the work alone. That's why the PTR Realm was created for specifics guilds. Giving a restricted access to the PTR gave us higher quality feedback than the feedback we could have gotten during Test sessions prior to the server release. On the other hand, a lot of complaints from all of the non-allowed guilds were addressed creating an elitist concept which was not good for the community. As a consequence, our next PTR session will be accessible for all players with a level 60 character with strict rules regarding how issues will be reported.

A streaming page has been created to display streams from Twitch and Hitbox.
For your stream to appear on our page, you just need to have "Nost" or "Nostalrius" in your channel status.

Nostalrius understands that communication is always a bone of contention in the gaming industry and we are working hard to free the server from this plague. We have already taken steps with our Twitter to give the community constant news that goes beyond what we post as announcements. As a consequence, a task force has been created to improve community management going forward.
If you have any suggestions/reviews concerning our communication,
feel free to PM our communication representative.
Extra work unscheduled : 3 FTE for 4 weeks, work still needed.

***

As you can see, a lot of unexpected tasks appeared, so the choice had to be made between releasing new content or improving the existing content (hardware, software, etc.). Nostalrius has actually learned a lot from these past 5 months and now has a better understanding of why retail made some of their choices in the past.
We continue to grow and learn as a team in experience and maturity, and even with the content being shifted back a month because of the reasons listed, you will still find very soon™ that we will provide an updated timeline. ;)

Best regards,
Nostalrius.
User avatar
Viper
Administrator
Administrator
 

Re: 5 months already!

by Souleàter » Fri Jul 24, 2015 7:41 pm

neat
thanks for the update viper 8-)
My pvp montage: https://www.youtube.com/watch?v=9omhqWtqqgo

Without a thought, Without a voice, Without a soul
User avatar
Souleàter
Private
Private
 

Re: 5 months already!

by DrearyYew » Fri Jul 24, 2015 8:38 pm

Glad to see these improvements coming down the pipeline, especially the bit about keeping the players in the loop more.

Let's hope we can stick to this
Dreary - Human "Needs No Mana" Holy Paladin
User avatar
DrearyYew
Sergeant Major
Sergeant Major
 

Re: 5 months already!

by Rhoen » Fri Jul 24, 2015 8:51 pm

Thank you viper =)
User avatar
Rhoen
Senior Sergeant
Senior Sergeant
 

Re: 5 months already!

by Trum » Fri Jul 24, 2015 9:08 pm

A * Amount of players

Solutions:
+ Core optimization to handle a great deal of players,
+ New hardware and a way to support the new upkeep,
+ Complete internal reorganization.

Upon release, our server was designed, for the first time in private server history, to support ~5.000 players online simultaneously without lagging. We reached much more players than expected (up to 9.100 players online, still growing). Unique core re-optimizations were done by the team after release to make the server be able to support +7.500 players without lag (+50% performance increase). Some additional server performance plans are in the backlog, but none of them are planned for now as we may have found another possible solution to allow a lot more players.
Extra work unscheduled : 1 FTE for 3 weeks.


The server still lagging a lot, 6k players yesterday and a delay around 2sec and everything appear only when you are under 5yards.

E * Itemization

Solutions:
+ Gathering all of the items evolutions for each patch since the release of retail,
+ Need to create tools to deal with big data information (up to 15 gigabits of data to handle),
+ False alarm detection regarding wrong itemization for a dedicated patch

Itemization was something very important for Nostalrius when the server was rebuilt. To be honest, it was (and still is) a very difficult task to handle. Creation of tools to deal with the big amount of data was kind of new for the team too. Issues regarding itemization are always difficult to check because the server is lacking resources to analyze/check that everything is on point before releasing the content.
Extra work unscheduled : 3 FTE for 2 weeks, work still needed.


Edited: Removed exploitable bug. Tickets that are flagged as exploitable bugs are hidden from public view.

As a consequence, our next PTR session will be accessible for all players with a level 60 character with strict rules regarding how issues will be reported.


Image
<Phoenix>
Trum
Senior Sergeant
Senior Sergeant
 

Re: 5 months already!

by ritjos71 » Fri Jul 24, 2015 9:19 pm

This is great info, as always the work is appreciated. The new time line will be very welcome as well. Thanks for the update please keep them coming =).
ritjos71
Sergeant
Sergeant
 

Re: 5 months already!

by Aklol » Fri Jul 24, 2015 9:24 pm

Still not apologizing for blaming guilds that tested for a bad AV release or the mistake of banning 3 premade leaders for a week for using a vanilla-era addon? Quite frankly, I'm disappointed.
Cabal Co-Conspirator
Aklol
Senior Sergeant
Senior Sergeant
 

Re: 5 months already!

by Cousy » Fri Jul 24, 2015 9:39 pm

Trum wrote:Edited: Removed exploitable bug. Tickets that are flagged as exploitable bugs are hidden from public view.


Or you know, maybe they hid the report from the general public so they wouldn't be aware of a bugged item before they could patch it.

Honestly, so many people on this forum are awful. Half the replies to this thread already are people who are trying to tear down the very post they've been waiting for the admins to bring. Stop biting the hand that feeds you and maybe it will feed you more.
User avatar
Cousy
Senior Sergeant
Senior Sergeant
 

Re: 5 months already!

by Armilus » Fri Jul 24, 2015 9:47 pm

Aklol wrote:Still not apologizing for blaming guilds that tested for a bad AV release or the mistake of banning 3 premade leaders for a week for using a vanilla-era addon? Quite frankly, I'm disappointed.


The ability to queue as a group was removed from the NPC's. They happened to miss one API function that would allow addons to automatically queue a group without confirmation dialog. You used an addon from vanilla wow that was created at a point when group queueing for AV was possible.

If the only way to queue as a group was with an addon, did it not occur to you that maybe you shouldn't be able to queue as a group? The fact that you could with this addon was obviously a bug and none of the people using it reported it, instead they chose to exploit it.

Maybe it just happened to be a coincidence that everyone who was exploiting this was also invited to test on the PTR, maybe it wasn't. Either way, they trusted your guild enough to give you access to the PTR and report any bugs that you found. At some point you found this bug, whether it was on the PTR or after release and chose to exploit it instead. If I was them, I'd be pissed. It seems to me that they exercised a shit load of restraint by only issuing a 1 week ban on the raid leaders. It could have easily been a lot more people.
Cadmus - Priest, Nostalrius Begins PvP
Stabsington - Rogue, Nostalrius Begins PvP
Armilus
Sergeant Major
Sergeant Major
 

Re: 5 months already!

by Badtank » Fri Jul 24, 2015 9:52 pm

Armilus wrote:
Aklol wrote:Still not apologizing for blaming guilds that tested for a bad AV release or the mistake of banning 3 premade leaders for a week for using a vanilla-era addon? Quite frankly, I'm disappointed.


The ability to queue as a group was removed from the NPC's. They happened to miss one API function that would allow addons to automatically queue a group without confirmation dialog. You used an addon from vanilla wow that was created at a point when group queueing for AV was possible.

If the only way to queue as a group was with an addon, did it not occur to you that maybe you shouldn't be able to queue as a group? The fact that you could with this addon was obviously a bug and none of the people using it reported it, instead they chose to exploit it.

Maybe it just happened to be a coincidence that everyone who was exploiting this was also invited to test on the PTR, maybe it wasn't. Either way, they trusted your guild enough to give you access to the PTR and report any bugs that you found. At some point you found this bug, whether it was on the PTR or after release and chose to exploit it instead. If I was them, I'd be pissed. It seems to me that they exercised a shit load of restraint by only issuing a 1 week ban on the raid leaders. It could have easily been a lot more people.



Can you stop this shit, please? It was not an exploit, it was not some secret witchcraft vodoo. It was a ten year old addon that has been used forever. Stop trying to frame an agenda, it just makes you look silly.



All this transparency is nice but do you have any plans to address some of the things affecting the server like herb nodes or the lack of fishing/seasonal/darkmoon decks? If not is the new community task force going to be doing that sort of thing going forward?
Image
User avatar
Badtank
Sergeant Major
Sergeant Major
 

Next

Return to Announcements