Welcome to the {KN} Kill Noobie Clan Forums!
Welcome Guest [Log In] [Register]
We hope you enjoy our servers enough to join us here and talk directly to us!

Feel free to register for our forums, take a seat, make yourself comfortable, stay awhile!

Shoot ya later!

{KN} Clan



Click HERE to reigister for our forums!

Username:   Password:
New Reply
Finally some candid information
Topic Started: Jan 27 2011, 07:55 AM (419 Views)
{KN}You're Damn Skippy
Member Avatar
KN Nerd
Finally some candid information from Gameservers. I wanted to pass this info along to those that might still be interested in the status of this.

These are quotes from Gameservers Zachary Williams ([gs] leviathan) on the Black Ops server stability problem.

Quote:
 
Hello,

To reply to your question, here is what the problem is, the triggers (as we know them to date) and the work that has been done.

1: The Problem:

The core issue is in the physics engine of the game. The crash happens during some physics calculation. We have confirmed that the crash comes from within the game binary, and it is very consistent as to the specific module that it is hitting (phsyics).

2: The Trigger
This issue seems to require multiple things to happen to trigger. One of the triggers, is RCON use. We know this because of the GameTracker scanning issue that happened. We increased scanning frequency, and crashes went through the roof. Se stopped that quickly, and now use a much more passive method of acquiring data (similar to that of what the server browser uses). Some other third party companies have tried to scan us regularly, and those have been blocked, to ensure as much stability as possible. This is also why we have made the suggestion to limit RCON use, as it may affect stability.

3: What has been done so far?
Every crash, on every server we have is logged. Analyzed, and recorded in a database. Treyarch has direct access to this information to assist them in finding and repairing the problem. This bug is elusive, and nearly impossible to replicate on demand (we have tried, a lot to force a server to crash, but have not been successful). This is important, because, if we can't replicate it on demand, 'testing' of new code becomes virtually impossible. New code requires client updates, as well as server updates, so the only way to know for sure, is to release the patch with the fixes that have been made.

This is not to say treyarch has not tried to verify the fixes! They have test servers up, the populate them, they keep them running for days while troubleshooting. But, as we can't replicate crashes on demand, their tests can show no problems, but when it gets out into the real world... then we see the results.

So, that's the real deal. The short of it is that we at least know where the crash is happening, we know its triggers, and have created a complete crash database and tracking system to assist in finding the solution. Treyarch is working continuously to try to repair this (and other issues in the game). For example, there was a crash exploit that could have allowed anyone to crash a server on demand (different thing than the crash bug). That was fixed in the latest release before it got into the wild, which is a very, very good thing.

If you have other questions, I will monitor this thread to try to answer them. Just note, that I do not speak directly to Treyarch developers, so I can primarily only answer questions that would be related to Gameservers.


Quote:
 
On demand replication is really critical to bug solving. When I say can't replicate, I mean, we throw GT requests at a box, repeatedly, for an hour or more, and can't get it to crash.

Trying to solve a problem requires more instant ****. We know we can have it crash here or there, but if we can duplicate the exact conditions and 'make' it crash, now there is a way to really narrow in on the exact piece of code that is the culprit.


Quote:
 
To address some speculation.

1: Rcon tool has shown to cause an 'increase' in crashes. Please don't mistake 'not using the tool' to 'no crashes'. Rcon use, in general, be it tool or scans is a trigger. We know this, we have verified this.

2: We have tried spamming rcon packets. We have spent a lot of time trying to replicate this. We know full servers have a higher probability of crashing. We have taken the servers that crash most often (by our crash log database), and spammed them to no end. We have not been able to make them crash on demand.

What this means, most likely, is that rcon is not the sole cause. This means that something, within the game must be present in a specific state before the crash happens. It probably ALSO means that it requires both game state 'and' some rcon command.

3: We are not saying you have to come to terms, and that it will never be fixed. That is very unlikely. What 'is' likely, is that it will take longer to fix.

I have done my absolute best to be as candid as possible. I'm not saying anyone should be happy with the information I've provided. We're not happy. However, we know that we are doing everything we can do. We're still working on reproducing the crash on demand. We are working on behalf of every server renter here to help get this solved.

Thank you for taking the time to read this.

Zachary Williams
Gameservers.com


Quote:
 
There are no other ways we are aware of to lower the rate of crashes, other that keeping rcon use to a minimum. Unfortunately, that is all that can be done currently.



Profile Quote Post Goto Top
 
{KN}NoobieDoobieDoo
Member Avatar
Commander
Thanks for the post Skippy.....Though not the answer we hoped for, it is at least a candid and perhaps truthful response. It's about time.

I just hope the boys at the Treyarch are doing overtime to fix it........ :huh:
Profile Quote Post Goto Top
 
{KN}You're Damn Skippy
Member Avatar
KN Nerd
BTW,

Once this info was released a community member managed to find a repeatable cause for crashing within hours. Not saying that is the ONLY source of crashes, but a repeatable cause is one that can be fixed. Hopefully this helps.

GS has not been able to verify this completely, but we'll see once they get the chance to work through it. Maybe some real headway will be made here very shortly.

Quote:
 
Excellent find. I wonder if the busy servers we tried to crash, just had people with fantastic aim.

I am going to get this information verified by our staff, and sent along to treyarch.

Just remember guys. This isn't an inability on the part of GS or Treyarch, this is a really great job done by a community member, based upon all the information we had at the time.

It's not fixed yet, but, I feel like we're a solid step closer to the goal.

I'll keep this thread updated as I get any additional information.

Well done.


Quote:
 
We are trying to reproduce this, however, no success as of yet.

Are you able to make this happen with one or two shots in a controlled environment?

So far, we do have the information from the logs, but, actually 'making' it happen by firing a shot directly, has not been successful so far.


Quote:
 
Yes. The RCON spam is there. We have tried this before, just so everyone is aware here. We have taken popular servers, spammed them with rcon requests, and have not been able to make them crash.

WE spent about an hour here this morning, spamming with rcon, and trying to shoot in lower torso with a variety of weapons, and were unable to make the server crash.

So, I need to get more information on how demonstrable this was, or, if it was that it crashed, and that was the log entry.

We have the same log entries, and they have been looking at that, but, without actually being able to force a crash by a specific and direct action, this still leaves us in the same position as before.


Profile Quote Post Goto Top
 
1 user reading this topic (1 Guest and 0 Anonymous)
« Previous Topic · Call Of Duty 7: Black Ops · Next Topic »
New Reply