|
|
#1 | |
|
Firesky Staff
![]() Join Date: Mar 2007
|
Quote:
![]() We talk about this stuff a lot at work and I want your thoughts on it now... What do YOU think would work for this type of thing? Tell me what your thoughts are on this ![]() Last edited by Scepter : 02-25-2008 at 09:44 AM. |
|
|
|
|
|
|
#2 | |
|
Banned
Join Date: Feb 2006
Location: The Castle
|
Quote:
Funny you should ask. I started typing something up not too long after I made the post. Be warned though, it's very, very rough, and not knowing details about the game and how it works, I could only make something that in of itself is very broad and general: Bug reporting tool (rough draft) 1) Character name automatically put in 2) World waypoint: automatically put in, or easily obtainable by the player via UI 3) Is this quest related? Yes / No. (If yes, then drop down to name of quest from characters quest log NOT every quest in the game). If no, then skip to 4. 3a) Drop down listing name of quests in user journal 3b) Is the bug repeatable? Drop down Yes/No 3c) Description (with character limit) 3d) Submit, with autogenerated response / confirmation of submission email (ingame as well as out of game) to the player with a confirmation number for reference. 4a) Nature of bug/incident. Drop down menu: Character/avatar issues, World Geometry, Abilities/specials, World interaction (switch not working, npc unresponsive, etc), Exploit (dupes, etc), GUI issues, etc. Each drop down could/should have subtopics for further refinement. The goal here is to narrow down the issue to determine which team it should go to for resolve. Ideally, all teams should be divided up (art, world builders, etc even if only on a broadcast email group) and bug reports parsed to the relevant dev teams if at all possible, or alternatively (and probably the best avenue), all bug reports should go first to QA for immediate follow up and replication, then sent to dev teams responsible for that area. QA should flag each report with a score indicating severity and urgency. The two are not directly related; a bug may be severe, but not urgent, and vice versa (example: a dupe exploit may be simple to replicate/fix, but its urgency is paramount). If QA is unable to replicate, then someone from QA should contact the person who submitted the bug for further follow up. 4b) Description (with character limit) - You don't want too much to where the info becomes convoluted and useless, but you don't want too little so that you can't get an accurate description. 5) Submit, with autogenerated response / confirmation of submission email (ingame as well as out of game) to the player with a confirmation number for reference. I would argue that the drop down menus need to be as thorough as possible to help YOU guys isolate the issue as quick as possible. The problem for you guys is that players WANT to submit a bug, but at times it can be overwhelming trying to accurately describe what is happening. In beta invites, I'd stress the importance of a pen/paper to keep notes, or alternatively have a /notepad command in game that players can use in leiu of (it'd be nice if this could somehow be attached to the bug report...). Quest Surveys: Pretty easily done. POTBS had a pretty nice one although at times it became a nuissence. The biggest problem with surveys is you depend TOTALLY on your players to be honest and unbiased, both to get an accurate answer as well as us not just answering <whatever> to get it out of the way. Too many surveys or too many questions and we tend to have our eyes gloss over and click random numbers and get it over with. The magic number of questions for <me> is three... 1) Did you find the quest entertaining? On a scale of 1 to 5 (with 5 being very entertaining), how would rate this quest? 2) Did you find the quest challenging? On a scale of 1 to 5 (with 5 being very challenging), how would rate this quest? 3) Did you find the reward for the quest be worth the risk and/or time involved? On a scale of 1 to 5 (with 5 being 'worth the risk/time'), how would rate this quest? It would do wonders for you guys to have a background process that snapshotted how many people were in the group, what levels they were, and what classes they were, as well as the level/class the person filling out the survey was. It would give you a reason as to why someone thought a quest was too easy (too high of level versus the intended level of the quest, was it an intentional solo quest that the player did with a full group? Did the player try a quest that was intentionally set for a group/higher level? This is, of course, dependent on whether or not NPCs will give out quests to players outside the level range of the quest line the NPC is giving...) You could possibly have an optional 'Comments' section where players can freely put thoughts about the quest down, with character limit. They don't have to fill out the comments to submit your survey answer, to avoid getting stuff like adkjfhakjhsdfkhaskdf and 'submit'. Probably more than what you wanted, but there you go... |
|
|
|
|
|
|
#3 | ||
|
Join Date: May 2007
Location: San Francisco Bay Area, CA
|
Quote:
Quote:
A standardized template and objectives that everyone must follows - have to be driven by CME Drop down menu or list of the key features or categories to report the bug under- along with a numerical ranking system that is develop by CME that the beta tester can assign like (1-5) example where five is severe, and the bug is a kernel fault or system crash type. A well thought out beta testing schedule of what key components/elements to test along with a Calendar that beta testers can access to see what components that the dev want the beta testers to focus on and test out on when they sign on that day and for how long (days or week). We here to help CME, but CME have to drive the process and standards and provide us the framework, tools and formats that will make it easy for the beta testers to report and for CME to track the bugs and maintain the data.
__________________
Encourage Positive Behavior and GamePlay Dial the Gate, Let the Adventure Begin !!! |
||
|
|
|
|
|
#4 |
|
Join Date: Jul 2007
|
If you go with drop downs just make sure to include one area where we can place a description of the issue. Lots of times the drop down doesnt suit the needs of the bug.
Also a beta working ingame bug knowledge base would be nice. This can also be including in the beta forums so we have access off line to update other issues associated with the bug. Alot of times I will /bug the same quest with multiple issues. This can also be used as a great tool to let us know to re-test fixed content. Consistant beta events.. a little off topic, but run a different event every day or even 2 to 3 times a week (based off level content, quest series, etc..) .I did this in only one other mmo and they did it once or twice but it was so fun and a lot was accomplished. |
|
|
|
|
|
#5 |
|
Join Date: Dec 2007
Location: Minneapolis
|
Continuing on this topic, it would be nice to have a /stuck command. Make this command do two things. First, it creates a log file that contains whatever information would be useful to you for finding the hole, like current coordinates and, if possible, pathing information for where the character was moving about for the past minute or so. The second thing the command does is teleport you to where you were physically one minute ago. This should get you unstuck without the potential of people just using the command as a quick teleport home.
I agree with SnakeCharmer, the more robust the bug report tool the better. Having it autopopulate with information like character race, class, level, location, current mission and team size is great. Also, make it clear that you are capturing that information, so folks don't feel the need to duplicate it. Also, please put a small textbox for suggestions on the quest survey, or at least on some of them. And if you are going to have quest arcs, only put one survey at the end of the arc, instead of one popping up after each piece of the arc. I always have a better feeling for an overall arc than I do for one part of it. Lastly, some manner of suggestion box. I would recommend a section of the forums that is specifically geared to gameplay, quest and UI suggestions. Make it known that the mods will have a heavy hand there, and things that are off topic, or generally not on point to the discussion at hand will be cleared out, so the powers that be can easily find and respond to suggestions.
__________________
The Old Timers Guild
"Laid back, not so serious, no drama. All about the fun!" www.oldtimersguild.com >> Open to the Public! << |
|
|
|
|
|
#6 | |
|
Join Date: Oct 2006
Location: Fayetteville, AR, U.S.A.
|
Quote:
Name of Quest: Did the story element of the quest grab you?: Did the story element of the quest fit into the overall storyline of SGW? (could be tough to know that in beta): Did the quest giver provide sufficient information?: Did the quest clues provide enough direction for you to locate quest objective?: Where the quest objectives logical (i.e. killing wolves has nothing to do with salvaging naquadah generators)?: Was the quest fun?: What was the best and worst feature of the quest?: Enter additional comments: The problem I've had with many games I've tested is that there's only room in the forms for short answers. I understand that devs don't necessarily have time to read a thousand novels written by testers, but by not taking the time to read all that the devs aren't getting complete answers to important questions. Please don't limit tester comments to short summaries, but instead let testers fully explain what they are trying to express in bug and content reports. |
|
|
|
|
|
|
#7 |
|
Join Date: Jul 2007
|
In my opinion forms aren't so good for feedback. I mean, problem is that not all feedback fits to pre-define form and selections. I have had previosly problems with that. If it fits to form then of course it's okey, but what if it doesn't?
I'm more for like free feedback person. Because sometimes, I report bug, when something isn't working, I can be agaist color used in somewhere, I just make suggestiong because something is missing or because I could not do some logical action, because quest or somethinng doesn't allow. I ques I'm developers nightmare as beta testers, because I'm not walking pre-define roads, I explore possibilities. I do how ever agree with others that the feedback tool should be something what takes alot of pre-define information about player and situation where player is. Also it would be good if beta testers could fill the base information about they systems only ones. I'm not gonna tell everytime in reports what my computer cpu or OS is. Just store it in database one time. I'm sure developers know what to ask from use, but remember that some of us, could do something unexpected. Like been outside the zone map. |
|
|
|
|
|
#8 |
|
Join Date: Apr 2006
Location: P3X-888
|
I love forums for feedbacks.
SWG changed its entire death system because of the argument on a forum by first closed beta people From a save state system where your clone if not updated you lose all that you gained to something less punishing. It was forums not reporting, it was discussions open to all beta testers and QA people. It wasnt opening a bug report and filling in a little form and then having that form go off to god knows where.
__________________
Just say NO to the Warcraft static storyline and YES to the Asheron's Call model where things change, end and progress. |
|
|
|
|
|
#9 |
|
Moderator
Join Date: Apr 2006
Location: Slovenia
|
In another thread one of our devs, Khestral, has asked us to give our opinion about various reporting tools for when in beta.
In my opinion this is an enough important topic to deserve it's own thread. So I have opened this one and copied all the relevant posts (if anyone is missing just PM me. ).So here you have it, fresh from the gate. ![]() Post away at your hearts content about the topic. ![]() magecu
__________________
Don't limit your imagination! Last edited by magecu : 12-11-2007 at 01:20 AM. |
|
|
|
|
|
#10 |
|
Join Date: Nov 2006
|
Be careful with forms- too many fields and it gets confusing. As others have suggested, as much information as possible should be automatically gathered (player stats, location, system specs, current settings, etc). Also, I think Yes/No should be implemented as option buttons rather than a dropdown box.
|
|
|
|
|
|
#11 |
|
Join Date: Nov 2007
Location: TX
|
Definitely have an "other" option to report bugs. I've run into a lot of bugs that don't fit any category while testing other games. It's very frustrating to decide what category some bug will belong to with only a few options. Also, it makes it harder to get the bug info to the correct person if certain bug types are being handled by different people.
As a side note, PLEASE don't blow off weird bugs that only a few people report. I found a bug in the WoW mail system that kept getting blown off because no one else had reported it. It took almost a week of emailing different Devs to have someone look into the problem thoroughly. After finding the right person the bug was fixed. |
|
|
|
|
|
#12 |
|
Join Date: Aug 2006
|
Forums all the way!...
While beta testing a few games i reported numerous amounts of bugs via the in-game tool...at release around 80% of those bugged remained ingame,they all varied from broken quests to npc's floating around in mid-air. I'd much prefer a forum site where we could actually get dev confirmation that they are looking at the bugs... |
|
|
|
|
|
#13 |
|
Join Date: Mar 2007
Location: Switzerland
|
A nice to have would be if you could right-click for example on a NPC and you could chose there the option "report bug". A window would pop up and there are then different options for the kind of bug (for example: the active quests and if its a vendor or a NPC with some other function these actions can be selected) and the problem shortly described. At the same time the variables that define the relation and the status between the NPC and you (also with respect to the quests) are attached to the report.
Something similar can also be used for other kind of bugs like menus that do not work properly or mobs that behave strange. Such a kind of system would reduce the complexity of the bug report tool and you would not have to look through a big amount of possible options in a drop down menu. On the other hand this reporting system can also have bugs itself.
__________________
Albert Einstein: "Science without religion is lame, religion without science is blind." Scientia est Licentia - Knowledge is Freedom - Join our Science and Archaeology focussed command. Admin of the German Stargate Worlds Wiki |
|
|
|
|
|
#14 |
|
Join Date: Feb 2006
Location: Aus
|
Bugs - Forms.
Game issues - Forums.
__________________
|
|
|
|
|
|
#15 |
|
Moderator
Join Date: Nov 2006
Location: Dublin, Ireland Guild: Scientia Est Licentia
|
It's not quite a tool, but one thing that is very important is having lots and lots of communication and interaction. There's nothing quite as frustrating as spending time filling out the appropriate forms and sending them off only to hear absolutely nothing about the issue for weeks, leaving you wondering whether the forms are even read. By far the most common complaint I've seen in beta's is not 'x is broken', it's 'they aren't listening'. A bug database as mentioned would definitely help.
|
|
|
|
|
|
#16 |
|
Join Date: Oct 2007
|
If you have a character limit for the "description" field, then please have a secondary field with no character limit, that dev's and read/ignore at their leasure. That way when really bored people, such as myself, start testing the bug, to see what/where/if/how/why/whatever causes it, and what doesn't, we can list all the random info. Instead of running out of space to type, after two or three sentences.
Oh yeah, a nice part of the tool would be a "search" function, so that you could search through all the currently "queued" bugs, and click on box on that bug, which would "link" your bug report to another. Not sure how many people would use it, but even if a few did, it'd help immensly. |
|
|
|
|
|
#17 |
|
Moderator
Join Date: Oct 2006
Location: Australia
|
From someone who has never really beta tested anything before I guess I can bring a different perspective.
Ease of use should be the number one priority, both for us and you QA folks. We have to want to report bugs, it has to be that easy. It could be fun. We're all provided with an aerosol can item "Bug Spray" which we use to "report" the bug, for example. (except what happens when the bug is with the bug spray.......) Perhaps competitions could help. Separate and clear channels for reporting bugs, and more subjective issues. A way for us to know our feedback is being valued and utilised(and not a generic "You bug report is important to us" message). |
|
|
|
|
|
#18 | |
|
Join Date: Mar 2007
Location: Switzerland
|
Quote:
__________________
Albert Einstein: "Science without religion is lame, religion without science is blind." Scientia est Licentia - Knowledge is Freedom - Join our Science and Archaeology focussed command. Admin of the German Stargate Worlds Wiki |
|
|
|
|
|
|
#19 |
|
Join Date: Feb 2006
Location: I dont know. You Tell Me
|
Forums for Beta players. So they all can chime in on what they think is wrong with the game.
And in game bug reporting tool. So Beta players can report on whats broke. I think that is all thats needed.
__________________
|
|
|
|
|
|
#20 | |
|
Banned
Join Date: Feb 2006
Location: The Castle
|
Quote:
|
|
|
|
|
|
|
#21 |
|
Join Date: Jun 2007
Location: Newcastle, England.
|
hmm interesting....
![]() Well first of all thanks CME for asking us and allowing to trying a help out any which way we can, I know some of these might be implemented and some might not but at least the Developers are always looking which I find refreshing and also reassuring about the game we all hope to play when it is complete. here are a few suggestions : 1.) 3 different types of beta ; I) Technical - making sure things work as they say they do and making sure any hardware issues are addressed, also trying to do more of what happen if we change the resolutions from a to b,c,d,e,f,g,a,b,e,g,f,a?/ creating and deleting characters / trying to bypass the operational start up icons and starting from main file folder / trying to bypass the name filter, that kinda thing not game play more testing the actual systems side of it. (reporting would play a part in this beta so we can get the best things for the game we are going to play and pay for )II) Stress test Beta - this is your bog standard let 5k people about the SGW and let them create havoc hehe, resetting characters, locking people into one zone and seeing what happens if 5k people turn up and can the servers coup with it, the general things we do in beta really play the game report the bugs. III) Mission Beta - This would be select people from Technical & Stress test beta or even just select members from the forums or even people just signing up to that part of the beta. This beta would enable CME to ask set group of people to do set missions or tasks to see if the in game system works and how it maybe improved or doing bugged missions / testing fixed bugged missions or doing things that most people wont see until later in the game, also major reporting on this and this would mean people didn't report didn't continue in the beta. I like the input from Blkjak and others who have input so far with the Reporting forms, and also the standard idea of Forums passworded or protected and of course the NDA(Non disclosure agreement)<-[encase anyone didn't know] people would have to sign up for, but I would like to see people removed from beta for breaking this agreement as too many times we see people breaking it and no action taken. 2) A private poll for beta users/testers in which people would vote and that would be there say and that it, no reply function just options to vote on good/bad, yes/no/maybe, Green/red/Blue/Yellow, things like that so the Developers get instant feedback with out all the hassle of reading through 400 posts just to see one part that hits the spot on target. 3) Developer TS2/Ventrillio Discussion with select members who have technical issue with the game aka hardware/software to resolve any problems, Or even just a open place where we can go and chat to Developer about issue we have, set times would be allowed for specific area's. 4) Developer TS2/Ventrillio Q&A on problematic area's / questions where people would respond threw IRC or selected forum area, so the CME team could keep control of the discussion and also have the desired effect of communicating with its beta testers. 5) Developer TS2/Ventrillio Feedback on how we are doing as testers and what we need to do to help the CME Developer teams and not just a case of us running through the game and then stop playing when the game comes out. 6) Changing the cover sheet on the TPS reports hehe Office space rocks! hehe if you ain't seen it WHERE HAVE YOU BEEN???? UNDER A ROCK? and you know the TPS report sheet is over due for a change. Q&A guys are going to love me ![]() Just a few thoughts. FreejackeR -out.
__________________
|
|
|
|
|
|
#22 |
|
Join Date: Nov 2007
Location: Renton, Wa
|
I would like to see in ingame bug reporting tool, as you may forget half the bugs you intend to report if you have to wait till you log out to report them on the forums. However, I believe it is important to also have a Web Based bug reporting tool in case the in game bug reporting tool is broken/bugged. I've beta'd a game where they had an in game bug reporting tool that crashed the client!
Definatly keep the tool's interface simple, checkboxes for yes/no, dropdown catagory listings, and ALWAYS include an "other" option! Autofills for things like Race/Class/Level, Location, and the X number of lines from a system message log or something like it. Also, I feel a quest rating system is a good idea. I don't, however want to see it every time I create a new character on every single newb or tutorial quest. Perhaps a flag that tells the system when you've filled out the survey once on the account making it less likely to get the survey on a different character on the same account. Certainly remove the survey from quests you've gotten enough feedback on. For a beginning tutorial wait till the tutorial is completely over before prompting for feedback, don't go asking us for feedback at each stage! I agree that background gathering info for Chacter Race/Class/Level and Group composition could help deturmine if the quest was being completed by a group that was too high/low for the quest's intended difficulty.
__________________
Sam: "Didn't that tape say there were no fish in your pond?" Jack: "Close enough." |
|
|
|
|
|
#23 | |
|
Join Date: May 2006
Location: Westland, MI
|
Quote:
![]()
__________________
"Hail to the King, Baby!" Ash - Army of Darkness |
|
|
|
|
|
|
#24 |
|
Join Date: Feb 2006
Location: Somewhere in the USA
|
All I can say is make the reporting easy, for both you to read and sort (database for you). Also simple to fill out, complicated systems don’t work out to well. Check boxes work well this way you don’t have 5 million forms
1) Character name automatically & stats are automatic 1. Chose all that apply with issue: a. Quest () b. Interface () c. geometry () d. GUI issues () e. Exploit () f. World interaction () g. Other () 2. Description of Issue: In Detail (box) 3. Repeatable; Yes () No () 4. Location of issue: In Detail( box) 5. Journal of what happened: What were you doing in the game and out of the game?(computer stuff did it ) 6. Computer status a. Froze up b. Crashed c. No issue d. Slow e. Other In Detail( box) Oh also a Escape command!!! we can use if we get stuck in a wall or tree. I have gotten stuck before and had to wait hours till it was corrected. I would rather start over at a home site than get stuck in a wall for hours.
__________________
Minganblack Maybe next time, you'll just wait and see." "And blow my last chance of ever being right?" -- Jack and Jacob, Enemies Last edited by MinganBlack : 12-11-2007 at 12:05 PM. Reason: Addtion |
|
|
|
|
|
#25 |
|
Join Date: May 2007
Location: San Francisco Bay Area, CA
|
Adding more to this.. put a limit or threshold on the number of times a certain bug is reported by beta testers for the same problem along with dev flag and alert system that tell the beta testers that the dev is aware of the problem and the status of it inside the game and in the beta forum, The dev flags can be color coded, like blue for awareness, green for completed, yellow testing, red - being work on -
Example a certain NPC does not give you a quest - after a certain number of beta testers reported it of let say 200 out of 500 beta testers, there is no need for the reminding beta testers to keep reporting the same bug until after a fix been done and you want to test it out. You had enough of the population input on it where you can close it, and evaluate it accordingly. This way it will save both parties time, also, it will help you become aware of the more critical bugs faster, if you set critical threshold numbers based on number of times the bugs is being reported by the beta testers population, like if more than 35% of beta testers is reporting the same problem then the dev alert will let you know the frequency and allow you to prioritizes the critical bugs to work on. Now how durable to do a system or provide something like this I do not know.
__________________
Encourage Positive Behavior and GamePlay Dial the Gate, Let the Adventure Begin !!! Last edited by blkjag : 02-28-2008 at 02:41 PM. |
|
|
|
![]() |
| Thread Tools | |
| Display Modes | |
|
|