Testing. It’s not Hard

There’s a point where this becomes petty/funny/sad.

zr5a1q1bleo11

The item on the left is a normal drop.  The item on the right is a Warforged version, which should be an upgrade but is in fact worse that a Normal BfA dungeon drop.

 

jnyv6wc6meo11

The items for sale on the Timewalking Vendor have not had their ilvl updated, but do need a 120 player.

These are item specific – not related to how the stat squish has impacted the overall balance.  End Times can’t seem to be completed right now, for example, as Sylvanas will 1 shot most tanks.

All of this was corrected in a hotfix on the 25th.

Timewalking is something that comes up every few months.  It opens old dungeons from a previous expansion, and applies scaling so that max level players can go back into them.  Completing these dungeons awards drops as well as tokens.  Those tokens can be exchanged for items that were previously on-part with heroic dungeons, or cosmetic gear, or heirlooms, or mounts.

It’s a neat side activity. This is the way most dungeons work in FF14, so it’s not like the tech is new or hard to do.  Blizz has done this a few times in the past.  An entirely different topic as to why this system is always available…

Bug Hunting

IT has bugs.  That’s life.  Some are obvious, some are not.  Some are easy to fix, some are not.  Fixing them is one conversation that I don’t think I need to get into.  This is about finding them.

The 3 examples put above would have been obvious from a single dungeon run, through every Timewalking dungeon.  Guaranteed, 100% reproduction.  That these were pushed into production means that either a) it was not tested or b) it was tested and the project manager accepted the test results.

If this happened in my team, this would merit disciplinary action.  After 3 times, they would either be dismissed or re-assigned to other work.

Reality

It’s a game.  It’s made to be relaxing for some, challenging for others, fair for all.  It’s just symptomatic of BfA in general.  Which is sad – and not what we should expect from Blizzard.  I should be focusing on the fun items, not these brain fart items that keep popping up.  I’m missing the polish.

And I can only fathom how Blizzard employees are feeling about this.  These items deflate overall confidence… and it takes a long time to build that back up.  It also causes finger pointing, and niche work.  No good ever comes from that.  Again, in most industry the lead of these teams gets replaced so that it at least appears something is being done to rectify the situation.

5 thoughts on “Testing. It’s not Hard

  1. Testing, it is very hard. Here is my experience with QA life over the last 25 years.

    There is a change planned, QA is asked how long it will take to test and everybody pushes back hard on the answer. The assumption is always, always, always that it can’t possibly be that hard to test something, just try it and give us the green light.

    So QA compromises. They have to. You cannot test everything on every build. So they reduce the scope of the testing, the ask dev about the details of the change and exactly what areas will be impacted. That allows them to focus. The build comes, they do a smoke test on most of the product to ensure basic functionality, then focus their effort on the areas that were changed. With a product as big as WoW you migh have a QA team for specific areas even.

    Then it goes live and it turns out dev was wrong, they actually forgot to mention that for the fix they actually had to touch something outside of the scope they previously reported and something broke. Meanwhile some end user goes off into a corner of the product that wasn’t supposed to change, finds a problem, and declares that QA is dumb and should be fired because look how easy it was to reproduce this issue.

    And everybody points at QA and tells them they suck and asks how hard could this be. Then, in the very next breath, they complain again about how testing takes too long and QA should work smarter, not harder and the cycle continues.

    Liked by 3 people

    • I agree with every statement about QA that you have listed here. I’ve spent the 10 past years juggling the need to meet a timeline vs the amount of testing.

      I always press on my team that UAT be the largest phase, so that select clients get to find the things that we missed. They are exceedingly good at finding weird functions that we never would have found for lack of time/resources.

      In 99% of cases, these are obscure failures that would not have been found without the additional help. Some odd dependency that only the devs ever knew about… if that. (I recall one bug that could not be replicated… until we visited the user and saw their mousepad on the the keypad…)

      Testing is hard, when you’re focusing on the obscure and flukes. That part, I have more than enough patience on. A spelling mistake, or some odd combination of things that causes X to occur (e.g. Rextroy). The 8.0 stat squish bugs are a good example too… no way to test every little bit for all content.

      My specific issue here is that these are not obscure bugs. They are immediately reproduce able, and occur 100% of the time. They were found in production within an hour of the patch being applied. It is astounding that a multi-billion dollar company has a QA process that allow something like this through. I am more than confident that the people doing QA want the best… but something in the overall process is clearly broken.

      Liked by 1 person

  2. It is abundantly clear now that Blizzard rushed Beta for Azeroth. All of Legion was on a fast clock, as they finally committed to not having a year-long content drought, after a decade of saying they’ll release expansions faster. BfA is the result.

    …except it’s even worse than that. The Timewalking bit is embarrassing, because it demonstrates that nobody bothered even running a dungeon once, or checking the vendor. But Ion has come out several times already specifically stating that they don’t care about data gathered through beta testing anyway – he wants Live data from millions of players before e.g. deciding that Azerite traits are broken, or whatever. It’s incredibly sloppy, and I can’t imagine what executive thought it’d be fine to screw around with Production like that.

    I dunno. I guess we’ll see how everything shakes out in the Grand Scheme© of things.

    Liked by 2 people

    • I can sort of understand the balancing question when it comes to live data. At least a little bit. Seeing hotfixes that say something like “rogues do 5% more damage”… that’s not small tweaks. That’s a large ax swing at a problem.

      There was a year between 7.3 and BfA launch. There was a year + 2 months between 6.2 and Legion. 2 months may not seem like long… but it’s an eternity while testing. We’re not even 2 months since BfA launch!

      Liked by 1 person

Leave a comment