Why not?

Topics: Rawr.Hunter
Nov 8, 2009 at 9:36 AM
Edited Nov 8, 2009 at 9:40 AM

Wouldn't there be something to win by actually keeping two separate models of the Hunter module. One "Hunters Spreadsheet" and one HunterSE?

By making a model that exactly copies the results of the spreadsheet would,

a) ...be fairly simple to maintain since its "just" a translation from ".xls" and VB to C# and all changes is delivered on a silver plate in the release notes of the spreadsheet.
b) ...provide an easy way to get some rough validation of the HunterSE model (as we were doing last night). There is quite a lot of ppl that scrutinize the spreadsheet and helps out getting the formulas correct in the EJ community.
c) ...provide a module that you can pinpoint "errors" down to the last digit. When I was looking for "errors" (i.e. errors being non spreadsheet sameness) i was in a very rare situation as a programmer. I had the "correct" answer that I knew the program should generate.
d) ...actually benefit the EJ work as well, I found 2 bugs of the spreadsheet implementation just by working with the Rawr code. (one which iamcal already had found and implemented in Rawr but was lacking in the spreadsheet code.)
e) ...allow ppl that are "followers" of the spreadsheet to get benefit from the Rawr interface as well.

Of course there is no "truth" out there since each fight is unique and even each try at the same boss is different due to your performance varying as well as your team members performance varies. I'm very interested in the Boss module that I understand is getting developed in the core code and I'm all for getting a Hunter module that by being a more "pure" Rawr model, could benefit from any central changes and enhancements may it be constants changing to boss modules to graphing.

However, you have in the spreadsheet an unique opportunity to validate parts of your module. I would guess that quite a few that uses the spreadsheet and Rawr are Theorycrafters to whom a difference of plus minus 10 DPS is regarded a significant "error" not to mention if the model were off by several %.

When using any type of support for helping you choose gear you need to have faith in the model you choose. Otherwise it does not help you and would have you second guess every choice you make. I have found that in my "loot journey" that my upgrades to a slot often is a matter of getting maybe 40, or if I'm lucky a 100 better theoretical dps out. And that, assuming a max DPS output of 8000, translates to a difference between a ½ to 1 %. To then say, "Bah, +- a ½ percent is less than the error margin of the model so I say no to roll on the just dropped "death's verdict" trinket.", would not really help you.

Even in Rawr you state that "Don't use stats weights to evaluate gear, it is highly in-accurate." Well, how do you validate the inner models of your modules to know that the inaccuracy in Rawr is less than the inaccuracy you would end up with by using rule of thumb stats weights?

In my view, having 2 models could actually help that challenge.

Coordinator
Nov 8, 2009 at 10:05 AM

Nope. It'd be a huge detriment to our users to have both.

Nov 8, 2009 at 10:15 AM

ok, nvm the users. How does the developers validate the models then?

Developer
Nov 8, 2009 at 12:27 PM
Edited Nov 8, 2009 at 12:32 PM

With Enhance it's easy as there exists an EnhSim - Enhancement Simulator which actually does a full scale simulation over thousands of hours of combat. Rawr is dramatically quicker and easier to use and I strive to have both producing similar results. However a simulator is naturally going to end up more accurate than a closed form solution where we are averaging everything. It does give me an excellent benchmark with which to compare the results however.

The design I've gone for is Rawr should be great for 95% of users and more than accurate enough for their needs, for the top theorycrafters min/maxers there is an export to EnhSim option so that they can use Rawr to generate the EnhSim config file, and run the sim to verify the results Rawr gives. I do this a LOT to compare and contrast the figures in both models. I can often be found having two debug sessions going one in Rawr.Enhance and one in EnhSim stepping through the code checking values of things at different stages to compare the two.

By using this method Rawr.Enhance has become dramatically more accurate lately and is very very similar to EnhSim now. It still favours hit a bit too much though compared with EnhSim output. My next major task after I've completed the Rawr3 options panel is to do various tests on the effects of hit above the spell hit cap and see where the variances are coming from.

I would concur that being able to scrutinise both models allows you to find bugs in either one. However I'd agree with Astrylian having two hunter models would be overly confusing for the vast bulk of users. Perhaps the hunter model can go down the route of Enhance and have an export option that outputs the necessary data to make the Hunter Spreadsheet work. With a spreadsheet it would be relatively simple to export a standard csv file and have a spreadsheet macro that imported the csv and copied the data from the csv to the relevant cells in the spreadsheet.

That way you could do the same rapid testing and development I'm used to in Enhance, and benefit from the insights of both communities.

Nov 8, 2009 at 1:44 PM

I guess that EnhSim is similar to simulationcraft in its approach. And I would say having those kind of validation possibilities is quite useful for model evaluation.

Actually, hehe, the more hours simulated in EnhSim the closer it should get to your "averaging"... accoding to good o'l statistics :)

I.e. averaging is not "THE EVIL"... it is what you have to do in order to get an answer out that does not fluctuate with each run... beacuse then you would have to to introduce use measures of variance to describe the resulting numbers... and that would probably get us unto the upper league if we wanna play the game of "confuse-an-user".

Regarding Spreadsheet export,well, not sure that it would benefit that much apart from as you say as quick validation for Rawr dev.

To start out from a blank Spreadsheet to getting figures out is just a matter of pressing the "Load from Armory" button and then set the rotation and buffs... 

Hunters are fairly "simple" creatures atm, we have more of a priority queue than a Rotation we follow since we have rather few "procs" apart from gear/trinket ones. Sure, the Improved steadyshot and Lock and Load talents adds some but not that much.

User friendliness/user understandability is a balancing act and always will be. But I must say I don't quite understand the choice of words in "huge detriment". Have you looked at the option tab for mages lately? 105 or so different places where I can tick boxes or enter values. (Not including the "Advanced" tab adding 20 more...). But I can't argue with the fact that there is only one choice under the Stats tab "Model:" pulldown.

I get a feeling that its the "Spreadsheet" word that tend to taint things... Things with "Sim" in the name = good. Things with "Spreadsheet" in the name = bad. I don't see Rawr as a simple spreadsheet when I make the comparisons... and never will... you have far to few cells where I can write formulas in for that, and your save files does not end with .xls ;). Nor do I see the current "Spreadsheet" at EJ as "simple" any more. Far too much VB and simulation going on behind the cells. I do however, see in Rawr an excellent potential to do a lot more especially when doing more of the synergy approach "good-things-in-one-module-get's-moved-to-rawr-base".

However, if the aim is to help users "...comparing and exploring gear..." and do a good job at that, to get what Levva talks about, a "EnhSim" equivalent for hunter model validation would be nice.

 

Developer
Nov 8, 2009 at 5:16 PM

The point becomes, if we want to validate against spreadsheet (or other) outputs, we can use the spreadsheet (or other) instead of trying to translate spreadsheet to our code and then compare the 2 modules. Huge step in one path, much smaller step in the other.

Developer
Nov 8, 2009 at 6:05 PM

Its also a lot simpler for a model to have an export option in a format suitable for use in spreadsheet so that anyone can use it, than it is to have an option to convert a spreadsheet to VB code then to C# code something that would utterly defeat 99% of the Rawr userbase and not something we'd want to be supporting.

 

Nov 8, 2009 at 6:21 PM

Ah, well, I accept the point of it being easier to maintain one model.

(... though you almost had 2, Iamcal had already translated the VB code of the spreadsheet model, the thing missing was updates since Iamcal went MIA.)

Anyway, I think It is probably one reason to why I have so difficult with letting the "old code" go is that there I had a possibility to contribute with small patches since it was more or less only a 1 to 1 row conversion and thereby easy to verify/compare. :)

 

 

Developer
Nov 8, 2009 at 6:29 PM

*insert a music video pertaining to "Let it Go" here*