[Bldg-sim] Comparing ASHRAE 90.1 App G Models to Real Buildings (UNCLASSIFIED)

Justin Spencer jspencer17 at gmail.com
Wed May 27 14:34:59 PDT 2015


Models are never going to agree, but that doesn't mean they aren't powerful
tools for decision-making. Look at the ASHRAE 140 results for highly
simplified geometries with different engines. Things easily vary by 20%.
Hell, things vary by 20% just between some versions of the same tool. I
didn't see it mentioned that the underlying models just aren't going to
give you the "right" answer.

That's why we need to calibrate. The models are not generally actually
calculating things based on first principles. They use shortcuts for
computation time and ease of specification everywhere. That's why it's no
less real for me to impose some sort of somewhat goofy setpoint schedule to
get things to work out well in the calibration of my model. Ideally, we'd
calibrate to one set of bills and then run it against a second set of bills
to see if we over-calibrated the models (the same as over-fitting a
regression model). Instead of arguing whether or not the models are right,
we should be going back to the fundamental question of whether or not the
models provide useful decision-making information, i.e. whether or not they
get the first order impacts right for different kinds of design choices in
new construction. Remember that uncertainty around each point and move
ahead making the design decisions you need to make.

There's still plenty of room to develop better analysis practices that will
improve the design decisions we make. I would love to see a study looking
at normalized consumption and key characteristics of 5-year-old commercial
buildings, and then again at 10-year-old commercial buildings. You could do
this with LEED submittals and regress against the ECMs included and other
characteristics. You wouldn't necessarily get to the why of how the
buildings failed to deliver without a lot of extra work, but I think you
could get the what just by looking at bills and data. You could also do
this on the residential side. We have some utility clients who run
commercial or residential new construction programs. Theoretically, we
could get bills and characteristics for a large number buildings and
regress. I know that would work on the residential side. On the commercial
side, I don't know if there is enough participant data.  Others should
consider doing the same. It would be a fascinating study and would make for
a great energy nerd parlor game. Which of these measures just didn't
deliver?

My gut says that failures in fancy control systems are to blame for the
most egregious differences. There are so many failure points -- sensors,
valves, dampers, actuators, not to mention the likelihood of a mistake in
the controls sequence. You need somebody on top of monitoring for each of
those potential failure points and correcting them. My gut also says that
good solid basic design will deliver the savings. Cutting down on
west-facing solar gains will save cooling energy. Installing more efficient
HVAC equipment, given solid ratings information, will also work. More
efficient lighting designs will work. Really fancy stuff will fail some of
the time. And some of the time it will work! And we need to celebrate those
pieces that work!

On Wed, May 27, 2015 at 3:02 PM, Fred Betz <fbetz at aeieng.com> wrote:

> Just catching up with this thread so I'm sorry if I'm repeating anything
> that's already been mentioned.
>
> Take a look at the paper from Pam Berkeley et al published last year at
> SimBuild.
>
> https://www.ashrae.org/membership--conferences/conferences/ashrae-ibpsa-usa-papers
>
> 10 experienced energy modelers modeling the same building in a 3hr period.
> Fascinating results.
>
>
> There are emerging methods to do more rigorous QC using a variation on
> Monte Carlo for energy models to calculate a confidence interval for the
> model rather than fully rely on modeler experience and 3rd party QC.
> Georgia Tech has integrated this into their version of e+, which I hope
> gets integrated into a future version of e+. It's computationally intense
> so cloud computing is probably the right way to do this, which I believe e+
> is heading towards.
>
>
> Fred
>
>
> FRED BETZ  PhD., LEED AP ®BD+C
> SENIOR SUSTAINABLE
> DESIGN CONSULTANT
>
> AEI | AFFILIATED ENGINEERS, INC.
> 5802 Research Park Blvd. | Madison, WI  53719
>
> P: 608.236.1175 | F: 608.238.2614
> fbetz at aeieng.com  |  www.aeieng.com
>
>
> -----Original Message-----
> From: Eurek, John S NWO [mailto:John.S.Eurek at usace.army.mil]
> Sent: Wednesday, May 27, 2015 11:28 AM
> To: bldg-sim at lists.onebuilding.org
> Subject: Re: [Bldg-sim] Comparing ASHRAE 90.1 App G Models to Real
> Buildings (UNCLASSIFIED)
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
> This is a slightly different question:
> How close do you expect 2 energy models to be created by 2 different
> modelers (using the same program) if you give them the same plans and
> information?
>
> (Ask Pablo Picasso, Salvador Dali and Rembrandt to draw a tree)
>
> I assume most companies don't double up on the energy modeling efforts
> which would show how consistent or non-consistent energy models are.
> (assuming the energy modelers are experienced and competent.)
>
> Somebody who teachers energy modeling may be able to provide insight and
> good examples.
>
> As far as an energy model matching the actual utilities bills..... If you
> have a 1000 modelers, making models on 1000 computers for 1000 years......
>
>
>
> -----Original Message-----
> From: Bldg-sim [mailto:bldg-sim-bounces at lists.onebuilding.org] On Behalf
> Of Jacob Dunn
> Sent: Friday, May 22, 2015 2:37 PM
> To: bldg-sim at lists.onebuilding.org
> Subject: [EXTERNAL] Re: [Bldg-sim] Comparing ASHRAE 90.1 App G Models to
> Real Buildings
>
> Thank you all for your thoughtful insight on this matter!  It’s an
> important debate – both in understanding the capabilities/limitations of
> our craft as energy modelers, but also to communicate our value to the
> community at large.
>
>
>
> To clarify slightly, my question revolved around how the specific modeling
> protocol of Appendix G could account for the “performance” gap between
> modeled and actual use.  Thus, the fact that buildings aren’t operated as
> the energy model specified and the lack of building commissioning, while
> true and important, are not inherent to the intent of Appendix G modeling.
> The most interesting question is, “If you model a LEED App G model
> perfectly according to protocol, AND the building was operated according to
> the modeled schedules, will it predict the right number?”  Lots of your
> responses lent insight into this question, thanks again!
>
>
>
> I’ve revised my list below based on your responses:
>
>
>
> Added:
>
> -   Plug load values are assumed, which can have a huge impact on overall
> energy (Thanks Christoph and Chris Hadlock for the insight)
>
> -  Insulation values are largely specified without thought to thermal
> bridging
>
> -  HVAC controls simulation is often simplified
>
> - Performance curves are often not simulated due to increased effort and
> unavailability of performance data from manufacturers
>
>
>
> Original:
>
> -          Appendix G does not take into account external shading, which
> can be critical in urban environments for accurate energy predictions
>
> -          Schedules are typically not created with the intent of being
> predictive.  Overall building hours are adhered to, but detailed schedule
> creation is not usually in the scope of a LEED model (or is it, in your
> experience?).  For instance, typical plug load base values during
> unoccupied times are .3, this is a pretty big assumption.
>
> -          The App G model uses a TMY weather file, which can vary from
> the current weather year (I wonder on average by how much?)
>
> -          Infiltration values are assumed, unless blower door testing has
> been done (which is rare for commercial buildings).
>
> -          Thermostat values are modeled as consistent across the
> building, which is rarely the case in an actual operating building
>
> Cheers,
>
>
>
> Jacob Dunn LEED AP BD+C
>
>
>
> ESKEW+DUMEZ+RIPPLE, APC
>
> 2014 AIA National Architecture Firm Award
>
>
>
> 365 Canal Street Suite 3150
>
> New Orleans LA 70130
>
> 504.561.8686
>
> eskewdumezripple.com <http://www.eskewdumezripple.com/>
>
>
>
> From: Bldg-sim [mailto:bldg-sim-bounces at lists.onebuilding.org] On Behalf
> Of Chris Hadlock
> Sent: Friday, May 22, 2015 1:08 PM
> To: Christoph Reinhart
> Cc: bldg-sim at lists.onebuilding.org
> Subject: Re: [Bldg-sim] Comparing ASHRAE 90.1 App G Models to Real
> Buildings
>
>
>
> All,
>
>
>
> I would agree with all of the factors mentioned that absolutely can result
> in deviations between actual and modeled building performance. I would also
> echo the sentiment that following modeling rules shouldn't necessarily
> preclude us from attempting to better predict actual building performance
> through the LEED process. Applying careful attention to important details
> and a healthy dose of experience (bringing together real life building
> performance knowledge as it relates to the grey areas - namely schedules,
> equipment controls, occupant behavior, etc) can truly help close the gap.
> At the end of the day, a rating system should be attempting to reward
> buildings that actually perform well, not theoretically perform well (and
> as modeler's we should take a leading role in making good (i.e. fair)
> assumptions).
>
>
>
> My colleague (Janine Vanry) has recently completed research (to be
> published soon) for her masters thesis at the University of Waterloo
> (Ontario, Canada) which studied how LEED certified academic buildings in
> southwestern Ontario performed in comparison to government energy intensity
> benchmarks, campus-wide energy intensities, and in general how LEED
> (modeled) results compare to actual building performance (as measured
> through M&V). Consistent with Dr Samuelson's (et al.) research findings,
> the discrepancies between the modeled results and the actual energy
> intensities showed that there was an under-prediction anywhere from 2% to
> 44%.
>
>
>
> While energy modeling professionals understand (as is evident by this
> thread) that there will be differences between the documented EAc1 energy
> savings and actual building energy usage, this isn't always communicated
> and understood by the building owners and the professionals we work with.
>
>
>
> Chris
>
>
>
> On Thu, May 21, 2015 at 10:56 AM, Christoph Reinhart <tito_ at mit.edu>
> wrote:
>
>         Dear Jacob,
>
>
>
>         This is an eternal debate and there are many reasons for moving
> away from the use of 90.1 Appendix G to evaluate the performance of a
> building designs.  To answer your question directly, we worked a few years
> ago with Enermodal in Canada on a comparison between design phase building
> energy models (BEM) prepared for LEED Canada certification (slightly
> different to Appendix G) to calibrated BEM and measured energy use for 18
> buildings. The main findings are quoted below:
>
>
>
>         Analysis of a Simplified Calibration Procedure for 18 Design-Phase
> Building Energy Models
>
>         H W Samuelson, A Ghorayshi and C F Reinhart
>
>         Journal of Building Performance Simulation, DOI:
> http://dx.doi.org/10.1080/19401493.2014.988752 <
> http://dx.doi.org/10.1080/19401493.2014.988752>
>
>         This paper evaluates the accuracy of 18 design-phase building
> energy models, built according to LEED Canada protocol, and investigates
> the effectiveness of model calibration steps to improve simulation
> predictions with respect to measured energy data. These calibration steps,
> applied in professional practice, included inputting actual weather data,
> adding unregulated loads, revising plug loads (often with submetered data),
> and other simple updates. In sum, the design-phase energy models
> underpredicted the total measured energy consumption by 36%. Following the
> calibration steps, this error was reduced to a net 7% underprediction. For
> the monthly energy use intensity (EUI), the coefficient of variation of the
> root mean square error improved from 45% to 24%. Revising plug loads made
> the largest impact in these cases. This step increased the EUI by 15%
> median (32% mean) in the models. This impact far exceeded that of
> calibrating the weather data, even in a sensitivity test using extreme
> weather years.
>
>
>
>         Best,
>
>
>
>         Christoph
>
>         Christoph Reinhart
>
>         Associate Professor
>
>         Department of Architecture
>
>         Massachusetts Institute of Technology
>
>         77 Massachusetts Ave, Rm 5-418, Cambridge, MA 02139, USA
>
>         t: 617 253 7714 <tel:617%20253%207714> , f: 617 253 6152
> <tel:617%20253%206152> , creinhart at mit.edu <mailto:creinhart at mit.edu>
>
>         Sustainable Design Lab <http://mit.edu/SustainableDesignLab/>  |
> DIVA <http://www.diva4rhino.com/>  | Daysim <http://daysim.ning.com/>  |
> mapdwell <http://www.mapdwell.com/>  | umi <http://www.urbanmodeling.net/>
>
>
> -------------------------------------------------------------------------
>
>         Events  Modeling Urban Sustainability <
> http://architecture.mit.edu/event/modeling-urban-sustainability-energy-daylight-and-walkability>
> | DIVA Day 2015 <http://diva4rhino.com/diva-day-2015>
>
>
>
>
>
>
>
>         From: Bldg-sim [mailto:bldg-sim-bounces at lists.onebuilding.org] On
> Behalf Of Brooks, Alamelu
>         Sent: Thursday, May 21, 2015 10:43 AM
>         To: Jim Dirkes; Nathan Kegel
>         Cc: bldg-sim at lists.onebuilding.org
>         Subject: Re: [Bldg-sim] Comparing ASHRAE 90.1 App G Models to Real
> Buildings
>
>
>
>         I believe Appendix G is not meant to measure the performance of
> the existing building. ASHRAE 90.1 Appendix G Technical Committee is the
> right source to answer this question. They can clarify the intention of the
> APP G modeling methodology.
>
>
>
>         Best,
>
>         Alamelu
>
>         Alamelu  Brooks LEED AP (BD+C), HBDP, BEAP, EIT| Senior Associate
> | +1.443.718.4881 <tel:%2B1.443.718.4881>  direct |
> Alamelu.Brooks at icfi.com <mailto:Alamelu.Brooks at icfi.com>  | icfi.com
>
>         ICF INTERNATIONAL | 7125 Thomas Edison Drive, Suite 100, Columbia,
> MD 21046 USA
>
>         Connect with us on social media <http://www.icfi.com/social> .
>
>
>
>
>
>         From: Bldg-sim [mailto:bldg-sim-bounces at lists.onebuilding.org] On
> Behalf Of Jim Dirkes
>         Sent: Thursday, May 21, 2015 10:36 AM
>         To: Nathan Kegel
>         Cc: bldg-sim at lists.onebuilding.org
>         Subject: Re: [Bldg-sim] Comparing ASHRAE 90.1 App G Models to Real
> Buildings
>
>
>
>         I agree fully with all of the above comments and would like to add
> these:
>
>         *       Even buildings that are commissioned properly will see
> their performance erode over time.  There are hundreds of reason for this,
> ranging from poor maintenance to well-intentioned maintenance people not
> having time to monitor operations well.  There is NO BUILDING that operates
> well for long.
>         *       Buildings often see changes in operation, occupancy and
> schedule.  These are oftimes gradual changes over a period of years, but
> can be substantial
>
>
>
>         On Thu, May 21, 2015 at 10:22 AM, Nathan Kegel <
> nathan.kegel at iesve.com> wrote:
>
>         Climate files used in the simulations versus the actual weather.
>
>
>
>         I’m in the midst of a project that shows a variance in EUI of up
> to 200% just by changing the climate file for the DOE primary school.  Full
> results to be presented in September.
>
>
>
>         Add in all the other factors already mentioned, and if your 90.1
> model comes anywhere close the real buildings’ it’s far more likely that
> the 90.1 model was extremely “lucky” than it is that the model used
> accurate assumptions.
>
>
>
>         Regards,
>
>
>
>         Nathan
>
>
>
>  <http://www.iesve.com/>
>
> Nathan Kegel
> Business Development Manager
>
> O:
>
>   763.276.9981
>
> M:
>
>   415.420.9314
>
> http://www.iesve.com <http://www.iesve.com/>
>
> Integrated Environmental Solutions Limited. Registered in Scotland No.
> SC151456 Registered Office - Helix Building, West Of Scotland Science Park,
> Glasgow G20 0SP
>
> Email Disclaimer <http://www.iesve.com/disclaimer.html>
>
>
>
>
>
>         From: Bldg-sim [mailto:bldg-sim-bounces at lists.onebuilding.org] On
> Behalf Of Maria-Lida Kou
>         Sent: Thursday, May 21, 2015 9:17 AM
>         To: Jacob Dunn
>         Cc: bldg-sim at lists.onebuilding.org
>         Subject: Re: [Bldg-sim] Comparing ASHRAE 90.1 App G Models to Real
> Buildings
>
>
>
>         Jacob,
>
>
>
>         Happy to hear that other people are thinking the same.
>
>
>
>         I was into this subject on my own thoughts recently.
>
>
>
>         I would like to add in your list: Occupants' behavior actually
> which is not in the stage to be included into the prediction.
>
>         I would add commissioning as well along with controls simulation
> and controls operation.
>
>
>
>         Apologies because I haven't worked with LEED projects but I think
> the above applied in general to "the performance gap".
>
>
>
>         Really looking forward to hearing more about this subject as I am
> not that experienced engineer yet, but really interested in "the
> performance" side of buildings.
>
>
>
>         Best,
>
>         Maria-Lida Kounadi
>
>
>
>
>
>         2015-05-21 15:04 GMT+01:00 Jacob Dunn <jdunn at eskewdumezripple.com
> >:
>
>                 Bldg-Sim Community –
>
>
>
>                 I’m trying to compile a list of why it might be
> inappropriate to compare Appendix G models to actual consumption data.
> This comes about because I recently got into a debate with one of my
> co-workers when looking at the infamous NBI chart/study that shows little
> correlation to predicted and actual energy values of LEED buildings.  I was
> trying to explain that the Appendix G model’s intent is NOT to be compared
> to actual consumption, as it is a modeling protocol aimed at creating
> consistent relative comparisons for LEED points.
>
>
>
>                 Here are the reasons thus far that support this notion
> (that App G models shouldn’t be compared to actual data).  Does anyone know
> of any resources out there that expand upon this?  Or can you think other
> reasons?
>
>
>
>                 -          Appendix G does not take into account external
> shading, which can be critical in urban environments for accurate energy
> predictions
>
>                 -          Schedules are typically not created with the
> intent of being predictive.  Overall building hours are adhered to, but
> detailed schedule creation is not usually in the scope of a LEED model (or
> is it, in your experience?).  For instance, typical plug load base values
> during unoccupied times are .3, this is a pretty big assumption.
>
>                 -          The App G model uses a TMY weather file, which
> can vary from the current weather year (I wonder on average by how much?)
>
>                 -          Infiltration values are assumed, unless blower
> door testing has been done (which is rare for commercial buildings).
>
>                 -          Thermostat values are modeled as consistent
> across the building, which is rarely the case in an actual operating
> building
>
>
>
>                 Any additional insight is much appreciated!
>
>
>
>
>
>                 Jacob Dunn LEED AP BD+C
>
>                 ESKEW+DUMEZ+RIPPLE, APC
>
>                 2014 AIA National Architecture Firm Award
>
>
>
>                 365 Canal Street Suite 3150
>
>                 New Orleans LA 70130
>
>                 504.561.8686
>
>                 eskewdumezripple.com <http://www.eskewdumezripple.com/>
>
>
>
>
>                 _______________________________________________
>                 Bldg-sim mailing list
>
> http://lists.onebuilding.org/listinfo.cgi/bldg-sim-onebuilding.org
>                 To unsubscribe from this mailing list send  a blank
> message to BLDG-SIM-UNSUBSCRIBE at ONEBUILDING.ORG <mailto:
> BLDG-SIM-UNSUBSCRIBE at ONEBUILDING.ORG>
>
>
>
>
>         _______________________________________________
>         Bldg-sim mailing list
>         http://lists.onebuilding.org/listinfo.cgi/bldg-sim-onebuilding.org
>         To unsubscribe from this mailing list send  a blank message to
> BLDG-SIM-UNSUBSCRIBE at ONEBUILDING.ORG <mailto:
> BLDG-SIM-UNSUBSCRIBE at ONEBUILDING.ORG>
>
>
>
>
>
>
>
>         --
>
>         James V Dirkes II, PE, BEMP, LEED AP
>         CEO/President
>         The Building Performance Team Inc.
>         1631 Acacia Dr, GR, Mi 49504
>
>         Direct: 616.450.8653
>         jim at buildingperformanceteam.com
>
>         Website <http://buildingperformanceteamcom> l  LinkedIn <
> https://www.linkedin.com/pub/jim-dirkes/7/444/413>
>
>         The truth is still the truth, even if nobody believes it.  A lie
> is still a lie, even if everyone believes it.
>
>
>         _______________________________________________
>         Bldg-sim mailing list
>         http://lists.onebuilding.org/listinfo.cgi/bldg-sim-onebuilding.org
>         To unsubscribe from this mailing list send  a blank message to
> BLDG-SIM-UNSUBSCRIBE at ONEBUILDING.ORG <mailto:
> BLDG-SIM-UNSUBSCRIBE at ONEBUILDING.ORG>
>
>
>
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
>
> _______________________________________________
> Bldg-sim mailing list
> http://lists.onebuilding.org/listinfo.cgi/bldg-sim-onebuilding.org
> To unsubscribe from this mailing list send  a blank message to
> BLDG-SIM-UNSUBSCRIBE at ONEBUILDING.ORG
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.onebuilding.org/pipermail/bldg-sim-onebuilding.org/attachments/20150527/757ff6fa/attachment.htm>


More information about the Bldg-sim mailing list