Q&A Session for V&V: Quantifying Prediction Uncertainty and Demonstrating Simulation Credibility _________________________________________________________________ Q: Could Dr.

Hemez recommend a text on this topic? To the best of my knowledge, there is no textbook that offers a comprehensive look at V&V in computational physics and engineering. There is no technical journal that deals specifically with V&V. Patrick Roaches book entitled Verification and Validation in Computational Science and Engineering (Hermosa Publishers, 1998) is the closest that one can find, even though it deals almost exclusively with code and calculation verification and does not address model validation, calibration, sensitivity analysis or uncertainty quantification. High-level overviews of V&V have been published as guides by the AIAA (AIAA-G-077-1998) and ASME (V&V in Computational Solid Mechanics). These documents are useful to understand what the various elements of V&V are; they do not, however, provide guidance on how to implement and manage a specific V&V study. Books that I found useful to learn statistical methods for sensitivity analysis and the quantification of uncertainty include: 1) Myers, R.H., Montgomery, D.C., Response Surface Methodology: Process and Product Optimization Using Designed Experiments, Wiley Inter-science, New York, 1995; 2) Saltelli, A., Chan, K., Scott, M., Editors, Sensitivity Analysis, John Wiley & Sons, 2000. There are many manuscripts published yearly at conferences and workshops in the U.S. and Europe that address technical aspects of V&V and its application to specific studies. For example: 1) AIAA Non-deterministic Applications (NDA) Conference; 2) NAFEMS meetings; 3) SEM International Modal Analysis Conference (IMAC); 4) University of Leuven, Belgium International Symposium on Modal Analysis (ISMA). This is to cite only a few. Since there is no single, centralized source of information (such as a textbook), I find that an efficient and cost-effective way to organize and start prioritizing all this information is to attend a short-course on V&V. The only two such short-courses that I know of are those offered by F. Hemez (usually through Los Alamos Dynamics or NAFEMS) or B. Oberkampf and C. Roy (usually through AIAA). _________________________________________________________________ Q: Because V&V can be a huge timely task and most projects are time sensitive, what V&V is worth doing? Any level of V&V that offers the potential of finding an algorithmic deficiency in the codes used, improving prediction accuracy, reducing prediction uncertainty, or improving the modeling rules is worth doing. Time, resources and budget constraints must, however, be taken into account. This is why there is no clear-cut answer to this question: it depends on the environment that one is in, the resources available (in terms of numerical modeling but also physical experimentation) and the needs of customers. An ideal situation would be to commit to V&V and define a multiyear plan that would help improve the predictive capability over the long term, with the understanding that not everything can be achieved at once. One could, for example, put the priority on code verification during the first year, as well as assess the performance of Computer Assisted Design (CAD) and meshing tools. The second year could be devoted to experimental protocols: What types of tests are needed? How many replicates? Etc. The starting point can be high-level meetings with all stakeholders to understand the needs of customers, types of decisions that must typically be made, and how this environment translates into requirements that the predictive capability must, eventually, be able to meet. A tool that I have found to be useful in starting establishing priorities is the PIRT (Phenomenon Identification and Ranking Table). A PIRT analysis of a modeling and predictive capability can provide insight into individual phenomena that are unknown/uncontrollable and, at the same time, can greatly

influence the predictions that one needs to make. For example, we think that energy dissipation due to this particular assembly procedure between two components is both unknown and it can influence the performance of the system to a great extent. If one encounters such combination of lack-of-knowledge and sensitivity (or lack of robustness), then this is where to focus the V&V effort because improving the predictive capability for this particular phenomenon offers the greatest potential in terms of reducing prediction uncertainty. _________________________________________________________________ Q: Please elaborate more on the selection of simple surrogate models for use in V&V. How to make sure they are relevant to the real problem? What I meant by thisand I apologize if this was not made clear during the webinaris that it is sometimes not possible to propagate uncertainty or perform parameter calibration, for example, when the physics-based numerical simulation is expensive to run. A finite element model that takes, for example, two hours to complete can be analyzed a few times a day with no problem. Wrapping, on the other hand, a sampling or optimization procedure around it can rapidly become unrealistic or impractical. Assume that, say, 1,000 runs are needed and it means nearly 3 months of run time! Nobody is going to wait three months for an answer that may not even be a good-quality answer. (A thousand runs may not be nearly enough if you are sampling, say, 10 dimensions or calibrating 10 parameters at once.) It is my experience that it is much more efficient to substitute, at this point, a fast-running surrogate to the physics-based simulation (such as a finite element model). The surrogate model simply builds a relationship between multiple inputs and a single (or multiple) outputs. It can be evaluated in a fraction of a second. Examples are polynomial functions, Component Mode Synthesis (CMS) models, Krigging models, neural networks, support vector machines, etc. Sure, they are approximate and do not include all the physical details of the predictive model but the level of approximation introduced can be controlled. They are only used to map the relationship between inputs and outputs, which is all that one cares about when sampling or optimizing parameters. I was not talking about substituting a surrogate problem to the real problem (reality) of interest to the V&V study. _________________________________________________________________ Q: Slides pertained to uncertainties in simulation, but V&V also critically depends on all stakeholders understanding the variability of test data (for 1 unit and across multiple units of the same design). Any comment? I could not agree more. Understanding the level of uncertainty/variability in the test data, and understanding what may control this variability, is critical to V&V. It should be the starting point of any test-analysis correlation effort because it makes no sense to compare a single prediction to a single measurement if repeating the physical test is likely to change the value of this measurement. Test-analysis comparison makes sense only in the context of the experimental variability. For example, which one of these two statements, (A) or (B), is more credible? (A) The simulation predicts a peak acceleration of 8.2 gs and the measurement is 8.7 gs. (B) The model predicts 8.2 gs and the measurement is 8.7 gs, on average, with a two-sigma variability of 0.4 gs. The test-analysis correlation pictured by statement (A) may look good because the prediction error is only a g. But statement (B) depicts an entirely different picture since the simulation falls outside the +/- 2-sigma bounds of measurements. Of course, there may be contexts where all that is available is a single experiment because a one-of-a-kind system, such as a satellite, is manufactured. But if there is any reason to believe that multiple tests (even on the same structure) will result in some level of variability, then such variability should be estimated and used as the baseline for test-analysis correlation, model updating, propagation of modeling uncertainty, etc.

_________________________________________________________________ Q: What if the simulation you pursue can not be verified through empirical data? Does this not force the dependence on the simulation to be greater? I am grouping the above two questions to answer them because they proceed from what, I think, is a similar logic. The logic is, if I understand correctly, that testing is impractical or too expensive, therefore pushing for the need to rely on modeling and simulation instead. But then, how can one V&V the predictions if test datasets are not available? My answer is that there cannot be any model validation without access to test data. However, a validation test is different from a qualification or certification test. Yes, it may be advantageous to replace expensive or dangerous experiments by a numerical simulation. No, simulations will never replace testing because model validation, if anything, requires a lot of testing. But what I am talking about is a different kind of testing, with more emphasis on replicate experiments and separate-effect tests. One may not be able to perform a system-level test but in the V& logic, this large-scale and expensive test can be broken down into multiple, small-scale tests. Some of these small-scale tests can focus on learning the behavior of materials (to decide which material model, which parameterization is more appropriate). Other small-scale tests can assess the severity of a non-linearity (to guide the development of modeling rules). Yet other small-scale tests can investigate the coupling between a thermal environment and the material response, for example. Etc. So I would claim that, at the end of the day, one has to perform even more testing to validate a predictive capability; but it is a different kind of testing. The small-scale or separate effect test should focus on one piece of mechanics or physics, or one coupling phenomenon, at a time. They should be designed to feed back what is learned directly to the modeling rules that will be used for simulating the large-scale, system level test (that can no longer be performed experimentally). So you will then ask: well, if one large-scale test is now broken down into multiple, small-scale tests, then where do we start? One could start by writing a PIRT (Phenomenon Identification and Ranking Table) to identify as many of these separate effect aspects of the simulation as possible. Implementing these phenomena, for example, a material model, in a numerical simulation will require information such as the values of material coefficients. A small-scale test can then be designed to learn this information and feed it to the simulation. Therefore I would recommend to start by going after those small-scale tests that offer the most bang for the buck, that have the greatest potential in terms of providing the information needed for the simulation. _________________________________________________________________ Q: How does one go about developing a V&V rating scheme for FEA modeling, analysis, etc.? I am not sure what is meant by rating scheme and will assume that it would be a score that rates, say, on a scale of 0-to-4 or color from red to green, the usefulness or numerical quality of the modeling and analysis tool. This goes back to the concept of predictive maturity scale that I briefly overviewed during the presentation. One would start by defining attributes that are important to the numerical simulation. Surely this list would include the representation of the geometry, initial and boundary conditions, etc. It would also include the physics-based models that the simulation requires and discretization on a computational grid or mesh (if there is such a thing). There may be many other attributes depending on the type of numerical simulation. Then one would need to decide what kind of V&V effort reaches a specific score for a given attribute. For example, performing mesh refinement (from which the bounds of solution uncertainty are estimated) would provide a score of 3 for the attribute verification. Likewise for the attribute sensitivity and uncertainty quantification, performing a local (finite difference-based) sensitivity calculation may be enough to provide a score of 2 but it would not be enough to reach a score

of 3, for which propagating a design of computer experiments and performing a statistical screening of important effects may be required. With such machinery in place, one could then score various attributes of a numerical simulation depending on the level of V&V implemented to support the simulation. Over time one would expect an overall improvement of these ratings, as more verification activities and validation tests become available. Note that there is, to the best of my knowledge, no universally agreed-upon standard for such rating scheme; what makes sense for one organization may be different for another. What is important is to define a scheme that makes sense for a given environment or application. And, even if it gets started as a modest and incomplete effort, developing such maturity model offers the benefit to pressure the analysts to address V&V and put down what they are doing about it (rather that eluding the discussion altogether with an answer along the lines of trust me, I know what I am doing, as it is too often the case). _________________________________________________________________ Q: According to your work, which kind simulation type is the more accuracy to the real world? The Stress Analysis, CFD, Vibration/PSD Analysis, EMC/EMI Analysis? I am sorry that I cannot offer an answer to this question. All numerical methods are based on assumptions and approximations. What is important, what V&V is all about, is to understand when these assumptions are justified and when they may start to break down. This depends on the numerical method itself, but also the question asked. The Navier Stokes equations of Computational Fluid Dynamics (CFD), for example, are generally thought of as being well-suited to describe internal or external flows. Whether they are appropriate to simulate turbulent flows, on the other hand, depends on the type of solver, the level of resolution in the calculation, and the type of sub-grid model used to feeds back the small-scale dynamics to the large-scale. So knowing whether CFD is accurate or even appropriate really depends on the problem it is applied to and the questions asked. One should select a modeling and simulation capability (finite element, CFD or other) depending on the types of mechanics or physics that need to be simulated (problems with stresses, fluids, thermal responses, etc.). Accuracy then depends on the ability of the simulation to provide the same overall balance of competing phenomena (kinetic forces vs. inertia forces vs. dissipation, for example) as the one observed in the real world. And this is the whole difficulty since we cannot possibly know what this balance is in the real world, other than through (limited) observations provided by physical testing. _________________________________________________________________ Q: In our organization we perform test-analysis modal correlation. Starting from testanalysis modal discrepancies we would like to estimate the prediction accuracy due to a transient response. Do you know about publications on this topic? If I understand correctly, one starts by correlating a finite element model to the modal response identified on a structure via Experimental Modal Analysis (EMA), then the model is used to predict the transient response of the same system or a modified version of it. The question is: how accurate are the predictions of transient response? Or, rephrased slightly differently, how does a given level of accuracy of modal parameters translate to the prediction of transient response? I am not aware of work that would address or answer these questions. The closest that I ever came to looking at something like thisand it is the e-mail that I sent you, Adriano is to bypass the modal analysis altogether and assess directly the ability of the finite element simulation to predict the transient response. I found it particularly useful to use the Shock Response Spectrum (SRS) and temporal moments to characterize the transient response. This study successfully demonstrated that the finite element simulation was able to predict the SRS and temporal moments with a level of accuracy that the customer found to be sufficient. But

back to your question: it is a problem of inference. One wishes to infer the level of accuracy of the (predicted) transient response from an observed level of accuracy of the modal response. It seems to me that they are two ways of addressing this question. The first one would be to build an inference model from the modal parameters to the time-domain response and propagate the accuracy from one to the other. This is difficult because, as you know, modal parameters are extracted from time signals through system identification and EMA, but going the other way around (given modal parameters, what are corresponding time series?) is not possible. Linking the accuracy of one to the accuracy of the other would be a wonderful breakthrough technology but I dont think that it is realistic. The second way of addressing the question may be counterintuitive given the culture of your organization, but it would be to bypass EMA altogether. If what you are really interested in is predicting the transient response, why bother with modal parameters? Just focus your modeling effort, and the V&V that supports it, on the transient response. This may mean abandoning modal superposition to reconstruct a time-domain response and going, instead, to an explicit time integration solver. Likewise, it may imply performing fewer modal tests and replacing them by vibration and transient tests. This may be a fundamental change in culture, but one that is worth undergoing if the first avenue mentioned above of inference is too difficult to pursue. _________________________________________________________________ Q: If we plan to do Fatigue Analysis, is there any breakthrough that can increase our confidence of Simulation on this field? Or we just pay more attention on experiments? I am not very knowledgeable in fatigue and stress analysis. But given that it falls within the realm of structural dynamics, I would recommend that you pay particular attention to the material modeling and boundary conditions. Material modeling is an obvious one because it is what controls the stress field locally. If damage (in one form or another) occurs and grows due to fatigue loading, then the ability to predict this damage with a finite element simulation will be greatly influenced by the material model. The description of the boundary condition may not be obvious but it is a phenomenon that, I think, is often overlooked. The boundary condition could, for example, induce residual stresses that may not be modeled, yet, have the potential to significantly influence the results of a field or laboratory test. There may be specialized software, that I am not knowledgeable of, that allows you to simulate fatigue efficiently. Starting with an experimental campaign, as you suggest, is always a good idea. There is a lot of useful information that can be learned from relatively inexpensive experiments. For example, the variability of materials you are working with can be characterized through coupon testing. Knowing this variability will be important when you want to propagate it through the numerical simulation to estimate the uncertainty of predictions. And, while performing these tests, you may learn that the material or structure has some preferred damage accumulation or failure modes. This knowledge may allow you to improve your modeling rules.