2.8. Annex B Prediction uncertainties for Part III models

In the following, the model development uncertainties associated with the prediction models presented in Part 3 - EEE, Part 4 - MEC and Part 5 - MIS of this handbook are summarized, using the format proposed in Section 2.6.3.1.

Note 27

The uncertainty of the prediction results depends not only on the uncertainties inherent in the model, but also on the use of these modelss, see Section 2.6.3.2 for details.

Model development uncertainties associated with the EEE models

The models provided in Part 3 - EEE, are based on the FIDES 2022 [NR_METHOD_15] approach, making use of a combined method with base failure rates derived from statistics (field return from on-ground applications) and scaling factors derived from the PoF. Table 2.6.1 and Table 2.6.2 summarize the uncertainties associated with these two modelling aspects.

Model development uncertainties associated with the mechanical models

Besides providing own models based on Structural Reliability Methods, Part 4 - MEC makes reference also to external data sources such as [NR_METHOD_12] and the [NR_METHOD_1], which is based on a combined method using both statistics and PoF. The limitations of these data sources when used for space applications are discussed more in detail in Part 4 - MEC. Here, only a quick overview is given using the common checklist format for the assessment of model development uncertainties from Section 2.6.3.1.

Model development uncertainties associated with the models for miscellaneous items

The “standard” model for miscellaneous items, as defined in Part 5 - MIS, has been developed based on IOR for an equivalent class of items. Model development uncertainties associated with this approach are listed in Table 2.8.7. For “non-standard” items,only a general process for model development is provided and the model development uncertainties should to be assessed by the user.

Table 2.8.1 Model uncertainties associated with the statistical modelling behind the EEE component models (base failure rates)

1 – Statistical approach: Inputs 1.1 – Data representativeness Is the data representative for the application? Basic failure rates are the main data issued from statistical approach. These data are issued from data from manufacturers, data from tests or data issued from other handbooks representative within the limits defined by the models for each technology and each quality level. NRPM study provides complement for specificities linked to space. However, some of the specificities such as vacuum, single atom like oxygen, UV, light, SEE are not considered. Partly 1.2 – Failure characteristics Is the data representative in terms of the failure characteristics observed? Failures are considered through their failure modes, their failure mechanisms and physical stresses providing the failures. Information on the failure root causes should be performed during the development of the FIDES methodology. Failure modes are relevant for all industrial domains, and surely for the space industry also. However, no repartition of failure modes is provided. For failure mechanisms specific to space constraints, such as Total Ionizing Dose (TID), this methodology is described in NRPM. Yes 1.3 – Data quality and content Does the data provide all the information required by the model? In FIDES, only the technologies with enough data of basic failure rates are relevant, other ones have not been considered. As models are based on PoF, it is not necessary to get numerous data. Yes 1.4 – Data sample size Is the sample size large enough to limit statistical uncertainty? Models are based on PoF, so it is not necessary to have numerous data of basic failure rates. Otherwise, it would not be possible to cover all technologies and especially the most recent ones. For basic failure rates, data are issued from data from manufacturers, data from tests or data issued from other handbooks with sufficient sampling to get good confidence. Yes 2 – Statistical approach: Modelling 2.1 – Model input variables Does modelling account for the variables driving the item’s reliability? The basic failure rates are released from statistics coming from in-service feedback, manufacturer data and different handbooks. These data are not linked to any quality factor and conditions of use. No 2.2 – Distributional assumption Is the distribution model (e.g. Exponential, Weibull) justified? Quantity of data cannot justify the use of distribution law. No 2.3 – Data processing Are assumptions made during data processing explicit and justified? Are based on physics of failures. No 2.4 – Estimation methods and results Are the estimation results based on sound methods and reported consistently? The basic failure rates are released from statistics coming from in-service feedback, manufacturer data and different handbooks. This data is adapted and corrected to fit with the FIDES guide. PEA REX and the Reliability Prediction Data Sources and Methodologies (RPDSM) have demonstrated good adequation of basic failure rates and models from field data return. Partly

Table 2.8.2 Model uncertainties associated with the PoF modelling behind the EEE component models (acceleration factors)

1 – Physics of Failure: Inputs 1.1 – Failure characteristics Is the choice of failure mechanisms underlying the model justified? Analysis has been made from a clear definition of failure modes, leading to the failure mechanisms. However, uncertainties and confidence levels are not known as not enough data are available. All failure modes are considered whatever the technology but limited to the specification of the equipment. Yes 1.2 – Physical variables information Is the selection and quantification of physical model input variables sound? The physical phenomenon considered are: Temperature, Thermal cycling, Humidity and Vibrations. This data is clearly measurable and can be quantified. Yes 2 – Physics of Failure: Modelling 2.1 – Failure mechanism modelling Is the reliability model based on sound failure mechanism modelling? Models are based on recent acceleration laws developed by Arrhenius, Coffin-Manson, Norris-Lanzberg, Peck and Hallberg. These laws have been defined in many publications over the last 50 years and are considered as operative and sound. No simplification is made. The chosen activation energies consider the most significant failure modes, but generally cover all other failure modes. Yes 2.2 – Uncertainty modelling Is the reliability model based on sound uncertainty modelling? There are no statistical studies for uncertainties and levels of confidence. However, studies such as the PEA REX and the RPDSM have demonstrated good adequation of models with data from field data return. Partly 2.3 – Mathematical modelling Are the mathematical methods used to derive reliability estimates suitable? Models are based on the same acceleration laws (Arrhenius, Coffin-Manson, Norris-Lanzberg, Peck and Hallberg). These laws are widely published, considered operative and sound, and no simplification is made. Yes 2.4 – Model validation Has the model been validated, using quantitative or qualitative information? No formal validation of each model has been made. However, studies such as the PEA REX and the RPDSM have demonstrated good adequation of models from field data return. The PEA REX has shown good correlation with the FIDES prediction on 14 products used in air, sea and land applications; an analysis of main component families level also showed a good correlation. Partly

Table 2.8.3 Uncertainties associated with the mechanical reliability models based on (simplified) structural reliability methods (PoF)

1 - Physics of Failure: Inputs 1.1 – Failure characteristics Is the choice of failure underlying the model justified? Part 4 – MEC of this handbook provides guidance on the identification and selection of failure mechanisms to be modelled and discusses the requirements for appropriate root causes coverage. The failure criterion is clearly defined together with the limit state function(s) used for the failure mechanism modelling. Can be 1.2 – Physical variables information Is the selection and quantification of physical model input variables sound? The relevant physical variables are defined by the limit state function(s) used. Each variable is clearly defined and quantification is possible, e.g. based on design information. Quantifying the uncertainties associated with each basic variable can in some cases be challenging. Yes 2 - Physics of Failure: Modelling 2.1 – Failure mechanism modelling Are the mathematical methods used to derive reliability estimates suitable? Limit state functions provided in the handbook are based on well-known physical laws and published information. All assumptions and simplifications inherent in the simplified models are made explicit and a model uncertainty variable is introduced for quantitative treatment. The models account for one failure mechanism at a time. Yes 2.2 – Uncertainty modelling Is the reliability model based on sound uncertainty modelling? The limit state functions, and distributional assumptions used for the simplified structural reliability methods were designed to achieve analytic solutions for the probability of failure. All assumptions and simplifications made are listed together with the models. Uncertainty quantification for the basic variables treated as random is left to the model user. Partly 2.3 – Mathematical modelling Are the mathematical methods used to derive reliability estimates suitable? The analytical formulas are suitable under the assumptions made in the modelling of the limit state function and basic variable distributions (see above). Yes 2.4 – Model validation Has the model been validated, using quantitative or qualitative information? No validation efforts have been made so far to prove the pertinence of the models and to check the behavior of the models. No

Table 2.8.4 Model uncertainties associated with the NPRD when considered as a data source for mechanical reliability prediction in space applications

1 – Statistical approach: Inputs 1.1 – Data representativeness Is the data representative for the application? Limited coverage of part types used in space and space-grade items. Most data come from terrestrial applications, and some data records are very old (early 1970’s). A search limited to parts listed with Space Flight (SF) environment can deliver data that is representative with respect to technology, quality, use conditions and environment (to be checked with the part details). Partly 1.2 – Failure characteristics Is the data representative in terms of the failure characteristics observed? Data recorded in NPRD is from field failure rate data sources. The failure criterion is not clearly specified and should not be applicable to the specific use of the data record. The discussion provided in the introduction is maintenance-related and should not be applicable to data from SF environment. Information on the coverage, e.g. failure root causes and failure modes are not provided. Unknown 1.3 – Data quality and content Does the data provide all the information required by the model? Even when restricted to SF environment, data comes from non-homogeneous population with respect to design characteristics and specific use conditions. Information on variables likely to drive the reliability of mechanical items is not provided. The records provide information on the number of failures observed and cumulated hours, without information on the individual TTF. No 1.4 – Data sample size Is the sample size large enough to limit statistical uncertainty? The sample size (cumulated hours and number of failures) is provided in each data sheet. Failure rates calculated for entries with zero failures are based on the conservative assumption that a single failure has occurred. Partly 2 – Statistical approach: Modelling 2.1 – Model input variables Does the modelling account for the variables driving the item’s reliability? Summary statistics are derived from a merging that considers information provided in the data records: part type, part quality and environment. Part details statistics provide failure rate estimates as derived from each data entry. Design details, specific use conditions and environmental variables cannot be considered. Partly 2.2 – Distributional assumption Is the distributional model (e.g. exponential, Weibull) justified? Failure rate estimates are based on the exponential distribution assumption without justification; due to the way the data has been collected. No 2.3 – Data processing Are assumptions made during data processing explicit and justified? Data entries are checked by Reliability Information Analysis Centre (RIAC) for completeness, consistency and adequacy, but the handbook provides no details on the criteria and assumptions used. No 2.4 – Estimation methods and results Are the estimation results based on sound methods and reported consistently? Failure rate estimates are recorded as average values except for entries without failures, where a conservative assumption is made (cf. 1.4 – Data sample size of this table). Statistical uncertainties can be quantified based on the information provided for each entry, but not for the summary statistics, due to the data merging process applied. Yes

---

Table 2.8.5 Model uncertainties associated with the statistical modelling behind the NSWC models when considered as a data source for mechanical reliability prediction in space applications.

1 – Statistical approach: Inputs 1.1 – Data representativeness Is the data representative for the application? The data used to estimate base failure rates is not representative (in terms of technology, quality level, use conditions and environment) for space applications. No 1.2 – Failure characteristics Is the data representative in terms of the failure characteristics observed? The handbook provides no detailed information on the failure characteristics considered for the estimation of base failure rates (failure criterion, root causes, coverage, observed failure modes). Unknown 1.3 – Data quality and content Does the data provide all the information required by the model? According to the handbook introduction, base failure rates have been derived (mostly) from laboratory testing, where the exact stress levels are known. Yes 1.4 – Data sample size Is the sample size large enough to limit statistical uncertainty? The handbook does not provide information on the sample size of the data used to derive base failure rates. Unknown 2 – Statistical approach: Modelling 2.1 – Model input variables Does the modelling account for the variables driving the item’s reliability? NSWC predictions can be tuned with the aid of scaling factors derived from the PoF. The uncertainties inherent in this approach are assessed separately. Partly 2.2 – Distributional assumption Is the distributional model (e.g. exponential, Weibull) justified? Time dependence on mechanical parts failure rates is considered in a simplified way. Base failure rates are given as constant (or time-averaged) values. Partly 2.3 – Data processing Are assumptions made during data processing explicit and justified? Assumptions made for the statistical modelling behind the NSWC models are not explicit. No 2.4 – Estimation methods and results Are the estimation results based on sound methods and reported consistently? The estimation methods used are not explained. No

Table 2.8.6 Model uncertainties associated with the PoF modelling behind the NSWC models when considered as a data source for mechanical reliability prediction in space applications

1 - Physics of Failure: Inputs 1.1 – Failure characteristics Is the choice of failure mechanisms underlying the model justified? Failure mechanisms relevant for naval applications are identified and described in the handbook. The relevance of the considered failure mechanism models for space applications is given only for some parts. Additional failure mechanisms and different failure root causes can be covered by the base failure rates (statistical approach); the PoF are only used to derive scaling factors to account for the detailed design, environment and use conditions. Partly 1.2 – Physical variables information Is the selection and quantification of physical model input variables sound? The selection of model input variables related to failure mechanisms appears to be sound. Each variable is clearly defined in the handbook, though with reference to US units. The required model input can be extracted from design information, at least in later project stages when detailed design information becomes available. Yes 2 - Physics of Failure: Modelling 2.1 – Failure mechanism modelling Is the reliability model based on sound failure mechanism modelling? The failure mechanism modelling behind the NSWC scaling factors is not fully described in the handbook. Model uncertainties and simplifications made are not discussed. Unknown 2.2 – Uncertainty modelling Is the reliability model based on sound uncertainty modelling? Uncertainties in the failure mechanism modelling due to the input variables entering the models are not quantified. The unknown model risks bias introduced by this omission can be considerable in case the underlying failure mechanism model is nonlinear. No 2.3 – Mathematical modelling Are the mathematical methods used to derive reliability estimates suitable? The mathematical formulas used behind the NSWC scaling factors are not described in the handbook. Unknown 2.4 – Model validation Has the model been validated, using quantitative or qualitative information? Some limited validation based on testing is mentioned in the handbook introduction. Partly

Table 2.8.7 Model uncertainties associated with the “standard model” for miscellaneous items

1 - Statistical approach: Inputs 1.1 – Data representativeness Is the data representative for the application? Since the method used to derive reliability is based on IOR from industrials, the data are fully representative for the items classified as standard in a certain frame defined by (1) Quality process, (2) Maturity, (3) Technology, (4) Complexity and (5) Operating conditions contribution to the reliability of the item. Yes 1.2 – Failure characteristics Is the data representative in terms of the failure characteristics observed? The observed failures in-orbit are used to support the reliability prediction of the standard miscellaneous items. All random failures are counted and failures due to systematic failures are counted once. The model reflects these categories of failures. No extrinsic failures are considered like those from IOR nor degradation. Partly 1.3 – Data quality and content Does the data provide all the information required by the model? The IOR data is homogeneous by definition of the model, i.e. the IOR merges in case of inconsistency (cumulated time, number of failures). The individual failure times are recorded but no statistical tests are conducted to verify the assumption that the failure rates are constant. The data provided by IOR are truncated (and sometimes censored) since the cumulated time is calculated for both fleets of satellites along which the majority is still operational. Partly 1.4 – Data sample size Is the sample size large enough to limit statistical uncertainty? Models for standard miscellaneous items are built with industrials’ fleets representing millions of hours of cumulated time. Most of the time failures have been observed, sometimes higher than 10 (e.g. Travelling Wave Tube Amplifier – TWTA), sometimes less than 10. In specific cases where no failure is observed there is no possibility of justifying the failure mechanisms. Partly 2 - Statistical approach: Modelling 2.1 – Model input variables Does the modelling account for the variables driving the item’s reliability? For the standard miscellaneous items (derived from in-orbit data) the model (per item) reflects a “standard” frame of Quality (high), Maturity (high), Technology and Complexity and a set of Operating conditions dependent on the item. No analytical justification is provided regarding the failure mechanisms related to this set of parameters. The approach is fully of statistical one. Partly 2.2 – Distributional assumption Is the distributional model (e.g. exponential, Weibull) justified? For the items presenting a high number of failures (higher than 10), the distribution of time does not reflect an evolution in time but the in-line use of the item. No study has been conducted to quantify this assumption in the case of the constant failure rate. No 2.3 – Data processing Are assumptions made during data processing explicit and justified? No specific assumption is other than the class of miscellaneous items and the functional criterion has been justified. The set of parameters contributing to the reliability is well mastered in some cases (e.g. TWTA) and non-mastered for various items in a same class with variability of design (e.g. deployment device). No 2.4 – Data processing Has the model been validated, using quantitative or qualitative information? The constant failure rates derived from field data return data are based on the interval estimation (one-bounded-side) at 60% level of confidence. No