What makes a good QSAR model?

in

According to REACH regulation (Annex XI) an assessment using a QSAR model is valid if:

  1. the model is recognised as scientifically valid;
  2. the substance is included in the applicability domain of the model;
  3. results are adequate for classification and labelling and for risk assessment;
  4. adequate documentation of the methods is provided.’

These four ‘conditions’ are specific to the regulatory context, and address important scientific and practical aspects of the use of in silico models.  It is important to note that the four conditions are not just about the model, but also about the particular chemicals it is used for in an assessment, the particular regulatory function it is used for, and how well the model and its use are documented.  

A QSAR model is built on a set of experimental data for particular chemicals (a ‘training set’) and it is trialled by testing other chemicals for which there is also experimental data available (the ‘test set’).  In this way a QSAR model is designed to evaluate a particular set of chemicals in relation to a particular endpoint.  So while it could be used for other chemicals outside this ‘applicability domain’, the evaluation of those chemicals will have a higher level of uncertainty.  

For this reason, REACH and ECHA do not provide approval for models: each use of QSAR models has to be evaluated on its own merit, on a case-by-case basis.  

The first of the four conditions above is nevertheless about the model itself.  In the implementation of REACH, ECHA and others usefully refer to the five OECD principles for QSAR models.  The OECD stated that ‘to facilitate the consideration of a (Q)SAR model for regulatory purposes, it should be associated with the following information:

  1. a defined endpoint;
  2. an unambiguous algorithm;
  3. a defined domain of applicability;
  4. appropriate measures of goodness-of-fit, robustness and predictivity;
  5. a mechanistic interpretation, if possible.’

These five principles refer to the rigour of the model and the transparency of the information provided by the model developer.  The REACH demand to provide ‘adequate documentation of the models’ and their use, is central to demanding and ensuring the quality of models and the quality of their use, now and in the future.  It is ultimately about enabling the regulator to evaluate the chemical safety assessments independently in the interest of human health and the environment.

Related FAQ

Are QSAR models expensive, or free to use?

The CAESAR and VEGA platforms have been produced from EC-funded research, and are therefore freely available for use.  The software can be downloaded. Furthermore, predicted values for more than four million chemicals will be made freely available.

The Danish Environmental Protection Agency has created and made available a database with predictions from more than 70 (Q)SAR models on endpoints for physico-chemical properties, fate, eco-toxicity, absorption, metabolism and toxicity.  More than half of all the estimates are for mammalian (human) toxicity endpoints and include commercial data sets from TOPKAT and MULTICASE as well as models developed in-house.  A structure set of about 166.000 discrete organic chemicals, including almost half of the (around) 100,000 EINECS substances, has been batched through the models, and the results are integrated in the database.  (EINECS: European INventory of Existing Commercial chemical Substances.)

The US Environmental Protection Agency made available a series of QSAR models, within the EPISuite and the T.E.S.T. platforms.

The EC funded ANTARES project is evaluating existing models which could be used for REACH, many of which are free to use.  Hundreds of possible models have been identified and listed, which theoretically could be used for tens of REACH endpoints.

The EU Joint Research Centre has promoted the availability of reliable computer-based estimation methods for use in the regulatory assessment of chemicals.  Within the JRC Institute for Health and Consumer Protection (IHCP), the European Chemicals Bureau (ECB) developed a range of freely available software.  One of the aims of the IHCP is to support the implementation of EU chemicals policy (including the safety assessment of industrial chemicals, chemicals in consumer products, pesticides and biocides) through the development, assessment and application of computational (in silico) methods. 

Typically in such QSAR models, the training set used to develop the model, the algorithm, and other aspects of the model are open for scrutiny by the regulator.   However, in commercially available software the full documentation of each model may not be available.  Indeed, the algorithm and the training set used to build the model are often confidential.  Despite this, the use of commercial programs has not been explicitly criticised by regulatory authorities.  In reality, commercial programs have been used in the USA and Europe, because of their ease of use, their availability and because they represent a major source of QSAR models.  We do not foresee restrictions in their use, and in our opinion it would be a pity to renounce these models.  However, it is likely that in the case of two similar models being available, one commercial and one freely available, the user will prefer the second one.  Within REACH, it also seems likely that the regulators will demand more transparency from the commercial models, and/or prefer the use of fully documented models.

Are in silico methods / QSAR models accepted by REACH?

REACH explicitly encourages innovation in toxicity evaluation.  The development of alternative methods is one of the purposes of REACH. 

The legislation sets out conditions specifically for the use of QSAR models, and the European Chemicals Agency (ECHA) offers detailed guidance.  Even the introductory ‘Guidance in a nutshell’ on substance registration advises industry to ‘collect QSAR estimated results for the substance if suitable models are available’ as an initial step.

However, acceptability depends on both the model and on how it is used in practice.  See ‘What makes a good QSAR model?’

Why are in silico methods not yet used widely in REACH?

QSARs have been used for decades in the development of pharmaceuticals, where a drug is to be developed to achieve a particular biological action.  Our interest here is in the reverse use, where the chemical is known, and the biological action is to be predicted.  That too has been the focus of research for decades.  But here we are concerned specifically with their use for evaluating toxicity within the regulatory framework of REACH, and it is early days.  

In practice, the use of in silico methods by European industry within REACH is still limited.  There are  three key practical issues that can delay their use.  They are highly inter-connected.

(i) Progress in developing models for regulatory use:  The limitations for model development are primarily the lack of good available experimental data as the basis for developing models.  The results of the many thousands of past animal experiments carried out or commissioned by industry are held as confidential by industry.   (The regulatory demand for the transparency of QSAR models inevitably requires that they not only reveal the experimental data used to develop the model, but also make available for independent review the details of how the experimental results were achieved.)  In his video interview (see documentary, part 4) Professor John Dearden made a plea to industry to release that data.  

A positive development is that it is now clearer to QSAR developers than ever before, exactly what is expected of QSAR models for regulatory use.  The regulatory framework of REACH, the OECD principles and the ECHA guidelines effectively work together to increase the demands on models in terms of rigour, reliability and transparency.    

(ii) Uptake by industry and consultants: uptake requires awareness of the models available, confidence in their reliability and confidence in regulatory acceptance if they are used.  (See the FAQ: ‘Do in silico methods require expertise, or can anyone use them?’)  The use of in silico methods for REACH also involves a simultaneous shift from simply using in vivo methods as an accepted form of evidence, to using different and complementary sources of evidence.  

(iii) Acceptance in practice by regulators (ECHA and the national competent authorities):  The ultimate responsibility of regulators is to protect human health and the environment, so there is some caution in accepting the use of alternative methods.  The experience of European regulators in using QSARs is also still limited.  

The regulation has made the shift to animal testing being ‘only as a last resort’, but process requires confidence, shared expertise and communication between these three major stakeholders.  It may be encouraged by wider interest among industry shareholders, publics, and policy makers, in realisation that there are alternatives to animal testing.  

What's next

Learn more about in silico, browse resources,
find out what's happening and get involved